Mitigating Data and System Heterogeneity and Achieving Linear Convergence Speedup in Federated Learning

In this talk, I will present our recent work on mitigating data and system heterogeneity to
achieve linear convergence speedup in federated learning. Federated learning (FL) is a distributed machine learning architecture that leverages a large number of workers to jointly learn a model with decentralized data. FL has received increasing attention in recent years thanks to its data privacy protection, communication efficiency and a linear speedup for convergence in training (i.e., convergence performance increases linearly with respect to the number of workers). However, existing studies on linear speedup for convergence are only limited to the assumptions of i.i.d. datasets across workers and/or full worker participation, both of which rarely hold in practice. In the first part of the talk, we will propose a generalized federated averaging (FedAvg) algorithm with two-sided learning rates to address the “client drift” challenges in federated learning with non-i.i.d. data. In the second part of the talk, we will further propose a new federated learning paradigm called “anarchic federated learning” (AFL), which features a loose coupling between the server and the workers to enable the workers to participate in the learning anytime in any way they want, thus addressing the system heterogeneity in federated learning.