Mitigating Data and System Heterogeneity and Achieving Linear Convergence Speedup in Federated Learning

Prof Jia Liu
Assistant Professor, Dept. of Electrical and Computer Engineering
The Ohio State University
JEC 3117
Wed, August 03, 2022 at 11:00 AM

In this talk, I will present our recent work on mitigating data and system heterogeneity to
achieve linear convergence speedup in federated learning. Federated learning (FL) is a distributed machine learning architecture that leverages a large number of workers to jointly learn a model with decentralized data. FL has received increasing attention in recent years thanks to its data privacy protection, communication efficiency and a linear speedup for convergence in training (i.e., convergence performance increases linearly with respect to the number of workers). However, existing studies on linear speedup for convergence are only limited to the assumptions of i.i.d. datasets across workers and/or full worker participation, both of which rarely hold in practice. In the first part of the talk, we will propose a generalized federated averaging (FedAvg) algorithm with two-sided learning rates to address the “client drift” challenges in federated learning with non-i.i.d. data. In the second part of the talk, we will further propose a new federated learning paradigm called “anarchic federated learning” (AFL), which features a loose coupling between the server and the workers to enable the workers to participate in the learning anytime in any way they want, thus addressing the system heterogeneity in federated learning.

Jia (Kevin) Liu is an Assistant Professor in the Dept. of Electrical and Computer Engineering at
The Ohio State University and an Amazon Visiting Academics (AVA). He received his Ph.D. degree from the Dept. of Electrical and Computer Engineering at Virginia Tech in 2010. From Aug. 2017 to Aug. 2020, he was an Assistant Professor in the Dept. of Computer Science at Iowa State University. His research areas include theoretical machine learning, stochastic network optimization and control, and performance analysis for data analytics infrastructure and cyber-physical systems. Dr. Liu is a senior member of IEEE and a member of ACM. He has received numerous awards at top venues, including IEEE INFOCOM'19 Best Paper Award, IEEE INFOCOM'16 Best Paper Award, IEEE INFOCOM'13 Best Paper Runner-up Award, IEEE INFOCOM'11 Best Paper Runner-up Award, IEEE ICC'08 Best Paper Award, and honors of long/spotlight presentations at ICML, NeurIPS, and ICLR. He is an NSF CAREER Award recipient in 2020 and a winner of the Google Faculty Research Award in 2020. He received the LAS Award for Early Achievement in Research at Iowa State University in 2020, and the Bell Labs President Gold Award. His research is supported by NSF, AFOSR, AFRL, and ONR.