Towards Exact Gradient-based Training for Analog In-memory Computing

Artificial intelligence (AI) models that have billions of parameters exacerbate poor energy efficiency on conventional general-purpose processors such as GPUs and CPUs. Analog in-memory computing, or simply analog AI, is a promising approach to addressing the challenge, which is predicted to enjoy energy efficiencies 40-140 times higher than those of cutting-edge GPUs. However, training AI models on analog devices is difficult and largely unexplored. Recent empirical studies have shown that the "workhorse" of AI training - stochastic gradient descent (SGD) algorithm performs poorly when applied to train models on non-ideal analog devices.

 In this talk, we will propose a mathematical model to accurately characterize training dynamics on analog devices. Building upon this, for the first time, we will uncover the role of underlying device physics on the training dynamics, and then demystify the non-convergence issues of vanilla SGD-based training on non-ideal analog devices that have asymmetric updates, reading noise, and device variability. We will further discuss how to algorithmically mitigate the asymmetric error induced by training on non-ideal analog devices and hence eliminate the asymptotic training error. We will conclude the talk by presenting some simulations that verify the correctness of the theoretical analyses and pointing out future directions in this promising area.