Many modern applications from camera arrays to federated learning depend on distributed collection, processing and communication of correlated data, requiring efficient compression techniques to minimize the induced communication overhead. While information-theoretic foundations of distributed compression are well-investigated, the impact of theory on practice has been somewhat limited. As the field of data compression is undergoing a significant transformation with the emergence of learning-based techniques, it is natural to ask whether machine learning can reap the benefits of distributed compression promised by information theory long ago.
In this talk we answer this question affirmatively by focusing on a simple distributed lossy compression setting, also known as the Wyner-Ziv problem, in which the decoder has direct access to correlated information that is unknown at the encoder. We show that for some well-studied source distributions, neural compression techniques mimic information theoretically optimal solutions such as “binning” or “grouping” in the source space as well as optimal combination of the quantization index and side information. These interpretable behaviors appear even though we neither impose a particular structure nor assume any prior knowledge about the source distributions. Binning is a widely-used tool in network information theory, and its emergence from data-driven learning can have implications beyond the setting considered in this talk.
Elza Erkip is an Institute Professor in the Electrical and Computer Engineering Department at New York University Tandon School of Engineering. She received the B.S. degree in Electrical and Electronics Engineering from Middle East Technical University, Ankara, Turkey, and the M.S. and Ph.D. degrees in Electrical Engineering from Stanford University, Stanford, CA, USA. Her research interests are in information theory, communication theory, and wireless communications.
Dr. Erkip is a member of the Science Academy of Turkey and is a Fellow of the IEEE. She received the NSF CAREER award in 2001, the IEEE Communications Society WICE Outstanding Achievement Award in 2016, the IEEE Communications Society Communication Theory Technical Committee (CTTC) Technical Achievement Award in 2018, and the IEEE Communications Society Edwin Howard Armstrong Achievement Award in 2021. She was the Padovani Lecturer of the IEEE Information Theory Society in 2022. Her paper awards include the IEEE Communications Society Stephen O. Rice Paper Prize in 2004, the IEEE Communications Society Award for Advances in Communication in 2013 and the IEEE Communications Society Best Tutorial Paper Award in 2019. She was a member of the Board of Governors of the IEEE Information Theory Society 2012-2020, where she was the President in 2018. She was a Distinguished Lecturer of the IEEE Information Theory Society from 2013 to 2014.