Big Data at Scale and Small Data at Low Latency: From Research Prototypes to Production Systems

The rapid growth of data and the increasing demand for intelligent, real-time services have made two seemingly different computing problems equally important: processing big data on a scale and processing small data at low latency. Big data systems must efficiently store, manage, and analyze massive datasets distributed across many machines, while many emerging applications require relatively small amounts of data to be processed with extremely low latency on a single node.  

In this talk, I will present two R&D cases illustrating how systems research can progress from research prototypes to widely deployed production systems. The first one is related to storage engines for big-data analytics, starting from academic work to a production system transformation. The second case focuses on low-latency spatial data analytics. We have explored an unconventional approach that repurposes Raytracing (RT) cores, a specialized hardware device originally designed for computer graphics, to accelerate spatial database operations. Industry engineers have merged them into Apache SedonaDB, providing a path from experimental hardware acceleration to a production spatial database system.

NSF recently announced a new university and industry partnership pilot initiative for Ph.D. programs to address a growing mismatch in American doctoral education, and this talk will echo this initiative.