When GenAI Meets Extended Reality: Enabling Scalable and Natural Interactions

Extended Reality (XR) offers immersive experiences and has broad applications across education, training, manufacturing, transportation, and healthcare. In this talk, we first present several of our developed XR prototypes and highlight the growing need for natural language interaction within XR environments -- such as generating and animating 3D scenes directly from spoken input. Achieving this vision requires seamless integration of Large Language Models (LLMs), such as GPT-4, with XR systems. However, the complexity of XR environments makes it difficult to accurately extract relevant contextual data and scene/object parameters from an overwhelming volume of XR artifacts. This not only increases costs under pay-per-use LLM models, but also elevates the risk of generation errors and privacy concerns. To address these challenges, we introduce a novel framework for creating interactive XR worlds using structured JSON data generated by LLMs. Unlike prior work that focuses on generating executable code, LLMER translates natural language input into lightweight and interpretable JSON representations, significantly reducing the risk of application crashes and minimizing processing latency. It adopts a multi-stage pipeline that delivers only essential, context-aware information tailored to the user’s request, and incorporates multiple modules optimized for various XR tasks.