EmbedLLM from legacy documents to embedded XR instructions: automatic context aware content generation to support manual task execution
Universiteit Hasselt
2026
This thesis addresses the cognitive friction associated with executing manual tasks using flat 2D documentation in spatial computing environments, where users must constantly shift attention between instructions, physical tools, and machinery. To solve this, we present EmbedLLM, the first automated, context-aware content reformatting framework that adapts both content design and presentation to the user’s environment and specific goal. EmbedLLM converts unstructured step-based PDF manuals into 3D spatially embedded instructions in Virtual Reality (VR). The system operates via a pipeline that abstracts the environment into a semantic scene graph, leverages a multi-stage Large Language Model (LLM) reasoning chain to construct step-based instruction nodes and semantic edges, and instantiates these panels using a novel, semantically aware force-directed 3D label placement algorithm to achieve spatial equilibrium and prevent visual clutter.
We evaluated EmbedLLM through a preliminary preference survey (n = 20) across diverse environments and a within-subjects user study (n = 12) comparing the framework to a traditional floating PDF window during 3D printer maintenance tasks. The preference survey showed that EmbedLLM achieves significantly higher environmental alignment than traditional baseline documents, with participants strongly favoring spatial orchestration for complex, structurally demanding procedures. However, the task execution study revealed a clear trade-off: active task support with EmbedLLM resulted in significantly longer completion times (996.93 s vs. 580.46 s) and increased cognitive load (3.00 vs. 2.31 on the NASA-TLX scale). Qualitative feed-back suggests these performance costs stem from the rigid sequential pacing of the generated interface and novices struggling with hardware terminology.
This work contributes the technical pipeline of EmbedLLM, a semantically enhanced physics layout algorithm, and empirical insights on context-aware XR instruction delivery. Future research directions include real-time situated action updates, user skill level adaptation, and integration with agentic GUI automation and text-to-3D models.
Meer lezen