Towards a New Generation of Augmented Reality Assistance Systems for Procedural Tasks

Procedural tasks are common in many professions and require precise execution, especially when working with complex machines, equipment, or patients. Augmented reality (AR) head-mounted displays (HMDs) have shown promise in effectively assisting operators with such tasks. However, state-of-the-art AR systems only provide static information to the operator without considering their current situation or level of experience. Context-aware AR support has been introduced to improve these systems’ support, especially for complex operations. By comparing user behavior inferred by real-time analysis of sensor data (e.g., eye tracking) with a process model, the current context and required support can be inferred to adapt displayed information and to provide real-time feedback.

The thesis Towards Advanced User Guidance and Context Awareness in Augmented Reality-guided Procedures includes three studies that investigated the effectiveness of augmented reality (AR) and context-aware AR support systems in performing procedural tasks. Study I compared the benefits of contextual information provided by AR with those of traditional media for ECMO cannulation training, demonstrating clear benefits of AR while highlighting areas for improvement with context-aware AR. Study II addressed effective visualization strategies in providing real-time feedback to operators in simulated spine surgery, demonstrating that real-time feedback levels the performance of novice and experienced operators while identifying preferred AR visualizations. Study III demonstrates the potential of eye and hand tracking features to predict and prevent operator errors before they occur, using a memory card game as an example.