Human Digital Twin Architecture for Ground Vehicle Scenario Integration

Ground vehicle commanders operate in scenarios which bare high cognitive load. They must be reactive to time-critical events where attention is divided between a variety of sensors, crew members, the physical world, and digital displays, which can result in missed situational cues. This paper presents a Human Digital Twin (HDT) architecture which provides real-time, embodied AI assistance to commanders in a military ground vehicle simulation scenario. The system integrates a data pipeline for combining a MetaHuman avatar in Unreal Engine with multi-modal data ingestion and a large language model (LLM). In addition, a retrieval-augmented generation approach grounds the LLM with mission-specific context, and a Big Five personality framework for prompt design constructs a consistent agent persona throughout the scenario. The architecture is demonstrated with a prisoner of war camp scouting mission, in which the HDT selectively intervenes when needed to alert the commander to critical events when missed. A system latency evaluation is provided to demonstrate viability for real-time integration. Results show the potential of integrated HDT systems to improve situational awareness and decision support in high stakes ground vehicle operations.