Multimodal Maintenance Application HCI

Human Computer Interface (HCI) in applications for the maintenance of complex machinery such as an aircraft can be enhanced by exploiting new developments in HCI. We have developed a multimodal HCI demonstration system for maintenance applications, incorporating Augmented Reality (AR), Speech Recognition, and 3- dimensional audio technologies. The Augmented Reality interface is based on an original dynamic tracking approach to provide rapid update of the scene with graphical overlays. We enhance the use of this interface with speech recognition to control the system and to add annotations using dictation-based text information. A combination of 3-D audio, graphic animations, and text displays is used to communicate information to the user.