Visual Speech Interface: Apparatus and Algorithms

To make speech recognition a viable input modality in the cockpit, we propose to include visual speech input to improve robustness of the approach in the presence of noise. The visual speech interface includes a headmounted lip imaging apparatus and algorithms to recognize spoken words visually. Our algorithms are based on a few components which address all issues related to lip localization, lip shape model extraction, tracking, feature extraction and recognition. We demonstrate the practicability of the concept with a visual speech recognizer for a discrete-word recognition task that is relatively simple but achievable in real time.