Integrated Multimodal System for Real-Time Driver Fatigue Detection and Cognitive Load Assessment

Paying attention to safety while traveling in a vehicle was essential. One of the greatest barriers to travel safety is driver fatigue. In order to tackle this issue, we designed an all-inclusive hardware and software environment for analyzing driver interaction with highly automated vehicles, providing a special emphasis on detection of driver’s cognitive load and providing appropriate alerts in advance. There are three essential types of measurements included in this system. First, the facial features of the driver are measured through the use of Driver Behavior Measurements (DBM) which are made from RGB and IR cameras that capture the eye aspect ratio (EAR), mouth aspect ratio (MAR), pupil circularity (PUC), and mouth to eye aspect ratio (MOE), Most of these measurements have been shown to work under normal as well as more extreme conditions including low light conditions or use of sunglasses by the drivers. Secondly, Vehicle Dynamics Measurements (VDM) is performed by an Inertial Measurement Unit (IMU) that is able to sense vehicle turns, accelerations or braking that help to detect regularity associated with tiredness. Thirdly, Due to external factors such as stress, heart rate increases which can be achieved through a wearable devise with a heart rate monitor and offers a measure of the Driver Physiological Measurements (DPM) Also measurement of fatigue. Data from the three measurements is processed using the previous study implemented Convolutional Neural Networks-Long Short-Term Memory (CNN-LSTM) model which assists in temporal features which improves fatigue measurement accuracy. The model achieved an accuracy of 91% in detecting fatigue. Preliminary tests conducted with one human driver shows that the system is robust and scalable, and has potential large-scale studies related to driver engagement which sets the direction for real-world applications of modern vehicular systems that aim at reducing the risk of drowsy driving accidents.