This content is not included in your SAE MOBILUS subscription, or you are not logged in.
A Robust Failure Proof Driver Drowsiness Detection System Estimating Blink and Yawn
ISSN: 0148-7191, e-ISSN: 2688-3627
Published April 14, 2020 by SAE International in United States
This content contains downloadable datasetsAnnotation ability available
The fatal automobile accidents can be attributed to fatigued and distracted driving by drivers. Driver Monitoring Systems alert the distracted drivers by raising alarms. Most of the image based driver drowsiness detection systems face the challenge of failure proof performance in real time applications. Failure in face detection and other important part (eyes, nose and mouth) detections in real time cause the system to skip detections of blinking and yawning in few frames. In this paper, a real time robust and failure proof driver drowsiness detection system is proposed. The proposed system deploys a set of detection systems to detect face, blinking and yawning sequentially. A robust Multi-Task Convolutional Neural Network (MTCNN) with the capability of face alignment is used for face detection. This system attained 97% recall in the real time driving dataset collected. The detected face is passed on to ensemble of regression trees to detect the 68 facial landmarks. The eye and mouth landmarks are isolated to detect the blink and yawning by identifying the open/closed state by calculating their aspect ratios. The proposed system combines the two detectors with failure proof strategies. A Minimization of Sum Squared Error based object tracker is used to track the face in the video when the face detector fails. A new binary classification neural network is deployed to reinforce the detection of eye and mouth state when the face landmark detection fails. A combination of the face, landmark detector and the failure proof systems such as the object tracker and classification neural network provides a driver drowsiness detection system with nearly 99.9% accuracy in detecting blinking and yawning.
CitationJesudoss, Y. and Park, J., "A Robust Failure Proof Driver Drowsiness Detection System Estimating Blink and Yawn," SAE Technical Paper 2020-01-1030, 2020, https://doi.org/10.4271/2020-01-1030.
Data Sets - Support Documents
|[Unnamed Dataset 1]|
|[Unnamed Dataset 2]|
|[Unnamed Dataset 3]|
|[Unnamed Dataset 4]|
|[Unnamed Dataset 5]|
|[Unnamed Dataset 6]|
- Arnold, L.S. and Tefft, B.C. , “Prevalence of Self - Reported Drowsy Driving , United States : 2015,” AAA Foundation for Traffic Safety, 2017.
- National Center for Statistics and Analysis , “Drowsy Driving 2015 Crash Stats Brief Statistical Summary,” Report No. DOT HS 812 446). Washington, DC: National Highway Traffic Safety Administration 2017.
- Sahayadhas, A., Sundaraj, K., and Murugappan, M. , “Detecting Driver Drowsiness Based on Sensors: A Review,” Sensors 12(12):16937-16953, 2012, doi:10.3390/s121216937.
- Vicente, J., Laguna, P., Bartra, A., and Bailon, R. , “Drowsiness Detection Using Heart Rate Variability,” Med Biol Eng Comput. 54:927-937, 2016, doi:10.1007/s11517-015-1448-7.
- Lin, F.-C., Ko, L.-W., Chuang, C.-H., Tung-Ping, S., and Lin, C.-T. , “Generalized EEG-Based Drowsiness Prediction System by Using a Self-Organizing Neural Fuzzy System,” IEEE Transactions on Circuits and Systems I: Regular Papers 59(9):2044-2055, 2012, doi:10.1109/tcsi.2012.2185290.
- Morris, D.M., Pilcher, J.J., and Switzer, F.S. , “Lane Heading Difference: An Innovative Model for Drowsy Driving Detection Using Retrospective Analysis around Curves," Accident Analysis & Prevention 80 (2015): 117-24. doi:10.1016/j.aap.2015.04.007.
- Li, K., Wang, S., Chang, D., Huang, Y. et al. , “Accurate Fatigue Detection Based on Multiple Facial Morphological Features,” Journal of Sensors 2019:1-10, 2019, doi:10.1155/2019/7934516.
- Knapik, M. and Cyganek, B. , “Driver’s Fatigue Recognition Based on Yawn Detection in Thermal Images,” Neurocomputing 338:274-292, 2019, doi:10.1016/j.neucom.2019.02.014.
- Zhang, Z. and Zhang, J. , J. Control Theory Appl. 8:181, 2010, doi:https://doi.org/10.1007/s11768-010-8043-0.
- Soukupova, T. and Cech, J. , “Real-Time Eye Blink Detection using Facial Landmarks,” Center for Machine Perception, Department of Cybernetics. Faculty of Electrical Engineering, Czech Technical University in Prague, 2016.
- Alioua, N., Amine, A., and Rziza, M. , “Driver’s Fatigue Detection Based on Yawning Extraction,” International Journal of Vehicular Technology, 2014, Article ID 678786, 7 pages, 2014, https://doi.org/10.1155/2014/678786.
- Sun, X., Wu, P., and Hoi, S. C.h. , "Face Detection Using Deep Learning: An Improved Faster RCNN Approach," Neurocomputing 299 (2018): 42-50. doi:10.1016/j.neucom.2018.03.030.
- Schroff, F., Kalenichenko, D., and Philbin, J. , "FaceNet: A Unified Embedding for Face Recognition and Clustering," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015. doi:10.1109/cvpr.2015.7298682.
- Zhang, K. and Zhang, Z. and Li, Z. and Qiao, Y. , “Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks,” in 2016 IEEE Signal Processing Letters, 23. doi:10.1109/LSP.2016.2603342.
- Simonyan, K. and Zisserman, A. , Very Deep Convolutional Networks for Large-Scale Image Recognition (CVPR, 2014).
- Yang, S., Luo, P., Loy, C.C., Tang, X. , "WIDER FACE: A Face Detection Benchmark," in IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- Do, N.-T., Kim, S.-H., Yang, H.-J., Lee, G.-S., and Na, I.-S. , “Face Tracking with Convolutional Neural Network Heat-map,” in Proceedings of the 2nd International Conference on Machine Learning and Soft Computing - ICMLSC 18, February 2018. doi:10.1145/3184066.3184081.
- Jain, V. and Learned-Miller, E. , “FDDB: A Benchmark for Face Detection in Unconstrained Settings,” Technical Report UM-CS-2010-009, Dept. of Computer Science, University of Massachusetts, Amherst, 2010.
- Ågren, S. , “Object Tracking Methods and their Areas of Application: A Meta-Analysis: A Thorough Review and Summary of Commonly used Object Tracking Methods,” 2017, http://www8.cs.umu.se/education/examina/Rapporter/SannaAgrenFinal.pdf
- Lukežič, A., Vojíř, T., Zajc, L.Č., Matas, J., and Kristan, M. , “Discriminative Correlation Filter Tracker with Channel and Spatial Reliability,” International Journal of Computer Vision 126(7):671-688, August 2018, doi:https://doi.org/10.1007/s11263-017-1061-3.
- Uzkent, B., and Seo, Y. , “EnKCF: Ensemble of Kernelized Correlation Filters for High-Speed Object Tracking,” in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), 2018.
- Bolme, D., Beveridge, J.R., Draper, B.A., Lui, Y.M. , “ Visual Object Tracking using Adaptive Correlation Filters,” in 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, doi:10.1109/cvpr.2010.5539960.
- Kazemi, V., and Sullivan, J. , “One Millisecond Face Alignment with an Ensemble of Regression Trees,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, https://doi.org/10.1109/cvpr.2014.241.
- Xiong, X., and De La Torre, F. , “Supervised Descent Method and Its Applications to Face Alignment,” in 2013 IEEE Conference on Computer Vision and Pattern Recognition, 2013, https://doi.org/10.1109/cvpr.2013.75.
- Faithfull, W.J. , “Blink Detection Dataset,” 2015, https://will.faithfull.me/blinks-dataset/. Bangor University, School of Computer Science.
- Yang, S., Luo, P., Loy, C.-C., and Tang, X. , “From Facial Parts Responses to Face Detection: A Deep Learning Approach,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015.https://doi.org/10.1109/iccv.2015.419.
- Diederik, P., Jimmy, B.A. , “Adam : A Method for Stochastic Optimization,” arXiv:1412.6980 [cs.LG], December 2014.
- Chen, D., Ren, S., Wei, Y., Cao, X., and Sun, J. , “Joint Cascade Face Detection and Alignment,” Computer Vision - ECCV 2014 Lecture Notes in Computer Science 109-122, 2014, doi:https://doi.org/10.1007/978-3-319-10599-4_8.
- Asthana, A., Zafeiriou, S., Cheng, S., and Pantic, M. , “Incremental Face Alignment in the Wild,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, https://doi.org/10.1109/cvpr.2014.240.
- Abtahi, S., Omidyeganeh, M., Shirmohammadi, S., and Hariri, B. , YawDD: A Yawning Detection Dataset (Singapore: Proc. ACM Multimedia Systems), 24-28, doi:March 19-21 2014, 10.1145/2557642.2563678.