This content is not included in your SAE MOBILUS subscription, or you are not logged in.
Drowsiness Detection Using Facial Expression Features
ISSN: 0148-7191, e-ISSN: 2688-3627
Published April 12, 2010 by SAE International in United States
Annotation ability available
This paper presents the method of detecting driver's drowsiness level from the facial expression. The motivation for this research is to realize the novel safety system which can detect the driver's slight drowsiness and keep the driver awake while driving.
The brain wave is commonly used as the drowsiness index. However, it is not suitable for the in-vehicle system since it is measured with sensors worn over the head. We precisely investigated the relationship between the change of brain wave and other drowsiness indices that can be measured without any contact; PERCLOS, heart rate, lane deviation, and facial expression. We found that the facial expression index had the highest linear correlation with the brain wave. Therefore, we selected the facial expression as the drowsiness-detection index and automated the drowsiness detection from the facial expression.
Three problems need to be solved for automation; (1) how to define the features of drowsy expression, (2) how to capture the features from the driver's video-recorded facial image, and (3) how to estimate the driver's drowsiness index from the features. First, we found that frontalis muscle, zygomaticus major muscle, and masseter muscle activated with increase of drowsiness in more than 75 percents of participants. According to the result, we determined the coordinates data of points on eyebrows, eyelids, and mouth as the features of drowsiness expression. Second, we calculated the 3D coordinates data of the features by image processing with Active Appearance Model (AAM). Third, we applied k-Nearest-Neighbor method to classify the driver's drowsiness level. Eleven participants' data of the features and the drowsiness level estimated by trained observers were used as the training data. We achieved the classification of the drivers' drowsiness in a driving simulator into 6 levels. The average Root Mean Square Errors (RMSE) among 12 participants was less than 1.0 level.
CitationHachisuka, S., Kimura, T., Ishida, K., Nakatani, H. et al., "Drowsiness Detection Using Facial Expression Features," SAE Technical Paper 2010-01-0466, 2010, https://doi.org/10.4271/2010-01-0466.
Intelligent Vehicle Initiative (IVI) Technology Advanced Controls and Navigation, 2010
Number: SP-2264; Published: 2010-04-13
Number: SP-2264; Published: 2010-04-13
- U.S. Department of Transportation “Saving lives through advanced safety technology. Intelligent Vehicle Initiative 2002 annual report,” http://www.itsdocs.fhwa.dot.gov//JPODOCS/REPTS_TE//13821.pdf May 2003
- The Ministry of Economy, Trade and Industry “Technological Strategy Map 2009,” http://www.meti.go.jp/policy/economy/gijutsu_kakushin/kenkyu_kaihatu/str2009/7_1.pdf April 2009
- Wierwille W. W. Ellsworth L. A. Wreggit S. S. Fairbanks R. J. Kirn C. L. “Research on Vehicle-Based Driver Status/Performance Monitoring; Development, Validation, and Refinement of Algorithms For Detection of Driver Drowsiness,” National Highway Traffic Safety Administration Final Report: DOT HS 808 247 1994
- Ishida Kenji Ito Akiko Kimura Teiyuu “A Study of Feature Factors on Sleepy Expressions based on Observational Analysis of Facial Images,” Transactions of Society of Automotive Engineers of Japan 39 3 251 256 2008
- Eoh Hong J. Chung Min K. Kim Seong-Han “Electroencephalographic study of drowsiness in simulated driving with sleep deprivation,” International Journal of Industrial Ergonomics 35 307 320 2005
- Saper Clifford B. Chou Thomas C. Scammell Thomas E. “The sleep switch: hypothalamic control of sleep and wakefulness,” TRENDS in Neurosciences 24 12 726 731 2001
- Kitajima Hiroki Numata Nakaho Yamamoto Keiichi Goi Yoshihiro “Prediction of Automobile Driver Sleepiness (1st Report, Rating of Sleepiness Based on Facial Expression and Examination of Effective Predictor Indexes of Sleepiness),” The Japan Society of Mechanical Engineers Journal 63 613 93 100 1997
- Sato Shuji Taoda Kazushi Kawamura Masanori Wakaba Kinzou Fukuchi Yasuma Nishiyama Katsuo “Heart rate variability during long truck driving work,” Journal of Human Ergology 30 235 240 2001
- Kozak Ksenia Pohl Jochen Birk Wolfgang Greenberg Jeff Artz Bruce Blommer Mike Cathey Larry Curry Reates “Evaluation of lane departure warnings for drowsy drivers,” Proceedings of the Human Factors and Ergonomics Society 50th annual meeting: 2400-2404 2006
- Liu Charles C. Hosking Simon G. Lenne Michael G. “Predicting driver drowsiness using vehicle measures: Recent insights and future challenges,” Journal of Safety Research 40 239 245 2009
- Ueno A. Manabe S. Uchikawa Y. “Acoustic Feedback System with Digital Signal Processor to Alert the Subject and Quantitative Visualization of Arousal Reaction Induced by the Sound Using Dynamic Characteristics of Saccadic Eye Movement: A Preliminary Study,” Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference 6149 6152 2005
- Bergasa Luis M. Nuevo Jesus Sotelo Miguel A. Barea Rafael Lopez Maria Elena “Real-Time System for Monitoring Driver Vigilance,” IEEE Transactions on Intelligent Transportation Systems 7 1 63 77 2006
- Ji Qiang Zhu Zhiwei Lan Peilin “Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue,” IEEE Transactions on Vehicular Technology 53 4 1052 1068 2004
- Arimitsu Satori Sasaki Ken Hosaka Hiroshi Itoh Michimasa Ishida Kenji Ito Akiko “Seat Belt Vibration as a Stimulating Device for awakening Drivers,” IEEE/ASME Transactions on Mechatronics 12 5 511 518 2007
- Ishida Kenji Hachisuka Satori Kimura Teiyuu Kamijo Masayoshi “Comparing Trends of Sleepiness Expressions Appearance with Performance and Physiological Change Caused by Arousal Level Declining,” Transactions of Society of Automotive Engineers of Japan 40 3 885 890 2009
- Ishida Kenji Ichimura Asami Kamijo Masayoshi “A Study of Facial Muscular Activities in Drowsy Expression,” Kansei Engineering International Journal 9 2 2010
- Kimura Teiyuu Ishida Kenji Ozaki Noriyuki “Feasibility Study of Sleepiness Detection Using Expression Features,” Review of Automotive Engineering 29 4 567 574 2008
- Cootes Timothy F. Edwards Gareth J. Taylor Christopher J. “Active appearance models,” IEEE Transactions on Pattern Analysis and Machine Intelligence 23 6 681 685 2001
- Vural Esra Cetin Mujdat Ercil Aytul Littlewort Gwen Bartlett Marian Movellan Javier “Automated Drowsiness Detection For Improved Driving Safety,” Proc. 4th International conference on Automotive Technologies Turkey Nov. 13-14 2008