Driver drowsiness detection using facial dynamic fusion information and a DBN

Driver drowsiness detection using facial dynamic fusion information and a DBN

For access to this article, please select a purchase option:

Buy article PDF
(plus tax if applicable)
Buy Knowledge Pack
10 articles for £75.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Your details
Why are you recommending this title?
Select reason:
IET Intelligent Transport Systems — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

Driver drowsiness is a frequent cause of traffic accidents. Research on driver drowsiness detection methods is important to improve road traffic safety. Previous driving fatigue detection methods frequently extracted single features such as eye or mouth changes and trained shallow classifiers, which limit the generalisation capability of these methods. This study proposes a framework for recognising driver drowsiness expression by using facial dynamic fusion information and a deep belief network (DBN) to address the aforementioned problem. First, the landmarks and textures of the facial region are extracted from videos captured using a high-definition camera. Then, a DBN is built to classify facial drowsiness expressions. Finally, the authors’ method is tested on a driver drowsiness dataset, which includes different genders, ages, head poses and illuminations. Certain experiments are also carried out to investigate the effects of different facial subregions and temporal resolutions on the accuracy of driver fatigue recognition. Results demonstrate the validity of the proposed method, which has an average accuracy of 96.7%.


    1. 1)
      • 1. Rau, P.S.: ‘Drowsy driver detection and warning system for commercial vehicle drivers: field operational test design, data analyses, and progress’. National Highway Traffic Safety Administration, 2005, pp. 050192.
    2. 2)
      • 2. Bergasa, L.M., Nuevo, J., Sotelo, M.A., et al: ‘Real-time system for monitoring driver vigilance’, IEEE Trans. Intell. Transp. Syst., 2006, 7, (1), pp. 6377.
    3. 3)
      • 3. Sahayadhas, A., Sundaraj, K., Murugappan, M.: ‘Detecting driver drowsiness based on sensors: a review’, Sensors (Basel), 2012, 12, (12), pp. 1693716953.
    4. 4)
      • 4. Kim, J.Y., Jeong, C.H., Jung, M.J., et al: ‘Highly reliable driving workload analysis using driver electroencephalogram (EEG) activities during driving’, Int. J. Autom. Technol., 2013, 14, (6), pp. 965970.
    5. 5)
      • 5. Wang, M., Jeong, N., Kim, K., et al: ‘Drowsy behavior detection based on driving information’, Int. J. Autom. Technol., 2016, 17, (1), pp. 165173.
    6. 6)
      • 6. Dinges, D.F., Grace, R.: ‘PERCLOS: a valid psychophysiological measure of alertness as assessed by psychomotor vigilance’. Publication Number FHWA[AQ5] -MCRT-98–006, US Department of Transportation, Federal Highway Administration, 1998.
    7. 7)
      • 7. Liu, C., Wechsler, H.: ‘Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition’, IEEE Trans. Image Process., 2002, 11, (4), pp. 467476.
    8. 8)
      • 8. Rong-Ben, W., Ke-You, G., Shu-Ming, S., et al: ‘A monitoring method of driver fatigue behavior based on machine vision’. Proc. Int. Conf. Intelligent Vehicles Symp., Columbus, OH, USA, June 2003, pp. 110113.
    9. 9)
      • 9. Fan, X., Yin, B., Sun, Y.: ‘Nonintrusive driver fatigue detection’. Proc. IEEE Int. Conf. Networking, Sensing and Control, Sanya, China, May 2008, pp. 905910.
    10. 10)
      • 10. Kim, K.W., Lee, W.O., Kim, Y.G., et al: ‘Segmentation method of eye region based on fuzzy logic system for classifying open and closed eyes’, Opt. Eng., 2015, 54, (3), p. 033103.
    11. 11)
      • 11. Ibrahim, M.M., Soraghan, J.J., Petropoulakis, L.: ‘Eye-state analysis using an interdependence and adaptive scale mean shift (IASMS) algorithm’, Biomed. Signal Process. Control, 2014, 11, pp. 5362.
    12. 12)
      • 12. Song, F., Tan, X., Liu, X., et al: ‘Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients’, Pattern Recognit., 2014, 47, (9), pp. 28252838.
    13. 13)
      • 13. Cyganek, B., Gruszczyński, S.: ‘Hybrid computer vision system for drivers, eye recognition and fatigue monitoring’, Neurocomputing, 2014, 126, pp. 7894.
    14. 14)
      • 14. Lee, S.J., Jo, J., Jung, H.G., et al: ‘Real-time gaze estimator based on driver's head orientation for forward collision warning system’, IEEE Trans. Intell. Transp. Syst., 2011, 12, (1), pp. 254267.
    15. 15)
      • 15. Oyini Mbouna, R., Kong, S.G., Chun, M.-G.: ‘Visual analysis of eye state and head pose for driver alertness monitoring’, IEEE Trans. Intell. Transp. Syst., 2013, 14, (3), pp. 14621469.
    16. 16)
      • 16. Gu, H., Ji, Q.: ‘An automated face reader for fatigue detection’. Proc. Sixth IEEE Int. Conf. Automatic Face and Gesture Recognition, Seoul, South Korea, June 2004, pp. 111116.
    17. 17)
      • 17. Fan, X., Sun, Y., Yin, B., et al: ‘Gabor-based dynamic representation for human fatigue monitoring in facial image sequences’, Pattern Recognit. Lett., 2010, 31, (3), pp. 234243.
    18. 18)
      • 18. Zhao, C., Zhang, X., Lian, J., et al: ‘Driver's fatigue expressions recognition by combined features from pyramid histogram of oriented gradient and contourlet transform with random subspace ensembles’, IET Intell. Transp. Syst., 2013, 7, (1), pp. 3645.
    19. 19)
      • 19. Zhang, Y., Hua, C.: ‘Driver fatigue recognition based on facial expression analysis using local binary patterns’, Opt., Int. J. Light Electron Opt., 2015, 126, (23), pp. 45014505.
    20. 20)
      • 20. Zhao, C., Lian, J., Dang, Q., et al: ‘Classification of driver fatigue expressions by combined curvelet features and Gabor features, and random subspace ensembles of support vector machines’, J. Intell. Fuzzy Syst., 2014, 26, (1), pp. 91100.
    21. 21)
      • 21. Hinton, G.E., Salakhutdinov, R.R.: ‘Reducing the dimensionality of data with neural networks’, Science, 2006, 313, (5786), pp. 504507.
    22. 22)
      • 22. LeCun, Y., Bottou, L., Bengio, Y., et al: ‘Gradient-based learning applied to document recognition’, Proc. IEEE, 1998, 86, (11), pp. 22782324.
    23. 23)
      • 23. LeCun, Y., Bengio, Y., Hinton, G.: ‘Deep learning’, Nature, 2015, 521, (7553), pp. 436444.
    24. 24)
      • 24. Srivastava, N., Salakhutdinov, R.: ‘Learning representations for multimodal data with deep belief nets’. Proc. Int. Conf. Machine Learning Workshop, Edinburgh, Scotland, UK, July 2012.
    25. 25)
      • 25. Kuremoto, T., Kimura, S., Kobayashi, K., et al: ‘Time series forecasting using a deep belief network with restricted Boltzmann machines’, Neurocomputing, 2014, 137, pp. 4756.
    26. 26)
      • 26. Kang, S., Qian, X., Meng, H.: ‘Multi-distribution deep belief network for speech synthesis’. Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, May 2013, pp. 80128016.
    27. 27)
      • 27. Viola, P., Jones, M.J.: ‘Robust real-time face detection’, Int. J. Comput. Vis., 2004, 57, (2), pp. 137154.
    28. 28)
      • 28. Asthana, A., Zafeiriou, S., Cheng, S., et al: ‘Incremental face alignment in the wild’. Proc. IEEE Conf. Computer Vision and Pattern Recognition, Columbus, OH, USA, June 2014, pp. 18591866.
    29. 29)
      • 29. Ji, S., Xu, W., Yang, M., et al: ‘3D convolutional neural networks for human action recognition’, IEEE Trans. Pattern Anal. Mach. Intell., 2013, 35, (1), pp. 221231.
    30. 30)
      • 30. Sanin, A., Sanderson, C., Harandi, M.T., et al: ‘Spatio-temporal covariance descriptors for action and gesture recognition’. Proc. IEEE Workshop on Applications of Computer Vision, Tampa, FL, January 2013, pp. 103110.
    31. 31)
      • 31. Picot, A., Charbonnier, S., Caplier, A., et al: ‘Using retina modelling to characterize blinking: comparison between EOG and video analysis’, Mach. Vis. Appl., 2012, 23, (6), pp. 11951208.
    32. 32)
      • 32. Akrout, B., Mahdi, W.: ‘Spatio-temporal features for the automatic control of driver drowsiness state and lack of concentration’, Mach. Vis. Appl., 2015, 26, (1), pp. 113.

Related content

This is a required field
Please enter a valid email address