- Marian Bartlett23,
- Gwen Littlewort23,
- Esra Vural23,25,
- Kang Lee24,
- Mujdat Cetin25,
- Aytul Ercil25 &
- …
- Javier Movellan23
Part of the book series:Lecture Notes in Computer Science ((LNAI,volume 5042))
1139Accesses
Abstract
The computer vision field has advanced to the point that we are now able to begin to apply automatic facial expression recognition systems to important research questions in behavioral science. The machine perception lab at UC San Diego has developed a system based on machine learning for fully automated detection of 30 actions from the facial action coding system (FACS). The system, called Computer Expression Recognition Toolbox (CERT), operates in real-time and is robust to the video conditions in real applications. This paper describes two experiments which are the first applications of this system to analyzing spontaneous human behavior: Automated discrimination of posed from genuine expressions of pain, and automated detection of driver drowsiness. The analysis revealed information about facial behavior during these conditions that were previously unknown, including the coupling of movements. Automated classifiers were able to differentiate real from fake pain significantly better than naïve human subjects, and to detect critical drowsiness above 98% accuracy. Issues for application of machine learning systems to facial expression analysis are discussed.
This is a preview of subscription content,log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek, C., Fasel, I., Movellan, J.R.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1(6), 22–35 (2006)
Cobb, W.: Recommendations for the practice of clinical neurophysiology. Elsevier, Amsterdam (1983)
Cohn, J.F., Schmidt, K.L.: The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution & Information Processing 2(2), 121–132 (2004)
Craig, K.D., Hyde, S., Patrick, C.J.: Genuine, supressed, and faked facial behaviour during exacerbation of chronic low back pain. Pain 46, 161–172 (1991)
Craig, K.D., Patrick, C.J.: Facial expression during induced pain. J Pers Soc Psychol. 48(4), 1080–1091 (1985)
Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying facial actions. IEEE Trans. Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)
DOT, Saving lives through advanced vehicle safety technology. USA Department of Transportation. (2001),http://www.its.dot.gov/ivi/docs/AR2001.pdf
Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)
Ekman, P.: Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W.W. Norton, New York (2001)
Ekman, P., Rosenberg, E.L. (eds.): What the face reveals: Basic and applied studies of spontaneous expression using the FACS. Oxford University Press, Oxford (2005)
Fasel, I., Fortenberry, B., Movellan, J.R.: A generative framework for real-time object detection and classification. Computer Vision and Image Understanding 98 (2005)
Fishbain, D.A., Cutler, R., Rosomoff, H.L., Rosomoff, R.S.: Chronic pain disability exaggeration/malingering and submaximal effort research. Clin J Pain 15(4), 244–274 (1999)
Fishbain, D.A., Cutler, R., Rosomoff, H.L., Rosomoff, R.S.: Accuracy of deception judgments. Pers Soc Psychol Rev. 10(3), 214–234 (2006)
Frank, M.G., Ekman, P., Friesen, W.V.: Behavioral markers and recognizability of the smile of enjoyment. J Pers Soc Psychol. 64(1), 83–93 (1993)
Grossman, S., Shielder, V., Swedeen, K., Mucenski, J.: Correlation of patient and caregiver ratings of cancer pain. Journal of Pain and Symptom Management 6(2), 53–57 (1991)
Gu, H., Ji, Q.: An automated face reader for fatigue detection. In: FGR, pp. 111–116 (2004)
Gu, H., Zhang, Y., Ji, Q.: Task oriented facial behavior recognition with selective sensing. Comput. Vis. Image Underst. 100(3), 385–415 (2005)
Hadjistavropoulos, H.D., Craig, K.D., Hadjistavropoulos, T., Poole, G.D.: Subjective judgments of deception in pain expression: accuracy and errors. Pain 65(2-3), 251–258 (1996)
Hill, M.L., Craig, K.D.: Detecting deception in pain expressions: the structure of genuine and deceptive facial displays. Pain 98(1-2), 135–144 (2002)
Hong, C.K.: Electroencephalographic study of drowsiness in simulated driving with sleep deprivation. International Journal of Industrial Ergonomics 35(4), 307–320 (2005)
Igarashi, K., Takeda, K., Itakura, F., Abut, H.: DSP for In-Vehicle and Mobile Systems. Springer, US (2005)
Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the fourth IEEE International conference on automatic face and gesture recognition (FG 2000), Grenoble, France, pp. 46–53 (2000)
Larochette, A.C., Chambers, C.T., Craig, K.D.: Genuine, suppressed and faked facial expressions of pain in children. Pain 126(1-3), 64–71 (2006)
Littlewort, G., Bartlett, M.S., Fasel, I., Susskind, J., Movellan, J.: Dynamics of facial expression extracted automatically from video. J. Image & Vision Computing 24(6), 615–625 (2006)
Morecraft, R.J., Louie, J.L., Herrick, J.L., Stilwell-Morecraft, K.S.: Cortical innervation of the facial nucleus in the non-human primate: a new interpretation of the effects of stroke and related subtotal brain trauma on the muscles of facial expression. Brain 124(Pt 1), 176–208 (2001)
Orden, K.F.V., Jung, T.P., Makeig, S.: Combined eye activity measures accurately estimate changes in sustained visual task performance. Biological Psychology 52(3), 221–240 (2000)
Pantic, M., Pentland, A., Nijholt, A., Huang, T.: Human Computing and machine understanding of human behaviour: A Survey. In: Proc. ACM Int’l Conf. Multimodal Interfaces, pp. 239–248 (2006)
Pantic, M.F.V., Rademaker, R., Maat, L.: Web- based Database for Facial Expression Analysis. In: Proc. IEEE Int’l Conf. Multmedia and Expo (ICME 2005), Amsterdam, The Netherlands (July 2005)
Prkachin, K.M.: The consistency of facial expressions of pain: a comparison across modalities. Pain 51(3), 297–306 (1992)
Prkachin, K.M., Schultz, I., Berkowitz, J., Hughes, E., Hunt, D.: Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav Res Ther. 40(5), 595–607
Rinn, W.E.: The neuropsyhology of facial expression: a review of the neurological and psychological mechanisms for producing facial expression. Psychol Bull 95, 52–77
Schmand, B., Lindeboom, J., Schagen, S., Heijt, R., Koene, T., Hamburger, H.L.: Cognitive complaints in patients after whiplash injury: the impact of malingering. J Neurol Neurosurg Psychiatry 64(3), 339–343
Schmidt, K.L., Cohn, J.F., Tian, Y.: Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol. 65(1), 49–66 (2003)
Schneiderman, H., Kanade, T.: Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 45–51 (1998)
Takei, Y., Furukawa, Y.: Estimate of driver’s fatigue through steering motion. In: Man and Cybernetics, 2005 IEEE International Conference, vol. 2, pp. 1765–1770 (2005)
Viola, P., Jones, M.: Robust real-time face detection. J. Computer Vision 57(2), 137–154 (2004)
Vural, E., Ercil, A., Littlewort, G.C., Bartlett, M.S., Movellan, J.R.: Machine learning systems for detecting driver drowsiness. In: Proceedings of the Biennial Conference on Digital Signal Processing for in-Vehicle and Mobile Systems (2007)
Zhang, Z., Shu Zhang, J.: Driver fatigue detection based intelligent vehicle control. In: Proceedings of the 18th International Conference on Pattern Recognition, Washington, DC, USA, pp. 1262–1265. IEEE Computer Society, Los Alamitos (2006)
Author information
Authors and Affiliations
Institute for Neural Computation, University of California, San Diego, La Jolla, CA 92093-0445, USA
Marian Bartlett, Gwen Littlewort, Esra Vural & Javier Movellan
Human Development and Applied Psychology, University of Toronto, Ontario, Canada
Kang Lee
Engineering and Natural Science, Sabanci University, Istanbul, Turkey
Esra Vural, Mujdat Cetin & Aytul Ercil
- Marian Bartlett
You can also search for this author inPubMed Google Scholar
- Gwen Littlewort
You can also search for this author inPubMed Google Scholar
- Esra Vural
You can also search for this author inPubMed Google Scholar
- Kang Lee
You can also search for this author inPubMed Google Scholar
- Mujdat Cetin
You can also search for this author inPubMed Google Scholar
- Aytul Ercil
You can also search for this author inPubMed Google Scholar
- Javier Movellan
You can also search for this author inPubMed Google Scholar
Editor information
Editors and Affiliations
Department of Psychology, Second University of Naples, and IIASS, Via Pellegrino 19, 84019, Vietri sul Mare (SA), Italy
Anna Esposito
ATRC Center, Wright State University, Dayton, OH, USA
Nikolaos G. Bourbakis
Human Computer Interaction Group, University of Patras, Rio Patras, Greece
Nikolaos Avouris
Department of Computer Engineering, University of Patras, Patras, Greece
Ioannis Hatzilygeroudis
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bartlett, M.et al. (2008). Data Mining Spontaneous Facial Behavior with Automatic Expression Coding. In: Esposito, A., Bourbakis, N.G., Avouris, N., Hatzilygeroudis, I. (eds) Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction. Lecture Notes in Computer Science(), vol 5042. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-70872-8_1
Download citation
Publisher Name:Springer, Berlin, Heidelberg
Print ISBN:978-3-540-70871-1
Online ISBN:978-3-540-70872-8
eBook Packages:Computer ScienceComputer Science (R0)
Share this paper
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative