Pain Facial Expression Recognition from Video Sequences Using Spatio-temporal Local Binary Patterns and Tracking Fiducial Points

Document Type : Original Article

Authors

1 Computer Engineering & IT Department, Shahrood University of Technology, Shahrood, Iran

2 Department of Strategic Management, Bank Melli Iran, Tehran, Iran

3 Faculty of Engineering, East Guilan, University of Guilan, Guilan, Iran

Abstract

Monitoring the facial expressions of patients in clinical environments is a necessity in addition to vital sign monitoring. Pain monitoring of patients by facial expressions from video sequences eliminates the need for another person to accompany patients. In this paper, a novel approach is presented to monitor the expression of face and notify in case of pain using tracking fiducial points of face in video sequences and spatio-temporal Local Binary Patterns (LBPs) for eyes and eyebrows. The motion of eight fiducial points on facial features such as mouth, eyes, eyebrows are tracked by Lucas-Kanade algorithm and the movement angles are recorded in a feature vector which along with the spatio-temporal histogram of LBPs creates a concatenated feature vector. Spatio-temporal LBPs boost the proposed algorithm to capture minor deformations on eyes and eyebrows. The feature vectors are then compared and classified using the Chi-square similarity measure. Experimental results show that leveraging spatio-temporal LBPs improves the accuracy by 12% on STOIC database.

Keywords



1. Hammal, Z. and Kunz, M., "Pain monitoring: A dynamic and
context-sensitive system", Pattern Recognition,  Vol. 45, No. 4,
(2012), 1265-1280. 
2. Littlewort, G.C., Bartlett, M.S. and Lee, K., "Faces of pain:
Automated measurement of spontaneousallfacial expressions of
genuine and posed pain", in Proceedings of the 9th international
conference on Multimodal interfaces. (2007), 15-21. 
3. Monwar, M.M. and Rezaei, S., "Pain recognition using artificial
neural network", in 2006 IEEE International Symposium on 
Signal Processing and Information Technology, IEEE., (2006),
28-33. 
4. Y.-I., T., T., K. and J.F., C., "Recognizing action units for facial
expression analysis", IEEE Transactions on Pattern Analysis
and Machine Intelligence,  Vol. 23, (2001), 97-115. 
5. Tian, Y.-l., "Evaluation of face resolution for expression
analysis", in 2004 Conference on Computer Vision and Pattern
Recognition Workshop, IEEE. (2004), 82-82. 
6. Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I.
and Movellan, J., "Recognizing facial expression: Machine
learning and application to spontaneous behavior", in 2005 IEEE
Computer Society Conference on Computer Vision and Pattern
Recognition (CVPR'05), IEEE. Vol. 2, (2005), 568-573. 
7. Ira, C., Nicu, S., Ashutosh, G., Lawrence S., C. and Thomas S.,
H., "Facial expression recognition from video sequences:
Temporal and static modeling", Computer Vision and Image
Understanding,  Vol. 91, No., (2003), 160-187. 
8. M, P. and L.J.M, R., "Expert system for automatic analysis of
facial expressions", Image and Vision Computing,  Vol. 18,
(2000), 881-905. 
9. M., P. and L.J.M., R., "Facial action recognition for facial
expression analysis from static face images", IEEE Transactions
on Systems, Man and Cybernetics, Part B (Cybernetics),  Vol.
34, (2004), 1449-1461. 
10. M., P. and I., P., "Dynamics of facial expression: Recognition of
facial actions and their temporal segments from face profile image
sequences", IEEE Transactions on Systems, Man and
Cybernetics, Part B (Cybernetics),  Vol. 36, (2006), 433-449. 
11. Y., Z., L.C., D.S. and C.C., K., "Using moment invariants and
hmm in facial expression recognition", Pattern Recognition
Letters,  Vol. 23, (2002), 83-91. 
12. Bumhwi, K., Sang-Woo, B. and Minho, L., Improving adaboost
based face detection using face-color preferable selective
attention. 2008, Springer Berlin Heidelberg.88-95. 
13. Irene, K., Stefanos, Z. and Ioannis, P., "Texture and shape
information fusion for facial expression and facial action unit
recognition", Pattern Recognition,  Vol. 41, (2008), 833-851. 
14. Caifeng, S., Shaogang, G. and Peter W., M., "Facial expression
recognition based on local binary patterns: A comprehensive
study", Image and Vision Computing,  Vol. 27, (2009), 803-816. 
15. Gwen C., L., Marian Stewart, B. and Kang, L., "Automatic coding
of facial expressions displayed during posed and genuine pain",
Image and Vision Computing,  Vol. 27, (2009), 1797-1803. 
16. Zahedi, M. and Firouzian, I., "Pain facial expression recognition
among 8 facial expressions using similarity measures", in
Machine Vision and Image Processing (MVIP), Zanjan, Iran.,
(2013). 
17. Y., Y. and L.S., D., "Recognizing human facial expressions from
long image sequences using optical flow", IEEE Transactions on
Pattern Analysis and Machine Intelligence,  Vol. 18, (1996),
636-642. 
18. I.A., E. and A.P., P., "Coding, analysis, interpretation, and
recognition of facial expressions", IEEE Transactions on
Pattern Analysis and Machine Intelligence,  Vol. 19, (1997),
757-763. 
19. Black, M.J. and Yacoob, Y., "Recognizing facial expressions in
image sequences using local parameterized models of image
motion", International Journal of Computer Vision,  Vol. 25,
No. 1, (1997), 23-48. 
20. Essa, I.A. and Pentland, A.P., "Coding, analysis, interpretation,
and recognition of facial expressions", IEEE Transactions on
Pattern Analysis and Machine Intelligence,  Vol. 19, No. 7,
(1997), 757-763. 
 
21. M., Y., B., B. and R., S., "From facial expression to level of
interest: A spatio-temporal approach, IEEE. 
22. J., H. and J.J., L., "Value directed learning of gestures and facial
displays, IEEE.  
23. R., E.K. and P., R., "Real-time inference of complex mental states
from facial expressions and head gestures, IEEE.  
24. , Y.Z. and , Q.J., "Active and dynamic information fusion for
facial expression understanding from image sequences", IEEE
Transactions on Pattern Analysis and Machine Intelligence, 
Vol. 27, No., (2005), 699-714. 
25. Firouzian, I. and Firouzian, N., "Face recognition by cognitive
discriminant features", International Journal of Nonlinear
Analysis and Applications,  Vol. 11, No. 1, (2020), 7-20. 
26. Chang, Y., Hu, C. and Turk, M., "Probabilistic expression
analysis on manifolds", in Proceedings of the 2004 IEEE
Computer Society Conference on Computer Vision and Pattern
Recognition, 2004. CVPR 2004., IEEE. Vol. 2, (2004), II-II. 
27. José M., B., Enrique, M. and Luis, B., "Recognising facial
expressions in video sequences", Pattern Analysis and
Applications,  Vol. 11, (2007), 101-116. 
28. Guoying, Z. and Matti, P., "Boosted multi-resolution
spatiotemporal descriptors for facial expression recognition",
Pattern Recognition Letters,  Vol. 30, (2009), 1117-1127. 
29. Zheng, Z., Zheng, Z. and Tiantian, Y., Expression recognition
based on multi-scale block local gabor binary patterns with
dichotomy-dependent weights. 2009, Springer Berlin
Heidelberg.895-903. 
30. Stephen, M. and Richard, B., "The effects of pose on facial
expression recognition, British Machine Vision Association.
(2009). 
31. Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Chew, S.
and Matthews, I., "Painful monitoring: Automatic pain
monitoring using the unbc-mcmaster shoulder pain expression
archive database", Image and Vision Computing,  Vol. 30, No.
3, (2012), 197-205. 
32. Werner, P., Al-Hamadi, A., Niese, R., Walter, S., Gruss, S. and
Traue, H.C., "Towards pain monitoring: Facial expression, head
pose, a new database, an automatic system and remaining
challenges", in Proceedings of the British Machine Vision
Conference. (2013), 1-13. 
33. Hashemi, S.M.R. and Faridpour, M., "Evaluation of the
algorithms of face identification", in 2015 2nd International 
Conference on Knowledge-Based Engineering and Innovation
(KBEI), IEEE. (2015), 1049-1052. 
34. Bartlett, M.S., Littlewort, G.C., Frank, M.G. and Lee, K.,
"Automatic decoding of facial movements reveals deceptive pain
expressions", Current Biology,  Vol. 24, No. 7, (2014), 738-743. 
35. Werner, P., Al-Hamadi, A. and Niese, R., "Comparative learning
applied to intensity rating of facial expressions of pain", 
International Journal of Pattern Recognition and Artificial
Intelligence,  Vol. 28, No. 05, (2014), 1451008. 
36. Sikka, K., Ahmed, A.A., Diaz, D., Goodwin, M.S., Craig, K.D.,
Bartlett, M.S. and Huang, J.S., "Automated assessment of
children’s postoperative pain using computer vision", Pediatrics, 
Vol. 136, No. 1, (2015), e124-e131. 
37. Liu, D., Peng, F., Shea, A. and Picard, R., "Deepfacelift:
Interpretable personalized models for automatic estimation of
self-reported pain", arXiv preprint arXiv:1708.04670,  (2017). 
38. Gruss, S., Geiger, M., Werner, P., Wilhelm, O., Traue, H.C., AlHamadi,
A.
and
Walter,
S.,
"Multi-modal
signals for analyzing
pain responses to thermal and electrical stimuli", JoVE (Journal
of Visualized Experiments),  No. 146, (2019), e59057. 
39. O'Neill, M.C., Ahola Kohut, S., Pillai Riddell, R. and Oster, H.,
"Ageā€related differences in the acute pain facial expression during
infancy", European Journal of Pain,  Vol. 23, No. 9, (2019),
1596-1607. 
40. Virrey, R.A., Liyanage, C.D.S., Petra, M.I.b.P.H. and Abas, P.E.,
"Visual data of facial expressions for automatic pain detection",
Journal of Visual Communication and Image Representation, 
Vol. 61, (2019), 209-217. 
41. Zamzmi, G., Paul, R., Goldgof, D., Kasturi, R. and Sun, Y., "Pain
assessment from facial expression: Neonatal convolutional neural
network (n-cnn)", in 2019 International Joint Conference on
Neural Networks (IJCNN), IEEE. (2019), 1-7. 
42. Roy, S., Roy, C., Éthier-Majcher, C., Fortin, I., Belin, P. and
Gosselin, F., "Stoic: A database of dynamic and static faces
expressing highly recognizable emotions", J. Vis,  Vol. 7, (2007),
944. 
43. Lucas, B.D. and Kanade, T., "An iterative image registration
technique with an application to stereo vision",  (1981). 
44. Pietikäinen, M., Hadid, A., Zhao, G. and Ahonen, T., Local binary
patterns for still images, in Computer vision using local binary
patterns. 2011, Springer.13-47.