Authors

1. Ms. Manju Lata Joshi
2. Ms. Suhani Agarwal

Affiliation

• Department of CS & IT, International School of Informatics & Management, Jaipur
• Department of CS & IT, IIS (Deemed to be University), Jaipur

Abstract

Recognition of facial expression has been an important and exciting area of research in the past
decade. It is still challenging due to high intraclass variations like facial expression, body posture,
speech recognition, etc. There is an immense stipulation for facial expression acknowledgement in
different areas like medical services, educational institutes, business, entertainment, e-commerce,
health, and security. Face expressions play an essential role in identifying a person's emotions. To
detect or identify someone's emotions, traditional approaches such as Local Binary Pattern (LBP),
Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradient (HOG) rely on
handcrafted features, followed by a trained classifier on a database of photos or videos. The majority
of the research studies perform reasonably well on datasets of images collected under controlled
conditions but cannot perform well on more complex datasets with more image variation and partial
faces. In recent years several research studies have suggested an end to end system for Facial
Emotion Recognition (FER) using deep learning models. These studies have exhibited outstanding
performance and show that deep learning-based FER performs better than a traditional approach
based FER. This study provides an in-depth analysis of the existing research for facial emotion
recognition through deep learning. Firstly, existing studies based on traditional approaches are
analyzed, and then deep learning-based approaches used for FER are discussed.

Keywords

Emotion Recognition, Facial Expression, Facial Expression Recognition, Deep Learning

References

1. Zhang, Y., &Ji, Q. (2005). Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on pattern analysis and machine intelligence, 27(5), 699-714.
2. Ying, Z., Wang, Z., and Huang, M. (2010). Advanced intelligent computing theories and applications. With Aspects of Artificial Intelligence, Lecture Notes in Computer Science, 6216, (2010),457-464.
3. Viola, P., & Jones, M. J. (2004). Robust real-time face detection.International Journal of computer vision, 57(2), 137-154.
4. Uddin, M. Z., Hassan, M. M., Almogren, A., Zuair, M., Fortino, G., & Torresen, J. (2017). A facial expression recognition system using robust face features from depth videos and deep learning. Computers & Electrical Engineering, 63, 114-125.
5. Tian, Y. L., Kanade, T., & Cohn, J. F. (2005). Facial expression analysis. In Handbook of face recognition (pp. 247-275).Springer, New York, NY.
6. Tian, Y. I., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence, 23(2), 97-115.
7. Tarnowski, P., Kołodziej, M., Majkowski, A., & Rak, R. J. (2017).Emotion recognition using facial expressions.Procedia Computer Science, 108, 1175-1184.
8. Stolar, M. N., Lech, M., Bolia, R. S., & Skinner, M. (2017, December). Real-time speech emotion recognition using RGB image classification and transfer learning. In 2017 11th International Conference on Signal Processing and Communication Systems (ICSPCS) (pp. 1-8). IEEE.
9. Sown, M. (1978).A preliminary note on pattern recognition of facial emotional expression.In The 4th International Joint Conferences on Pattern Recognition, 1978.
10. Shan, C., Gong, S., & McOwan, P. W. (2009). Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 27(6), 803-816.
11. Singh P. &Kaur A. (2018).Deep Learning-based Face Emotion Recognition, JETIR.5(12).
12. Roopa, N. (2019). S “Emotion Recognition from Facial Expression using Deep Learning”. International Journal of Engineering and Advanced Technology (IJEAT) ISSN, 2249-8958.
13. Plutchik, R. E., & Conte, H. R. (1997).Circumplex models of personality and emotions (pp. xi- 484). American Psychological Association.
14. Pantic, M., &Rothkrantz, L. J. M. (2000). Automatic analysis of facial expressions: The state of the art. IEEE Transactions on pattern analysis and machine intelligence, 22(12), 1424-1445.
15. Mollahosseini, A., Chan, D., &Mahoor, M. H. (2016). Going deeper in facial expression recognition using deep neural networks.In 2016 IEEE Winter conference on applications of computer vision (WACV) (pp. 1-10). IEEE
16. Minaee, S., Minaei, M., &Abdolrashidi, A. (2021). Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors, 21(9), 3046.
17. Milborrow, Stephen &Nicolls, Fred. (2008). Locating Facial Features with an Extended Active Shape Model. Proc. of European Conf. on Computer Vision. 5305. 504-513. 10.1007/978-3-540- 88693-8_37.
18. Mahoor, M. H., Zhou, M., Veon, K. L., Mavadati, S. M., & Cohn, J. F. (2011). Facial action unit recognition with sparse representation.In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) (pp. 336-342).IEEE.
19. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The extended Cohn-Kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE computer society conference on computer vision and pattern recognition-workshops (pp. 94-101). IEEE.
20. Liu, W., & Wang, Z. (2006). Facial expression recognition based on fusion of multiple Gabor features. In 18th International Conference on Pattern Recognition (ICPR
21. Li, S., & Deng, W. (2020). Deep facial expression recognition: A survey. IEEE transactions on affective computing.
22. Li, C. E. J., & Zhao, L. (2019).Emotion recognition using convolutional neural networks.
23. Liliana, D. Y. (2019).Emotion recognition from facial expression using deep convolutional neural network. In Journal of physics: conference series (Vol. 1193, No. 1, p. 012004). IOP Publishing.
24. Khorrami, P., Paine, T., & Huang, T. (2015). Do deep neural networks learn facial action units when doing expression recognition?. In Proceedings of the IEEE international conference on computer vision workshops (pp. 19-27).
25. Kaliouby, R. E., & Robinson, P. (2005).Real-time inference of complex mental states from facial expressions and head gestures.In Real-time vision for human-computer interaction (pp. 181- 200).Springer, Boston, MA.
26. Jung, H., Lee, S., Park, S., Kim, B., Kim, J., Lee, I., &Ahn, C. (2015).Development of deep learning-based facial expression recognition system.In 2015 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV) (pp. 1-4). IEEE
27. Huang, K. Y., Wu, C. H., Yang, T. H., Su, M. H., & Chou, J. H. (2016). Speech emotion recognition using autoencoder bottleneck features and LSTM. The 2016 International Conference on Orange Technologies (ICOT) (pp. 1-4).IEEE.
28. Giannopoulos, P., Perikos, I., &Hatzilygeroudis, I. (2018). Deep learning approaches for facial emotion recognition: A case study on FER-2013. In Advances in the hybridization of intelligent methods (pp. 1-16).Springer, Cham.
29. Fasel, B. (2002, August). Robust face analysis using convolutional neural networks.In Object recognition supported by user interaction for service robots (Vol. 2, pp. 40-43). IEEE
30. Fasel, B., &Luettin, J. (2003). Automatic facial expression analysis: a survey. Pattern recognition, 36(1), 259-275.
31. Ekman, P., & Friesen, W. V. (1976). Measuring facial movement. Environmental psychology and nonverbal behaviour, 1(1), 56-75.
32. Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003). Facial expression recognition from video sequences: temporal and static modelling. Computer Vision and image understanding, 91(1-2), 160-187.
33. Clore, G. L., &Ortony, A. (2013).Psychological construction in the OCC model of emotion. Emotion Review, 5(4), 335-343.
34. Chen, J., Chen, D., Gong, Y., Yu, M., Zhang, K., & Wang, L. (2012). Facial expression recognition using geometric and appearance features. In Proceedings of the 4th international conference on internet multimedia computing and service (pp. 29-33).
35. Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2006). Fully automatic facial action recognition in spontaneous behaviour. In 7th International Conference on Automatic Face and Gesture Recognition (FGR06) (pp. 223-230). IEEE.
36. Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., &Movellan, J. (2005). Recognizing facial expression: machine learning and application to spontaneous behaviour. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR
37. Aneja, D., Colburn, A., Faigin, G., Shapiro, L., & Mones, B. (2016). Modelling stylized character expressions via deep learning. In Asian conference on computer vision (pp. 136-153).Springer, Cham.
38. Alhindi, T. J., Kalra, S., Ng, K. H., Afrin, A., &Tizhoosh, H. R. (2018). Comparing LBP, HOG and indepth features for classification of histopathology images. In 2018 joint international conference on neural networks (IJCNN) (pp. 1-7). IEEE.