Real Time Emotion Recognition with AD8232 ECG Sensor for Classwise Performance Evaluation of Machine Learning Methods

Document Type : Original Article

Authors

1 Department of Electronics & Telecommunication Engineering, AISSMS Institute of Information Technology, Pune, India

2 Department of Electronics & Telecommunication Engineering, Bharti Vidyapeeth College of Engineering, Pune, India

Abstract

Emotions are the accelerators of human intellect and innovation and creativity, so the ability to recognize emotions is in high demand. Real-time hardware has hurdles of Noise and hardware factors as compared to simulations. An electrocardiogram (ECG) sensor (AD8232), a temperature sensor (LM35), and a signal processing circuit is  hardware of the proposed real-time emotion identification. The RR intervals are calculated from the ECG data. Emotions prediction using machine learning makes use of RR intervals and body temperature as features. One of the four emotions (namely 1. Happy 2. Stressed 3. Neutral 4. Sad.) is visualized at the serial port of the processor by using WESAD benchmark dataset and the HRV, serial, and pickle libraries.This article's innovation factors are (1) Use of ECG for emotion detection rather than disease detection with Emotion induction method, RR interval capturing and design of RR interval GUI for real time capture of temperature and ECG (2) Display of current emotion on Arduino serial port. (3) Measurement of Class performance using F1 score, macro average, and weighted average instead of general term accuracy. (4) Use of the probability based  Navies Bayes as compared to traditional KNN, SVM, Random Forest nethods  (5) Class wise performance for example Navies Bayes' specificity or accuracy is lower than SVM's (0.96), but its recall or sensitivity is higher (0.97) vs. (0.94) for stress.In this article, we presented performance parameters in terms of interactive computations, tabular form and graphical display. 

Keywords

Main Subjects


  1. Lewis, K., "The influence of emotions on our shopping habits", Siegel Institute Ethics Research Scholars, Vol. 1, No. 1, (2017), 3. https://digitalcommons.kennesaw.edu/siers/vol1/iss1/3
  2. Brundin, E., Liu, F. and Cyron, T., "Emotion in strategic management: A review and future research agenda", Long Range Planning, Vol. 55, No. 4, (2022), 102144. https://doi.org/10.1016/j.lrp.2021.102144
  3. Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X. and Yang, X., "A review of emotion recognition using physiological signals", Sensors, Vol. 18, No. 7, (2018), 2074. https://doi.org/10.1016/j.lrp.2021.102144
  4. Dzedzickis, A., Kaklauskas, A. and Bucinskas, V., "Human emotion recognition: Review of sensors and methods", Sensors, Vol. 20, No. 3, (2020), 592. https://doi.org/10.3390/s20030592
  5. Basharirad, B. and Moradhaseli, M., "Speech emotion recognition methods: A literature review", in AIP Conference Proceedings, AIP Publishing LLC. Vol. 1891, (2017), 020105.
  6. Hui, T.K. and Sherratt, R.S., "Coverage of emotion recognition for common wearable biosensors", Biosensors, Vol. 8, No. 2, (2018), 30. doi: 10.3390/bios8020030.
  7. Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M. and Pollak, S.D., "Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements", Psychological Science in the Public Interest, Vol. 20, No. 1, (2019), 1-68. https://doi.org/10.1177/1529100619832930
  8. Fernández-Caballero, A., Martínez-Rodrigo, A., Pastor, J.M., Castillo, J.C., Lozano-Monasor, E., López, M.T., Zangróniz, R., Latorre, J.M. and Fernández-Sotos, A., "Smart environment architecture for emotion detection and regulation", Journal of Biomedical Informatics, Vol. 64, (2016), 55-73. doi: 10.1016/j.jbi.2016.09.015.
  9. Shu, L., Yu, Y., Chen, W., Hua, H., Li, Q., Jin, J. and Xu, X., "Wearable emotion recognition using heart rate data from a smart bracelet", Sensors, Vol. 20, No. 3, (2020), 718. https://doi.org/10.3390/s20030718
  10. Roshani, S., Coccia, M. and Mosleh, M., "Sensor technology for opening new pathways in diagnosis and therapeutics of breast, lung, colorectal and prostate cancer", medRxiv, Vol., No., (2022), 2022.2002. 2018.22271186. doi: 10.28991/HIJ-2022-03-03-010
  11. Egger, M., Ley, M. and Hanke, S., "Emotion recognition from physiological signal analysis: A review", Electronic Notes in Theoretical Computer Science, Vol. 343, (2019), 35-55. doi: 10.1016/j.entcs.2019.04.009.
  12. Schmidt, P., Reiss, A., Duerichen, R., Marberger, C. and Van Laerhoven, K., "Introducing wesad, a multimodal dataset for wearable stress and affect detection", in Proceedings of the 20th ACM international conference on multimodal interaction. (2018), 400-408.
  13. Goshvarpour, A., Abbasi, A. and Goshvarpour, A., "An accurate emotion recognition system using ecg and gsr signals and matching pursuit method", Biomedical journal, Vol. 40, No. 6, (2017), 355-368.
  14. Valderas, M.T., Bolea, J., Laguna, P., Vallverdú, M., Bailón, R. , "Human emotion recognition using heart rate variability analysis with spectral bands based on respiration", in In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Vol. 343, (2015 of Conference), 6134-6137.
  15. Dissanayake, T., Rajapaksha, Y., Ragel, R. and Nawinne, I., "An ensemble learning approach for electrocardiogram sensor based human emotion recognition", Sensors, Vol. 19, No. 20, (2019), 4495. https://doi.org/10.3390/s19204495
  16. Rodríguez, R., Mexicano, A., Bila, J., Cervantes, S. and Ponce, R., "Feature extraction of electrocardiogram signals by applying adaptive threshold and principal component analysis", Journal of applied research and technology, Vol. 13, No. 2, (2015), 261-269. doi: 10.1016/j.jart.2015.06.008.
  17. Manullang, M.C.T., Simanjuntak, J. and Ramdani, A.L., "Implementation of ad8232 ecg signal classification using peak detection method for determining rst point", Indonesian Journal of Artificial Intelligence and Data Mining, Vol. 2, No. 2, (2019), 61-66. doi: 10.24014/ijaidm.v2i2.7593.
  18. Dhuheir, M., Albaseer, A., Baccour, E., Erbad, A., Abdallah, M. and Hamdi, M., "Emotion recognition for healthcare surveillance systems using neural networks: A survey", in 2021 International Wireless Communications and Mobile Computing (IWCMC), IEEE. (2021), 681-687.
  19. Patil, V.K. and Pawar, V.R., "How can emotions be classified with ecg sensors, ai techniques and iot setup?", in 2022 International Conference on Signal and Information Processing (IConSIP), IEEE. (2022), 1-6.
  20. Londhe, S. and Borse, R., "Emotion recognition based on various physiological signals-a review", ICTACT Journal on Communication Technology, Vol. 9, No. 3, (2018), 1815-1822. doi: 10.21917/ijct.2018.0265.
  21. Haag, A., Goronzy, S., Schaich, P. and Williams, J., "Emotion recognition using bio-sensors: First steps towards an automatic system", in Affective Dialogue Systems: Tutorial and Research Workshop, ADS 2004, Kloster Irsee, Germany, June 14-16, 2004. Proceedings, Springer., (2004), 36-48.
  22. Butkevičiūtė, E., Michalkovič, A. and Bikulčienė, L., "Ecg signal features classification for the mental fatigue recognition", Mathematics, Vol. 10, No. 18, (2022), 3395. https://doi.org/10.3390/math10183395
  23. Sun, B. and Lin, Z., "Emotion recognition using machine learning and ecg signals", arXiv preprint arXiv:2203.08477, (2022). https://doi.org/10.48550/arXiv.2203.08477
  24. Kumar, S. and Sahoo, G., "A random forest classifier based on genetic algorithm for cardiovascular diseases diagnosis (research note)", International Journal of Engineering, Transactions B: Applications, Vol. 30, No. 11, (2017), 1723-1729. doi: 10.5829/ije.2017.30.11b.13.
  25. Hasnul, M.A., Aziz, N.A.A., Alelyani, S., Mohana, M. and Aziz, A.A., "Electrocardiogram-based emotion recognition systems and their applications in healthcare—a review", Sensors, Vol. 21, No. 15, (2021), 5015. https://doi.org/10.3390/s21155015
  26. Bravo-Zanoguera, M., Cuevas-González, D., Reyna, M.A., García-Vázquez, J.P. and Avitia, R.L., "Fabricating a portable ecg device using ad823x analog front-end microchips and open-source development validation", Sensors, Vol. 20, No. 20, (2020), 5962. https://doi.org/10.3390/s20205962
  27. Shin, D., Shin, D. and Shin, D., "Development of emotion recognition interface using complex eeg/ecg bio-signal for interactive contents", Multimedia Tools and Applications, Vol. 76, (2017), 11449-11470. https://doi.org/10.1007/s11042-016-4203-7
  28. Bulagang, A.F., Weng, N.G., Mountstephens, J. and Teo, J., "A review of recent approaches for emotion classification using electrocardiography and electrodermography signals", Informatics in Medicine Unlocked, Vol. 20, (2020), 100363. https://doi.org/10.1016/j.imu.2020.100363
  29. Xianhai, G., "Study of emotion recognition based on electrocardiogram and rbf neural network", Procedia Engineering, Vol. 15, (2011), 2408-2412. https://doi.org/10.1016/j.proeng.2011.08.452
  30. Zenonos, A., Khan, A., Kalogridis, G., Vatsikas, S., Lewis, T. and Sooriyabandara, M., "Healthyoffice: Mood recognition at work using smartphones and wearable sensors", in 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops), IEEE. (2016), 1-6.
  31. Patil, V.K., Hadawale, O., Pawar, V.R. and Gijre, M., "Emotion linked aiot based cognitive home automation system with sensovisual method", in 2021 IEEE Pune Section International Conference (PuneCon), IEEE. (2021), 1-7.
  32. Zhang, Z., Wang, X., Li, P., Chen, X. and Shao, L., "Research on emotion recognition based on ecg signal", in Journal of Physics: Conference Series, IOP Publishing. Vol. 1678, (2020), 012091.
  33. Emami, N. and Kuchaki Rafsanjani, M., "Extreme learning machine based pattern classifiers for symbolic interval data", International Journal of Engineering, Transactions B: Applications, Vol. 34, No. 11, (2021), 2545-2556. doi: 10.5829/IJE.2021.34.11B.17.
  34. Tivatansakul, S. and Ohkura, M., "Emotion recognition using ecg signals with local pattern description methods", International Journal of Affective Engineering, Vol. 15, No. 2, (2016), 51-61.