Identifying dominant emotional state using handwriting and drawing samples by fusing features
Tóm tắt
The expertise to quickly and non-invasively determine a subject’s emotional state can contribute to a milestone in research on emotionally intelligent computing systems. The ability to identify emotions via everyday activities such as writing and drawing is beneficial to one’s well-being. Tablet devices and other human-machine interfaces have made collecting handwriting and drawing samples simpler. To understand more about writing and drawing signals there is a need to investigate them in the temporal, spectral, and cepstral domains for discovering new insights. Extracting more information will help to improve classification accuracy. This study combines temporal, spectral, and Mel Frequency Cepstral Coefficient (MFCC) methods to extract features from such signals, and finds its correlation with depression, anxiety, and stress emotional state of the humans. To examine spatial features, velocities are also computed as variations of displacement in the x- and y-directions while performing the tasks. Bidirectional Long-Short Term Memory (BiLSTM) network is used to classify the generated features’ vectors. To evaluate the proposed work, multiple publically available benchmark datasets are utilized. This work determine which activities and features help describe a specific emotional state through in-depth investigation. The results of the experiments demonstrate that fusing several features improve recognition accuracy significantly. For emotions identification like depression, anxiety, and stress states, this work achieved a higher classification improvement ranging from 5.32% to 8.9% as compared to the baseline approaches.
Tài liệu tham khảo
Likforman-Sulem L, Esposito A, Faundez-Zanuy M, Clémençon S (2015) Extracting style and emotion from handwriting. In: Advances in neural networks: computational and theoretical issues. Springer, Cham, pp 347–355
Champa HN, AnandaKumar KR (2010) Automated human behavior prediction through handwriting analysis. In first international conference on integrated intelligent computing. IEEE: 160-165
Marcelli A, Parziale A, Santoro A (2012) Modeling handwriting style: a preliminary investigation. In international conference on Frontiers in handwriting recognition. IEEE: 411-416
Likforman-Sulem L, Esposito A, Faundez-Zanuy M, Clémençon S, Cordasco G (2017) EMOTHAW: a novel database for emotional state recognition from handwriting and drawing. IEEE Transactions on Human-Machine Systems 47:273–284
Kedar SV, Bormane DS, Dhadwal A, Alone S, Agarwal R (2015) Automatic emotion recognition through handwriting analysis: a review. In2015 international conference on computing communication control and automation. IEEE: 811-816
Alonso-Martinez C, Faundez-Zanuy M, Mekyska J (2017) A comparative study of in-air trajectories at short and long distances in online handwriting. Cogn Comput 9:712–720
Mekhaznia T, Djeddi C, Sarkar S (2020) Personality traits identification through handwriting analysis. InMediterranean Conference on Pattern Recognition and Artificial Intelligence Springer Cham: 155–169
Fallah B, Khotanlou H (2016) Identify human personality parameters based on handwriting using neural network. In artificial intelligence and robotics (IRANOPEN) IEEE: 120-126
De Stefano C, Fontanella F, Impedovo D, Pirlo G, di Freca AS (2019) Handwriting analysis to support neurodegenerative diseases diagnosis: a review. Pattern Recogn Lett 121:37–45
Fairhurst M, Erbilek M, Li C (2014) Enhancing the forensic value of handwriting using emotion prediction. In2nd international workshop on biometrics and forensics. IEEE: 1-6
Chaudhari K, Thakkar A (2019) Survey on handwriting-based personality trait identification. Expert systems with applications : 282-308
Fisher J, Maredia A, Nixon A, Williams N, Leet J (2012) Identifying personality traits, and especially traits resulting in violent behavior through automatic handwriting analysis. Proceedings of student-faculty research day. CSIS Pace University: 1-8
Mutalib S, Ramli R, Rahman SA, Yusoff M, Mohamed A (2008) Towards emotional control recognition through handwriting using fuzzy inference. In2008 international symposium on information technology. IEEE: 1-5
Gavrilescu M, Vizireanu N (2018) Predicting the big five personality traits from handwriting. EURASIP Journal on Image and Video Processing 2018:1–7
Taleb C, Khachab M, Mokbel C, Likforman-Sulem L (2017) Feature selection for an improved Parkinson's disease identification based on handwriting. In2017 1st international workshop on Arabic script analysis and recognition. IEEE: 52-56
Esposito A, Amorese T, Buonanno M, Cuciniello M, Esposito AM (2019) Handwriting and drawing features for detecting personality traits. In2019 10th IEEE international conference on cognitive Infocommunications. CogInfoCom. IEEE: 79-84
Al Maadeed S, Hassaine A (2014) Automatic prediction of age, gender, and nationality in offline handwriting. EURASIP Journal on Image and Video Processing:1–10
Mostafa MA, Al-Qurishi M, Mathkour HI (2019) Towards personality classification through Arabic handwriting analysis. In The International Research and Innovation Forum. Springer, Cham: 557–565
Fairhurst M, Erbilek M, Li C (2015) Study of automatic prediction of emotion from handwriting samples. IET Biometrics 4:90–97
Han J, Chernyshov G, Zheng D, Gao P, Narumi T (2019) Sentiment pen: recognizing emotional context based on handwriting features. InProceedings of the 10th augmented human international conference: 1-8
Slim MA, Abdelkrim A, Benrejeb M (2015) An efficient handwriting velocity modelling for electromyographic signals reconstruction using radial basis function neural networks. In2015 7th international conference on modelling, identification and control. IEEE: 1-6
Faundez-Zanuy M, Fierrez J, Ferrer MA, Diaz M, Tolosana R (2020) Handwriting biometrics: applications and future trends in e-security and e-health. Cogn Comput 12:940–953
Plamondon R, Srihari SN (2000) Online and off-line handwriting recognition: a comprehensive survey. IEEE Trans Pattern Anal Mach Intell 22:63–84
Chen Z, Lin T (2017) Automatic personality identification using writing behaviours: an exploratory study. Behav Inform Technol 36:839–845
Halim Z, Waqar M, Tahir M (2020) A machine learning-based investigation utilizing the in-text features for the identification of dominant emotion in an email. Knowledge-based systems: 1-17
Cordasco G, Scibelli F, Faundez-Zanuy M, Likforman-Sulem L, Esposito A (2017) Handwriting and drawing features for detecting negative moods. InItalian Workshop on Neural Nets. Springer, Cham: 73–86
Prange A, Barz M, Sonntag D (2018) A categorisation and implementation of digital pen features for behaviour characterisation. arXiv preprint arXiv:1810.03970
Impedovo D, Pirlo G, Vessio G, Angelillo MT (2019) A handwriting-based protocol for assessing neurodegenerative dementia. Cognitive computation: 576-86
Bougamouza F, Hazmoune S, Benmohammed M (2016) Using Mel frequency cepstral coefficient method for online Arabic characters handwriting recognition. In 5th international conference on multimedia computing and systems. ICMCS IEEE: 87-92
Ayzeren YB, Erbilek M, Çelebi E (2019) Emotional state prediction from online handwriting and signature biometrics. IEEE Access 7:164759–164774
Cordasco G, Buonanno M, Faundez-Zanuy M, Riviello MT, Likforman-Sulem L (2020) Gender identification through handwriting: an online approach. In 11th IEEE international conference on cognitive Infocommunications. CogInfoCom IEEE: 000197-000202
Nolazco-Flores JA, Faundez-Zanuy M, De La Cueva VM, Mekyska J (2021) Exploiting spectral and cepstral handwriting features on diagnosing Parkinson’s disease. IEEE Access 9:141599–141610
Nolazco-Flores JA, Faundez-Zanuy M, Velázquez-Flores OA, Cordasco G, Esposito A (2021) Emotional state recognition performance improvement on a handwriting and drawing task. IEEE Access 9:28496–28504
Srihari SN, Singer K (2014) Role of automation in the examination of handwritten items. Pattern Recogn 47:1083–1095
Zagorskis V, Kapenieks A, Gorbunovs A (2019) Emotions identification utilizing periodic handwriting on mobile surfaces. Periodicals of Engineering and Natural Sciences PEN 7:228–237
Fallah A, Jamaati M, Soleamani A (2011) A new online signature verification system based on combining Mellin transform, MFCC and neural network. Digital signal processing: 404-16
Rosenblum S, Werner P, Dekel T, Gurevitz I, Heinik J (2010) Handwriting process variables among elderly people with mild major depressive disorder: a preliminary study. Aging Clin Exp Res 22:141–147
Ekman P (1992) Facial expressions of emotion: new findings, new questions, Psychological Sciences, Sage Journals: 34–38
Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 1161:1161–1178
Choudhury H, Prasanna SM (2019) Representation of online handwriting using multi-component sinusoidal model. Pattern Recogn 91:200–215
Lovibond PF, Lovibond SH (1995) The structure of negative emotional states: comparison of the depression anxiety stress scales (DASS) with the Beck depression and anxiety inventories. Behav Res Ther 33:335–343
Buechel S, Hahn U (2017) Emobank: studying the impact of annotation perspective and representation format on dimensional emotion analysis. InProceedings of the 15th conference of the European chapter of the Association for Computational Linguistics: 578–585
Yu K, Epps J, Chen F (2011) Cognitive load evaluation of handwriting using stroke-level features. InProceedings of the 16th international conference on intelligent user interfaces: 423-426
Hochreiter S, Schmidhuber J (1997) Long short-term memory. In neural computation, IEEE: 1735–1780
Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45:2673–2681
Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw 18:602–610
Chitlangia A, Malathi G (2019) Handwriting analysis based on histogram of oriented gradient for predicting personality raits using SVM. Procedia computer science: 384-90