Abstract: Affective brain–computer interfaces have become an increasingly important topic to achieve emotional intelligence in human–machine collaboration. However, due to the complexity of electroencephalogram (EEG) signals and the individual differences in emotional response, it is still a great challenge to design a reliable and effective model. Considering the influence of personality traits on emotional response, it would be helpful to integrate personality information and EEG signals for emotion recognition. This study proposes a personality-guided attention neural network that can use personality information to learn effective EEG representations for emotion recognition. Specifically, we first use a convolutional neural network to extract rich temporal and regional representations of EEG signals, and a special convolution kernel is designed to learn inter- and intra-regional correlations simultaneously. Second, inspired by the fact that electrodes within distinct brain scalp regions play different roles in emotion recognition, a personality-guided regional-attention mechanism is proposed to further explore the contributions of electrodes within a region and between regions. Finally, attention-based long short-term memory is designed to explore the temporal dynamics of EEG signals. Experiments on the AMIGOS dataset, which is a dataset for multimodal research for affect, personality traits, and mood on individuals and groups, show that the proposed method can significantly improve the performance of subject-independent emotion recognition and outperform state-of-the-art methods.

一种面向机器情感智能的人格引导型情感脑机接口

[1]Alarcão SM, Fonseca MJ, 2019. Emotions recognition using EEG signals: a survey. IEEE Trans Affect Comput, 10(3):374-393.

[2]Alhagry S, Fahmy AA, El-Khoribi RA, 2017. Emotion recognition based on EEG using LSTM recurrent neural network. Int J Adv Comput Sci Appl, 8(10):355-358.

[3]Alhussein M, 2016. Automatic facial emotion recognition using Weber local descriptor for e-Healthcare system. Clust Comput, 19(1):99-108.

[4]Ayata D, Yaslan Y, Kamasak ME, 2018. Emotion based music recommendation system using wearable physiological sensors. IEEE Trans Consum Electron, 64(2):196-203.

[5]Barkana BD, Ozkan Y, Badara JA, 2022. Analysis of working memory from EEG signals under different emotional states. Biomed Signal Process Contr, 71:103249.

[6]Bhardwaj A, Gupta A, Jain P, et al., 2015. Classification of human emotions from EEG signals using SVM and LDA classifiers. 2^nd Int Conf on Signal Processing and Integrated Networks, p.180-185.

[7]Chaplin TM, 2015. Gender and emotion expression: a developmental contextual perspective. Emot Rev, 7(1):14-21.

[8]Chen H, Song Y, Li XL, 2019. A deep learning framework for identifying children with ADHD using an EEG-based brain network. Neurocomputing, 356:83-96.

[9]Chevalier P, Martin JC, Isableu B, et al., 2015. Impact of personality on the recognition of emotion expressed via human, virtual, and robotic embodiments. 24^th IEEE Int Symp on Robot and Human Interactive Communication, p.229-234.

[10]Cui H, Liu AP, Zhang X, et al., 2020. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl Based Syst, 205:106243.

[11]Ding Y, Robinson N, Zeng QH, et al., 2020. TSception: a deep learning framework for emotion detection using EEG. Int Joint Conf on Neural Networks, p.1-7.

[12]Ding Y, Hu X, Xia ZY, et al., 2021. Inter-brain EEG feature extraction and analysis for continuous implicit emotion tagging during video watching. IEEE Trans Affect Comput, 12(1):92-102.

[13]Ding Y, Robinson N, Zhang S, et al., 2022. TSception: capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. IEEE Trans Affect Comput, early access.

[14]Duan RN, Zhu JY, Lu BL, 2013. Differential entropy feature for EEG-based emotion classification. 6^th Int IEEE/EMBS Conf on Neural Engineering, p.81-84.

[15]Etkin A, Egner T, Kalisch R, 2011. Emotional processing in anterior cingulate and medial prefrontal cortex. Trends Cogn Sci, 15(2):85-93.

[16]Farahat A, Reichert C, Sweeney-Reed CM, et al., 2019. Convolutional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization. J Neur Eng, 16(6):066010.

[17]Fiterau M, Bhooshan S, Fries J, et al., 2017. ShortFuse: biomedical time series representations in the presence of structured information. Proc 2^nd Machine Learning for Healthcare Conf, p.59-74.

[18]Fossum TA, Barrett LF, 2000. Distinguishing evaluation from description in the personality-emotion relationship. Pers Soc Psychol Bull, 26(6):669-678.

[19]Furnes D, Berg H, Mitchell RM, et al., 2019. Exploring the effects of personality traits on the perception of emotions from prosody. Front Psychol, 10:184.

[20]He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. IEEE Conf on Computer Vision and Pattern Recognition, p.770-778.

[21]Kehoe EG, Toomey JM, Balsters JH, et al., 2012. Personality modulates the effects of emotional arousal and valence on brain activation. Soc Cogn Affect Neurosci, 7(7):858-870.

[22]Klados MA, Konstantinidi P, Dacosta-Aguayo R, et al., 2020. Automatic recognition of personality profiles using EEG functional connectivity during emotional processing. Brain Sci, 10(5):278.

[23]Koelstra S, Muhl C, Soleymani M, et al., 2012. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 3(1):18-31.

[24]Kragel PA, LaBar KS, 2016. Decoding the nature of emotion in the brain. Trends Cogn Sci, 20(6):444-455.

[25]Lan ZR, Sourina O, Wang LP, et al., 2016. Real-time EEG-based emotion monitoring using stable features. Vis Comput, 32(3):347-358.

[26]Larsen RJ, Ketelaar T, 1991. Personality and susceptibility to positive and negative emotional states. J Pers Soc Psychol, 61(1):132-140.

[27]Lawhern VJ, Solon AJ, Waytowich NR, et al., 2018. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neur Eng, 15(5):056013.

[28]Li WY, Hu X, Long XF, et al., 2020. EEG responses to emotional videos can quantitatively predict Big-Five personality traits. Neurocomputing, 415:368-381.

[29]Li X, Song DW, Zhang P, et al., 2016. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. IEEE Int Conf on Bioinformatics and Biomedicine, p.352-359.

[30]Li Y, Zheng WM, Zong Y, et al., 2021. A bi-hemisphere domain adversarial neural network model for EEG emotion recognition. IEEE Trans Affect Comput, 12(2):494-504.

[31]Li Y, Zheng WM, Wang L, et al., 2022. From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Trans Affect Comput, 13(2):568-578.

[32]Lindquist KA, Barrett LF, 2012. A functional architecture of the human brain: emerging insights from the science of emotion. Trends Cogn Sci, 16(11):533-540.

[33]Lotfi E, Akbarzadeh-T MR, 2014. Practical emotional neural networks. Neur Netw, 59:61-72.

[34]Ma WF, Gong YF, Zhou GX, et al., 2021. A channel-mixing convolutional neural network for motor imagery EEG decoding and feature visualization. Biomed Signal Process Contr, 70:103021.

[35]Maaoui C, Pruski A, 2010. Emotion recognition through physiological signals for human-machine communication. In: Kordic V (Ed.), Cutting Edge Robotics 2010. IntechOpen, Rijeka, Croatia.

[36]Martínez-Tejada LA, Maruyama Y, Yoshimura N, et al., 2020. Analysis of personality and EEG features in emotion recognition using machine learning techniques to classify arousal and valence labels. Mach Learn Knowl Extr, 2(2):99-124.

[37]Miranda-Correa JA, Abadi MK, Sebe N, et al., 2021. AMIGOS: a dataset for affect, personality and mood research on individuals and groups. IEEE Trans Affect Comput, 12(2):479-493.

[38]Mohammadi Z, Frounchi J, Amiri M, 2017. Wavelet-based emotion recognition system using EEG signal. Neur Comput Appl, 28(8):1985-1990.

[39]Mühl C, Allison B, Nijholt A, et al., 2014. A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges. Brain Comput Interf, 1(2):66-84.

[40]Niemic CP, 2002. Studies of emotion: a theoretical and empirical review of psychophysiological studies of emotion. J Undergrad Res, 1:15-18.

[41]Orgeta V, 2009. Specificity of age differences in emotion regulation. Aging Ment Health, 13(6):818-826.

[42]Özerdem MS, Polat H, 2017. Emotion recognition based on EEG features in movie clips with channel selection. Brain Inform, 4(4):241-252.

[43]Perugini M, Di Blas L, 2002. Analyzing personality related adjectives from an eticemic perspective: the Big Five marker scales (BFMS) and the Italian AB5C taxonomy. In: de Raad B, Perugini M (Eds.), Big Five Assessment. Hogrefe & Huber Publishers, Seattle, USA, p.281-304.

[44]Picard RW, Vyzas E, Healey J, 2001. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Patt Anal Mach Intell, 23(10):1175-1191.

[45]Rao HC, Wang SQ, Hu XP, et al., 2021. A self-supervised gait encoding approach with locality-awareness for 3D skeleton based person re-identification. IEEE Trans Patt Anal Mach Intell, early access.

[46]Revelle WR, Scherer KR, 2009. Personality and emotion. In: Sander D, Scherer KR (Eds.), Oxford Companion to Emotion and the Affective Sciences. Oxford University Press, Oxford, UK, p.304-305.

[47]Rukavina S, Gruss S, Hoffmann H, et al., 2016. Affective computing and the impact of gender and age. PLoS ONE, 11(3):e0150584.

[48]Salovey P, Mayer JD, 1990. Emotional intelligence. Imaginat Cognit Pers, 9(3):185-211.

[49]Schirrmeister RT, Springenberg JT, Fiederer LDJ, et al., 2017. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 38(11):5391-5420.

[50]Schmidtke JI, Heller W, 2004. Personality, affect and EEG: predicting patterns of regional brain activity related to extraversion and neuroticism. Pers Individ Differ, 36(3):717-732.

[51]Shen J, Zhang XW, Wang G, et al., 2022. An improved empirical mode decomposition of electroencephalogram signals for depression detection. IEEE Trans Affect Comput, 13(1):262-271.

[52]Shen T, Zhou TY, Long GD, et al., 2018. DiSAN: directional self-attention network for RNN/CNN-free language understanding. Proc 32^nd AAAI Conf on Artificial Intelligence, p.5446-5455.

[53]Shu YY, Wang SF, 2017. Emotion recognition through integrating EEG and peripheral signals. IEEE Int Conf on Acoustics, Speech and Signal Processing, p.2871-2875.

[54]Steinert S, Friedrich O, 2020. Wired emotions: ethical issues of affective brain-computer interfaces. Sci Eng Ethics, 26(1):351-367.

[55]Topic A, Russo M, Stella M, et al., 2022. Emotion recognition using a reduced set of EEG channels based on holographic feature maps. Sensors, 22(9):3248.

[56]van Leeuwen KG, Sun H, Tabaeizadeh M, et al., 2019. Detecting abnormal electroencephalograms using deep convolutional networks. Clin Neurophysiol, 130(1):77-84.

[57]Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. Proc 32^st Int Conf on Neural Information Processing Systems, p.6000-6010.

[58]Vuoskoski JK, Eerola T, 2011. The role of mood and personality in the perception of emotions represented by music. Cortex, 47(9):1099-1106.

[59]Waytowich N, Lawhern VJ, Garcia JO, et al., 2018. Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials. J Neur Eng, 15(6):066031.

[60]Wen ZY, Xu RF, Du JC, 2017. A novel convolutional neural networks for emotion recognition based on EEG signal. Int Conf on Security, Pattern Analysis, and Cybernetics, p.672-677.

[61]Wolpaw JR, Birbaumer N, McFarland DJ, et al., 2002. Brain–computer interfaces for communication and control. Clin Neurophysiol, 113(6):767-791.

[62]Xu SH, Rao HC, Peng H, et al., 2021. Attention-based multilevel co-occurrence graph convolutional LSTM for 3-D action recognition. IEEE Internet Things J, 8(21):15990-16001.

[63]Yang YL, Wu QF, Qiu M, et al., 2018. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. Int Joint Conf on Neural Networks, p.1-7.

[64]Zhang DL, Yao LN, Zhang X, et al., 2018. Cascade and parallel convolutional recurrent neural networks on EEG-based intention recognition for brain computer interface. Proc 32^nd AAAI Conf on Artificial Intelligence, p.1703-1710.

[65]Zhang DL, Yao LN, Chen KX, et al., 2019. A convolutional recurrent attention model for subject-independent EEG signal analysis. IEEE Signal Process Lett, 26(5):715-719.

[66]Zhang GH, Yu MJ, Liu YJ, et al., 2021. SparseDGCNN: recognizing emotion from multichannel EEG signals. IEEE Trans Affect Comput, early access.

[67]Zhang T, Cui Z, Xu CY, et al., 2020. Variational pathway reasoning for EEG emotion recognition. Proc 34^th AAAI Conf on Artificial Intelligence, 2709-2716.

[68]Zhang XW, Li JL, Hou KC, et al., 2020. EEG-based depression detection using convolutional neural network with demographic attention mechanism. 42^nd Annual Int Conf of the IEEE Engineering in Medicine & Biology Society, p.128-133.

[69]Zhang XW, Liu JY, Shen J, et al., 2021. Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine. IEEE Trans Cybern, 51(9):4386-4399.

[70]Zhang XW, Pan J, Shen J, et al., 2022. Fusing of electroencephalogram and eye movement with group sparse canonical correlation analysis for anxiety detection. IEEE Trans Affect Comput, 13(2):958-971.

[71]Zhang YH, Prasad S, Kilicarslan A, et al., 2017. Multiple kernel based region importance learning for neural classification of gait states from EEG signals. Front Neurosci, 11:170.

[72]Zhao GZ, Zhang YL, Ge Y, et al., 2018a. Asymmetric hemisphere activation in tenderness: evidence from EEG signals. Sci Rep, 8(1):8029.

[73]Zhao GZ, Ge Y, Shen BY, et al., 2018b. Emotion analysis for personality inference from EEG signals. IEEE Trans Affect Comput, 9(3):362-371.

[74]Zhao GZ, Zhang YL, Ge Y, 2018c. Frontal EEG asymmetry and middle line power difference in discrete emotions. Front Behav Neurosci, 12:225.

[75]Zhao SC, Ding GG, Han JG, et al., 2018. Personality-aware personalized emotion recognition from physiological signals. Proc 27^th Int Joint Conf on Artificial Intelligence, p.1660-1667.

[76]Zheng WL, Lu BL, 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev, 7(3):162-175.