Concealed emotion analysis and recognition method

doi:10.3724/SP.J.1042.2020.01426

Abstract

Abstract:

It is of great value to recognize concealed emotions for early warning of public security issues. Micro-expression is a vital channel to reveal concealed emotions. However, there are relatively few studies on concealed emotions, and micro-expressions are challenging to recognize because of their subtle magnitude and short duration. Existing Laboratory studies of micro-expression have few practical applications. Therefore, the perception and expression of concealed emotion should be systematically investigated by collecting micro-expression samples in an ecological situation, while synchronically collecting EEG signals for better labeling of micro-expressions. We spot and recognize concealed emotions mainly through micro-expressions, accompanied by face color analysis, gaze estimation, and contactless physiological signals measurement. Then, we verify and modify our system and method in two realistic public security related application scenarios: a Recognition Assistant System for the aggressive and suicidal behaviors of psychiatric patients and a Concealed Emotion Detection System for prisoners interview.

Key words: pattern recognition, micro-expression spotting and recognition, concealed emotion, deep learning, color space

CLC Number:

B842

WANG Su-Jing, ZOU Bochao, LIU Rui, LI Zhen, ZHAO Guozhen, LIU Ye, FU Xiaolan. Concealed emotion analysis and recognition method[J]. Advances in Psychological Science, 2020, 28(9): 1426-1436.

Figures/Tables 1

References 73

[1]	曹杏田, 张丽华. (2018). 青少年情绪调节自我效能感和自我控制在自尊与攻击性的关系中的链式中介作用. 中国心理卫生杂志, 32(7), 574-579.
[2]	范萌, 傅可月. (2016). 2200例重性精神疾病患者出院信息分析. 四川精神卫生, 29(5), 442-445.
[3]	厉爱婷, 赵丽萍, 汪健健, 彭德珍, 盛丽娟. (2015). 男性住院精神疾病患者攻击行为发生率及相关因素分析. 当代护士: 专科版(下旬刊), (11), 113-115.
[4]	梁静, 李开云, 曲方炳, 陈宥辛, 颜文靖, 傅小兰. (2014). 说谎的非言语视觉线索. 心理科学进展, 22(6), 995-1005.
[5]	梁静, 颜文靖, 吴奇, 申寻兵, 王甦菁, 傅小兰. (2013). 微表情研究的进展与展望. 中国科学基金, 27(2), 75-78, 82.
[6]	卢婉波, 徐银儿, 史尧胜. (2015). 重性精神病住院患者攻击及自杀行为的评估与防范. 中医药管理杂志, 23(14), 148-150.
[7]	苏光大. (2015). 人脸识别在社会公共安全领域的应用. 中国安防, 1(14), 12-14.
[8]	吴奇, 申寻兵, 傅小兰. (2010). 微表情研究及其应用. 心理科学进展, 18(9), 1359-1368.
[9]	吴晓薇, 何晓琴, 唐海波, 胡青竹, 蒲唯丹. (2015). 攻击性: 关于感觉寻求和情绪调节自我效能感的可选模型. 中国临床心理学杂志, 23(2), 196-200.
[10]	Abler, B., Hofer, C., Walter, H., Erk, S., Hoffmann, H., Traue, H. C., & Kessler, H. (2010). Habitual emotion regulation strategies and depressive symptoms in healthy subjects predict fMRI brain activation patterns related to major depression. Psychiatry Research: Neuroimaging, 183(2), 105-113. URL pmid: 20630713
[11]	Akehurst, L., Köhnken, G., Vrij, A., & Bull, R. (2010). Lay persons' and police officers' beliefs regarding deceptive behaviour. Applied Cognitive Psychology, 10(6), 461-471.
[12]	Bassili, J. N. (1979). Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. Journal of Personality and Social Psychology, 37(11), 2049-2058. doi: 10.1037//0022-3514.37.11.2049 URL pmid: 521902
[13]	Benitez-Quiroz, C. F., Srinivasan, R., & Martinez, A. M. (2018). Facial color is an efficient mechanism to visually transmit emotion. Proceedings of the National Academy of Sciences, 115(14), 3581-3586.
[14]	Brausch, A. M., & Woods, S. E. (2019). Emotion regulation deficits and nonsuicidal self-injury prospectively predict suicide ideation in adolescents. Suicide and Life-Threatening Behavior, 49(3), 868-880. URL pmid: 29900570
[15]	Carroll, J. D., & Chang, J. J. (1970). Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition. Psychometrika, 35(3), 283-319.
[16]	Chen, D.-Y., Wang, J.-J., Lin, K.-Y., Chang, H.-H., Wu, H.-K., Chen, Y.-S., & Lee, S.-Y. (2015). Image sensor- based heart rate evaluation from face reflectance using Hilbert-Huang transform. IEEE Sensors Journal, 15(1), 618-627.
[17]	Chen, X., Cheng, J., Song, R. C., Liu, Y., Ward, R., & Wang, Z. J. (2019). Video-based heart rate measurement: Recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement, 68(10), 3600-3615.
[18]	Cheng, J., Chen, X., Xu, L. X., & Wang, Z. J. (2017). Illumination variation-resistant video-based heart rate measurement using joint blind source separation and ensemble empirical mode decomposition. IEEE Journal of Biomedical and Health Informatics, 21(5), 1422-1433. URL pmid: 27723609
[19]	Cheng, Y. H., Lu, F., & Zhang, X. Z. (2018). Appearance-based gaze estimation via evaluation-guided asymmetric regression. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV), Munich, German.
[20]	DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). Cues to deception. Psychological Bulletin, 129(1), 74-118. URL pmid: 12555795
[21]	Derrick, D. C., Moffitt, K., & Nunamaker Jr, J. F. (2011). Eye gaze behavior as a guilty knowledge test: Initial exploration for use in automated, kiosk-based screening. Paper presented at the Hawaii International Conference on System Sciences, Hawaii, America.
[22]	Ekman, P. (2003a). Darwin, deception, and facial expression. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. DeWaal (Eds.), Emotions inside Out: 130 Years after Darwin's the Expression of the Emotions in Man and Animals (pp. 205-221). New York, America: New York Academy of Sciences.
[23]	Ekman, P. (2003b). Darwin, deception, and facial expression. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. DeWaal (Eds.), Emotions inside Out: 130 Years after Darwin's the Expression of the Emotions in Man and Animals (pp. 205-221). New York, America: New York Academy of Sciences.
[24]	Ekman, P. (2009). Lie catching and microexpressions. In Martin (Ed.), The Philosophy of Deception (pp. 118-133). Britain: Oxford University.
[25]	Ekman, P., & Friesen, W. V. (1969). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88-106. doi: 10.1080/00332747.1969.11023575 URL pmid: 5779090
[26]	Ekman, P., & Friesen, W. V. (1974). Detecting deception from the body or face. Journal of Personality and Social Psychology, 29(3), 288-298.
[27]	Ekman, P., Friesen, W. V., & Hagar, J. C. (2002). Facial Action Coding System (pp. 15-464). Salt Lake City, America: Research Nexus division of Network Information Research Corporation.
[28]	Ekman, P., & O'Sullivan, M. (2006). From flawed self-assessment to blatant whoppers: The utility of voluntary and involuntary behavior in detecting deception. Behavioral Science and the Law, 24(5), 673-686.
[29]	Ekman, P., & Sullivan, M. O. (2006). From flawed self- assessment to blatant whoppers: The utility of voluntary and involuntary behavior in detecting deception. Behavioral Science and the Law, 24(5), 673-686.
[30]	Fischer, T., Jin Chang, H., & Demiris, Y. (2018). Rt-gene: Real-time eye gaze estimation in natural environments. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV).
[31]	Fitzgerald, J. M., MacNamara, A., Kennedy, A. E., Rabinak, C. A., Rauch, S. A. M., Liberzon, I., & Phan, K. L. (2017). Individual differences in cognitive reappraisal use and emotion regulatory brain function in combat-exposed veterans with and without PTSD. Depress Anxiety, 34(1), 79-88. URL pmid: 27559724
[32]	Forkmann, T., Scherer, A., Bocker, M., Pawelzik, M., Gauggel, S., & Glaesmer, H. (2014). The relation of cognitive reappraisal and expressive suppression to suicidal ideation and suicidal desire. Suicide Life Threat Behav, 44(5), 524-536. doi: 10.1111/sltb.12076 URL pmid: 24494723
[33]	Frank, M. G., & Svetieva, E. (2015). Microexpressions and deception. Understanding facial expressions in communication (pp. 227-242). Berlin, Germany: Springer-Verlag.
[34]	Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348-362. URL pmid: 12916575
[35]	Haggard, E. A., & Isaacs, K. S. (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy, 154-165. URL pmid: 22122035
[36]	He, J. C., Hu, J. F., Lu, X., & Zheng, W. S. (2017). Multi-task mid-level feature learning for micro-expression recognition. Pattern Recognition, 66, 44-52.
[37]	Hjortskov, N., Rissén, D., Blangsted, A. K., Fallentin, N., Lundberg, U., & Søgaard, K. (2004). The effect of mental stress on heart rate variability and blood pressure during computer work. European Journal of Applied Physiology, 92(1-2), 84-89. doi: 10.1007/s00421-004-1055-z URL pmid: 14991326
[38]	Huang, Q., Veeraraghavan, A., & Sabharwal, A. (2017). TabletGaze: dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets. Machine Vision and Applications, 28(5-6), 445-461.
[39]	Huang, X. H., Wang, S.-J., Liu, X., Zhao, G. Y., Feng, X. Y., & Pietikainen, M. (2019). Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Transactions on Affective Computing, 10(1), 32-47.
[40]	Jimenez, J., Scully, T., Barbosa, N., Donner, C., Alvarez, X., Vieira, T., … Weyrich, T. (2010). A practical appearance model for dynamic facial color. ACM Transactions Graphics, 29(6), 141.
[41]	Kettenring, J. R. (1971). Canonical analysis of several sets of variables. Biometrika, 58(3), 433-451.
[42]	Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., & Torralba, A. (2016). Eye tracking for everyone. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
[43]	le Ngo, A. C., See, J., & Phan, R. C. W. (2017). Sparsity in dynamics of spontaneous subtle emotions: Analysis and application. IEEE Transactions on Affective Computing, 8(3), 396-411.
[44]	Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G. Y., & Fu, X. L. (2016). A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 7(4), 299-310.
[45]	McIntosh, D. N. (1996). Facial feedback hypotheses: Evidence, implications, and directions. Motivation and Emotion, 20(2), 121-147.
[46]	Meservy, T. O., Jensen, M. L., Kruse, J., Burgoon, J. K., & Nunamaker, J. F. (2005). Automatic extraction of deceptive behavioral cues from video. In P. Kantor, G. Muresan, F. Roberts, D. D. Zeng, F. Y. Wang, H. Chen, & R. C. Merkle (Eds.), Intelligence and Security Informatics, Proceedings (Vol. 3495, pp. 198-208).
[47]	Miles, S. R., Sharp, C., Tharp, A. T., Stanford, M. S., Stanley, M., Thompson, K. E., & Kent, T. A. (2017). Emotion dysregulation as an underlying mechanism of impulsive aggression: Reviewing empirical data to inform treatments for veterans who perpetrate violence. Aggression and Violent Behavior, 34, 147-153.
[48]	Moretti, G., Ellis, R. A., & Mescon, H. (1959). Vascular patterns in the skin of the face. The Journal of Investigative Dermatology, 33(3), 103-112.
[49]	Moses, Z. B., Luecken, L. J., & Eason, J. C. (2007). Measuring task-related changes in heart rate variability. In 2007 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vols 1-16 (pp. 644-647).
[50]	Phillips, M. L., & Swartz, H. A. (2014). A critical appraisal of neuroimaging studies of bipolar disorder: Toward a new conceptualization of underlying neural circuitry and a road map for future research. American Journal of Psychiatry, 171(8), 829-843. URL pmid: 24626773
[51]	Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010). Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762-10774. URL pmid: 20588929
[52]	Pompili, M., Gonda, X., Serafini, G., Innamorati, M., Sher, L., Amore, M., ... Girardi, P. (2013). Epidemiology of suicide in bipolar disorders: A systematic review of the literature. Bipolar Disorders, 15(5), 457-490. doi: 10.1111/bdi.12087 URL pmid: 23755739
[53]	Porter, S., & Brinke, L. T. (2008). Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 19(5), 508-514. doi: 10.1111/j.1467-9280.2008.02116.x URL pmid: 18466413
[54]	Smith, F. W., & Schyns, P. G. (2009). Smile through your fear and sadness: Transmitting and identifying facial expression signals over a range of viewing distances. Psychological Science, 20(10), 1202-1208. doi: 10.1111/j.1467-9280.2009.02427.x URL pmid: 19694983
[55]	ten Brinke, L., MacDonald, S., Porter, S., & O'connor, B. (2012). Crocodile tears: Facial, verbal and body language behaviours associated with genuine and fabricated remorse. Law and Human Behavior, 36(1), 51-59. URL pmid: 22471385
[56]	Vaillant, G. E. (2004). Aging well: surprising guideposts to a happier life from the landmark Harvard study of adult development. American Journal of Psychiatry, 161(1), 178-179.
[57]	Vanderhasselt, M. A., Baeken, C., van Schuerbeek, P., Luypaert, R., & de Raedt, R. (2013). Inter-individual differences in the habitual use of cognitive reappraisal and expressive suppression are associated with variations in prefrontal cognitive control for emotional information: An event related fMRI study. Biological Psychollgy, 92(3), 433-439.
[58]	Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434-21445. doi: 10.1364/oe.16.021434 URL pmid: 19104573
[59]	Vrij, A., & Granhag, P. A. (2012). Eliciting cues to deception and truth: What matters are the questions asked. Journal of Applied Research in Memory and Cognition, 1(2), 110-117. doi: 10.1016/j.jarmac.2012.02.004 URL
[60]	Wang, K., & Ji, Q. (2016a). Hybrid model and appearance based eye tracking with Kinect. Paper presented at the Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications.
[61]	Wang, K., & Ji, Q. (2016b). Real time eye gaze tracking with kinect. Paper presented at the 2016 23rd International Conference on Pattern Recognition (ICPR).
[62]	Wang, K., & Ji, Q. (2017). Real time eye gaze tracking with 3d deformable eye-face model. Paper presented at the Proceedings of the IEEE International Conference on Computer Vision.
[63]	Wang, S.-J., Li, B.-J., Liu, Y.-J., Yan, W.-J., Ou, X. Y., Huang, X. H., ... Fu, X. L. (2018). Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing, 312(2018), 251-226.
[64]	Wang, S.-J., Wu, S. H., Qian, X. S., Li, J. X., & Fu, X. L. (2017). A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing, 230, 382-389.
[65]	Wang, S.-J., Yan, W.-J., Sun, T. K., Zhao, G. Y., & Fu, X. L. (2016). Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing, 214, 218-232.
[66]	Wang, W. J., den Brinker, A. C., Stuijk, S., & de Haan, G. (2017). Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479-1491. URL pmid: 28113245
[67]	Weiss, J. (2011). Telling lies: Clues to deceit in the marketplace, politics, and marriage. American Journal of Clinical Hypnosis, 53(4), 287-288.
[68]	Xiong, X. H., Liu, Z. C., Cai, Q., & Zhang, Z. Y. (2014). Eye gaze tracking using an RGBD camera: A comparison with a RGB solution. Paper presented at the Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication.
[69]	Xu, F., Zhang, J. P., & Wang, J. Z. (2017). Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing, 8(2), 254-267.
[70]	Yap, M. H., See, J., Hong, X. P., & Wang, S.-J. (2018, 15-19 May 2018). Facial micro-expressions grand challenge 2018 summary. Paper presented at the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).
[71]	Zhang, X. C., Sugano, Y., Fritz, M., & Bulling, A. (2015). Appearance-based gaze estimation in the wild. Paper presented at the 2015 IEEE Conference on Computer Vision and Pattern Recognition.
[72]	Zong, Y., Huang, X. H., Zheng, W. M., Cui, Z., & Zhao, G. Y. (2018). Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Transactions on Multimedia, 20(11), 3160-3172.
[73]	Zong, Y., Zheng, W. M., Huang, X. H., Shi, J. G., Cui, Z., & Zhao, G. Y. (2018). Domain regeneration for cross-database micro-expression recognition. IEEE Transactions on Image Processing, 27(5), 2484-2498.