隐藏情绪分析与识别方法

doi:10.3724/SP.J.1042.2020.01426

心理科学进展 ›› 2020, Vol. 28 ›› Issue (9): 1426-1436.doi: 10.3724/SP.J.1042.2020.01426

隐藏情绪分析与识别方法

王甦菁¹^,²(), 邹博超³^,⁴, 刘瑞⁴^,⁵, 李振¹^,², 赵国朕¹^,², 刘烨²^,⁶, 傅小兰²^,⁶

¹中国科学院行为科学重点实验室(中国科学院心理研究所), 北京 100101
²中国科学院大学心理学系, 北京 100039
³中国电子科学研究院, 社会安全风险感知与防控大数据应用国家工程实验室, 北京 100041
⁴首都医科大学,人脑保护高精尖创新中心, 北京 100069
⁵首都医科大学附属北京安定医院, 国家精神心理疾病临床医学研究中心, 精神疾病诊断与治疗北京市重点实验室, 北京 100088
⁶中国科学院心理研究所脑与认知科学国家重点实验室, 北京 100039

收稿日期:2020-03-02 出版日期:2020-09-15 发布日期:2020-07-24
通讯作者: 王甦菁 E-mail:wangsujing@psych.ac.cn
基金资助:
* 国家自然科学基金项目(U19B2032);国家自然科学基金项目(61772511);社会安全风险感知与防控大数据应用国家工程实验室主任基金项目(18112403)

Concealed emotion analysis and recognition method

WANG Su-Jing¹^,²(), ZOU Bochao³^,⁴, LIU Rui⁴^,⁵, LI Zhen¹^,², ZHAO Guozhen¹^,², LIU Ye²^,⁶, FU Xiaolan²^,⁶

¹CAS Key Laboratory of Behavioral Science, Institute of Psychology, Beijing 100101, China
²Department of Psychology, University of Chinese Academy of Sciences, Beijing 100049, China
³National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data (PSRPC), China Academy of Electronics and Information Technology, Beijing 100041, China
⁴Advanced Innovation Center for Human Brain Protection, Capital Medical University, Beijing 100069, China
⁵The National Clinical Research Center for Mental Disorders & Beijing Key Laboratory of Mental Disorders, Capital Medical University, Beijing 100088, China
⁶State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China

Received:2020-03-02 Online:2020-09-15 Published:2020-07-24
Contact: WANG Su-Jing E-mail:wangsujing@psych.ac.cn

摘要/Abstract

摘要：

隐藏情绪识别对公共安全防范与预警具有重要的意义。微表情是揭示隐藏情绪的一条重要通道。但目前隐藏情绪研究较少且微表情因其细微幅度与快速出现等特性难以识别, 其研究尚未在实际中广泛应用。因为, 隐藏情绪的认知与表达机理亟需系统的研究,采集实际场景中的微表情数据, 并以脑电信号辅助微表情的精确标注是提高微表情标注效率的有效途径。深入研究微表情识别方法, 并辅以人脸颜色、注视估计和非接触生理信号等多通道数据, 以检测与识别隐藏情绪。社会公共安全是隐藏情绪分析和识别的典型场景。面向精神疾病患者两害行为(即危害自身或他人的危险行为)风险评估和服刑人员会见场景隐藏情绪检测, 可以有效地对相应系统和方法进行验证和修正。

关键词: 模式识别, 微表情检测和识别, 隐藏情绪, 深度学习, 颜色空间

Abstract:

It is of great value to recognize concealed emotions for early warning of public security issues. Micro-expression is a vital channel to reveal concealed emotions. However, there are relatively few studies on concealed emotions, and micro-expressions are challenging to recognize because of their subtle magnitude and short duration. Existing Laboratory studies of micro-expression have few practical applications. Therefore, the perception and expression of concealed emotion should be systematically investigated by collecting micro-expression samples in an ecological situation, while synchronically collecting EEG signals for better labeling of micro-expressions. We spot and recognize concealed emotions mainly through micro-expressions, accompanied by face color analysis, gaze estimation, and contactless physiological signals measurement. Then, we verify and modify our system and method in two realistic public security related application scenarios: a Recognition Assistant System for the aggressive and suicidal behaviors of psychiatric patients and a Concealed Emotion Detection System for prisoners interview.

Key words: pattern recognition, micro-expression spotting and recognition, concealed emotion, deep learning, color space

中图分类号:

B842

王甦菁, 邹博超, 刘瑞, 李振, 赵国朕, 刘烨, 傅小兰. (2020). 隐藏情绪分析与识别方法. 心理科学进展 , 28(9), 1426-1436.

WANG Su-Jing, ZOU Bochao, LIU Rui, LI Zhen, ZHAO Guozhen, LIU Ye, FU Xiaolan. (2020). Concealed emotion analysis and recognition method. Advances in Psychological Science, 28(9), 1426-1436.

图/表 1

参考文献 73

[1]	曹杏田, 张丽华. (2018). 青少年情绪调节自我效能感和自我控制在自尊与攻击性的关系中的链式中介作用. 中国心理卫生杂志, 32(7), 574-579.
[2]	范萌, 傅可月. (2016). 2200例重性精神疾病患者出院信息分析. 四川精神卫生, 29(5), 442-445.
[3]	厉爱婷, 赵丽萍, 汪健健, 彭德珍, 盛丽娟. (2015). 男性住院精神疾病患者攻击行为发生率及相关因素分析. 当代护士: 专科版(下旬刊), (11), 113-115.
[4]	梁静, 李开云, 曲方炳, 陈宥辛, 颜文靖, 傅小兰. (2014). 说谎的非言语视觉线索. 心理科学进展, 22(6), 995-1005.
[5]	梁静, 颜文靖, 吴奇, 申寻兵, 王甦菁, 傅小兰. (2013). 微表情研究的进展与展望. 中国科学基金, 27(2), 75-78, 82.
[6]	卢婉波, 徐银儿, 史尧胜. (2015). 重性精神病住院患者攻击及自杀行为的评估与防范. 中医药管理杂志, 23(14), 148-150.
[7]	苏光大. (2015). 人脸识别在社会公共安全领域的应用. 中国安防, 1(14), 12-14.
[8]	吴奇, 申寻兵, 傅小兰. (2010). 微表情研究及其应用. 心理科学进展, 18(9), 1359-1368.
[9]	吴晓薇, 何晓琴, 唐海波, 胡青竹, 蒲唯丹. (2015). 攻击性: 关于感觉寻求和情绪调节自我效能感的可选模型. 中国临床心理学杂志, 23(2), 196-200.
[10]	Abler, B., Hofer, C., Walter, H., Erk, S., Hoffmann, H., Traue, H. C., & Kessler, H. (2010). Habitual emotion regulation strategies and depressive symptoms in healthy subjects predict fMRI brain activation patterns related to major depression. Psychiatry Research: Neuroimaging, 183(2), 105-113. URL pmid: 20630713
[11]	Akehurst, L., Köhnken, G., Vrij, A., & Bull, R. (2010). Lay persons' and police officers' beliefs regarding deceptive behaviour. Applied Cognitive Psychology, 10(6), 461-471.
[12]	Bassili, J. N. (1979). Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. Journal of Personality and Social Psychology, 37(11), 2049-2058. doi: 10.1037//0022-3514.37.11.2049 URL pmid: 521902
[13]	Benitez-Quiroz, C. F., Srinivasan, R., & Martinez, A. M. (2018). Facial color is an efficient mechanism to visually transmit emotion. Proceedings of the National Academy of Sciences, 115(14), 3581-3586.
[14]	Brausch, A. M., & Woods, S. E. (2019). Emotion regulation deficits and nonsuicidal self-injury prospectively predict suicide ideation in adolescents. Suicide and Life-Threatening Behavior, 49(3), 868-880. URL pmid: 29900570
[15]	Carroll, J. D., & Chang, J. J. (1970). Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition. Psychometrika, 35(3), 283-319.
[16]	Chen, D.-Y., Wang, J.-J., Lin, K.-Y., Chang, H.-H., Wu, H.-K., Chen, Y.-S., & Lee, S.-Y. (2015). Image sensor- based heart rate evaluation from face reflectance using Hilbert-Huang transform. IEEE Sensors Journal, 15(1), 618-627.
[17]	Chen, X., Cheng, J., Song, R. C., Liu, Y., Ward, R., & Wang, Z. J. (2019). Video-based heart rate measurement: Recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement, 68(10), 3600-3615.
[18]	Cheng, J., Chen, X., Xu, L. X., & Wang, Z. J. (2017). Illumination variation-resistant video-based heart rate measurement using joint blind source separation and ensemble empirical mode decomposition. IEEE Journal of Biomedical and Health Informatics, 21(5), 1422-1433. URL pmid: 27723609
[19]	Cheng, Y. H., Lu, F., & Zhang, X. Z. (2018). Appearance-based gaze estimation via evaluation-guided asymmetric regression. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV), Munich, German.
[20]	DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). Cues to deception. Psychological Bulletin, 129(1), 74-118. URL pmid: 12555795
[21]	Derrick, D. C., Moffitt, K., & Nunamaker Jr, J. F. (2011). Eye gaze behavior as a guilty knowledge test: Initial exploration for use in automated, kiosk-based screening. Paper presented at the Hawaii International Conference on System Sciences, Hawaii, America.
[22]	Ekman, P. (2003a). Darwin, deception, and facial expression. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. DeWaal (Eds.), Emotions inside Out: 130 Years after Darwin's the Expression of the Emotions in Man and Animals (pp. 205-221). New York, America: New York Academy of Sciences.
[23]	Ekman, P. (2003b). Darwin, deception, and facial expression. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. DeWaal (Eds.), Emotions inside Out: 130 Years after Darwin's the Expression of the Emotions in Man and Animals (pp. 205-221). New York, America: New York Academy of Sciences.
[24]	Ekman, P. (2009). Lie catching and microexpressions. In Martin (Ed.), The Philosophy of Deception (pp. 118-133). Britain: Oxford University.
[25]	Ekman, P., & Friesen, W. V. (1969). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88-106. doi: 10.1080/00332747.1969.11023575 URL pmid: 5779090
[26]	Ekman, P., & Friesen, W. V. (1974). Detecting deception from the body or face. Journal of Personality and Social Psychology, 29(3), 288-298.
[27]	Ekman, P., Friesen, W. V., & Hagar, J. C. (2002). Facial Action Coding System (pp. 15-464). Salt Lake City, America: Research Nexus division of Network Information Research Corporation.
[28]	Ekman, P., & O'Sullivan, M. (2006). From flawed self-assessment to blatant whoppers: The utility of voluntary and involuntary behavior in detecting deception. Behavioral Science and the Law, 24(5), 673-686.
[29]	Ekman, P., & Sullivan, M. O. (2006). From flawed self- assessment to blatant whoppers: The utility of voluntary and involuntary behavior in detecting deception. Behavioral Science and the Law, 24(5), 673-686.
[30]	Fischer, T., Jin Chang, H., & Demiris, Y. (2018). Rt-gene: Real-time eye gaze estimation in natural environments. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV).
[31]	Fitzgerald, J. M., MacNamara, A., Kennedy, A. E., Rabinak, C. A., Rauch, S. A. M., Liberzon, I., & Phan, K. L. (2017). Individual differences in cognitive reappraisal use and emotion regulatory brain function in combat-exposed veterans with and without PTSD. Depress Anxiety, 34(1), 79-88. URL pmid: 27559724
[32]	Forkmann, T., Scherer, A., Bocker, M., Pawelzik, M., Gauggel, S., & Glaesmer, H. (2014). The relation of cognitive reappraisal and expressive suppression to suicidal ideation and suicidal desire. Suicide Life Threat Behav, 44(5), 524-536. doi: 10.1111/sltb.12076 URL pmid: 24494723
[33]	Frank, M. G., & Svetieva, E. (2015). Microexpressions and deception. Understanding facial expressions in communication (pp. 227-242). Berlin, Germany: Springer-Verlag.
[34]	Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348-362. URL pmid: 12916575
[35]	Haggard, E. A., & Isaacs, K. S. (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy, 154-165. URL pmid: 22122035
[36]	He, J. C., Hu, J. F., Lu, X., & Zheng, W. S. (2017). Multi-task mid-level feature learning for micro-expression recognition. Pattern Recognition, 66, 44-52.
[37]	Hjortskov, N., Rissén, D., Blangsted, A. K., Fallentin, N., Lundberg, U., & Søgaard, K. (2004). The effect of mental stress on heart rate variability and blood pressure during computer work. European Journal of Applied Physiology, 92(1-2), 84-89. doi: 10.1007/s00421-004-1055-z URL pmid: 14991326
[38]	Huang, Q., Veeraraghavan, A., & Sabharwal, A. (2017). TabletGaze: dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets. Machine Vision and Applications, 28(5-6), 445-461.
[39]	Huang, X. H., Wang, S.-J., Liu, X., Zhao, G. Y., Feng, X. Y., & Pietikainen, M. (2019). Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Transactions on Affective Computing, 10(1), 32-47.
[40]	Jimenez, J., Scully, T., Barbosa, N., Donner, C., Alvarez, X., Vieira, T., … Weyrich, T. (2010). A practical appearance model for dynamic facial color. ACM Transactions Graphics, 29(6), 141.
[41]	Kettenring, J. R. (1971). Canonical analysis of several sets of variables. Biometrika, 58(3), 433-451.
[42]	Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., & Torralba, A. (2016). Eye tracking for everyone. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
[43]	le Ngo, A. C., See, J., & Phan, R. C. W. (2017). Sparsity in dynamics of spontaneous subtle emotions: Analysis and application. IEEE Transactions on Affective Computing, 8(3), 396-411.
[44]	Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G. Y., & Fu, X. L. (2016). A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 7(4), 299-310.
[45]	McIntosh, D. N. (1996). Facial feedback hypotheses: Evidence, implications, and directions. Motivation and Emotion, 20(2), 121-147.
[46]	Meservy, T. O., Jensen, M. L., Kruse, J., Burgoon, J. K., & Nunamaker, J. F. (2005). Automatic extraction of deceptive behavioral cues from video. In P. Kantor, G. Muresan, F. Roberts, D. D. Zeng, F. Y. Wang, H. Chen, & R. C. Merkle (Eds.), Intelligence and Security Informatics, Proceedings (Vol. 3495, pp. 198-208).
[47]	Miles, S. R., Sharp, C., Tharp, A. T., Stanford, M. S., Stanley, M., Thompson, K. E., & Kent, T. A. (2017). Emotion dysregulation as an underlying mechanism of impulsive aggression: Reviewing empirical data to inform treatments for veterans who perpetrate violence. Aggression and Violent Behavior, 34, 147-153.
[48]	Moretti, G., Ellis, R. A., & Mescon, H. (1959). Vascular patterns in the skin of the face. The Journal of Investigative Dermatology, 33(3), 103-112.
[49]	Moses, Z. B., Luecken, L. J., & Eason, J. C. (2007). Measuring task-related changes in heart rate variability. In 2007 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vols 1-16 (pp. 644-647).
[50]	Phillips, M. L., & Swartz, H. A. (2014). A critical appraisal of neuroimaging studies of bipolar disorder: Toward a new conceptualization of underlying neural circuitry and a road map for future research. American Journal of Psychiatry, 171(8), 829-843. URL pmid: 24626773
[51]	Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010). Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762-10774. URL pmid: 20588929
[52]	Pompili, M., Gonda, X., Serafini, G., Innamorati, M., Sher, L., Amore, M., ... Girardi, P. (2013). Epidemiology of suicide in bipolar disorders: A systematic review of the literature. Bipolar Disorders, 15(5), 457-490. doi: 10.1111/bdi.12087 URL pmid: 23755739
[53]	Porter, S., & Brinke, L. T. (2008). Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 19(5), 508-514. doi: 10.1111/j.1467-9280.2008.02116.x URL pmid: 18466413
[54]	Smith, F. W., & Schyns, P. G. (2009). Smile through your fear and sadness: Transmitting and identifying facial expression signals over a range of viewing distances. Psychological Science, 20(10), 1202-1208. doi: 10.1111/j.1467-9280.2009.02427.x URL pmid: 19694983
[55]	ten Brinke, L., MacDonald, S., Porter, S., & O'connor, B. (2012). Crocodile tears: Facial, verbal and body language behaviours associated with genuine and fabricated remorse. Law and Human Behavior, 36(1), 51-59. URL pmid: 22471385
[56]	Vaillant, G. E. (2004). Aging well: surprising guideposts to a happier life from the landmark Harvard study of adult development. American Journal of Psychiatry, 161(1), 178-179.
[57]	Vanderhasselt, M. A., Baeken, C., van Schuerbeek, P., Luypaert, R., & de Raedt, R. (2013). Inter-individual differences in the habitual use of cognitive reappraisal and expressive suppression are associated with variations in prefrontal cognitive control for emotional information: An event related fMRI study. Biological Psychollgy, 92(3), 433-439.
[58]	Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434-21445. doi: 10.1364/oe.16.021434 URL pmid: 19104573
[59]	Vrij, A., & Granhag, P. A. (2012). Eliciting cues to deception and truth: What matters are the questions asked. Journal of Applied Research in Memory and Cognition, 1(2), 110-117. doi: 10.1016/j.jarmac.2012.02.004 URL
[60]	Wang, K., & Ji, Q. (2016a). Hybrid model and appearance based eye tracking with Kinect. Paper presented at the Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications.
[61]	Wang, K., & Ji, Q. (2016b). Real time eye gaze tracking with kinect. Paper presented at the 2016 23rd International Conference on Pattern Recognition (ICPR).
[62]	Wang, K., & Ji, Q. (2017). Real time eye gaze tracking with 3d deformable eye-face model. Paper presented at the Proceedings of the IEEE International Conference on Computer Vision.
[63]	Wang, S.-J., Li, B.-J., Liu, Y.-J., Yan, W.-J., Ou, X. Y., Huang, X. H., ... Fu, X. L. (2018). Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing, 312(2018), 251-226.
[64]	Wang, S.-J., Wu, S. H., Qian, X. S., Li, J. X., & Fu, X. L. (2017). A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing, 230, 382-389.
[65]	Wang, S.-J., Yan, W.-J., Sun, T. K., Zhao, G. Y., & Fu, X. L. (2016). Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing, 214, 218-232.
[66]	Wang, W. J., den Brinker, A. C., Stuijk, S., & de Haan, G. (2017). Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479-1491. URL pmid: 28113245
[67]	Weiss, J. (2011). Telling lies: Clues to deceit in the marketplace, politics, and marriage. American Journal of Clinical Hypnosis, 53(4), 287-288.
[68]	Xiong, X. H., Liu, Z. C., Cai, Q., & Zhang, Z. Y. (2014). Eye gaze tracking using an RGBD camera: A comparison with a RGB solution. Paper presented at the Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication.
[69]	Xu, F., Zhang, J. P., & Wang, J. Z. (2017). Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing, 8(2), 254-267.
[70]	Yap, M. H., See, J., Hong, X. P., & Wang, S.-J. (2018, 15-19 May 2018). Facial micro-expressions grand challenge 2018 summary. Paper presented at the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).
[71]	Zhang, X. C., Sugano, Y., Fritz, M., & Bulling, A. (2015). Appearance-based gaze estimation in the wild. Paper presented at the 2015 IEEE Conference on Computer Vision and Pattern Recognition.
[72]	Zong, Y., Huang, X. H., Zheng, W. M., Cui, Z., & Zhao, G. Y. (2018). Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Transactions on Multimedia, 20(11), 3160-3172.
[73]	Zong, Y., Zheng, W. M., Huang, X. H., Shi, J. G., Cui, Z., & Zhao, G. Y. (2018). Domain regeneration for cross-database micro-expression recognition. IEEE Transactions on Image Processing, 27(5), 2484-2498.

隐藏情绪分析与识别方法

Concealed emotion analysis and recognition method

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 1

参考文献 73

相关文章 1

编辑推荐

Metrics

本文评价