多模态大语言模型动态社会互动情景下的情感能力测评

doi:10.3724/SP.J.1041.2025.1988

摘要/Abstract

摘要：

多模态大语言模型(MLLMs)能够处理并整合图像、文本等多模态数据信息, 为理解人类心理与认知行为提供了强有力工具。结合经典的情绪心理学范式, 本研究通过比较两种主流MLLMs与人类被试在动态社会互动情景下情绪识别与情绪推理的表现, 分离出人物对话视觉特征(图像)和对话内容(文本)在识别与推理相关人物情绪中的不同作用。结果表明, 基于人物对话图像和对话内容的MLLMs已经初步展现出和人类被试类似的情绪识别与情绪推理能力。之后进一步比较仅基于人物对话图像、仅基于对话内容以及基于两者结合共三种条件下MLLMs的情绪识别与情绪推理表现, 发现人物对话视觉特征一定程度上制约MLLMs基本情绪识别的表现, 但能够有效促进复合情绪识别, 对情绪推理则未产生显著影响。通过对比两种主流MLLMs及其不同版本(GPT-4-vision/turbo vs. Claude-3-haiku)的表现, 发现相较于单纯扩大训练数据规模, 技术原理框架的创新对提升MLLMs在社会互动中情绪识别与推理能力更为重要。本研究结果对理解社会互动中情绪识别与推理的心理学机制、启发类人的情感计算与智能算法具有重要科学价值和意义。

关键词: 多模态大语言模型, 社会互动, 情绪识别, 情绪推理

Abstract:

Multimodal Large Language Models (MLLMs) can process and integrate multimodal data, such as images and text, providing a powerful tool for understanding human psychology and behavior. Combining classic emotional behavior experimental paradigms, this study compares the emotion recognition and prediction abilities of human participants and two mainstream MLLMs in dynamic social interaction contexts, aiming to disentangle the distinct roles of visual features of conversational characters (images) and conversational content (text) in emotion recognition and prediction.

The results indicate that the emotion recognition and prediction performance of MLLMs, based on character images and conversational content, exhibits moderate or lower correlations with human participants. Despite a notable gap, MLLMs have begun to demonstrate preliminary capabilities in emotion recognition and prediction similar to human participants in dyadic interactions. Using human performance as a benchmark, the study further compares MLLMs under different conditions: integrating both character images and conversational content, using only character images, or relying solely on conversational content. The results suggest that visual features of character interactions somewhat constrain MLLMs’ basic emotion recognition but effectively facilitate the recognition of complex emotions, while having no significant impact on emotion prediction.

Additionally, by comparing the emotion recognition and prediction performance of two mainstream MLLMs and different versions of GPT-4, the study finds that, rather than merely increasing the scale of training data, innovations in the underlying technical framework play a more crucial role in enhancing MLLMs’ emotional capabilities in dynamic social interaction contexts. Overall, this study deepens the understanding of the interaction between human visual features and conversational content, fosters interdisciplinary integration between psychology and artificial intelligence, and provides valuable theoretical and practical insights for developing explainable affective computing models and general artificial intelligence.

Key words: multimodal large language model, social interaction, emotion recognition, emotion prediction

中图分类号:

B842

周子森, 黄琪, 谭泽宏, 刘睿, 曹子亨, 母芳蔓, 樊亚春, 秦绍正. (2025). 多模态大语言模型动态社会互动情景下的情感能力测评. 心理学报, 57(11), 1988-2000.

ZHOU Zisen, HUANG Qi, TAN Zehong, LIU Rui, CAO Ziheng, MU Fangman, FAN Yachun, QIN Shaozheng. (2025). Emotional capabilities evaluation of multimodal large language model in dynamic social interaction scenarios. Acta Psychologica Sinica, 57(11), 1988-2000.

图/表 36

参考文献 40

[1]	Barrett, L. F. (2006). Are emotions natural kinds? Perspectives on Psychological Science, 1(1), 28-58. doi: 10.1111/j.1745-6916.2006.00003.x pmid: 26151184
[2]	Barrett, L. F., Mesquita, B., & Gendron, M. (2011). Context in emotion perception. Current Directions in Psychological Science, 20(5), 286-290.
[3]	Buck, R. (1985). Prime theory: An integrated view of motivation and emotion. Psychological Review, 92(3), 389-413.
[4]	Busso, C., Bulut, M., Lee, C. C., Kazemzadeh, A., Mower, E., Kim, S.,... Narayanan, S. S. (2008). IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42, 335-359.
[5]	Clark, H. H., & Schaefer, E. F. (1989). Contributing to discourse. Cognitive Science, 13(2), 259-294.
[6]	Cordaro, D. T., Sun, R., Kamble, S., Hodder, N., Monroy, M., Cowen, A.,... Keltner, D. (2020). The recognition of 18 facial-bodily expressions across nine cultures. Emotion, 20(7), 1292-1300. doi: 10.1037/emo0000576 pmid: 31180692
[7]	Cordaro, D. T., Sun, R., Keltner, D., Kamble, S., Huddar, N., & McNeil, G. (2018). Universals and cultural variations in 22 emotional expressions across five cultures. Emotion, 18(1), 75-93. doi: 10.1037/emo0000302 pmid: 28604039
[8]	Cowen, A. S., & Keltner, D. (2020). What the face displays: Mapping 28 emotions conveyed by naturalistic expression. American Psychologist, 75(3), 349-364. doi: 10.1037/amp0000488 pmid: 31204816
[9]	Cowen, A. S., Keltner, D., Schroff, F., Jou, B., Adam, H., & Prasad, G. (2021). Sixteen facial expressions occur in similar contexts worldwide. Nature, 589(7841), 251-257.
[10]	Cowen, A. S., Laukka, P., Elfenbein, H. A., Liu, R., & Keltner, D. (2019). The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures. Nature Human Behaviour, 3(4), 369-382. doi: 10.1038/s41562-019-0533-6 pmid: 30971794
[11]	De Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition & Emotion, 14(3), 289-311.
[12]	Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3-4), 169-200.
[13]	Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384-392. pmid: 8512154
[14]	Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System (FACS) [Database record]. APA PsycTests.
[15]	Ekman, P., & Friesen, W. V. (2003). Unmasking the face: A guide to recognizing emotions from facial clues (Vol. 10). Ishk.
[16]	Fayek, H. M., Lech, M., & Cavedon, L. (2016). Modeling subjectiveness in emotion recognition with deep neural networks: Ensembles vs soft labels. In 2016 International Joint Conference on Neural Networks (IJCNN) (pp. 566-570). IEEE.
[17]	Ghosal, D., Majumder, N., Poria, S., Chhaya, N., & Gelbukh, A. (2019). Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. arXiv preprint arXiv:1908.11540.
[18]	Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
[19]	Gross, J. J. (2015). Emotion regulation: Current status and future prospects. Psychological Inquiry, 26(1), 1-26.
[20]	Kosti, R., Alvarez, J. M., Recasens, A., & Lapedriza, A. (2017). Emotion recognition in context. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1667-1675). IEEE.
[21]	Lazarus, R. S. (1991). Emotion and adaptation. Oxford University Press.
[22]	Li, S., & Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing, 13(3), 1195-1215.
[23]	Lindquist, K. A., Barrett, L. F., Bliss-Moreau, E., & Russell, J. A. (2006). Language and the perception of emotion. Emotion, 6(1), 125-138. pmid: 16637756
[24]	Matsumoto, D., Yoo, S. H., & Nakagawa, S. (2008). Culture, emotion regulation, and adjustment. Journal of Personality and Social Psychology, 94(6), 925-937. doi: 10.1037/0022-3514.94.6.925 pmid: 18505309
[25]	McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748.
[26]	Mehrabian, A. (2017). Nonverbal communication. New York: Routledge.
[27]	Plutchik,, R. (1980). A general psychoevolutionary theory of emotion. In R.Plutchik & H.Kellerman (Eds.), Theories of emotion (pp. 3-33). Academic Press.
[28]	Pollmann, M. M., & Finkenauer, C. (2009). Empathic forecasting: How do we predict other people's feelings? Cognition and Emotion, 23(5), 978-1001.
[29]	Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98-125.
[30]	Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145-172. doi: 10.1037/0033-295x.110.1.145 pmid: 12529060
[31]	Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., & Vogeley, K. (2013). Toward a second- person neuroscience. Behavioral and Brain Sciences, 36(4), 393-414. doi: 10.1017/S0140525X12000660 pmid: 23883742
[32]	Sridhar, K., Lin, W. C., & Busso, C. (2021). Generative approach using soft-labels to learn uncertainty in predicting emotional attributes. In2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 1-8). IEEE.
[33]	Strack, F., & Deutsch, R. (2004). Reflective and impulsive determinants of social behavior. Personality and Social Psychology Review, 8(3), 220-247. doi: 10.1207/s15327957pspr0803_1 pmid: 15454347
[34]	Su, P. H., Gasic, M., Mrksic, N., Rojas-Barahona, L., Ultes, S., Vandyke, D.,... Young, S.(2016). On-line active reward learning for policy optimisation in spoken dialogue systems. arXiv preprint arXiv:1605.07669.
[35]	Van Kleef, G. A., & Côté, S. (2022). The social effects of emotions. Annual Review of Psychology, 73, 629-658.
[36]	Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N.,... Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 6000-6010.
[37]	Vinciarelli, A., Pantic, M., & Bourlard, H. (2009). Social signal processing: Survey of an emerging domain. Image and Vision Computing, 27(12), 1743-1759.
[38]	Wang, W., Zheng, V. W., Yu, H., & Miao, C. (2019). A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2), 1-37.
[39]	Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024). Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601.
[40]	Zhao, S., Yao, X., Yang, J., Jia, G., Ding, G., & Chua, T. S. (2021). Affective image content analysis: Two decades review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 6729-6751.

类型	电影名	双人对话场景
类型	电影名	时长	轮数	说话次数
喜剧	国产凌凌漆	48秒	5	10
	甲方乙方	59秒	3	6
	泰囧	1分18秒	13	27
	夏洛特烦恼	1分12秒	3	6
爱情	北京遇上西雅图	1分45秒	7	15
	非诚勿扰	1分12秒	4	9
	山楂树之恋	1分19秒	3	7
	失恋33天	1分57秒	4	8
悬疑	催眠大师	1分44秒	4	10
	风声	46秒	4	9
	神探	30秒	4	8
剧情	芳华	44秒	4	9
	让子弹飞	1分4秒	5	10
	我不是药神	38秒	3	7
	无间道	1分5秒	3	8

类型	电影名	双人对话场景
类型	电影名	时长	轮数	说话次数
喜剧	国产凌凌漆	48秒	5	10
	甲方乙方	59秒	3	6
	泰囧	1分18秒	13	27
	夏洛特烦恼	1分12秒	3	6
爱情	北京遇上西雅图	1分45秒	7	15
	非诚勿扰	1分12秒	4	9
	山楂树之恋	1分19秒	3	7
	失恋33天	1分57秒	4	8
悬疑	催眠大师	1分44秒	4	10
	风声	46秒	4	9
	神探	30秒	4	8
剧情	芳华	44秒	4	9
	让子弹飞	1分4秒	5	10
	我不是药神	38秒	3	7
	无间道	1分5秒	3	8

类型	电影名	双人对话场景
类型	电影名	地点	人物代码	人物关系
喜剧	国产凌凌漆	餐厅	A：凌凌漆 B：李香琴	B和A表面合作, 背负刺杀A的任务
	甲方乙方	轿车上	D：钱康 K：尤老板	K想吃苦, D让K在乡下吃苦两个月
	泰囧	飞机上	A：徐朗 F：王宝	A和F是陌生人
	夏洛特烦恼	学校停车棚	A：夏洛 D：秋雅	A和D是同学, A在追求D
爱情	北京遇上西雅图	B的家里	A：文佳佳 B：Frank/郝志	B是西雅图月子中心的司机, A是从中国飞过去的待产妈妈
	非诚勿扰	B打给A的电话	A：秦奋 B：梁笑笑	A和B相过一次亲
	山楂树之恋	B和M的家里	B：静秋 M：静秋母亲	M是B的妈妈
	失恋33天	餐厅门口	F：陆然 I：黄小仙	F是I的前男友
悬疑	催眠大师	A的心理咨询工作室	A：徐瑞宁 B：任小妍	A是B的心理咨询师
	风声	A和B的临时住处	A：顾晓梦 B：李宁玉	A和B都在伪军司令部工作, 被上级怀疑是中共间谍“老鬼”
	神探	警车里	C：高志伟 E：王国柱	C和E是警察同事
剧情	芳华	A的寝室	A：刘峰 B：何小萍	A是文工团的男兵, B是文工团的女兵
	让子弹飞	县长办公室	A：张牧之 B：马邦德	A是县长, B是师爷
	我不是药神	办公室	A：程勇 J：曹玲	J是A的前妻
	无间道	E的心理咨询工作室	B：陈永仁 E：李心儿	E是B的心理咨询师

类型	电影名	双人对话场景
类型	电影名	地点	人物代码	人物关系
喜剧	国产凌凌漆	餐厅	A：凌凌漆 B：李香琴	B和A表面合作, 背负刺杀A的任务
	甲方乙方	轿车上	D：钱康 K：尤老板	K想吃苦, D让K在乡下吃苦两个月
	泰囧	飞机上	A：徐朗 F：王宝	A和F是陌生人
	夏洛特烦恼	学校停车棚	A：夏洛 D：秋雅	A和D是同学, A在追求D
爱情	北京遇上西雅图	B的家里	A：文佳佳 B：Frank/郝志	B是西雅图月子中心的司机, A是从中国飞过去的待产妈妈
	非诚勿扰	B打给A的电话	A：秦奋 B：梁笑笑	A和B相过一次亲
	山楂树之恋	B和M的家里	B：静秋 M：静秋母亲	M是B的妈妈
	失恋33天	餐厅门口	F：陆然 I：黄小仙	F是I的前男友
悬疑	催眠大师	A的心理咨询工作室	A：徐瑞宁 B：任小妍	A是B的心理咨询师
	风声	A和B的临时住处	A：顾晓梦 B：李宁玉	A和B都在伪军司令部工作, 被上级怀疑是中共间谍“老鬼”
	神探	警车里	C：高志伟 E：王国柱	C和E是警察同事
剧情	芳华	A的寝室	A：刘峰 B：何小萍	A是文工团的男兵, B是文工团的女兵
	让子弹飞	县长办公室	A：张牧之 B：马邦德	A是县长, B是师爷
	我不是药神	办公室	A：程勇 J：曹玲	J是A的前妻
	无间道	E的心理咨询工作室	B：陈永仁 E：李心儿	E是B的心理咨询师

情绪标签	人类被试M ± SD	GPT-4-vision-imag eM ± SD	df	t	p	Cohen’s d	95% CI
乐趣	0.07 ± 0.08	0.04 ± 0.07	296	−3.11	0.004	−0.36	[−0.05, −0.01]
气愤	0.11 ± 0.17	0.04 ± 0.07	296	−4.75	0.000	−0.55	[−0.10, −0.04]
悲伤	0.05 ± 0.07	0.04 ± 0.06	296	−1.45	0.178	−0.17	[−0.03, 0.00]
惊讶	0.05 ± 0.09	0.03 ± 0.05	296	−1.93	0.080	−0.22	[−0.03, 0.00]
敬畏	0.01 ± 0.02	0.03 ± 0.03	296	6.66	0.000	0.77	[0.01, 0.02]
专注	0.12 ± 0.12	0.17 ± 0.14	296	3.41	0.001	0.40	[0.02, 0.08]
困惑	0.09 ± 0.11	0.13 ± 0.10	296	3.96	0.000	0.46	[0.02, 0.07]
轻蔑	0.09 ± 0.11	0.05 ± 0.07	296	−3.52	0.001	−0.41	[−0.06, −0.02]
知足	0.03 ± 0.05	0.08 ± 0.10	296	4.93	0.000	0.57	[0.03, 0.06]
渴望	0.05 ± 0.08	0.04 ± 0.05	296	−1.42	0.178	−0.16	[−0.03, 0.00]
失望	0.07 ± 0.08	0.06 ± 0.05	296	−0.27	0.796	−0.03	[−0.02, 0.01]
疑虑	0.06 ± 0.07	0.12 ± 0.09	296	7.23	0.000	0.84	[0.05, 0.08]
兴高采烈	0.05 ± 0.10	0.03 ± 0.08	296	−2.43	0.025	−0.28	[−0.05, 0.00]
兴趣	0.08 ± 0.08	0.08 ± 0.09	296	0.26	0.796	0.03	[−0.02, 0.02]
痛苦	0.04 ± 0.06	0.03 ± 0.05	296	−1.73	0.112	−0.20	[−0.02, 0.00]
胜利	0.04 ± 0.06	0.01 ± 0.03	296	−4.60	0.000	−0.53	[−0.04, −0.02]