奖赏预测误差对项目和联结记忆影响的分离：元记忆的作用

doi:10.3724/SP.J.1041.2023.00877

摘要/Abstract

摘要：

选取奖赏预测误差(reward prediction error, RPE)效价和凸显性为自变量, 通过3个实验考察RPE对项目和联结记忆影响的差异及其元记忆机制。被试在对图片的奖赏猜测−奖赏反馈中形成RPE, 且需要同时记忆图片(项目)以及图片−奖赏联结, 最后进行记忆测试。结果表明, (1)联结记忆成绩存在RPE正效价和低凸显性优势, 其信心判断准确性在RPE正效价时更高, 而项目记忆成绩存在RPE负效价和高凸显性优势; (2)在编码过程中, RPE正效价和低凸显性提高了个体的瞳孔变化均值和峰值; RPE低凸显性增加了分值注视时间, 缩短了图片注视时间; (3)增加RPE水平后, RPE对项目和联结记忆成绩的分离影响仍稳定存在。这些结果表明, RPE对项目和联结记忆的影响存在分离: 编码阶段中, 个体以RPE效价和凸显性为线索, 通过元记忆控制对项目和联结记忆加工中的认知资源进行差异性分配; 提取阶段中, RPE正效价提高了对联结记忆提取的元记忆监测水平。

关键词: 奖赏预测误差, 联结记忆, 眼动, 情景记忆, 元记忆

Abstract:

Episodic memory consists of item memory and associative memory. Individual cognitive resources are typically allocated to more valuable information during encoding through metamemory, leading to competitive processing of item and associative information. Reward prediction error (RPE), defined as the difference between reward results and reward expectations, has two properties: valence (positive or negative) and salience (degree of difference). To examine the impact of reward prediction error valence and salience on item and associative memory, and how reward prediction error influences memory based on metamemory, three experiments were conducted.
In the learning stage, participants were presented with indoor and outdoor scene pictures. They were asked to predict the score of each picture and then received feedback on the actual score. Through this reinforcement learning process, participants had to find out which type of pictures is more valuable, and 30% of the scores were accumulated into the total score. To induce the effect of reward motivation on memory, participants were introduced to the opportunity to choose between two pictures and receive the value of the selected picture, although the actual program did not include a decision-making stage. After the learning stage, participants were tested on item and reward associative memory.
The findings of the study showed that: (1) There were advantages in associative memory performance for positive reward prediction error valence and low salience, with higher accuracy of JOCs at positive valence. In contrast, there were advantages in item memory performance for negative valence and high salience. (2) In the eye-tracking results during the encoding process, positive valence and low salience of reward prediction error resulted in increased mean and peak pupil dilation after feedback presentation, as well as longer value fixation duration and shorter picture fixation duration at low salience. (3) When the reward prediction error level was increased to reduce overlap between reward results and reward prediction error effects, the separation effect of reward prediction error on item and associative memory performance remained stable.
The results of the study suggest that the effects of reward prediction error on item and associative memory are distinct. During the encoding stage, individuals use the valence and salience of reward prediction error as cues to allocate cognitive resources differently in item and associative memory encoding through metamemory control. In the retrieval stage, positive valence of reward prediction error enhances the metamemory monitoring level of associative memory retrieval.

Key words: reward prediction error, associative memory, eye movements, episodic memory, metamemory

中图分类号:

B842

龙翼婷, 姜英杰, 崔璨, 岳阳. (2023). 奖赏预测误差对项目和联结记忆影响的分离：元记忆的作用. 心理学报, 55(6), 877-891.

LONG Yiting, JIANG Yingjie, CUI Can, YUE Yang. (2023). The distinct effects of reward prediction error on item and associative memory: The influence of metamemory. Acta Psychologica Sinica, 55(6), 877-891.

图/表 19

参考文献 27

[1]	Aberg K. C., Muller J., & Schwartz S. (2017). Trial-by-trial modulation of associative memory formation by reward prediction error and reward anticipation as revealed by a biologically plausible computational model. Frontiers in Human Neuroscience, 11, 56. doi: 10.3389/fnhum.2017.00056 pmid: 28261071
[2]	Ariel R., & Castel A. D. (2014). Eyes wide open: Enhanced pupil dilation when selectively studying important information. Experimental Brain Research, 232(1), 337-344. doi: 10.1007/s00221-013-3744-5 pmid: 24162863
[3]	Bethus I., Tse D., & Morris R. G. M. (2010). Dopamine and memory: Modulation of the persistence of memory for novel hippocampal nmda receptor-dependent paired associates. Journal of Neuroscience, 30(5), 1610-1618. doi: 10.1523/JNEUROSCI.2721-09.2010 pmid: 20130171
[4]	Calderon C. B., De Loof E., Ergo K., Snoeck A., Boehler C. N., & Verguts T. (2021). Signed reward prediction errors in the ventral striatum drive episodic memory. Journal of Neuroscience, 41(8), 1716-1726. doi: 10.1523/JNEUROSCI.1785-20.2020 pmid: 33334870
[5]	Da Silva Castanheira K., Lalla A., Ocampo K., Otto A. R., & Sheldon S. (2021). Reward at encoding but not retrieval modulates memory for detailed events. Cognition, 219, 104957. doi: 10.1016/j.cognition.2021.104957 URL
[6]	Ergo K., De Loof E., Debra G., Pastotter B., & Verguts T. (2020). Failure to modulate reward prediction errors in declarative learning with theta (6 Hz) frequency transcranial alternating current stimulation. Plos One, 15(12), e0237829.
[7]	Ergo K., De Vilder L., De Loof E., & Verguts T. (2021). Reward prediction errors drive declarative learning irrespective of agency. Psychonomic Bulletin & Review, 28(6), 2045-2056. doi: 10.3758/s13423-021-01952-7
[8]	Guo Y., Song G., Zhao P., & Ma Y. (2011). Revision of the sensitivity to punishment and sensitivity to reward questionnaire. Journal of Jinan Vocational College, (1), 91-94+97.
	[郭永香, 宋广文, 赵平平, 马玉花. (2011). 大学生惩罚和奖励敏感性问卷(SPSRQ)的修订. 济南职业学院学报, (1), 91-94+97. ]
[9]	Höltje G., & Mecklinger A. (2020). Feedback timing modulates interactions between feedback processing and memory encoding: Evidence from event-related potentials. Cognitive Affective & Behavioral Neuroscience, 20(2), 250-264.
[10]	Jang A. I., Nassar M. R., Dillon D. G., & Frank M. J. (2019). Positive reward prediction errors during decision-making strengthen memory encoding. Nature Human Behaviour, 3(7), 719-732. doi: 10.1038/s41562-019-0597-3 pmid: 31061490
[11]	Jiang Y., Wang Z., Zheng M., & Jin X. (2016). How value- based agendas affect study time allocation: An eye tracking study. Acta Psychologica Sinica, 48(10), 1229-1238. doi: 10.3724/SP.J.1041.2016.01229 URL
	[姜英杰, 王志伟, 郑明玲, 金雪莲. (2016). 基于价值的议程对学习时间分配影响的眼动研究. 心理学报, 48(10), 1229-1238.]
[12]	Koriat A. (1997). Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126, 349-370. doi: 10.1037/0096-3445.126.4.349 URL
[13]	Mao X., Xu H., & Guo C. (2015). Emotional memory enhancement effect in dual-processing recognition retrieval. Acta Psychologica Sinica. 47(9), 1111-1123. doi: 10.3724/SP.J.1041.2015.01111
	[毛新瑞, 徐慧芳, 郭春彦. (2015). 双加工再认提取中的情绪记忆增强效应. 心理学报, 47(9), 1111-1123.]
[14]	Mason A., Lorimer A., & Farrell S. (2019). Expected value of reward predicts episodic memory for incidentally learnt reward-item associations. Collabra-Psychology, 5(1), 40.
[15]	Murty V. P., FeldmanHall O., Hunter L. E., Phelps E. A., & Davachi L. (2016). Episodic memories predict adaptive value-based decision-making. Journal of Experimental Psychology: General, 145(5), 548-558. doi: 10.1037/xge0000158 URL
[16]	Pessoa L. (2009). How do emotion and motivation direct executive control? Trends in Cognitive Sciences, 13(4), 160-166. doi: 10.1016/j.tics.2009.01.006 pmid: 19285913
[17]	Rouhani N., & Niv Y. (2019). Depressive symptoms bias the prediction-error enhancement of memory towards negative events in reinforcement learning. Psychopharmacology, 236(8), 2425-2435. doi: 10.1007/s00213-019-05322-z pmid: 31346654
[18]	Rouhani N., & Niv Y. (2021). Signed and unsigned reward prediction errors dynamically enhance learning and memory. Elife, 10, e61077.
[19]	Rouhani N., Norman K. A., & Niv Y. (2018). Dissociable effects of surprising rewards on learning and memory. Journal of Experimental Psychology: Learning, Memory and Cognition, 44(9), 1430-1443. doi: 10.1037/xlm0000518 URL
[20]	Rouhani N., Norman K. A., Niv Y., & Bornstein A. M. (2020). Reward prediction errors create event boundaries in memory. Cognition, 203, 104269. doi: 10.1016/j.cognition.2020.104269 URL
[21]	Schultz W., Dayan P., & Montague P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593-1599. doi: 10.1126/science.275.5306.1593 pmid: 9054347
[22]	Souchay C., Guillery-Girard B., Pauly-Takacs K., Wojcik D. Z., & Eustache F. (2013). Subjective experience of episodic memory and metacognition: A neurodevelopmental approach. Frontiers in Behavioral Neuroscience, 7, 212. doi: 10.3389/fnbeh.2013.00212 pmid: 24399944
[23]	Takeuchi T., Duszkiewicz A. J., Sonneborn A., Spooner P. A., Yamasaki M., Watanabe M., …., Morris R. G. M. (2016). Locus coeruleus and dopaminergic consolidation of everyday memory. Nature, 537(7620), 357.
[24]	Tulving E. (2001). Episodic memory and common sense: How far apart? Philosophical Transactions of the Royal Society B-Biological Sciences, 356(1413), 1505-1515. doi: 10.1098/rstb.2001.0937 pmid: 11571040
[25]	Wimmer G. E., Braun E. K., Daw N. D., & Shohamy D. (2014). Episodic memory encoding interferes with reward learning and decreases striatal prediction errors. Journal of Neuroscience, 34(45), 14901-14912. doi: 10.1523/JNEUROSCI.0204-14.2014 pmid: 25378157
[26]	Woroch B., & Gonsalves B. D. (2010). Event-related potential correlates of item and source memory strength. Brain Research, 1317, 180-191. doi: 10.1016/j.brainres.2009.12.074 pmid: 20051237
[27]	Yang X., Wang F., Wang Y., Zhao T., Gao C., & Hu X. (2020). Are pupils the window of our mind? Pupil-related application in psychology and pupillometry. Advances in Psychological Science, 28(7), 1029-1041. doi: 10.3724/SP.J.1042.2020.01029
	[杨晓梦, 王福兴, 王燕青, 赵婷婷, 高春颍, 胡祥恩. (2020). 瞳孔是心灵的窗口吗?——瞳孔在心理学研究中的应用及测量. 心理科学进展, 28(7), 1029-1041.] doi: 10.3724/SP.J.1042.2020.01029

记忆类型	RPE
记忆类型	−6	−3	0	3	6
项目记忆	0.73 (0.21)	0.73 (0.16)	0.71 (0.13)	0.75 (0.17)	0.79 (0.25)
联结记忆	0.32 (0.32)	0.43 (0.17)	0.62 (0.13)	0.43 (0.18)	0.42 (0.36)

记忆类型	RPE
记忆类型	−6	−3	0	3	6
项目记忆	0.73 (0.21)	0.73 (0.16)	0.71 (0.13)	0.75 (0.17)	0.79 (0.25)
联结记忆	0.32 (0.32)	0.43 (0.17)	0.62 (0.13)	0.43 (0.18)	0.42 (0.36)

变量	项目记忆击中率				联结记忆击中率
变量	b	z	p	95% CI	b	z	p	95% CI
intercept	1.11	7.18	<0.001	[0.34, 0.68]	−0.17	−1.21	0.227	[−0.46, 0.11]
RPE	0.03	1.23	0.217	[−0.02, 0.07]	0.14	3.54	<0.001	[0.06, 0.22]
奖赏结果	−0.02	−0.51	0.610	[−0.07, 0.04]	0.09	3.15	0.002	[0.04, 0.15]
交互作用					−0.04	−5.03	0.001	[−0.06, −0.02]
intercept	0.89	0.64	<0.001	[0.60, 1.17]	0.12	0.96	0.335	[−0.12, 0.36]
URPE	0.05	1.81	0.070	[−0.00, 0.09]	−0.19	−7.13	<0.001	[−0.24, −0.13]
奖赏结果	0.01	0.64	0.525	[−0.03, 0.05]	0.08	3.54	<0.001	[0.03, 0.12]

变量	项目记忆击中率				联结记忆击中率
变量	b	z	p	95% CI	b	z	p	95% CI
intercept	1.11	7.18	<0.001	[0.34, 0.68]	−0.17	−1.21	0.227	[−0.46, 0.11]
RPE	0.03	1.23	0.217	[−0.02, 0.07]	0.14	3.54	<0.001	[0.06, 0.22]
奖赏结果	−0.02	−0.51	0.610	[−0.07, 0.04]	0.09	3.15	0.002	[0.04, 0.15]
交互作用					−0.04	−5.03	0.001	[−0.06, −0.02]
intercept	0.89	0.64	<0.001	[0.60, 1.17]	0.12	0.96	0.335	[−0.12, 0.36]
URPE	0.05	1.81	0.070	[−0.00, 0.09]	−0.19	−7.13	<0.001	[−0.24, −0.13]
奖赏结果	0.01	0.64	0.525	[−0.03, 0.05]	0.08	3.54	<0.001	[0.03, 0.12]

变量	项目记忆反应时(ms)				联结记忆反应时(ms)
变量	b	t	p	95% CI	b	t	p	95% CI
intercept	1914.56	28.84	<0.001	[1784.32, 2044, 79]	1949.60	19.03	<0.001	[1748.41, 2150, 78]
RPE	8.23	−0.98	0.327	[−7, 80, 24.24]	2.13	0.16	0.874	[−23.83, 28.99]
奖赏结果	−10.23	−1.01	0.314	[−30.67, 10.22]	−49.95	−3.18	0.002	[−80.55, −18.77]
intercept	1856.27	29.24	<0.001	[1731.749, 1980, 79]	1916.96	19.47	<0.001	[1723.65, 2110.28]
URPE	10.77	1.21	0.228	[−6.74, 28.27]	11.51	0.80	0.426	[−16.88, 39.91]
奖赏结果	−2.44	−0.32	0.746	[−17.26, 12.37]	−47.31	−3.86	0.003	[−71.37, −23.24]