Dual system theory of physical activity: A reinforcement learning perspective

doi:10.3724/SP.J.1042.2020.01337

Abstract

Abstract:

Exercise behavior theory based on rational decision making is regarded as the dominant framework for understanding physical activity, which provides valuable information on cognitive constructs linked to physical activity. Behavioral interventions based on social ecological model have attracted much attention due to its better performance. However, recent studies show that exercise benefits cognition and current sport environment do not promote the formation of individual exercise habit, so it is necessary to explore a new theoretical framework to clarify the formation mechanism of individual exercise habit. The latest framework for explaining physical activity is dual system model, which promises to provide a broader perspective on motivation by considering the non-conscious and hedonic determinants of physical activity. Multiple representative dual system models associated with physical activity, on the one hand, from simple spontaneous path to context cues and exercise habit, and then to complex concept model highlighting the role of automatic affective evaluations, the construct of system 1 is elaborated. Combined with system 2, which is concerned by exercise behavior theory, they offer a basis on how to build dual system model for physical activity. On the other hand, through analysis of the principle for competitive, cooperative and hierarchical control of dual systems, proposals are provided for the control of dual system model. Canonical reinforcement learning framework explains the principle for construct and control of dual system model: in terms of construct of the model, model-free and model-based reinforcement learning represent system 1 and system 2, respectively. In terms of control of the model, Dyna cooperative architecture and hierarchical reinforcement learning provide a reasonable explanation for physical activity is more likely to be a complex combination of actions which is hierarchically operated with a cooperative form than competitive control. Finally, the exerciser-sport environment interaction in reinforcement learning is put forward to explore exercise behavior from a brand-new perspective.

Key words: physical activity, dual system model, theory of planned behavior, exercise habit, reinforcement learning

CLC Number:

B849

CHU Xin-Yu, WANG Ze-Jun, XIAO Huan-Yu. Dual system theory of physical activity: A reinforcement learning perspective[J]. Advances in Psychological Science, 2020, 28(8): 1337-1350.

Figures/Tables 7

References 80

[1]	艾炎, 胡竹菁. (2018). 推理判断中双重加工过程的协作与转换机制. 心理科学进展, 26(10), 1794-1806.
[2]	段文婷, 江光荣. (2008). 计划行为理论述评. 心理科学进展, 16(2), 315-320.
[3]	冯玉娟, 毛志雄. (2014). 高中生身体活动意向和行为的促进策略:自我决定动机对TPB的贡献. 体育科学, 34(8), 64-69.
[4]	郭强, 汪晓赞, 蒋健保. (2017). 我国儿童青少年身体活动与久坐行为模式特征的研究. 体育科学, 37(7), 17-21.
[5]	韩慧, 郑家鲲. (2016). 西方国家青少年体力活动相关研究述评——基于社会生态学视角的分析. 体育科学, 36(5), 62-70.
[6]	沈梦英, 毛志雄, 张一民. (2010). 中国成年人锻炼行为的影响因素——HAPA与TPB两个理论模型的整合. 体育科学, 30(12), 48-54.
[7]	司琦. (2007). 身体活动的行为科学理论综述. 体育科学, 27(9), 72-80.
[8]	许昭, 毛志雄. (2015). 身体活动熟虑-冲动双系统模型的构建与检验. 体育科学, 35(8), 16-23.
[9]	叶娜, 佐斌. (2007). 联想-命题评价模型——态度改变的新解释. 心理科学进展, 15(5), 834-839.
[10]	张加林, 唐炎, 胡月英. (2017). 我国儿童青少年体育环境特征与存在问题研究. 体育科学, 37(3), 21-34.
[11]	张业安. (2018). 青少年体质健康促进的媒介责任: 概念、目标及机制. 体育科学, 38(6), 14-26.
[12]	张展嘉, 王正珍, 于洪军, 陈蔚云. (2018). 第65届美国运动医学会年会关于身体活动促进的研究热点与进展综述. 北京体育大学学报, 41(8), 72-76.
[13]	周文吉, 俞扬. (2017). 分层强化学习综述. 智能系统学报, 12(5), 590-594.
[14]	朱为模. (2009). 从进化论、社会-生态学角度谈环境、步行与健康. 体育科研, 30(5), 12-16.
[15]	Aarts, H., Paulussen, T., & Schaalma, H. (1997). Physical exercise habit: On the conceptualization and formation of habitual health behaviours. Health Education Research, 12(3), 363-374. URL pmid: 10174219
[16]	Aarts, H., Verplanken, B., & van Knippenberg, A. (1997). Habit and information use in travel mode choices. Acta Psychologica, 96(1-2), 1-14.
[17]	Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior & Human Decision Processes, 50(2), 179-211.
[18]	Ajzen, I. (2011). The theory of planned behaviour: Reactions and reflections. Psychology & Health, 26(9), 1113-1127. doi: 10.1080/08870446.2011.613995 URL pmid: 21929476
[19]	Althoff, T., Sosič, R., Hicks, J. L., King, A. C., Delp, S. L., & Leskovec, J. (2017). Large-scale physical activity data reveal worldwide activity inequality. Nature, 547(7663), 336-339. doi: 10.1038/nature23018 URL pmid: 28693034
[20]	Antoniewicz, F., & Brand, R. (2014). Automatic evaluations and exercise setting preference in frequent exercisers. Journal of Sport & Exercise Psychology, 36(6), 631-636. doi: 10.1123/jsep.2014-0033 URL pmid: 25602145
[21]	Antoniewicz, F., & Brand, R. (2016a). Learning to like exercising: Evaluative conditioning changes automatic evaluations of exercising and influences subsequent exercising behavior. Journal of Sport & Exercise Psychology, 38(2), 138-148. doi: 10.1123/jsep.2015-0125 URL pmid: 27385674
[22]	Antoniewicz, F., & Brand, R. (2016b). Dropping out or keeping up? Early-dropouts, late-dropouts, and maintainers differ in their automatic evaluations of exercise already before a 14-week exercise course. Frontiers in Psychology, 7, 838. URL pmid: 27313559
[23]	Aral, S., & Nicolaides, C. (2017). Exercise contagion in a global social network. Nature Communication, 8, 14753.
[24]	Betsch, T., Haberstroh, S., Glockner, A., Haar, T., & Fiedler, K. (2001). The effects of routine strength on adaptation and information search in recurrent decision making. Organizational Behavior and Human Decision Processes, 84(1), 23-53.
[25]	Biddle, S. J. H., Hagger, M. S., Chatzisarantis, N. L. D., & Lippke, S. (2007). Theoretical frameworks in exercise psychology. In G. Tenenbaum & R. C. Eklund (Eds.). Handbook of sport psychology (3rd ed., pp. 537-559). Hoboken, NJ: Wiley.
[26]	Bluemke, M., Brand, R., Schweizer, G., & Kahlert, D. (2010). Exercise might be good for me, but I don’t feel good about it: Do automatic associations predict exercise behavior? Journal of Sport & Exercise Psychology, 32(2), 137-153. doi: 10.1123/jsep.32.2.137 URL pmid: 20479475
[27]	Botvinick, M. M., Niv, Y., & Barto, A. G. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113(3), 262-280. doi: 10.1016/j.cognition.2008.08.011 URL pmid: 18926527
[28]	Botvinick, M., & Weinstein, A. (2014). Model-based hierarchical reinforcement learning and human action control. Philosophical Transactions of the Royal Society B, 369(1655), 20130480.
[29]	Brand, R., & Antoniewicz, F. (2016). Affective evaluations of exercising: The role of automatic-reflective evaluation discrepancy. Journal of Sport & Exercise Psychology, 38(6), 631-638. doi: 10.1123/jsep.2016-0171 URL pmid: 28033023
[30]	Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge, Massachusetts: Harvard University Press.
[31]	Calitri, R., Lowe, R., Eves, F. F., & Bennett, P. (2009). Associations between visual attention, implicit and explicit attitude and behaviour for physical activity. Psychology & Health, 24(9), 1105-1123. doi: 10.1080/08870440802245306 URL pmid: 20205048
[32]	Chevance, G., Caudroit, J., Romain, A. J., & Boiche, J. (2017). The adoption of physical activity and eating behaviors among persons with obesity and in the general population: The role of implicit attitudes within the Theory of Planned Behavior. Psychology, Health & Medicine, 22(3), 319-324. doi: 10.1080/13548506.2016.1159705 URL pmid: 26952746
[33]	Conroy, D. E., & Berry, T. R. (2017). Automatic affective evaluations of physical activity. Exercise & Sport Sciences Reviews, 45(4), 230-237. URL pmid: 28704217
[34]	Cooper, R. P., & Shallice, T. (2006). Hierarchical schemas and goals in the control of sequential behavior. Psychological Review, 113(4), 887-916. URL pmid: 17014307
[35]	Cushman, F., & Morris, A. (2015). Habitual control of goal selection in humans. Proceedings of the National Academy of Science of the United States of America, 112(45), 13817-13822.
[36]	Daw, N. D. (2015). Of goals and habits. Proceedings of the National Academy of Science of the United States of America, 112(45), 13749-13750.
[37]	Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron, 69(6), 1204-1215. doi: 10.1016/j.neuron.2011.02.027 URL pmid: 21435563
[38]	Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704-1711. doi: 10.1038/nn1560 URL
[39]	Dezfouli, A., & Balleine, B. W. (2012). Habits, action sequences and reinforcement learning. European Journal of Neuroscience, 35(7), 1036-1051. doi: 10.1111/j.1460-9568.2012.08050.x URL
[40]	Dezfouli, A., & Balleine, B. W. (2013). Actions, action sequences and habits: Evidence that goal-directed and habitual action control are hierarchically organized. PLoS Computational Biology, 9(12), e1003364. doi: 10.1371/journal.pcbi.1003364 URL pmid: 24339762
[41]	Dolan, R. J., & Dayan, P. (2013). Goals and habits in the brain. Neuron, 80(2), 312-325. URL pmid: 24139036
[42]	Ekkekakis, P. (2017). People have feelings! Exercise psychology in paradigmatic transition. Current Opinion in Psychology, 16, 84-88. doi: 10.1016/j.copsyc.2017.03.018 URL pmid: 28813362
[43]	Ekkekakis, P., Hargreaves, E. A., & Parfitt, G. (2013). Invited Guest Editorial: Envisioning the next fifty years of research on the exercise-affect relationship. Psychology of Sport & Exercise, 14(5), 751-758.
[44]	Endrighi, R., Basen-Engquist, K., Szeto, E., Perkins, H., Baum, G., Cox-Martin, M., … Waters, A. J. (2016). Self-reported and automatic cognitions are associated with exercise behavior in cancer survivors. Health Psychology, 35(8), 824-828. URL pmid: 27505202
[45]	Evans, J., & Stanovich, K. E. (2013). Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8(3), 223-241. doi: 10.1177/1745691612460685 URL pmid: 26172965
[46]	Gardner, B., de Bruijn, G.-J., & Lally, P. (2011). A systematic review and meta-analysis of applications of the self-report habit index to nutrition and physical activity behaviours. Annals of Behavioral Medicine, 42(2), 174-187. URL pmid: 21626256
[47]	Gardner, B., Phillips, L. A., & Judah, G. (2016). Habitual instigation and habitual execution: Definition, measurement, and effects on behaviour frequency. British Journal of Health Psychology, 21(3), 613-630. doi: 10.1111/bjhp.12189 URL pmid: 26991427
[48]	Gawronski, B., & Bodenhausen, G. V. (2006). Associative and propositional processes in evaluation: An integrative review of implicit and explicit attitude change. Psychological Bulletin, 132(5), 692-731. URL pmid: 16910748
[49]	Gershman, S. J., Markman, A. B., & Otto, A. R. (2014). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General, 143(1), 182-194.
[50]	Graybiel, A. M. (2008). Habits, rituals, and the evaluative brain. Annual Review of Neuroscience, 31, 359-387. doi: 10.1146/annurev.neuro.29.051605.112851 URL pmid: 18558860
[51]	Hagger, M. S., & Chatzisarantis, N. L. (2014). An integrated behavior change model for physical activity. Exercise & Sport Sciences Reviews, 42(2), 62-69. URL pmid: 24508739
[52]	Kaushal, N., Rhodes, R. E., Meldrum, J. T., & Spence, J. C. (2017). The role of habit in different phases of exercise. British Journal of Health Psychology, 22(3), 429-448. doi: 10.1111/bjhp.12237 URL pmid: 28374463
[53]	Keatley, D., Clarke, D. D., & Hagger, M. S. (2012). Investigating the predictive validity of implicit and explicit measures of motivation on condom use, physical activity, and healthy eating. Psychology & Health, 27(5), 550-569. URL pmid: 21895458
[54]	Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology, 7(5), e1002055. URL pmid: 21637741
[55]	Kiviniemi, M. T., Voss-Humke, A. M., & Seifert, A. L. (2007). How do I feel about the behavior? The interplay of affective associations with behaviors and cognitive beliefs as influences on physical activity behavior. Health Psychology, 26(2), 152-158. URL pmid: 17385966
[56]	Kurdi, B., Gershman, S. J., & Banaji, M. R. (2019). Model-free and model-based learning processes in the updating of explicit and implicit evaluations. Proceedings of the National Academy of Science of the United States of America, 116(13), 6035-6044.
[57]	Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014). Neural computations underlying arbitration between model-based and model-free learning. Neuron, 81(3), 687-699. doi: 10.1016/j.neuron.2013.11.028 URL pmid: 24507199
[58]	Maher, J. P., & Conroy, D. E. (2015). Habit strength moderates the effects of daily action planning prompts on physical activity but not sedentary behavior. Journal of Sport & Exercise Psychology, 37(1), 97-107. doi: 10.1123/jsep.2014-0258 URL pmid: 25730895
[59]	Markland, D., Hall, C. R., Duncan, L. R., & Simatovic, J. (2015). The effects of an imagery intervention on implicit and explicit exercise attitudes. Psychology of Sport and Exercise, 17, 24-31. doi: 10.1016/j.psychsport.2014.11.007 URL
[60]	Marteau, T. M., Hollands, G. J., & Fletcher, P. C. (2012). Changing human behavior to prevent disease: The importance of targeting automatic processes. Science, 337(6101), 1492-1495. doi: 10.1126/science.1226918 URL pmid: 22997327
[61]	McEachan, R. R. C., Conner, M., Taylor, N. J., & Lawton, R. J. (2011). Prospective prediction of health-related behaviours with the theory of planned behaviour: A meta-analysis. Health Psychology Review, 5(2), 97-144. doi: 10.1080/17437199.2010.521684 URL
[62]	Neal, D. T., Wood, W., Labrecque, J. S., & Phillippa, L. (2012). How do habits guide behavior? Perceived and actual triggers of habits in daily life. Journal of Experimental Social Psychology, 48(2), 492-498. doi: 10.1016/j.jesp.2011.10.011 URL
[63]	Phillips, L. A., & Gardner, B. (2016). Habitual exercise instigation (vs. execution) predicts healthy adults’ exercise frequency. Health Psychology, 35(1), 69-77.
[64]	Rebar, A. L., Dimmock, J. A., Jackson, B., Rhodes, R. E., Kates, A., Starling, J., & Vandelanotte, C. (2016). A systematic review of the effects of non-conscious regulatory processes in physical activity. Health Psychology Review, 10(4), 395-407. URL pmid: 27118430
[65]	Rebar, A. L., Elavsky, S., Maher, J. P., Doerksen, S. E., & Conroy, D. E. (2014). Habits predict physical activity on days when intentions are weak. Journal of Sport & Exercise Psychology, 36(2), 157-165. URL pmid: 24686952
[66]	Rhodes, R. E. (2014). Adding depth to the next generation of physical activity models. Exercise & Sport Sciences Reviews, 42(2), 43-44. doi: 10.1249/JES.0000000000000014 URL pmid: 24657940
[67]	Rhodes, R. E., Fiala, B., & Conner, M. (2009). A review and meta-analysis of affective judgments and physical activity in adult populations. Annals of Behavioral Medicine, 38(3), 180-204. doi: 10.1007/s12160-009-9147-y URL pmid: 20082164
[68]	Rhodes, R. E., Janssen, I., Bredin, S. S. D., Warburton, D. E. R., & Bauman, A. (2017). Physical activity: Health impact, prevalence, correlates and interventions. Psychology & Health, 32(8), 942-975. doi: 10.1080/08870446.2017.1325486 URL pmid: 28554222
[69]	Rhodes, R. E., & Rebar, A. L. (2018). Physical activity habit: Complexities and controversies. In Verplanken, B. (Eds.), The psychology of habit: Theory, mechanisms, change, and contexts
[70]	Rhodes, R. E., Saelens, B. E., & Sauvage-Mar, C. (2018). Understanding physical activity through interactions between the built environment and social cognition: A systematic review. Sports Medicine, 48(8), 1893-1912.
[71]	Sala, M., Baldwin, A. S., & Williams, D. M. (2016). Affective and cognitive predictors of affective response to exercise: Examining unique and overlapping variance. Psychology of Sport & Exercise, 27, 1-8.
[72]	Sniehotta, F. F., Presseau, J., & Araujo-Soares, V. (2014). Time to retire the theory of planned behaviour. Health Psychology Review, 8(1), 1-7. URL pmid: 25053004
[73]	Strack, F., & Deutsch, R. (2004). Reflective and impulsive determinants of social behavior. Personality and Social Psychology Review, 8(3), 220-247. URL pmid: 15454347
[74]	Sutton, R. S., & Barto, A. G. (2018) . Reinforcement learning: An introduction Cambridge, Massachusetts: Massachusetts Institute of Technology Press An introduction. Cambridge, Massachusetts: Massachusetts Institute of Technology Press.
[75]	Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2), 181-211.
[76]	Tappe, K. A., & Glanz, K. (2013). Measurement of exercise habits and prediction of leisure-time activity in established exercise. Psychology, Health & Medicine, 18(5), 601-611.
[77]	Verplanken, B., Aarts, H., & van Knippenberg, A. (1997). Habit, information acquisition, and the process of making travel mode choices. European Journal of Social Psychology, 27(5), 539-560.
[78]	Verplanken, B., & Melkevik, O. (2008). Predicting habit: The case of physical exercise. Psychology of Sport & Exercise, 9(1), 15-26.
[79]	Williams, D. M., & Evans, D. R. (2014). Current emotion research in health behavior science. Emotion Review, 6(3), 277-287.
[80]	Wood, W., & Runger, D. (2016). Psychology of habit. Annual Reviews of Psychology, 67(1), 289-314.