Effects of embodied simulation on understanding Chinese body action verbs
WANG Bin1,LI Zhirui2,WU Limei3,ZHANG Jijia1()
1 Department of Psychology, Renmin University of China; The State Ethnic Affairs Commission Key Research Center for Language, Cultural, and Psychology; Key Research Center for National Psychology and Education, the National Education Development Center of the Ministry of Education, Beijing 100872, China 2 Mental Health Education and Counseling Center, Huaiyin Normal University, Huaian 223300, China 3 College of Chinese Language and Culture, Jinan University, Guangzhou 510610, China
Recent approaches on embodied cognition and theories of semantic and conceptual “grounding” have emphasized the role of perceptual and motor skills in language comprehension and action understanding. Evidence on the role of sensorimotor information in language processing has been obtained from behavioral and neuroimaging studies. These findings have been taken as a support for the claim that language is understood through sensorimotor simulations of actions and events being described. The theory on Perceptual Symbol Systems holds that the sensorimotor system participates in the language comprehension process, which is an empirical simulation of a situation through a series of complex language cues. Chinese characters may show different characteristics from English words because semantic radicals are linked with the meaning of these characters. These semantic radicals may affect the embodied effect of Chinese characters. In the study, authors used single-character body action verbs in Chinese as experimental materials. Body action verbs are words that use body parts to perform mechanical movements. The directional semantic feature of body action verbs reflects the direction of physical space and can be perceived. Experiment 1 investigated the influence of semantic direction on the spatial judgment of an arrow direction in the motion channel under whole word priming. The repeated measurement design of 2 (Chinese character embodied direction: up/down) × 2 (arrow direction: up/down) was adopted. Thirty-five participants volunteered in this study. Experiment 2 investigated the influence of semantic orientation on the spatial judgment of letter orientation in visual channels under whole word initiation. The repeated measurement design of 2 (Chinese character direction: up/down) × 2 (letter position: up/down) design was used. Thirty-eight participants took part in this experiment. Experiment 3 investigated the embodied simulation of the semantic radical and whole character under the radical priming paradigm. Chinese body action verbs formed by semantic radicals “扌” and “?” were used as experimental materials. A within-subject design of 2 (priming type: semantic radical’s priming/control priming) × 3 (character type: consistent/inconsistent) × 3 (SOA: 43 ms/72 ms/243 ms) was used. Before the behavioral experiment, the participants were asked to hold their hands up for one minute to reinforce the bodybuilding experience of having their hands above and their feet below. Character type shows the direction and orientation between the character and its semantic radical. Consistent character type means that the character and its semantic radical have a similar direction and orientation. For example, the semantic radical “?” means “foot, ” which is below the body. Thus, “蹲” is a consistent character, whereas “跳” is an inconsistent character. Results suggest the following: (1) When Chinese characters are in a downward semantic direction, the downward arrow is judged faster than the upward arrow. When characters are in an upward semantic direction, the direction of the arrow has a null effect. (2) Chinese characters with upward semantic movement can be used to recognize upper letters more quickly and characters with downward semantic movement can be used to recognize lower letters more quickly. (3) In the middle and late stages of Chinese character processing, the response of consistent and inconsistent Chinese characters is significantly different, indicating that the semantic radical is activated from the middle stage of the Chinese character processing until the late stage. The present findings demonstrate the following: (1) An action-character compatibility effect is present in Chinese body movement verbs in the movement and visual channels. Understanding Chinese body movement verbs is a cross-channel embodied simulation process. (2) The semantic radical activation of the phonogram starts from the middle stage to the late stage of Chinese body action verb processing. A semantic understanding of Chinese body action verbs has an embodied simulation at the whole character and component (semantic radical) levels.
Aziz-Zadeh L., & Damasio A . (2008). Embodied semantics for actions: Findings from functional brain imaging. Journal of Physiology-Paris, 102(1-3), 35-39.
Barsalou L. W . (1999a). Language comprehension:Archival memory or preparation for situated action? Discourse Processes, 28(1), 61-80.
Barsalou L. W . (1999b). Perceptual symbol systems. Behavioral and Brain Sciences, 22(4), 577-660.
Barsalou L.W., Simmons W. K., Barbey A. K., & Wilson C. D . (2003). Grounding conceptual knowledge in modality- specific systems. Trends in Cognitive Sciences, 7(2), 84-91.
Beauchamp M. S., Lee K. E., Haxby J. V., & Martin A . (2002). Parallel visual motion processing streams for manipulable objects and human movements. Neuron, 34(1), 149-159.
Beauchamp M. S., Lee K. E., Haxby J. V., & Martin A . (2003). FMRI responses to video and point-light displays of moving humans and manipulable objects. Journal of Cognitive Neuroscience, 15(7), 991-1001.
Bedny M., Caramazza A., Grossman E., Pascual-Leone A., & Saxe R . (2008). Concepts are more than percepts: The case of action verbs. Journal of Neuroscience, 28(44), 11347-11353.
Borghi A. M., & Riggio L . (2009a). Sentence comprehension and simulation of object temporary, canonical and stable affordances. Brain Research, 1253, 117-128.
Borghi A. M., & Scorolli C . (2009b). Language comprehension and dominant hand motion simulation. Human Movement Science, 28(1), 12-27.
Boulenger V., Roy A. C., Paulignan Y., Deprez V., Jeannerod M., & Nazir T. A . (2006). Cross-talk between language processes and overt motor behavior in the first 200 msec of processing. Journal of Cognitive Neuroscience, 18(10), 1607-1615.
Cardona J. F., Kargieman L., Sinay V., Gershanik O., Gelormini C., Amoruso L., …& Ibáñez A . (2014). How embodied is action language? Neurological evidence from motor diseases. Cognition, 131(2), 311-322.
Chen X. K., & Zhang J. J . (2008). Role of familiarity of semantic radicals in the recognition of highly familiar Chinese characters. Acta Psychologica Sinica, 40(2), 148-159.
Cohen J . (1992). A power primer. Psychological Bulletin, 112(1), 155-159.
Ding G. S., Peng D. L., & Taft M . (2004). The nature of the mental representation of radicals in Chinese:A priming study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(2), 530-539.
Dudschig C., Souman J., Lachmair M., Vega I. D. L., & Kaup B . (2013). Reading “sun” and looking up: The influence of language on saccadic eye movements in the vertical dimension. Plos One, 8(2), e56872.
Dunn B. M., Kamide Y., & Scheepers C . (2014). Hearing “moon” and looking up: Word-related spatial associations facilitate saccades to congruent locations. Proceedings of the Annual Meeting of the Cognitive Science Society, 36(36), 433-438.
Feldman, J., Narayanan S . (2004). Embodied meaning in a neural theory of language. Brain and Language, 89(2), 385-392.
Feldman L. B., & Siok W. W. T . (1999a). Semantic radicals contribute to the visual identification of Chinese characters. Journal of Memory and Language, 40(4), 559-576.
Feldman L. B., & Siok W. W. T . (1999b). Semantic radicals in phonetic compounds: Implications for visual character recognition in Chinese. In J. Wang, A. W. Inhoff, & H.-C. Chen (Eds). Reading Chinese script (pp. 19-35). Mahwah, NJ: Erlbaum.
Fernandino L., Binder J. R., Desai R. H., Pendl S. L., Humphries C. J., Gross W. L., … & Seidenberg M. S . (2016). Concept representation reflects multimodal abstraction:A framework for embodied semantics. Cerebral Cortex, 26(5), 2018-2034.
Gallese V., & Sinigaglia C . (2011). What is so special about embodied simulation? Trends in Cognitive Sciences, 15(11), 512-519.
Gallese V., & Sinigaglia C . (2012). Response to de bruin and Gallagher: Embodied simulation as reuse is a productive explanation of a basic form of mind-reading. Trends in Cognitive Sciences, 16(2), 99-100.
Giudice M. D., Manera V., & Keysers C . (2009). Programmed to learn? The ontogeny of mirror neurons. Developmental Science, 12(2), 350-363.
Glenberg A. M . (1997). What memory is for. Behavioral and Brain Sciences, 20(1), 1-19.
Glenberg A. M., & Kaschak M. P . (2002). Grounding language in action. Psychonomic Bulletin & Review, 9(3), 558-565.
Hald L. A., de Nooijer J., van Gog T., & Bekkering H . (2016). Optimizing word learning via links to perceptual and motoric experience. Educational Psychology Review, 28(3), 495-522.
Hauk O., Johnsrude I., & Pulvermüller F . (2004). Somatotopic representation of action words in human motor and premotor cortex. Neuron, 41(2), 301-307.
Horoufchin H., Bzdok D., Buccino G., Borghi A. M., & Binkofski F . (2018). Action and object words are differentially anchored in the sensory motor system - A perspective on cognitive embodiment. Scientific Reports, 8(1), 1-11.
Juárez F. P-G., Labrecque D., & Frak V . (2019). Assessing language-induced motor activity through event related potentials and the grip force sensor, an exploratory study. Brain and Cognition, 135, DOI: 10.1016/j.bandc.2019.05.010.
Keysers D., & Castillo J. G . (2010). The two-level theory of verb meaning: An approach to integrating the semantics of action with the mirror neuron system. Brain and Language, 112(1), 54-76.
Keysers C., & Perrett D. I . (2004). Demystifying social cognition: A Hebbian perspective. Trends in Cognitive Science, 8(11), 501-507.
Klepp A., Weissler H., Niccolai V., Terhalle A., Geisler H., Schnitzler A., & Biermann-Ruben K . (2014). Neuromagnetic hand and foot motor sources recruited during action verb processing. Brain and Language, 128(1), 41-52.
Law S. P., Yeung O., Wong W., & Chiu K. M. Y . (2005). Processing of semantic radicals in writing Chinese characters: Data from a Chinese dysgraphic patient. Cognitive Neuropsychology, 22(7), 885-903.
Li Y., & Kang J. S . (1993). Analysis of phonetics of the ideophonetic characters in modern Chinese. In: Y. Chen (eds.), Information analysis of usage of characters in modern Chinese (pp.84-98). Shanghai: Shanghai Education Press.
[ 李燕, 康家深 . (1993). 现代汉语用字信息分析. 上海: 上海教育出版社. ]
Lin N., Lu X. M., Fang F., Han Z. Z., & Bi Y. C . (2011). Is the semantic category effect in the lateral temporal cortex due to motion property differences? NeuroImage, 55(4), 1853-1864.
Liu C., Tardif T., Mai X. Q., Gehring W. J., Simms N., & Luo Y. J . (2010). What’s in a name? Brain activity reveals categorization processes differ across languages. Human Brain Mapping, 31(11), 1786-1801.
Lu Z. Y., Gao Z. H., Duan X. L., & Liu X. H . (2007). Views of embodied language comprehension. Advances in Psychological Science, 15(2), 275-281.
Richardson D. C., Spivey M. J., Barsalou L. W., & Mcrae K . (2003). Spatial representations activated during real-time comprehension of verbs. Cognitive Science, 27(5), 767-780.
Richardson D. C., Spivey M. J., Edelman S., Naples A. J . (2001). “Language is spatial”: Experimental evidence for image schemas of concrete and abstract verbs. In：Proceedings of the 23ed annual meeting of the cognitive science society (pp. 845-850). Erlbaum [poster presentation], Mahwah.
Rodriguez A. D., Mccabe M. L., Nocera J. R., & Reilly J . (2012). Concurrent word generation and motor performance: Further evidence for language-motor interaction. Plos One, 7(5):e37094.
She X. J., & Zhang B. Y . (1997). The effects of semantic and phonetic clues in puctophonetic character mental lexicon. Journal of Psychology Science, 20(2), 142-145.
Stanfield R. A., & Zwaan R. A . (2001). The effect of implied orientation derived from verbal context on picture recognition. Psychological Science, 12(2), 153-156.
Strozyk J. V., Dudschig C., & Kaup B . (2017). Do I need to have my hands free to understand hand-related language? Investigating the functional relevance of experiential simulations. Psychological Research, 83(3), 406-418.
Su D. Q., Zhong Y., Zeng H., & Ye H. S . (2013). Embodied semantic processing of Chinese action idioms: Evidence from fMRI study. Acta Psychologica Sinica, 45(11), 1187-1199.
Wang X. S., Pei. M., Wu. Y., & Su Y. J . (2017). Semantic radicals contribute more than phonetic radicals to the recognition of Chinese phonograms: Behavioral and ERP evidence in a factorial study. Frontiers in Psychology, DOI: 10.3389/fpsyg.2017.02230.
Wang X. X., Ma X., Tao Y., Tao Y. C., & Li H . (2018). How semantic radicals in Chinese characters facilitate hierarchical category-based induction. Scientific Reports, 8(1), 1-10.
Williams C . (2013). Emerging development of semantic and phonological routes to character decoding in Chinese as a foreign language learners. Reading and Writing, 26(2), 293-315.
Williams C., & Bever T . (2010). Chinese character decoding: A semantic bias? Reading and Writing, 23(5), 589-605.
Winkielman P., Niedenthal P., Wielgosz J., Eelen J., & Kavanagh L. C . (2015). Embodiment of cognition and emotion. Mario Mikulincer, (4), 151-175.
Wu H., Mai X., Tang H., Ge Y., Luo Y. J., & Liu C . (2013). Dissociable somatotopic representations of Chinese action verbs in the motor and premotor cortex. Scientific Reports, 3(1), 1-12.
Wu H. Y., Tang H. H., Ge Y., Yang S. Y., Mai X. Q., Luo Y. J., & Liu C . (2017). Object words modulate the activity of the mirror neuron system during action imitation. Brain and Behavior, 7(11), 1-13.
Wu L. M., Mo L., & Wang R. M . (2007). Dynamic mental simulation during comprehension of motion descriptions. Advances in Psychological Science, 15(4), 605-612.
Zhang J. J., Wang J., & Chen X. K . (2014). Semantic radicals’ research in twenty years: Theoretical exploration, experimental evidence and processing model. Advances in Psychological Science, 22(3), 381-399.
Zhang J. J., Wang J., & Yin C . (2014). The role of phonetic radicals and semantic radicals in phonetics and semantics extraction of phonogram characters: An eye movement study on components perception. Acta Psychologica Sinica, 46(7), 885-900.
Zhang Y. Z., & Zhang J. J . (2017). The effects of neighborhood size and category consistency of the semantic radical on semantic radical’s semantic activation under radical priming paradigm. Acta Psychologica Sinica, 49(8), 1041-1052.