The influence of language and context on sensorimotor simulation of concrete concepts

doi:10.3724/SP.J.1041.2022.00583

Abstract

Abstract:

Sensorimotor simulation in concrete concept processing is the core process of concept representation. In our study, the influence of linguistic factors (language type: first language and second language) and situational factors (perceptual situation: spatial perceptual situation and semantic perceptual situation) on sensorimotor simulation in concrete concept processing were investigated.
In Experiment 1, the semantic relevance judgment paradigm is used to test whether sensorimotor simulation is involved in the second language processing and whether there is any difference between the first language and second language processing. The results of Experiment 1 show that sensorimotor simulation has participated in the second language processing. However, there is an accuracy advantage in the sensorimotor simulation in the first language. In Experiment 2, two experiments are conducted to test the influence of the perceptual situation on sensorimotor simulation. Experiment 2a tests the influence of perceptual situations on sensorimotor simulation by changing the intensity of individual perception in the vertical spatial axis. The results show that the sensorimotor simulation was found in both strong and weak spatial perception. The effect of semantic processing level on perceptual motion simulation is tested in Experiment 2b. The experiment manipulates the level of semantic processing by using the semantic relevance judgment task and true-and-false word judgment task. The results show that sensorimotor simulation participates in the processing of concepts, while semantic situations did not modulate this effect. The results of Experiment 2 suggest that sensorimotor stimulation may be automatic in conceptual processing.
The present study conducted two experiments to test the role of sensorimotor simulation in conceptual processing. Results found that sensorimotor simulation participates in the conceptual representation, supporting the perceptual symbol theory. Furthermore, sensorimotor simulation takes place in both Chinese and English. This finding extends the perceptual symbol theory. Finally, results found that sensorimotor stimulation may be automatic in conceptual representation and is not affected by spatial information and semantic processing.

Key words: concrete concept, conceptual representation, sensorimotor simulation, perceptual symbol theory

SHI Rubin, XIE Jiushu, YANG Mengqing, WANG Ruiming. (2022). The influence of language and context on sensorimotor simulation of concrete concepts. Acta Psychologica Sinica, 54(6), 583-594.

Figures/Tables 12

Figure 1A. The flow chart of Chinese semantic relevance judgment task in Experiment 1.

Figure 1B. The flow chart of English semantic relevance judgment task in Experiment 1.

Table 1 The response time of different spatial relationships under the condition of language type (M ± SD)

Language type	Match (ms)	Mismatch (ms)
Chinese	784.00 ± 82.32	799.02 ± 94.00
English	1004.66 ± 110.66	1030.88 ± 144.82

Table 2 The correct rate of different spatial relationships under the condition of language type (M ± SD)

Language type	Match	Mismatch
Chinese	0.97 ± 0.04	0.95 ± 0.04
English	0.92 ± 0.07	0.93 ± 0.07

Figure 2A. The flow chart of strong spatial perception in Experiment 2a.

Figure 2B. The flow chart of weak spatial perception in Experiment 2a.

Table 3 The response time of different spatial relationships under spatial perceptual situation (M ± SD)

Spatial perception	Match (ms)	Mismatch (ms)
Strong spatial perception	732.53 ± 97.06	744.35 ± 88.56
Weak spatial perception	672.34 ± 78.68	679.83 ± 75.84

Table 4 The accuracy of different spatial relationships in the spatial perceptual situation (M ± SD)

Spatial perception	Match	Mismatch
Strong spatial perception	0.94 ± 0.07	0.94 ± 0.07
Weak spatial perception	0.96 ± 0.05	0.97 ± 0.03

Figure 3A. The flow chart of semantic relevance judgment task in Experiment 2b.

Figure 3B. The flow chart of true-and-false word judgement task in Experiment 2b.

Table 5 Response time of different spatial relationships in the semantic perceptual situation (M ± SD)

Task Type	Match (ms)	Mismatch (ms)
Semantic relevance judgment task	776.95 ± 99.58	791.62 ± 108.60
True-and-false word judgment task	839.05 ± 116.82	853.53 ± 101.66

Table 6 The accuracy of different spatial relationships in the semantic perceptual situation (M ± SD)

Task Type	Match	Mismatch
Semantic relevance judgment task	0.93 ± 0.10	0.95 ± 0.08
True-and-false word judgment task	0.98 ± 0.05	0.97 ± 0.05

References 45

[1]	Barsalou L. W. (1999). Perceptions of perceptual symbols. Behavioral and Brain Sciences, 22(4), 637-660. doi: 10.1017/S0140525X99532147 URL
[2]	Barsalou L. W. (2008). Grounded cognition. Annual Review of Psychology, 59(1), 617-645. doi: 10.1146/annurev.psych.59.103006.093639 URL
[3]	Barsalou L. W., Dutriaux L., & Scheepers C. (2018). Moving beyond the distinction between concrete and abstract concepts. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1752).
[4]	Baumeister J. C., Foroni F., Conrad M., Rumiati R. I., & Winkielman P. (2017). Embodiment and emotional memory in first vs. second language. Frontiers in Psychology, 8(8), 394.
[5]	Bergen B., Lau T. T. C., Narayan S., Stojanovic D., & Wheeler K. (2010). Body part representations in verbal semantics. Memory & Cognition, 38(7), 969-981. doi: 10.3758/MC.38.7.969 URL
[6]	Chen D., Wang R., Zhang J., & Liu C. (2020). Perceptual representations in L1, L2 and L3 comprehension: Delayed sentence-picture verification. Journal of Psycholinguistic Research, 49(1), 41-57. doi: 10.1007/s10936-019-09670-x URL
[7]	Costa A., Santesteban M. & Cano A. (2005). On facilitatory effect cognate words in bilingual speech produefion. Brain and Language, 94(9): 94-103. doi: 10.1016/j.bandl.2004.12.002 URL
[8]	Dudschig C., de la Vega I., de Filippis M., & Kaup B. (2014). Language and vertical space: On the automaticity of language action interconnections. Cortex, 58, 151-160. doi: 10.1016/j.cortex.2014.06.003 pmid: 25016318
[9]	Dudschig C., de la Vega I., & Kaup B. (2014). Embodiment and second-language: Automatic activation of motor responses during processing spatially associated L2 words and emotion L2 words in a vertical stroop paradigm. Brain and Language, 132, 14-21. doi: 10.1016/j.bandl.2014.02.002 pmid: 24681402
[10]	Dudschig C., & Kaup B. (2017). Is it all task-specific? The role of binary responses, verbal mediation, and saliency for eliciting language-space associations. Journal of Experimental Psychology: Learning, Memory, and Cognition, 43(2), 259-270. doi: 10.1037/xlm0000297 URL
[11]	Dudschig C., Lachmair M., de la Vega I., de Filippis M., & Kaup B. (2012). From top to bottom: Spatial shifts of attention caused by linguistic stimuli. Cognitive Processing, 13(1), 151-154.
[12]	Dudschig C., Souman J., Lachmair M., de la Vega I., & Kaup B. (2013). Reading “sun” and looking up: The influence of language on saccadic eye movements in the vertical dimension. Plos One, 8(2), e56872.
[13]	Dunn B., Kamide Y., & Scheepers C. (2014). Hearing “moon” and looking up:Word-related spatial associations facilitate saccades to congruent locations. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 36, No. 36).
[14]	Eilola T. M., & Havelka J. (2010). Behavioural and physiological responses to the emotional and taboo stroop tasks in native and non-native speakers of English. International Journal of Bilingualism, 15(3), 353- 369. doi: 10.1177/1367006910379263 URL
[15]	Estes Z., & Barsalou L. W. (2018). A comprehensive meta-analysis of spatial interference from linguistic cues: Beyond Petrova et al. (2018). Psychological Science, 29(9), 1558-1564. doi: 10.1177/0956797618794131 URL
[16]	Fodor J. A. (1975). The language of thought. Cambridge, MA: Harvard University Press.
[17]	Glenberg A. M., & Gallese V. (2012). Action-based language: A theory of language acquisition, comprehension, and production. Cortex, 48(7), 905-922. doi: 10.1016/j.cortex.2011.04.010 pmid: 21601842
[18]	Glenberg A. M., & Kaschak M. P. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9(3), 558-565. doi: 10.3758/BF03196313 URL
[19]	Grauwe S. D., Willems R. M., Rueschemeyer S. A., Lemhöfer K., & Schriefers H. (2014). Embodied language in first- and second-language speakers: Neural correlates of processing motor verbs. Neuropsychologia, 56, 334-349. doi: 10.1016/j.neuropsychologia.2014.02.003 URL
[20]	Günther F., Dudschig C., & Kaup B. (2018). Symbol grounding without direct experience: Do words inherit sensorimotor activation from purely linguistic context?. Cognitive Science, 42, 336-374. doi: 10.1111/cogs.12549 URL
[21]	Harnad S. (1990). The symbol grounding problem. Physica D Nonlinear Phenomena, 42(1-3), 335-346. doi: 10.1016/0167-2789(90)90063-U URL
[22]	Harris C. L., Ayçíçeğí A., & Gleason J. B. (2003). Taboo words and reprimands elicit greater autonomic reactivity in a first language than in a second language. Applied Psycholinguistics, 24(4), 561-579. doi: 10.1017/S0142716403000286 URL
[23]	Jared D., Pei Yun Poh R., & Paivio A. (2013). L1 and L2 picture naming in Mandarin-English bilinguals: A test of bilingual dual coding theory. Bilingualism: Language and Cognition, 16(2), 383-396. doi: 10.1017/S1366728912000685 URL
[24]	Lachmair M., Dudschig C., Filippis M. D., Vega I., & Kaup B. (2011). Root versus roof: Automatic activation of location information during word processing. Psychonomic Bulletin & Review, 18(6), 1180-1188. doi: 10.3758/s13423-011-0158-x URL
[25]	Lakens D. (2012). Polarity correspondence in metaphor congruency effects: Structural overlap predicts categorization times for bipolar concepts presented in vertical space. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38(3), 726-736. doi: 10.1037/a0024955 URL
[26]	Lakoff G., & Johnson M. (1980). Metaphors we live by. Chicago, IL: University of Chicago Press.
[27]	Lebois L. A., Wilson-Mendenhall C. D., & Barsalou L. W. (2014). Are automatic conceptual cores the gold standard of semantic processing? The context - dependence of spatial meaning in grounded congruency effects. Cognitive Science, 39(8), 1764-1801. doi: 10.1111/cogs.12174 URL
[28]	Lewis M. P., Simons G. F., & Fennig C. D. (2015). Ethnologue: Languages of the world (18th ed.). SIL International.
[29]	Liu W. J., Li Y., & Wang R. M. (2015). The role of sensorimotor information in conceptual representation. Journal of Psychological Science, 38(6), 1347-1352.
[30]	Liu W. J., Shen M. Q., Li Y., & Wang R. M. (2016). The interaction between emotional concept processing and emotional face perception. Acta Psychologica Sinica, 48(2), 163-173. doi: 10.3724/SP.J.1041.2016.00163 URL
[31]	Mo L., Wu L. M., & Wang R. M. (2006). Spatial iconicity affects semantic relatedness judgment. Journal of Psychological Science, 29(4), 770-773.
[32]	Nielsen A. K. S., & Rendall D. (2013). Parsing the role of consonants versus vowels in the classic Takete-Maluma phenomenon. Canadian Journal of Experimental Psychology, 67(2), 153-163. doi: 10.1037/a0030553 URL
[33]	ÖTTL B., Dudschig C., & Kaup B. (2017). Forming associations between language and sensorimotor traces during novel word learning. Language and Cognition, 9(1), 156-171. doi: 10.1017/langcog.2016.5 URL
[34]	Proctor R. W., & Xiong A. (2015). Polarity correspondence as a general compatibility principle. Current Directions in Psychological Science, 24(6), 446-451. doi: 10.1177/0963721415607305 URL
[35]	Pylyshyn Z. W. (1984). Computation and cognition. Cambridge, MA: MIT Press.
[36]	Strozyk J. V., Dudschig C., & Kaup B. (2019). Do I need to have my hands free to understand hand-related language? Investigating the functional relevance of experiential simulations. Psychological Research, 83(3), 406-418. doi: 10.1007/s00426-017-0900-8 URL
[37]	Vukovic N., & Shtyrov Y. (2014). Cortical motor systems are involved in second-language comprehension: Evidence from rapid mu-rhythm desynchronisation. NeuroImage, 102, 695-703. doi: 10.1016/j.neuroimage.2014.08.039 URL
[38]	Wang R. M., Mo L., Li L., Wang S. P., & Wu J. (2005). Perceptual symbol representation and propositional symbol representation in language comprehension. Acta Psychologica Sinica, 37(2), 143-150.
[39]	Wang R. M., Mo L., Wu L. M., & Li L. (2006). The influence of spatial information on semantic relatedness judgment. Journal of Psychological Science, 29(6), 1320-1323.
[40]	Wang R. M., Zhang J. T., Li L., & Mo L. (2010). The role of bilinguals^,second language in semantic access of third language. Journal of Psychological Science, 33(4), 853-856.
[41]	Xie J. S., Shen G. Y., & Huang Y. L. (2022). Debates and integration of bilinguals’ cross-language orthographic transfer. Journal of Psychological Science, in press.
[42]	Zhao T. Y., Huang Y. L., Chen D. G., Jiao L., Ramos F. M., Wang R. M., & Xie J. S. (2020). The modality switching costs of Chinese - English bilinguals in the processing of L1 and L2. Quarterly Journal of Experimental Psychology, 73(3) 396-412. doi: 10.1177/1747021819878089 URL
[43]	Zwaan R. A., & Madden C. J. (2005). Embodied sentence comprehension. In D. Pecher & R. Zwaan (Eds.),Grounding cognition: The role of perception and action in memory, language, and thinking (pp.224-245). Cambridge: Cambridge University Press.
[44]	Zwaan R. A., Stanfield R., & Yaxley R. H.,(2002). Language comprehenders mentally represent the shapes of objects. Psychological Science, 13(2), 168-171. pmid: 11934002
[45]	Zwaan R. A., & Yaxley R. H. (2003). Spatial iconicity affects semantic relatedness judgments. Psychonomic Bulletin & Review, 10(4), 954-958. doi: 10.3758/BF03196557 URL