The influential factors and neural mechanisms of McGurk effect

doi:10.3724/SP.J.1042.2018.01935

Abstract

Abstract:

The McGurk effect is a typical audiovisual integration phenomenon, influenced by characteristics of physical stimuli, attentional allocation, the extent that individuals rely on visual or auditory information in processing, the ability of audiovisual integration, and language/culture differences. Key visual information that leads to the McGurk effect is mainly extracted from the mouth area of the talker. The McGurk effect implicates both audiovisual integration (which occurs in the early processing stage and is related to the activation of superior temporal cortex) and the conflict of the incongruent audiovisual stimuli (which occurs in the late processing stage and is related to the activation of inferior frontal cortex). Future studies should further investigate the influence of social factors on the McGurk effect, pay attention to the relationship between unimodal information processing and audiovisual integration in the McGurk effect, and explore the neural mechanisms of McGurk effect with computational modeling.

Key words: McGurk effect, audiovisual speech perception, audiovisual integration, multisensory integration

CLC Number:

B842

LUO Xiaoxiao, KANG Guanlan, ZHOU Xiaolin. The influential factors and neural mechanisms of McGurk effect[J]. Advances in Psychological Science, 2018, 26(11): 1935-1951.

References 92

[1]	雷江华, 方俊明 . ( 2005). 聋人唇读的大脑机制研究. 心理科学, 28( 1), 10-12.
[2]	李燕芳, 梅磊磊, 董奇 . ( 2008). 汉语母语者视听双通道言语知觉的特点及发展研究. 心理发展与教育, 24( 3), 43-47.
[3]	李燕芳, 梅磊磊, 董奇 . ( 2009). 视觉言语在汉语母语儿童和成人英语语音知觉中的作用. 心理科学, 32( 5), 1038-1041.
[4]	朴永馨 . ( 2006). 特殊教育辞典 (第二版). 北京: 华夏出版社.
[5]	钱浩悦, 黄逸慧, 高湘萍 . ( 2018). Gamma神经振荡和信息整合加工. 心理科学进展, 26( 3), 433-441.
[6]	石涯, 王永华, 李文靖 . ( 2016). 唇读对听障儿童语音识别的帮助作用. 听力学及言语疾病杂志, 24( 5), 482-485. doi: 10.3969/j.issn.1006-7299.2016.05.015 URL
[7]	文小辉, 李国强, 刘强 . ( 2011). 视听整合加工及其神经机制. 心理科学进展, 19( 7), 976-982.
[8]	文小辉, 刘强, 孙弘进, 张庆林, 尹秦清, 郝明洁, 牟海蓉 . ( 2009). 多感官线索整合的理论模型. 心理科学进展, 17( 4), 659-666.
[9]	辛昕, 任桂琴, 李金彩, 唐晓雨 . ( 2017). 早期视听整合加工——来自MMN的证据. 心理科学进展, 25( 5), 757-768.
[10]	徐诚 . ( 2013). 唇读研究回顾:从聋人到正常人. 华东师范大学学报(教育科学版), 31( 1), 56-61.
[11]	张明, 陈骐 . ( 2003). 听觉障碍人群的言语机制. 心理科学进展, 11( 5), 486-493.
[12]	Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. ( 2005). Audiovisual integration of speech falters under high attention demands. Current Biology, 15( 9), 839-843.
[13]	Alsius, A., Navarra, J., & Soto-Faraco, S. ( 2007). Attention to touch weakens audiovisual speech integration. Experimental Brain Research, 183( 3), 399-404.
[14]	Alsius, A., Paré, M., & Munhall, K. G. ( 2018). Forty years after hearing lips and seeing voices: The McGurk effect revisited. Multisensory Research, 31( 1-2), 111-144.
[15]	Bayard, C., Colin, C., & Leybaert, J. ( 2014). How is the McGurk effect modulated by cued speech in deaf and hearing adults? Frontiers in Psychology, 5, 416. doi: 10.3389/fpsyg.2014.00416 URL pmid: 4032946
[16]	Beauchamp, M. S., Nath, A. R., & Pasalar, S. ( 2010). fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. The Journal of Neuroscience, 30( 7), 2414-2417.
[17]	Bertelson, P., Vroomen, J., & de Gelder, B. ( 2003). Visual recalibration of auditory speech identification: A McGurk after effect. Psychological Science, 14( 6), 592-597. doi: 10.1046/j.0956-7976.2003.psci_1470.x URL pmid: 14629691
[18]	Besle, J., Fort, A., Delpuech, C., & Giard, M. ( 2004). Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journal of Neuroscience, 20( 8), 2225-2234.
[19]	Brancazio, L., & Miller, J.L. ( 2005). Use of visual information in speech perception: Evidence for a visual rate effect both with and without a McGurk effect. Perception & Psychophysics, 67( 5), 759-769.
[20]	Buchan, J.N., & Munhall, K.G . ( 2012). The effect of a concurrent working memory task and temporal offsets on the integration of auditory and visual speech information. Seeing and Perceiving, 25( 1), 87-106.
[21]	Burnham, D., & Dodd, B.( 2004). Auditory-visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect. Developmental Psychobiology, 45( 4), 204-220.
[22]	Burnham, D., & Dodd, B.( 2018). Language-general auditory- visual speech perception: Thai-English and Japanese- English McGurk effects. Multisensory Research, 31( 1-2), 79-110.
[23]	Colin, C., Radeau, M., Soquet, A., & Deltenre, P. ( 2004). Generalization of the generation of an MMN by illusory McGurk percepts: Voiceless consonants. Clinical Neurophysiology, 115( 9), 1989-2000. doi: 10.1016/j.clinph.2004.03.027 URL pmid: 15294201
[24]	Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., & Deltenre, P. ( 2002). Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology, 113( 4), 495-506.
[25]	de Gelder, B.., & Vroomen, J.( 2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14( 3), 289-311. doi: 10.1080/026999300378824 URL
[26]	de Gelder, B., Vroomen, J.,& van der Heide, L.( 1991). Face recognition and lip-reading in autism. European Journal of Cognitive Psychology, 3( 1), 69-86.
[27]	Eskelund, K., MacDonald, E. N., & Andersen, T. S. ( 2015). Face configuration affects speech perception: Evidence from a McGurk mismatch negativity study. Neuropsychologia, 66, 48-54.
[28]	Fang, F., & He, S.( 2005). Cortical responses to invisible objects in the human dorsal and ventral pathways. Nature Neuroscience, 8( 10), 1380-1385. doi: 10.1038/nn1537 URL pmid: 16136038
[29]	Fernández, L. M., Macaluso, E., & Soto-Faraco, S. ( 2017). Audiovisual integration as conflict resolution: The conflict of the McGurk illusion. Human Brain Mapping, 38( 11), 5691-5705.
[30]	Gau, R., & Noppeney, U.( 2016). How prior expectations shape multisensory perception. Neuroimage, 124, 876-886.
[31]	Gurler, D., Doyle, N., Walker, E., Magnotti, J., & Beauchamp, M. ( 2015). A link between individual differences in multisensory speech perception and eye movements. Attention, Perception, & Psychophysics, 77( 4), 1333-1341.
[32]	Hisanaga, S., Sekiyama, K., Igasaki, T., & Murayama, N. ( 2016). Language/culture modulates brain and gaze processes in audiovisual speech perception. Scientific Reports, 6, 35265.
[33]	Hockley, N.S., & Polka, L.( 1994). A developmental study of audiovisual speech perception using the McGurk paradigm. The Journal of the Acoustical Society of America, 96( 5), 3309-3318.
[34]	Irwin, J., Avery, T., Brancazio, L., Turcios, J., Ryherd, K., & Landi, N. ( 2018). Electrophysiological indices of audiovisual speech perception: Beyond the McGurk effect and speech in noise. Multisensory Research, 31( 1-2), 39-56.
[35]	Jones, J.A., & Callan, D.E . ( 2003). Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. NeuroReport, 14( 8), 1129-1133.
[36]	Jordan, T. R., McCotter, M. V., & Thomas, S. M. ( 2000). Visual and audiovisual speech perception with color and gray-scale facial images. Perception & Psychophysics, 62( 7), 1394-1404. doi: 10.3758/BF03212141 URL pmid: 11143451
[37]	Jordan, T.R., & Sergeant, P.( 2000). Effects of distance on visual and audiovisual speech recognition. Language and Speech, 43( 1), 107-124. doi: 10.1177/00238309000430010401 URL
[38]	Jordan, T.R., & Thomas, S.M . ( 2011). When half a face is as good as a whole: Effects of simple substantial occlusion on visual and audiovisual speech perception. Attention, Perception, & Psychophysic, 73( 7), 2270-2285.
[39]	Kaiser, J., Hertrich, I., Ackermann, H., Mathiak, K., & Lutzenberger, W. ( 2005). Hearing lips: Gamma-band activity during audiovisual speech perception. Cerebral Cortex, 15( 5), 646-653.
[40]	Keil, J., Müller, N., Ihssen, N., & Weisz, N. ( 2012). On the variability of the McGurk effect: Audiovisual integration depends on prestimulus brain states. Cerebral Cortex, 22( 1), 221-231.
[41]	Lange, J., Christian, N., & Schnitzler, A. ( 2013). Audio- visual congruency alters power and coherence of oscillatory activity within and between cortical areas. Neuroimage, 79, 111-120.
[42]	Lüttke, C. S., Ekman, M., van Gerven, M. A., & de Lange, F. P.( 2015). Preference for audiovisual speech congruency in superior temporal cortex. Journal of Cognitive Neuroscience, 28( 1), 1-7.
[43]	Lüttke, C. S., Ekman, M., van Gerven, M. A. J., & de Lange, F. P.( 2016). McGurk illusion recalibrates subsequent auditory perception. Scientific Reports, 6, 32891. doi: 10.1038/srep32891 URL pmid: 5017187
[44]	MacDonald, J.( 2018). Hearing lips and seeing voices: The origins and development of the 'McGurk effect' and reflections on audio-visual speech perception over the last 40 years. Multisensory Research, 31( 1-2), 7-18.
[45]	MacDonald, J., Andersen, S., & Bachmann, T. ( 2000). Hearing by eye: How much spatial degradation can be tolerated? Perception, 29( 10), 1155-1168. doi: 10.1068/p3020 URL pmid: 11220208
[46]	Macsweeney, M., Amaro, E., Calvert, G. A., Campbell, R., David, A. S., McGuire, P., ... Brammer, M. J. ( 2000). Silent speechreading in the absence of scanner noise: An event-related fMRI study. Neuroreport, 11( 8), 1729-1733. doi: 10.1097/00001756-200006050-00026 URL
[47]	Macsweeney, M., Calvert, G. A., Campbell, R., McGuire, P. K., David, A. S., Williams, S. C. R., ... Brammer, M. J. ( 2002). Speechreading circuits in people born deaf. Neuropsychologia, 40( 7), 801-807. doi: 10.1016/S0028-3932(01)00180-4 URL
[48]	Magnotti, J.F., & Beauchamp, M.S . ( 2015). The noisy encoding of disparity model of the McGurk effect. Psychonomic Bulletin & Review, 22( 3), 701-709.
[49]	Magnotti, J.F., & Beauchamp, M.S . ( 2017). A causal inference model explains perception of the McGurk effect and other incongruent audiovisual speech. PLoS Computational Biology, 13( 2), e1005229.
[50]	Magnotti, J. F., Mallick, D. B., & Beauchamp, M. S. ( 2018). Reducing playback rate of audiovisual speech leads to a surprising decrease in the McGurk effect. Multisensory Research, 31( 1-2), 19-38.
[51]	Magnotti, J. F., Mallick, D. B., Feng, G., Zhou, B., Zhou, W., & Beauchamp, M. S. ( 2015). Similar frequency of the McGurk effect in large samples of native Mandarin Chinese and American English speakers. Experimental Brain Research, 233( 9), 2581-2586.
[52]	Mallick, D. B., Magnotti, J. F., & Beauchamp, M. S. ( 2015). Variability and stability in the McGurk effect: Contributions of participants, stimuli, time, and response type. Psychonomic Bulletin & Review, 22( 5), 1299-1307.
[53]	Marques, L. M., Lapenta, O. M., Costa, T. L., & Boggio, P. S. ( 2016). Multisensory integration processes underlying speech perception as revealed by the McGurk illusion. Language, Cognition and Neuroscience, 31( 9), 1115-1129.
[54]	Marques, L. M., Lapenta, O. M., Merabet, L. B., Bolognini, N., & Boggio, P. S. ( 2014). Tuning and disrupting the brain-modulating the McGurk illusion with electrical stimulation. Frontiers in Human Neuroscience, 8, 533.
[55]	McGurk, H., & MacDonald, J.( 1976). Hearing lips and seeing voices. Nature, 264( 5588), 746-748.
[56]	Miller, L.M., & D'Esposito, M.( 2005). Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. The Journal of Neuroscience, 25( 25), 5884-5893. doi: 10.1523/JNEUROSCI.0896-05.2005 URL pmid: 15976077
[57]	Moro, S.S., & Steeves, J. K.E . ( 2018). Audiovisual plasticity following early abnormal visual experience: Reduced McGurk effect in people with one eye. Neuroscience Letters, 672, 103-107. doi: 10.1016/j.neulet.2018.02.031 URL
[58]	Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. ( 1996). Temporal constraints on the McGurk effect. Perception, & Psychophysics, 58( 3), 351-362.
[59]	Munhall, K. G., ten Hove, M. W., Brammer, M., & Paré, M. ( 2009). Audiovisual integration of speech in a bistable illusion. Current Biology, 19( 9), 735-739.
[60]	Nath, A.R., & Beauchamp, M.S . ( 2012). A neural basis for interindividual differences in the McGurk Eeffect, a multisensory speech illusion. NeuroImage, 59( 1), 781-787.
[61]	Nath, A. R., Fava, E. E., & Beauchamp, M. S. ( 2011). Neural correlates of interindividual differences in children's audiovisual speech perception. The Journal of Neuroscience, 31( 39), 13963-13971.
[62]	Olasagasti, I., Bouton, S., & Giraud, A. L. ( 2015). Prediction across sensory modalities: A neurocomputational model of the McGurk effect. Cortex, 68, 61-75.
[63]	Palmer, T.D., & Ramsey, A.K . ( 2012). The function of consciousness in multisensory integration. Cognition, 125( 3), 353-364.
[64]	Paré, M., Richler, R. C., ten Hove, M., & Munhall, K. G. ( 2003). Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect. Perception, & Psychophysics, 65( 4), 553-567.
[65]	Proverbio, A. M., Massetti, G., Rizzi, E., & Zani, A. ( 2016). Skilled musicians are not subject to the McGurk effect. Scientific Reports, 6, 30423.
[66]	Quinto, L., Thompson, W. F., Russo, F. A., & Trehub, S. E. ( 2010). A comparison of the McGurk effect for spoken and sung syllables. Attention, Perception, & Psychophysics, 72( 6), 1450-1454. doi: 10.3758/APP.72.6.1450 URL pmid: 20675792
[67]	Romero, Y. R., Senkowski, D., & Keil, J. ( 2015). Early and late beta-band power reflect audiovisual perception in the McGurk illusion. Journal of Neurophysiology, 113( 7), 2342-2350. doi: 10.1152/jn.00783.2014 URL pmid: 25568160
[68]	Rosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. ( 1997). The McGurk effect in infants. Perception & Psychophysics, 59( 3), 347-357.
[69]	Rosenblum, L. D., Yakel, D. A., & Green, K. P. ( 2000). Face and mouth inversion effects on visual and audiovisual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 26( 2), 806-819.
[70]	Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. ( 2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17( 5), 1147-1153.
[71]	Rouger, J., Fraysse, B., Deguine, O., & Barone, P. ( 2008). McGurk effects in cochlear-implanted deaf subjects. Brain Research, 1188( 1), 87-99. doi: 10.1016/j.brainres.2007.10.049 URL pmid: 18062941
[72]	Saint-Amour, D., De Sanctis, P., Molholma, S., Ritter, W., & Foxe, J. J. ( 2007). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia, 45( 3), 587-597.
[73]	Samuel, A.G. ( 2011). Speech perception. Annual Review of Psychology, 62( 1), 49-72.
[74]	Sekiyama, K.( 1997). Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception & Psychophysics, 59( 1), 73-80.
[75]	Sekiyama, K., Soshi, T., & Sakamoto, S. ( 2014). Enhanced audiovisual integration with aging in speech perception: A heightened McGurk effect in older adults. Frontiers in Psychology, 5, 323.
[76]	Sekiyama, K., & Tohkura, Y.( 1993). Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21( 4), 427-444.
[77]	Soto-Faraco, S., & Alsius, A.( 2009). Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance, 35( 2), 580-587. doi: 10.1037/a0013483 URL pmid: 19331510
[78]	Stein, B.E., & Stanford, T.R . ( 2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9, 255-266.
[79]	Stevenson, R. A., Zemtsov, R. K., & Wallace, M. T. ( 2012). Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. Journal of Experimental Psychology: Human Perception and Performance, 38( 6), 1517-1529.
[80]	Strand, J., Cooperman, A., Rowe, J., & Simenstad, A. ( 2014). Individual differences in susceptibility to the McGurk effect: Links with lipreading and detecting audiovisual incongruity. Journal of Speech Language and Hearing Research, 57( 6), 2322-2331.
[81]	Summerfield, Q.( 1992). Lipreading and audio-visual speech perception. Philosophical Transactions: Biological Sciences, 335( 1273), 71-78. doi: 10.1098/rstb.1992.0009 URL
[82]	Thomas, S.M., & Jordan, T.R . ( 2002). Determining the influence of Gaussian blurring on inversion effects with talking faces. Perception & Psychophysics, 64( 6), 932-944. doi: 10.3758/BF03196797 URL pmid: 12269300
[83]	Thomas, S.M., & Jordan, T.R . ( 2004). Contributions of oral and extraoral facial movement to visual and audiovisual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 30( 5), 873-888. doi: 10.1037/0096-1523.30.5.873 URL pmid: 15462626
[84]	Tiippana, K.( 2014). What is the McGurk effect? Frontiers in Psychology, 5, 725. doi: 10.3389/fpsyg.2014.00725 URL pmid: 4091305
[85]	Tiippana, K., Andersen, T. S., & Sams, M. ( 2004). Visual attention modulates audiovisual speech perception. European Journal of Cognitive Psychology, 16( 3), 457-472.
[86]	Tsuchiya, N., & Koch, C.( 2005). Continuous flash suppression reduces negative afterimages. Nature Neuroscience, 8( 8), 1096-1101.
[87]	Ujiie, Y., Asai, T., & Wakabayashi, A. ( 2015). The relationship between level of autistic traits and local bias in the context of the McGurk effect. Frontiers in Psychology, 6, 891.
[88]	Ujiie, Y., Asai, T., & Wakabayashi, A. ( 2018). Individual differences and the effect of face configuration information in the McGurk effect. Experimental Brain Research, 236( 4), 973-986. doi: 10.1007/s00221-018-5188-4 URL pmid: 29383400
[89]	Van Engen, K. J., Xie, Z., & Chandrasekaran, B. ( 2017). Audiovisual sentence recognition not predicted by susceptibility to the McGurk effect. Attention Perception & Psychophysics, 79( 2), 396-403.
[90]	Walker, S., Bruce, V., & O'Malley, C. ( 1995). Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect. Perception & Psychophysics, 57( 8), 1124-1133. doi: 10.3758/BF03208369 URL pmid: 8539088
[91]	Wilson, A. H., Alsius, A., Paré, M., & Munhall, K. G. ( 2016). Spatial frequency requirements and gaze strategy in visual-only and audiovisual speech perception. Journal of Speech, Language, and Hearing Research, 59( 4), 601-615.
[92]	Zhu, L.L., & Beauchamp, M.S . ( 2017). Mouth and voice: A relationship between visual and auditory preference in the human superior temporal sulcus. The Journal of Neuroscience, 37( 10), 2697-2708.