Sound-induced flash illusion in multisensory integration

doi:10.3724/SP.J.1042.2020.01662

Abstract

Abstract:

Sound-induced Flash Illusion (SiFI) is an auditory-dominated multisensory integration phenomenon that occurs when flashes presented in conjunction with an incongruent number of auditory sounds results in the flashes being perceived as equal in number to the auditory sounds. Here we summarize several factors that could impact on SiFI: (1) Bottom-up variance factors comprising physical properties (i.e., spatial characteristics of stimuli, differences in stimulus intensity, and time interval between audiovisual stimuli), top-down variance factors comprisingcognitive processes (i.e., attention, feedback patterns, and cognitive expectations), and between-subjects variance factors, such as the dependence for audiovisual stimuli, the ability to integrate audiovisual stimuli, and perceptual sensitivity to audiovisual stimuli; (2) Cognitive processing stages and neural mechanisms of SiFI (i.e., related brain regions). Future studies should focus on the effects of cognitive processes such as attention, reward, audiovisual integration on SiFI. The effects of SiFI on memory and learning should also be explored in future research. In addition, future studies combining computational models and neuroscience should further explore the cognitive and neural mechanisms of SiFI.

Key words: sound induced flash illusion (SiFI), fission illusion, fusion illusion, multisensory integration

CLC Number:

B842

WANG Aijun, HUANG Jie, LU Feifei, HE Jiaying, TANG Xiaoyu, ZHANG Ming. Sound-induced flash illusion in multisensory integration[J]. Advances in Psychological Science, 2020, 28(10): 1662-1677.

Figures/Tables 1

References 80

[1]	罗霄骁, 康冠兰, 周晓林. (2018). McGurk效应的影响因素与神经基础. 心理科学进展, 26(11), 1935-1951.
[2]	文小辉, 李国强, 刘强. (2011). 视听整合加工及其神经机制. 心理科学进展, 19(7), 976-982.
[3]	于薇, 王爱君, 张明. (2017). 集中和分散注意对多感觉整合中听觉主导效应的影响. 心理学报, 49(2), 164-173. doi: 10.3724/SP.J.1041.2017.00164 URL
[4]	张明, 唐晓雨, 于薇, 宁波, 王智楠, 王爱君. (2018). 基于通道的内源性注意对声音诱发闪光错觉的影响. 心理学报, 50(11), 20-29.
[5]	Abadi, R. V., & Murphy, J. S. (2014). Phenomenology of the sound-induced flash illusion. Experimental Brain Research, 232(7), 2207-2220. doi: 10.1007/s00221-014-3912-2 URL pmid: 24691756
[6]	Andersen, T. S., Tiippana, K., & Sams, M. (2004). Factors influencing audiovisual fission and fusion illusions. Cognitive Brain Research, 21(3), 301-308. doi: 10.1016/j.cogbrainres.2004.06.004 URL pmid: 15511646
[7]	Balz, J., Keil, J., Romero, Y. R., Mekle, R., Schubert, F., Aydin, S., ... Senkowski, D. (2016). GABA concentration in superior temporal sulcus predicts gamma power and perception in the sound-induced flash illusion. NeuroImage, 125, 724-730. URL pmid: 26546865
[8]	Balz, J., Roa Romero, Y., Keil, J., Krebber, M., Niedeggen, M., Gallinat, J., & Senkowski, D. (2016). Beta/Gamma oscillations and event-related potentials indicate aberrant multisensory processing in schizophrenia. Frontiers in Psychology, 7, 1896. URL pmid: 27999553
[9]	Bao, V. A., Doobay, V., Mottron, L., Collignon, O., & Bertone, A. (2017). Multisensory integration of low-level information in autism spectrum disorder: Measuring susceptibility to the flash-beep illusion. Journal of Autism and Developmental Disorders, 47(8), 2535-2543. doi: 10.1007/s10803-017-3172-7 URL
[10]	Barutchu, A., Sahu, A., Humphreys, G. W., & Spence, C. (2019). Multisensory processing in event-based prospective memory. Acta Psychologica, 192, 23-30. URL pmid: 30391627
[11]	Bertelson, P., & Ascherslebenc, G. (2003). Temporal ventriloquism: Crossmodal interaction on the time dimension: Evidence from auditory-visual temporal order judgment. International Journal of Psychophysiology, 50(1-2), 147-155. doi: 10.1016/s0167-8760(03)00130-2 URL pmid: 14511842
[12]	Bidelman, G. M. (2016). Musicians have enhanced audiovisual multisensory binding: Experience-dependent effects in the double-flash illusion. Experimental Brain Research, 234(10), 3037-3047. doi: 10.1007/s00221-016-4705-6 URL pmid: 27334887
[13]	Bolognini, N., Rossetti, A., Casati, C., Mancini, F., & Vallar, G. (2011). Neuromodulation of multisensory perception: A tDCS study of the sound-induced flash illusion. Neuropsychologia, 49(2), 231-237. URL pmid: 21094177
[14]	Brighina, F., Bolognini, N., Cosentino, G., Maccora, S., Paladino, P., Baschi, R., ... Fierro, B. (2015). Visual cortex hyperexcitability in migraine in response to sound-induced flash illusions. Neurology, 84(20), 2057-2061. URL pmid: 25888559
[15]	Brunetti, R., Indraccolo, A., Mastroberardino, S., Spence, C., & Santangelo, V. (2017). The impact of cross-modal correspondences on working memory performance. Journal of Experimental Psychology: Human Perception and Performance, 43(4), 819-831. doi: 10.1037/xhp0000348 URL pmid: 28345948
[16]	Cecere, R., Rees, G., & Romei, V. (2015). Individual differences in alpha frequency drive crossmodal illusory perception. Current Biology, 25(2), 231-235. doi: 10.1016/j.cub.2014.11.034 URL pmid: 25544613
[17]	Chan, J. S., Connolly, S. K., & Setti, A. (2018). The number of stimulus-onset asynchronies affects the perception of the sound-induced flash illusion in young and older adults. Multisensory Research, 31(3-4), 175-190. doi: 10.1163/22134808-00002605 URL pmid: 31264623
[18]	Chan, J. S., Kaiser, J., Brandl, M., Matura, S., Prvulovic, D., Hogan, M., & Naumer, M. J. (2015). Expanded temporal binding windows in people with mild cognitive impairment. Current Alzheimer Research, 12(1), 61-68. URL pmid: 25523426
[19]	Chan, J. S., Wibral, M., Wollstadt, P., Stawowsky, C., Brandl, M., Helbling, S., ... Kaiser, J. (2017). Predictive coding over the lifespan:Increased reliance on perceptual priors in older adults — A magnetoencephalography and dynamic causal modelling study. BioRxiv, 178095.
[20]	Cuppini, C., Magosso, E., Bolognini, N., Vallar, G., & Ursino, M. (2014). A neurocomputational analysis of the sound- induced flash illusion. NeuroImage, 92, 248-266. doi: 10.1016/j.neuroimage.2014.02.001 URL pmid: 24518261
[21]	de Haas, B., Kanai, R., Jalkanen, L., & Rees, G. (2012). Grey matter volume in early human visual cortex predicts proneness to the sound-induced flash illusion. Proceedings of the Royal Society B: Biological Sciences, 279(1749), 4955-4961. URL pmid: 23097516
[22]	DeLoss, D. J., & Andersen, G. J. (2015). Aging, spatial disparity, and the sound-induced flash illusion. PLoS ONE, 10(11), e0143773. doi: 10.1371/journal.pone.0143773 URL pmid: 26619352
[23]	DeLoss, D. J., Pierce, R. S., & Andersen, G. J. (2013). Multisensory integration, aging, and the sound-induced flash illusion. Psychology Aging, 28(3), 802-812. doi: 10.1037/a0033289 URL pmid: 23978009
[24]	Georgios, M., & Julian, K. (2018). High cognitive load enhances the susceptibility to non-speech audiovisual illusions. Scientific Reports, 8(1), 11530. doi: 10.1038/s41598-018-30007-6 URL pmid: 30069059
[25]	Gieseler, A., Tahden, M. A. S., Thiel, C. M., & Colonius, H. (2018). Does hearing aid use affect audiovisual integration in mild hearing impairment? Experimental Brain Research, 236, 1161-1179. URL pmid: 29453491
[26]	Haß, K., Sinke, C., Reese, T., Roy, M., Wiswede, D., Dillo, W., ... Szycik, G. R. (2017). Enlarged temporal integration window in schizophrenia indicated by the double-flash illusion. Cognitive Neuropsychiatry, 22(2), 145-158. doi: 10.1080/13546805.2017.1287693 URL pmid: 28253091
[27]	Hernández, B., Setti, A., Kenny, R. A., & Newell, F. N. (2019). Individual differences in ageing, cognitive status, and sex on susceptibility to the sound-induced flash illusion: A large-scale study. Psychology and Aging, 34(7), 978-990. doi: 10.1037/pag0000396 URL pmid: 31621358
[28]	Innes-Brown, H., & Crewther, D. (2009). The impact of spatial incongruence on an auditory-visual illusion. PLoS ONE, 4(7), e6450. URL pmid: 19649293
[29]	Jiang, Y., & Han, S. H. (2007). Perceptual gain and perceptual loss: Distinct neural mechanisms of audiovisual interactions. International Journal of Magnetic Resonance Imaging, 1(1), 1749-8023.
[30]	Kaiser, M., Senkowski, D., Busch, N. A., Balz, J., & Keil, J. (2019). Single trial prestimulus oscillations predict perception of the sound-induced flash illusion. Scientific Reports, 9(1), 5983. doi: 10.1038/s41598-019-42380-x URL pmid: 30979927
[31]	Kamke, M. R., Vieth, H. E., Cottrell, D., & Mattingley, J. B. (2012). Parietal disruption alters audiovisual binding in the sound-induced flash illusion. NeuroImage, 62(3), 1334-1341. doi: 10.1016/j.neuroimage.2012.05.063 URL pmid: 22658974
[32]	Kawabe, T. (2009). Audiovisual temporal capture underlies flash fusion. Experimental Brain Research, 198(2-3), 195-208. doi: 10.1007/s00221-009-1877-3 URL pmid: 19521693
[33]	Keil, J., Müller, N., Hartmann, T., & Weisz, N. (2013). Prestimulus beta power and phase synchrony influence the sound-induced flash illusion. Cerebral Cortex, 24(5), 1278-1288. URL pmid: 23300109
[34]	Keil, J., & Senkowski, D. (2017). Individual alpha frequency relates to the sound-induced flash illusion. Multisensory Research, 30(6), 565-578. URL pmid: 31287091
[35]	Kim, R., Peters, M. A. K., & Shams, L. (2011). 0 + 1 > 1: How adding noninformative sound improves performance on a visual task. Psychological Science, 23(1), 6-12. doi: 10.1177/0956797611420662 URL pmid: 22127367
[36]	Klatzky, R. L., Marston, J. R., Giudice, N. A., Golledge, R. G., & Loomis, J. M. (2006). Cognitive load of navigating without vision when guided by virtual sound versus spatial language. Journal of Experimental Psychology: Applied, 12(4), 223-232. doi: 10.1037/1076-898X.12.4.223 URL pmid: 17154771
[37]	Kostaki, M., & Vatakis, A. (2016). Crossmodal binding rivalry: A “race” for integration between unequal sensory inputs. Vision Research, 127, 165-176. URL pmid: 27591367
[38]	Kumpik, D. P., Roberts, H. E., King, A. J., & Bizley, J. K. (2014). Visual sensitivity is a stronger determinant of illusory processes than auditory cue parameters in the sound-induced flash illusion. Journal of Vision, 14(7), 12. URL pmid: 24961249
[39]	Laurienti, P. J., Burdette, J. H., Maldjian, J. A., & Wallace, M. T. (2006). Enhanced multisensory integration in older adults. Neurobiology of Aging, 27(8), 1155-1163. doi: 10.1016/j.neurobiolaging.2005.05.024 URL pmid: 16039016
[40]	Lewald, J., & Guski, R. (2003). Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cognitive Brain Research, 16(3), 468-478. doi: 10.1016/s0926-6410(03)00074-0 URL pmid: 12706226
[41]	Maccora, S., Giglia, G., Bolognini, N., Cosentino, G., Gangitano, M., Salemi, G., & Brighina, F. (2019). Cathodal occipital tDCS is unable to modulate the sound induced flash illusion in migraine. Frontiers in Human Neuroscience, 13, 247. doi: 10.3389/fnhum.2019.00247 URL pmid: 31379542
[42]	Magnotti, J. F., & Beauchamp, M. S. (2015). The noisy encoding of disparity model of the McGurk effect. Psychonomic Bulletin & Review, 22(3), 701-709. doi: 10.3758/s13423-014-0722-2 URL pmid: 25245268
[43]	Mahoney, J. R., Li, P. C. C., Oh-park, M., Verghese, J., & Holtzer, R. (2011). Multisensory integration across the senses in young and old adults. Brain Research, 1426(17), 43-53. doi: 10.1016/j.brainres.2011.09.017 URL
[44]	Marques, L. M., Lapenta, O. M., Costa, T. L., & Boggio, P. S. (2016). Multisensory integration processes underlying speech perception as revealed by the McGurk illusion. Language, Cognition and Neuroscience, 31(9), 1115-1129. doi: 10.1080/23273798.2016.1190023 URL
[45]	Máté, A., Anette, G., Verena, C., Masataka, W., & Uta, N. (2015). A spatially collocated sound thrusts a flash into awareness. Frontiers in Integrative Neuroscience, 9, 1-8. doi: 10.3389/fnint.2015.00001 URL pmid: 25698945
[46]	McCormick, D., & Mamassian, P. (2008). "What does the illusory-flash look like?". Vision Research, 48(1), 63-69. doi: 10.1016/j.visres.2007.10.010 URL pmid: 18054372
[47]	Mcgovern, D. P., Roudaia, E., Stapleton, J., McGinnity, T. M., & Newell, F. N. (2014). The sound-induced flash illusion reveals dissociable age-related effects in multisensory integration. Frontiers in Aging Neuroscience, 6, 250. doi: 10.3389/fnagi.2014.00250 URL pmid: 25309430
[48]	Merriman, N. A., Whyatt, C., Setti, A., Craig, C., & Newell, F. N. (2015). Successful balance training is associated with improved multisensory function in fall-prone older adults. Computers in Human Behavior, 45, 192-203. doi: 10.1016/j.chb.2014.12.017 URL
[49]	Mishra, J., Martinez, A., & Hillyard, S. A. (2008). Cortical processes underlying sound-induced flash fusion. Brain Research, 1242(4), 102-115. doi: 10.1016/j.brainres.2008.05.023 URL
[50]	Mishra, J., Martínez, A., & Hillyard, S. A. (2010). Effect of attention on early cortical processes associated with the sound-induced extra flash illusion. Journal of Cognitive Neuroscience, 22(8), 1714-1729. URL pmid: 19583464
[51]	Mishra, J., Martinez, A., Sejnowski, T. J., & Hillyard, S. A. (2007). Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. Journal of Neuroscience, 27(15), 4120-4131. URL pmid: 17428990
[52]	Moro, S. S., & Steeves, J. K. E. (2018). Normal temporal binding window but no sound-induced flash illusion in people with one eye. Experimental Brain Research, 236(6), 1825-1834. URL pmid: 29675714
[53]	Narinesingh, C., Goltz, H. C., & Wong, A. M. F. (2017). Temporal binding window of the sound-induced flash illusion in amblyopia. Investigative Opthalmology & Visual Science, 58(3), 1442-1448.
[54]	O’Brien, J., Ottoboni, G., Tessari, A., & Setti, A. (2017). One bout of open skill exercise improves cross-modal perception and immediate memory in healthy older adults who habitually exercise. PLoS ONE, 12(6), e0178739. URL pmid: 28570704
[55]	Olasagasti, I., Bouton, S., & Giraud, A. L. (2015). Prediction across sensory modalities: A neurocomputational model of the McGurk effect. Cortex, 68, 61-75. URL pmid: 26009260
[56]	Quak, M., London, R. E., & Talsma, D. (2015). A multisensory perspective of working memory. Frontiers in Human Neuroscience, 9, 197. doi: 10.3389/fnhum.2015.00197 URL pmid: 25954176
[57]	Rosenthal, O., Shimojo, S., & Shams, L. (2009). Sound- induced flash illusion is resistant to feedback training. Brain Topography, 21(3-4), 185-192. URL pmid: 19340570
[58]	Samuel, A. G. (2011). Speech perception. Annual Review of Psychology, 62(1), 49-72. doi: 10.1146/annurev.psych.121208.131643 URL
[59]	Setti, A., & Chan, J. S. (2011). Familiarity of objects affects susceptibility to the sound-induced flash illusion. Neuroscience Letters, 492(1), 19-22. URL pmid: 21262319
[60]	Shams, L., & Beierholm, U. R. (2010). Causal inference in perception. Trends in Cognitive Sciences, 14(9), 425-432. doi: 10.1016/j.tics.2010.07.001 URL
[61]	Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408(6814), 788. URL pmid: 11130706
[62]	Shams, L., Kamitani, Y., & Shimojo, S. (2002). Visual illusion induced by sound. Cognitive Brain Research, 14(1), 147-152. URL pmid: 12063138
[63]	Shams, L., Kamitani, Y., Thompson, S. M., & Shimojo, S. (2001). Sound alters visual evoked potentials in humans. Neuroreport, 12(17), 3849-3852. doi: 10.1097/00001756-200112040-00049 URL pmid: 11726807
[64]	Shams, L., Ma, W. J., & Beierholm, U. (2005). Sound- induced flash illusion as an optimal percept. Neuroreport, 16(17), 1923-1927. doi: 10.1097/01.wnr.0000187634.68504.bb URL pmid: 16272880
[65]	Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9(4), 255-266. doi: 10.1038/nrn2331 URL pmid: 18354398
[66]	Sterzer, P., Kleinschmidt, A., & Rees, G. (2009). The neural bases of multistable perception. Trends in Cognitive Sciences, 13(7), 310-318. doi: 10.1016/j.tics.2009.04.006 URL
[67]	Sun, Y. W., Liu, X. L., Li, B. L., Sava-Segal, C., Wang, A., & Zhang, M. (2020). Effects of repetition suppression on sound induced flash illusion with aging. Frontiers in Psychology. 11, 216. doi: 10.3389/fpsyg.2010.00216 URL pmid: 21833271
[68]	Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14(9), 400-410. doi: 10.1016/j.tics.2010.06.008 URL
[69]	Tang, X. Y., Wu, J. L., & Shen, Y. (2016). The interactions of multisensory integration with endogenous and exogenous attention. Neuroscience & Biobehavioral Reviews, 61, 208-224. doi: 10.1016/j.neubiorev.2015.11.002 URL pmid: 26546734
[70]	Wang, A. J., Sang, H. B., He, J. Y., Sava-Segal, C., Tang, X. Y., & Zhang, M. (2019). Effects of cognitive expectation on sound-induced flash illusion. Perception, 48(12), 1214-1234. doi: 10.1177/0301006619885796 URL pmid: 31808371
[71]	Wang, M., Arteaga, D., & He, B. J. (2013). Brain mechanisms for simple perception and bistable perception. Proceedings of the National Academy of Sciences of the United States of America, 110(35), E3350-3359. doi: 10.1073/pnas.1221945110 URL pmid: 23942129
[72]	Watkins, S., Shams, L., Josephs, O., & Rees, G. (2007). Activity in human V1 follows multisensory perception. NeuroImage, 37(2), 572-578. doi: 10.1016/j.neuroimage.2007.05.027 URL pmid: 17604652
[73]	Watkins, S., Shams, L., Tanaka, S., Haynes, J.-D., & Rees, G. (2006). Sound alters activity in human V1 in association with illusory visual perception. NeuroImage, 31(3), 1247-1256. doi: 10.1016/j.neuroimage.2006.01.016 URL pmid: 16556505
[74]	Whittingham, K. M., Mcdonald, J. S., & Clifford, C. W. G. (2014). Synesthetes show normal sound-induced flash fission and fusion illusions. Vision Research, 105, 1-9. URL pmid: 25173429
[75]	van Erp, J. B. F., Philippi, T. G., & Werkhoven, P. (2013). Observers can reliably identify illusory flashes in the illusory flash paradigm. Experimental Brain Research, 226(1), 73-79. doi: 10.1007/s00221-013-3413-8 URL
[76]	Vanes, L. D., White, T. P., Wigton, R. L., Joyce, D., Collier, T., & Shergill, S. S. (2016). Reduced susceptibility to the sound-induced flash fusion illusion in schizophrenia. Psychiatry Research, 245, 58-65. doi: 10.1016/j.psychres.2016.08.016 URL pmid: 27526318
[77]	Xie, Y. J., Xu, Y. Y., Bian, C., & Li, M. (2017). Semantic congruent audiovisual integration during the encoding stage of working memory: An ERP and sLORETA study. Scientific Reports, 7(1), 5112. doi: 10.1038/s41598-017-05471-1 URL pmid: 28698594
[78]	Yalachkov, Y., Bergmann, H. J., Soydaş, D., Buschenlange, C., Fadai Motlagh, L. Y., Naumer, M. J., ... Gehrig, J. (2019). Cognitive impairment in multiple sclerosis is reflected by increased susceptibility to the sound-induced flash illusion. Frontiers in Neurology, 10, 373. URL pmid: 31031699
[79]	Zhang, N. Y., & Chen, W. (2006). A dynamic fMRI study of illusory double-flash effect on human visual cortex. Experimental Brain Research, 172(1), 57-66. doi: 10.1007/s00221-005-0304-7 URL pmid: 16369788
[80]	Zhou, W., Jiang, Y., He, S., & Chen, D. (2010). Olfaction modulates visual perception in binocular rivalry. Current Biology, 20(15), 1356-1358. URL pmid: 20598540