The influence of endogenous spatial cue validity on audiovisual integration

doi:10.3724/SP.J.1041.2020.00835

Abstract

Abstract:

Previous studies usually used the instruction to control the attention of the participants to different spatial locations to investigate the influence of endogenous spatial attention on the audiovisual integration, which found that endogenous spatial attention enhanced audiovisual integration. However, in reality, it is difficult to assign attention in an all-or-none manner. In present study, we used endogenous spatial cue-target paradigm to investigate the effect of endogenous spatial cue validity on audiovisual integration.

The current study was a 2 (cue validities: valid, invalid) × 3 (target modalities: visual, auditory, audiovisual) factorial design. A total of 62 undergraduate students were recruited as paid volunteers. Experiment 1 recruited 35 participants (5 males; age range: 18~26 years; mean age: 21.26 ± 1.77 years). Experiment 2 recruited 27 participants (6 males; age range: 19~25 years; mean age: 22.19 ± 1.75 years). The visual (V) target was a white pentagonal asterisk block (2°×2°). The auditory (A) target was a 1600 Hz sinusoidal tone presented by speakers. The audiovisual (AV) target was composed by the simultaneous presentation of both the visual and the auditory stimuli. At the beginning of each trial, the fixation stimulus was presented for 600~800 ms in the center of the monitor. Following the fixation stimulus, the cue was presented for 200 ms, which could predict (50% or 80%) the location of the target. Before the target that appeared for 100 ms, the inter stimulus interval (ISI) lasted for 700 ms. The target (A, V, or AV) randomly appeared for 100 ms in the left or right locations. At last, the fixation stimuli appeared for 600 ms to wait for the correspondence responses to targets. During the experiment, the participants were asked to locate targets by pressing buttons (F/J) as quickly and accurately as possible.

The results showed that the responses to AV targets were faster than V or A targets, indicating the appearance of the bimodal advancement effect in both experiments. Each target modality showed significant cueing effect. And the cueing effect in Experiment 2 was significantly larger than the cueing effect in Experiment 1. In addition, in Experiment 1, relative multisensory response enhancement (rMRE) and race model (probability difference) showed no significant difference at valid compared to invalid locations. However, in Experiment 2, rMRE and race model (probability difference) increased at valid compared to invalid locations. The results suggested that endogenous spatial attention enhanced audiovisual integration in high spatial validity condition.

In summary, endogenous spatial attention had different effects on the audiovisual integration under different spatial cue validity. High spatial cue validity enhanced audiovisual integration. The current study provided direct behavioral evidence for endogenous spatial attention to enhance audiovisual integration.

Key words: cue validity, audiovisual integration, endogenous spatial attention, cue-target paradigm, race model

TANG Xiaoyu, WU Yingnan, PENG Xing, WANG Aijun, LI Qi. (2020). The influence of endogenous spatial cue validity on audiovisual integration. Acta Psychologica Sinica, 52(7), 835-846.

Figures/Tables 7

Figure 1. Example of the stimuli and experimental procedure. (a) Schematic diagram of the target stimulus presentation position; (b) Procedure of a single trial. The visual cue (central arrow) points to the left, and the target stimulus (audiovisual) appears to the left (i. e., the valid cue position). Subjects are required to respond both quickly and accurately to the target stimulus. Target (V/A/AV) represents visual, auditory and audiovisual targets respectively. The ISI is the inter-stimulus interval. ITI is the inter-interval trial.

Table 1 Average of median reaction times (RTs, in milliseconds), accuracy (ACC, percent correct), and standard deviation (SD) for all combinations of target type (A, V, and AV) and cue validity (valid and invalid) in each experiment

Target modality	Cue type	Experiment 1		Experiment 2
Target modality	Cue type	RT (ms)	ACC (%)	RT (ms)	ACC (%)
AV	valid	302 ± 34	99.2 ± 1.2	289 ± 42	98.7 ± 1.2
	invalid	312 ± 36	99.2 ± 1.2	333 ± 51	95.8 ± 4.8
A	valid	341 ± 39	97.1 ± 2.8	318 ± 51	97.1 ± 2.6
	invalid	353 ± 42	96.8 ± 2.7	363 ± 56	92.7 ± 4.5
V	valid	334 ± 42	98.0 ± 2.3	324 ± 41	97.5 ± 2.3
	invalid	345 ± 47	97.7 ± 2.9	362 ± 46	95.4 ± 4.7

Figure 2. Reaction time results in Experiment 1. (a) Average of median RT in each condition; (b) Magnitude of relative multisensory response enhancement (rMRE) in both cue validity conditions. The error bars represent the standard errors of the mean (***p < 0.001)

Figure 3. Race model results in Experiment 1. (a) Mean probability enhancement over the race model across the full range of response times (RTs) for the valid cue (dashed red line), and the invalid cue condition (solid black line). (b) Significant violations time window. * Represents the peak value, and the arrows represent the starting time and the peak value of the violation of the race model time window.

Figure 4. Reaction time results in Experiment 2. (a) Average of median RT in each condition; (b) Magnitude of relative multisensory response enhancement (rMRE) in both cue validity conditions. The error bars represent the standard errors of the mean (***p < 0.001, * p < 0.05)

Table 2 Comparison of cueing effect in different target modalities

Experiment	Conditions	M	95% CI		t	p
Experiment	Conditions	M	Lower	Upper	t	p
Experiment 1	Cueing effect
	A	12.23	5.97	18.49	3.97	<0.001
	AV	9.44	5.48	13.41	4.84	<0.001
	V	10.23	5.62	14.84	4.51	<0.001
	Cueing effect comparison
	A vs. AV	2.79	-1.31	6.88	1.38	0.176
	A vs. V	2.00	-3.28	7.28	0.77	0.447
	AV vs. V	-0.78	-4.37	2.80	-0.45	0.659
Experiment 2	Cueing effect
	A	45.04	29.77	60.30	6.06	<0.001
	AV	43.54	29.75	57.32	6.49	<0.001
	V	38.70	26.00	51.41	6.26	<0.001
	Cueing effect comparison
	A vs. AV	1.50	-3.14	6.14	0.67	0.512
	A vs. V	6.33	0.21	12.46	2.13	0.043
	AV vs. V	4.83	-0.88	10.54	1.74	0.094

Figure 5. Race model results in Experiment 2. (a) Mean probability enhancement over the race model across the full range of response times (RTs) for the valid cue (dashed red line), and the invalid cue condition (solid black line). (b) Significant violations time window. * Represents the peak value, and the arrows represent the starting time and the peak value of the violation of the race model time window.

References 48

1	Arjona, A., Escudero, M., & Gómez, C. M . ( 2016). cue validity probability influences neural processing of targets. Biological Psychology, 119, 171-183. doi: 10.1016/j.biopsycho.2016.07.001 URL pmid: 27430935
2	Baluch, F., & Itti, L. ( 2011). Mechanisms of top-down attention. Trends in Neurosciences, 34( 4), 210-224. doi: 10.1016/j.tins.2011.02.003 URL pmid: 21439656
3	Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J . ( 2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62( 2), 321-332.
4	Buschman, T. J., & Miller, E. K. ( 2007). Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices. Science, 315( 5820), 1860-1862. doi: 10.1126/science.1138071 URL pmid: 17395832
5	Busse, L., Roberts, K. C., Crist, R. E., Weissman, D. H., & Woldorff, M. G . ( 2005). The spread of attention across modalities and space in a multisensory object. Proceedings of the National Academy of Sciences of the United States of America, 102( 51), 18751-18756. doi: 10.1073/pnas.0507704102 URL pmid: 16339900
6	Carrasco, I. ( 2014). Gender gap in innovation: An institutionalist explanation. Management Decision, 52( 2), 410-424.
7	Chica, A. B., & Lupiáñez, J. ( 2009). Effects of endogenous and exogenous attention on visual processing: An Inhibition of Return study. Brain Research, 1278, 75-85. doi: 10.1016/j.brainres.2009.04.011 URL pmid: 19374885
8	Dombert, P. L., Kuhns, A., Mengotti, P., Fink, G. R., & Vossel, S . ( 2016). Functional mechanisms of probabilistic inference in feature- and space-based attentional systems. Neuroimage, 142, 553-564. doi: 10.1016/j.neuroimage.2016.08.010 URL pmid: 27523448
9	Eimer, M. ( 1997). Uninformative symbolic cues may bias visual-spatial attention: Behavioral and electrophysiological evidence. Biological Psychology, 46( 1), 67-71. doi: 10.1016/s0301-0511(97)05254-x URL pmid: 9255432
10	Fairhall, S. L., & Macaluso, E. ( 2009). Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites. European Journal of Neuroscience, 29( 6), 1247-1257. URL pmid: 19302160
11	Giard, M. H., & Peronnet, F. ( 1999). audiovisual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11( 5), 473-490. doi: 10.1162/089892999563544 URL pmid: 10511637
12	Jonides, J. ( 1983). Towards a model of the mind's eye's movement. Bulletin of the Psychonomic Society, 21( 4), 247-250.
13	Koelewijn, T., Bronkhorst, A., & Theeuwes, J . ( 2010). Attention and the multiple stages of multisensory integration: A review of audiovisual studies. Acta Psychologica, 134( 3), 372-384. doi: 10.1016/j.actpsy.2010.03.010 URL pmid: 20427031
14	Kuhns, A. B., Dombert, P. L., Mengotti, P., Fink, G. R., & Vossel, S . ( 2017). Spatial attention, motor intention, and Bayesian cue predictability in the human brain. Journal of Neuroscience, 37( 21), 5334-5344. doi: 10.1523/JNEUROSCI.3255-16.2017 URL pmid: 28450541
15	Laurienti, P. J., Burdette, J. H., Maldjian, J. A., & Wallace, M. T . ( 2006). Enhanced multisensory integration in older adults. Neurobiology of Aging, 27( 8), 1155-1163. doi: 10.1016/j.neurobiolaging.2005.05.024 URL pmid: 16039016
16	Lewald, J., & Guski, R. ( 2003). Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cognitive Brain Research, 16( 3), 468-478. doi: 10.1016/s0926-6410(03)00074-0 URL pmid: 12706226
17	Li, Q., Wu, J. L., & Touge, T . ( 2010). Audiovisual interaction enhances auditory detection in late stage: an event-related potential study. Neuroreport, 21( 3), 173-178. doi: 10.1097/WNR.0b013e3283345f08 URL pmid: 20065887
18	Lunn, J., Sjoblom, A., Ward, J., Soto-Faraco, S., & Forster, S . ( 2019). Multisensory enhancement of attention depends on whether you are already paying attention. Cognition, 187, 38-49. URL pmid: 30825813
19	McCracken, H. S., Murphy, B. A., Glazebrook, C. M., Burkitt, J. J., Karellas, A. M., & Yielder, P. C . ( 2019). Audiovisual multisensory integration and evoked potentials in young adults with and without attention-deficit/hyperactivity disorder. Frontiers in Human Neuroscience, 13, 95. doi: 10.3389/fnhum.2019.00095 URL pmid: 30941026
20	Mengotti, P., Boers, F., Dombert, P. L., Fink, G. R., & Vossel, S . ( 2018). Integrating modality-specific expectancies for the deployment of spatial attention. Scientific Reports, 8( 1), 1210. doi: 10.1038/s41598-018-19593-7 URL pmid: 29352145
21	Meredith, M. A., Nemitz, J. W., & Stein, B. E . ( 1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. Journal of Neuropathology and Experimental Neurology, 7( 10), 3215-3229.
22	Meredith, M. A., & Stein, B. E. ( 1986a). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365( 2), 350-354. doi: 10.1016/0006-8993(86)91648-3 URL pmid: 3947999
23	Meredith, M. A., & Stein, B. E. ( 1986b). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56( 3), 640-662. doi: 10.1152/jn.1986.56.3.640 URL pmid: 3537225
24	Meyer, K. N., Du, F., Parks, E., & Hopfinger, J. B . ( 2018). Exogenous vs. endogenous attention: Shifting the balance of fronto-parietal activity. Neuropsychologia, 111, 307-316. doi: 10.1016/j.neuropsychologia.2018.02.006 URL pmid: 29425803
25	Miller, J. ( 1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14( 2), 247-279.
26	Miller, J. ( 1986). Timecourse of coactivation in bimodal divided attention. Perception & Psychophysics, 40( 5), 331-343. doi: 10.3758/bf03203025 URL pmid: 3786102
27	Peelen, M. V., & Kastner, S. ( 2014). Attention in the real world: Toward understanding its neural basis. Trends in Cognitive Sciences, 18( 5), 242-250. doi: 10.1016/j.tics.2014.02.004 URL pmid: 24630872
28	Posner, M. I . ( 1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32( 1), 3-25. doi: 10.1080/00335558008248231 URL pmid: 7367577
29	Raab, D. H . ( 1962). Statistical facilitation of simple reaction times. Transactions of the New York Academy of Sciences, 24( 5), 574-590.
30	Riggio, L., & Kirsner, K. ( 1997). The relationship between central cues and peripheral cues in covert visual orientation. Perception & Psychophysics, 59( 6), 885-899.
31	Santangelo, V., Fagioli, S., & Macaluso, E . ( 2010). The costs of monitoring simultaneously two sensory modalities decrease when dividing attention in space. Neuroimage, 49( 3), 2717-2727. doi: 10.1016/j.neuroimage.2009.10.061 URL pmid: 19878728
32	Santangelo, V., Ho, C., & Spence, C . ( 2008). Capturing spatial attention with multisensory cues. Psychonomic Bulletin & Review, 15( 2), 398-403. URL pmid: 18488658
33	Santangelo, V., & Spence, C. ( 2007). Multisensory cues capture spatial attention regardless of perceptual load. Journal of Experimental Psychology: Human Perception and Performance, 33( 6), 1311-1321. doi: 10.1037/0096-1523.33.6.1311 URL pmid: 18085945
34	Stein, B. E., Meredith, M. A., & Wallace, M. T . ( 1993). The visually responsive neuron and beyond: Multisensory integration in cat and monkey. Progress in Brain Research, 95, 79-90.
35	Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G . ( 2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14( 9), 400-410. doi: 10.1016/j.tics.2010.06.008 URL pmid: 20675182
36	Talsma, D., & Woldorff, M. G. ( 2005). Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity. Journal of Cognitive Neuroscience, 17( 7), 1098-1114. URL pmid: 16102239
37	Tang, X. Y., Gao, Y. L., Yang, W. P., Ren, Y. N., Wu, J. L., Zhang, M., & Wu, Q . ( 2019). Bimodal-divided attention attenuates visually induced inhibition of return with audiovisual targets. Experimental Brain Research, 237( 4), 1093-1107. doi: 10.1007/s00221-019-05488-0 URL pmid: 30770958
38	Ulrich, R., Miller, J., & Schröter, H . ( 2007). Testing the race model inequality: An algorithm and computer programs. Behavioural Brain Research, 39( 2), 291-302.
39	van den Brink, R. L., Cohen, M. X., van der Burg, E., Talsma, D., Vissers, M. E., & Slagter, H. A . ( 2014). Subcortical, modality-specific pathways contribute to multisensory processing in humans. Cerebral Cortex, 24( 8), 2169-2177. doi: 10.1093/cercor/bht069 URL pmid: 23529004
40	van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J . ( 2008). Pip and pop: Nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance, 34( 5), 1053-1065. doi: 10.1037/0096-1523.34.5.1053 URL pmid: 18823194
41	van der Burg, E., Talsma, D., Olivers, C. N. L., Hickey, C., & Theeuwes, J . ( 2011). Early multisensory interactions affect the competition among multiple visual targets. Neuroimage, 55( 3), 1208-1218. doi: 10.1016/j.neuroimage.2010.12.068 URL
42	van der Stigchel, S., Meeter, M., & Theeuwes, J . ( 2007). Top-down influences make saccades deviate away: The case of endogenous cues. Acta Psychologica, 125( 3), 279-290. doi: 10.1016/j.actpsy.2006.08.002 URL pmid: 17022930
43	van der Stoep, N., van der Stigchel, S., & Nijboer, T. C. W . ( 2015). Exogenous spatial attention decreases audiovisual integration. Attention Perception & Psychophysics, 77( 2), 464-482.
44	van der Stoep, N., van der Stigchel, S., Nijboer, T. C. W., & Spence, C . ( 2017). Visually induced inhibition of return affects the integration of auditory and visual information. Perception, 46( 1), 6-17. doi: 10.1177/0301006616661934 URL pmid: 27484341
45	Vossel, S., Mathys, C., Daunizeau, J., Bauer, M., Driver, J., Friston, K. J., & Stephan, K. E . ( 2014). Spatial attention, precision, and Bayesian inference: A study of saccadic response speed. Cerebral Cortex, 24( 6), 1436-1450. doi: 10.1093/cercor/bhs418 URL pmid: 23322402
46	Vossel, S., Mathys, C., Stephan, K. E., & Friston, K. J . ( 2015). Cortical coupling reflects Bayesian belief updating in the deployment of spatial attention. Journal of Neuroscience, 35( 33), 11532-11542. doi: 10.1523/JNEUROSCI.1382-15.2015 URL pmid: 26290231
47	Vossel, S., Thiel, C. M., & Fink, G. R . ( 2006). cue validity modulates the neural correlates of covert endogenous orienting of attention in parietal and frontal cortex. Neuroimage, 32( 3), 1257-1264. doi: 10.1016/j.neuroimage.2006.05.019 URL pmid: 16846742
48	Zou, H., Müller, H. J., & Shi, Z . ( 2012). Non-spatial sounds regulate eye movements and enhance visual search. Journal of Vision, 12( 5), 2. URL pmid: 22562709

[1]	ZHANG Ming, WANG Tingting, WU Xiaogang, ZHANG Yue’e, WANG Aijun. Effects of integration of facial expression and emotional voice on inhibition of return [J]. Acta Psychologica Sinica, 2022, 54(4): 331-342.
[2]	TANG Xiaoyu, TONG Jiageng, YU Hong, WANG Aijun. Effects of endogenous spatial attention and exogenous spatial attention on multisensory integration [J]. Acta Psychologica Sinica, 2021, 53(11): 1173-1188.
[3]	Xiaoyu TANG, Jiaying SUN, Xing PENG. The effect of bimodal divided attention on inhibition of return with audiovisual targets [J]. Acta Psychologica Sinica, 2020, 52(3): 257-268.
[4]	PENG Xing, CHANG Ruosong, LI Qi, WANG Aijun, TANG Xiaoyu. Visually induced inhibition of return affects the audiovisual integration under different SOA conditions [J]. Acta Psychologica Sinica, 2019, 51(7): 759-771.
[5]	CHEN Airui, WANG Aijun, WANG Tianqi, TANG Xiaoyu, ZHANG Ming. The primary visual cortex modulates attention oscillation [J]. Acta Psychologica Sinica, 2018, 50(2): 158-167.
[6]	Sui Guangyuan,Wu Yan. Overt Visual-Spatial Attention Shifts in Children [J]. , 2006, 38(06): 841-848.