Abstract: PURPOSE: Different dimensions of visual and auditory stimuli can map to each other to influence human behavioral responses, a phenomenon known as audiovisual cross-modal correspondence. A common audiovisual cross-modal correspondence is between auditory tones and visual spatial locations, with individuals tending to map high-pitched sounds to high spatial location and low-pitched sounds to low spatial location. When a high-pitch sound is accompanied or preceded by a visual stimulus, the participants respond faster to visual stimuli presented in the high spatial location than to visual stimuli presented in the low spatial location, and vice versa. Researchers have different views on the level at which audiovisual cross-modal correspondence occurs. Some argue that audiovisual cross-modal correspondence occurs at the perceptual level, increasing the perceptual saliency of the stimulus, while others argue that audiovisual cross-modal correspondence occurs at a later semantic or decision level. As inhibition of return (IOR) in the attentional system can affect human perception, this study used a cue-target paradigm to explore the interaction between IOR and audiovisual cross-modal correspondence to elucidate the occurrence level and mechanism of audiovisual cross-modal correspondence. Audiovisual cross-modal correspondence between auditory tones and visual spatial locations was expected to occur at the perceptual level and therefore would be subject to the IOR effect occurring at the same processing level.
METHODS: The present study consisted of 3 experiments. Experiment 1 had a 2 × 2 within-subjects design; we manipulated the spatial cue validity (valid cue vs. invalid cue) and audiovisual cross-modal correspondence (congruent vs. incongruent). During the experiment, a fixation point was first presented in the middle of the screen for 750 ms. The box above or below the fixation point was then bolded for 50 ms, but this cue was not predictive of the spatial location of the target. After a time interval of 250 ms, a fixation point was presented in bold as a central cue. A central cue is commonly used in spatial IOR research, as it facilitates stable occurrence of IOR. The central cue was presented for 50 ms, and then the auditory stimulus (either high or low pitch) was presented for 50 ms. After a 200-ms interval, the visual target was presented for 100 ms in the box above or below the fixation point. The participants were instructed to perform a detection task for the presence of a visual target. The experimental design and procedure of Experiment 2 were identical to those of Experiment 1, except that the sound presented before the visual target was a single tone that was present or absent. Experiment 3 had a 2 × 2 × 2 within-subjects design. Experiment 3 added a factor to Experiment 1, namely, stimulus onset asynchrony (SOA) between the cue and the target (600 ms vs. 1300 ms).
RESULTS: In all three experiments, the overall accuracy (ACC) was very high; thus, no further statistical analysis was conducted for the ACC. In terms of reaction time (RT), the results of Experiment 1 showed that both spatial IOR and audiovisual cross-modal correspondence occurred. Importantly, there was an interaction between spatial cue validity and audiovisual cross-modal correspondence. Specifically, when the cue was valid, audiovisual cross-modal correspondence occurred (322 ms vs. 327 ms); and when the cue was invalid, there was no audiovisual cross-modal correspondence. The results of Experiment 2 showed that the interaction between cue validity and sound presentation was not significant, and there was no evidence that IOR influenced the sound-induced facilitation effect. The results of Experiment 3 showed that the interaction among spatial cue validity, cross-modal correspondence congruency, and SOA was significant. Specifically, at an SOA of 600 ms, the interaction between spatial cue validity and cross-modal correspondence congruency was significant. When the cue was valid, audiovisual cross-modal correspondence occurred (350 ms vs. 361 ms); and when the cue was invalid, there was no audiovisual cross-modal correspondence. At an SOA of 1300 ms, the interaction between cue validity and cross-modal correspondence congruency was not significant, and cross-modal correspondence occurred in both valid-cue and invalid-cue conditions.
CONCLUSIONS: In conclusion, the present results suggested that the IOR effect, occurring at the perceptual level, moderated audiovisual cross-modal correspondence. When the IOR effect occurred, audiovisual cross-modal correspondence occurred in the cued location, but not in the non-cued location. The alerting effect induced by the sound did not interact with IOR. With the weakening of the IOR effect, the audiovisual cross-modal correspondence in the cued location decreased, and the moderating effect of the IOR effect on audiovisual cross-modal correspondence weakened.

Key words: audiovisual cross-modal correspondence, inhibition of return, cue-target paradigm, alerting effect