Object category differences regulate the sensory dominance of the response level in an audiovisual cross-modal conflict

doi:10.3724/SP.J.1041.2025.1001

Abstract

Abstract:

The sensory dominance refers to the phenomenon where the brain prioritizes the processing of information from a specific sensory modality when faced with multisensory modalities. The cognitive processing level hypothesis posits that the sensory dominance effect is determined by different levels of cognitive processing, with early perceptual processing favoring visual dominance and late response processing favoring auditory dominance. However, existing research has not focused on how the intermediate processing levels between early and late stages of cognitive processing influence the sensory dominance effect. This study manipulated object category differences at the intermediate processing level, employing a 2-1 mapping paradigm to examine how object category representation between early perceptual and late response levels affects cross-modal sensory dominance effects through three experiments. Experiment 1 found that object category differences can modulate the sensory dominance effect at the response level, with visual dominance when category differences are small and auditory dominance when they are large. Experiment 2 indicated that this effect is not related to different processing depths of visual stimuli, confirming that the effect is specific to the visual channel. Experiment 3, using transcranial direct current stimulation (tDCS) to inhibit the brain area responsible for category processing, the left anterior temporal lobe, found that the auditory dominance at the response level disappeared. The study demonstrates that the object category representation at the intermediate processing level within cognitive processing levels regulates the sensory dominance effect, thereby refining the cognitive processing level hypothesis's explanation of cross-modal sensory dominance effects.

Key words: object category, visual dominance, auditory dominance, pre-response level, response level

ZHOU Heng, WANG Ai-Jun, YUAN Xiang-Yong, JIANG Yi. (2025). Object category differences regulate the sensory dominance of the response level in an audiovisual cross-modal conflict. Acta Psychologica Sinica, 57(6), 1001-1012.

Figures/Tables 4

Figure 1. Illustrates the stimulus examples and experimental procedure for Experiment 1c. The audiovisual stimuli were composed of three conditions: congruent condition: the audiovisual stimuli were identical. Pre-response incongruent condition: the audiovisual stimuli were not consistent, but the corresponding key presses were the same. Response incongruent condition: the audiovisual stimuli and corresponding key presses were mismatched. Pre-response conflict was defined as the difference between the reaction time in the pre-response incongruent condition and the reaction time in the congruent condition. Response conflict was calculated as the difference between the reaction time in the response incongruent condition and the reaction time in the pre-response inconsistent condition.

Table 1 The mean reaction times and standard errors of the participants across all experiments. (ms)

	Attended to the visual			Attended to the auditory
	Congruent	Pre-response incongruent	Response incongruent	Congruent	Pre-response incongruent	Response incongruent
Ex 1a	494 ± 20	542 ± 29	539 ± 29	756 ± 25	850 ± 29	884 ± 32
Ex 1b	522 ± 32	571 ± 40	602 ± 45	787 ± 26	873 ± 339	907 ± 34
Ex 1c	496 ± 30	529 ± 37	562 ± 45	777 ± 27	862 ± 30	867 ± 30
Ex 2	503 ± 27	535 ± 33	559 ± 36	791 ± 24	874 ± 26	880 ± 29
Ex 3-sham	520 ± 22	522 ± 25	591 ± 28	704 ± 34	761 ± 38	807 ± 33
Ex 3-cathodal	510 ± 24	539 ± 27	580 ± 38	679 ± 26	734 ± 29	773 ± 27

Figure 2. presents the results of Experiments 1 and 2. A to C depict the outcomes of Experiments 1a to 1c, respectively; D illustrates the distribution of sample mean values obtained by resampling the sensory dominance indices (the difference between auditory conflict and visual conflict) from each condition of Experiment 1; E shows the results of Experiment 2.
Note: ***p < 0.001, **p < 0.01, and *p < 0.05. Error bars indicate standard errors. Color figures are available in the electronic version.

Figure 3. A. Diagram showing the localization of the left anterior temporal lobe cortex on the international 10-20 EEG system, situated between T7 and FT7. B. Electrode field model of the left anterior temporal lobe cortex simulated in the HD-explore software on a standard brain model. C. Experimental results under the pseudo-stimulation condition. D. Experimental results under the cathodal stimulation condition.
Note: ***p < 0.001, *p < 0.05. Error bars represent the standard error of the mean. Color figures are available in the electronic version.

References 56

[1]	Akbıyık, S., Goksun, T., & Balcı, F. (2020). Cathodal tDCS stimulation of left anterior temporal lobe eliminates cross-category color discrimination response time advantage. Behavioural Brain Research, 391, 112682. doi: 10.1016/j.bbr.2020.112682 pmid: 32445777
[2]	Amedi, A., Hofstetter, S., Maidenbaum, S., & Heimler, B. (2017). Task selectivity as a comprehensive principle for brain organization. Trends in Cognitive Sciences, 21(5), 307-310. doi: S1364-6613(17)30050-5 pmid: 28385460
[3]	Baddeley, A. D., & Hitch, G. J. (2017). Is the levels of processing effect language-limited? Journal of Memory and Language, 92, 1-13.
[4]	Bi, Y. C., Wang, X. Y., & Caramazza, A. (2016). Object domain and modality in the ventral visual pathway. Trends in Cognitive Sciences, 20(4), 282-290. doi: S1364-6613(16)00043-7 pmid: 26944219
[5]	Binney, R. J., Ashaie, S. A., Zuckerman, B. M., Hung, J. Y., & Reilly, J. (2018). Frontotemporal stimulation modulates semantically-guided visual search during confrontation naming: A combined tDCS and eye tracking investigation. Brain and Language, 180-182, 14-23. doi: S0093-934X(17)30362-0 pmid: 29655024
[6]	Butera, I. M., Stevenson, R. A., Gifford, R. H., & Wallace, M. T. (2023). Visually biased perception in cochlear implant users: A study of the McGurk and Sound-induced flash illusions. Trends in Hearing, 27, 23312165221076681.
[7]	Callan, A., Callan, D., & Ando, H. (2015). An fMRI study of the ventriloquism effect. Cerebral Cortex, 25(11), 4248-4258.
[8]	Chen, A. T., Liu, Q., Chen, C. M., & Li, H. (2010). The effect of Intermediate processing level on response conflict in Stroop- like tasks. Journal of Psychological Science, 33(3), 569-572.
[9]	Chen, Q., & Zhou, X. L. (2013). Vision dominates at the preresponse level and audition dominates at the response level in cross-modal interaction: Behavioral and neural evidence. Journal of Neuroscience, 33(17), 7109-7121. doi: 10.1523/JNEUROSCI.1985-12.2013 pmid: 23616521
[10]	Colavita, F. B. (1974). Human sensory dominance. Perception & Psychophysics, 16(2), 409-412.
[11]	Dietze, N., & Poth, C. H. (2023). Vision rivals audition in alerting humans for fast action. Acta Psychologica, 238, 103991.
[12]	Dirani, J., & Pylkkänen, L. (2023). The time course of cross-modal representations of conceptual categories. NeuroImage, 277, 120254.
[13]	Donohue, S. E., Todisco, A. E., & Woldorff, M. G. (2013). The rapid distraction of attentional resources toward the source of incongruent stimulus input during multisensory conflict. Journal of Cognitive Neuroscience, 25(4), 623-635. doi: 10.1162/jocn_a_00336 pmid: 23249355
[14]	Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using GPower 3.1: Tests for correlation and regression analyses. Behavior Research Methods*, 41(4), 1149-1160. doi: 10.3758/BRM.41.4.1149 pmid: 19897823
[15]	Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). GPower 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods*, 39, 175-191. doi: 10.3758/bf03193146 pmid: 17695343
[16]	Haselton, M. G., & Buss, D. M. (2000). Error management theory: A new perspective on biases in cross-sex mind reading. Journal of personality and social psychology, 78(1), 81-91. pmid: 10653507
[17]	Hockley, W. E., & Bancroft, T. (2011). Extensions of the picture superiority effect in as-sociative recognition. Canadian Journal of Experimental Psychology, 65(4), 236-244.
[18]	Higdon, K. F., Neath, I., Surprenant, A. M., & Ensor, T. M. (2024). Distinctiveness, not dual coding, explains the picture-superiority effect. Quarterly Journal of Experimental Psychology. https://doi.org/10.1177/17470218241235520
[19]	Hirst, R. J., Cragg, L., & Allen, H. A. (2018). Vision dominates audition in adults but not children: A meta-analysis of the Colavita effect. Neuroscience & Biobehavioral Reviews, 94, 286-301.
[20]	Hirst, R. J., McGovern, D. P., Setti, A., Shams, L., & Newell, F. N. (2020). What you see is what you hear: Twenty years of research using the sound-induced flash illusion. Neuroscience & Biobehavioral Reviews, 118, 759-774.
[21]	Kang, G. L., & Luo, X. X. (2020). The integration and conflict control in audiovisual processing. Journal of Psychological Science, 43(5), 1072-1078.
[22]	Kato, M., & Konishi, Y. (2006). Auditory dominance in the error correction process: A synchronized tapping study. Brain Research, 1084(1), 115-122. pmid: 16556436
[23]	Keil, J. (2020). Double flash illusions: Current findings and future directions. Frontiers in Neuroscience, 14, 298-304. doi: 10.3389/fnins.2020.00298 pmid: 32317920
[24]	Koppen, C., Levitan, C. A., & Spence, C. (2009). A signal detection study of the Colavita visual dominance effect. Experimental Brain Research, 196(3), 353-360. doi: 10.1007/s00221-009-1853-y pmid: 19488743
[25]	Lambon Ralph, M. A., & Patterson, K. (2008). Generalization and differentiation in semantic memory:Insights from semantic dementia. Annals of the New York Academy of Sciences, 1124(1), 61-76.
[26]	Laws, K. R. (2000). Category-specific naming errors in normal subjects: The influence of evolution and experience. Brain and Language, 75(1), 123-133. pmid: 11023642
[27]	Li, X., Cai, S. Z., Chen, Y., Tian, X. M., & Wang, A. J. (2024). Enhancement of visual dominance effects at the response level in children with attention-deficit/hyperactivity disorder. Journal of Experimental Child Psychology. 242, 105897.
[28]	Li, Z., Gu, R., Qi, M., Cen, J., Zhang, S., Gu, J., Zeng, X., & Chen, Q. (2019). Loss of vision dominance at the preresponse level in tinnitus patients: Preliminary behavioral evidence. Frontiers in Neuroscience, 13, 482. doi: 10.3389/fnins.2019.00482 pmid: 31139048
[29]	Martin, A. (2007). The representation of object concepts in the brain. Annual Review of Psychology, 58, 25-45. pmid: 16968210
[30]	Martin, A., & Chao, L. (2001). Semantic memory and the brain: Structure and processes. Current Opinion in Neurobiology, 11(2), 194-201. doi: 10.1016/s0959-4388(00)00196-3 pmid: 11301239
[31]	Mayer, A. R., Franco, A. R., Canive, J., & Harrington, D. L. (2009). The effects of stimulus modality and frequency of stimulus presentation on cross-modal distraction. Cerebral Cortex, 19(5), 993-1007.
[32]	Mesulam, M. M., Wieneke, C., Hurley, R., Rademaker, A., Thompson, C. K., Weintraub, S., & Rogalski, E. J. (2013). Words and objects at the tip of the left temporal lobe in primary progressive aphasia. Brain, 136(2), 601-618.
[33]	Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14(4), 452-465. doi: 10.1093/cercor/bhh007 pmid: 15028649
[34]	Morgenstern, Y., Storrs, K. R., Schmidt, F., Hartmann, F., Tiedemann, H., Wagemans, J., & Fleming, R. W. (2024). High-level aftereffects reveal the role of statistical features in visual shape encoding. Current Biology, 34(5), 1098-1106.
[35]	Nieznański, M. (2020). Levels-of-processing effects on context and target recollection for words and pictures. Acta Psychologica, 209, 103127.
[36]	Odegaard, B., Wozny, D. R., & Shams, L. (2016). The effects of selective and divided attention on sensory precision and integration. Neurosciences Letters, 614, 24-28.
[37]	Potter, M. C., & Fox, L. F. (2009). Detecting and remembering simultaneous pictures in a rapid serial visual presentation. Journal of Experimental Psychology-Human Perception and Pertypyance, 35(1), 28-38.
[38]	Roberts, B. R. T., MacLeod, C. M., & Fernandes, M. A. (2023). Symbol superiority: Why $ is better remembered than ‘dollar’. Cognition, 238, 105435.
[39]	Robinson, C. W., Chandra, M., & Sinnett, S. (2016). Existence of competing modality dominances. Attention, Perception, & Psychophysics, 78(4), 1104-1114.
[40]	Schubert, E. (2021). Creativity is optimal novelty and maximal positive affect: A new definition based on the spreading activation model. Frontiers in Neuroscience, 15, 612379.
[41]	Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature, 408(6814), 788-788.
[42]	Snodgrass, J. G., & Vanderwart, M. (1980). A standardized set of 260 picture: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 6(2), 174-215.
[43]	Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9(4), 255-266. doi: 10.1038/nrn2331 pmid: 18354398
[44]	Tang, X. Y., Wu, J. L., & Shen, Y. (2016). The interactions of multisensory integration with endogenous and exogenous attention. Neuroscience & Biobehavioral Reviews, 61, 208-224.
[45]	Ubaldi, S., & Fairhall, S. L. (2021). fMRI response to automatic and purposeful familiar-face processing in perceptual and nonperceptual cortical regions. Journal of Neurophysiology, 125(4), 1058-1067. doi: 10.1152/jn.00481.2020 pmid: 33596739
[46]	Velasquez, A. G., Gazzaley, A., Toyoda, H., Ziegler, D. A., & Morsella, E. (2021). The generation of involuntary mental imagery in an ecologically-valid task. Frontiers in Psychology. 12, 759685.
[47]	Visconti di Oleggio Castello, M., Haxby, J. V., & Gobbini, M. I. (2021). Shared neural codes for visual and semantic information about familiar faces in a common representational space. Proceedings of the National Academy of Sciences, 118(45), e2110474118.
[48]	Vogler, J. N., & Titchener, K. (2011). Cross-modal conflicts in object recognition: Determining the influence of object category. Experimental Brain Research, 214, 597-605. doi: 10.1007/s00221-011-2858-x pmid: 21912929
[49]	Wang, A. J., Huang, J., Lu, F. F. He, J. Y., Tang, X. Y., & Zhang, M. (2020). Sound-induced flash illusion in multisensory integration. Advances in Psychological Science, 28(10), 1662-1677. doi: 10.3724/SP.J.1042.2020.01662
[50]	Wang, A. J., Sang, H. B., He, J. Y., Sava-Segal, C., Tang, X. Y., & Zhang, M. (2019). Effects of cognitive expectation on sound-induced flash illusion. Perception, 48(12), 1214-1234. doi: 10.1177/0301006619885796 pmid: 31808371
[51]	Wang, A., Zhou, H., Zhang, F., Sang, H., Yu, W., Tang, X., Zhang, T., & Zhang, M. (2021). Repetition suppression in visual and auditory modalities affects the sound-induced flash illusion. Perception, 50(6), 489-507.
[52]	Wang, Y. H., Wang, Y. F., & Zhou, X. L. (2006). Conflict Control at different periods of processing in children with two subtypes of ADHD. Acta Psychologica Sinica, 38(2), 181-188.
[53]	Wiggett, A. T., Pritchard, R. C., & Downing, P. E. (2009). Animate and inanimate objects in human visual cortex: Evidence for task-independent category effects. Neuropsychologia, 47(14), 3111-3117. doi: 10.1016/j.neuropsychologia.2009.07.008 pmid: 19631673
[54]	Wong, C., & Gallate, J. (2012). The function of the anterior temporal lobe: A review of the empirical evidence. Brain Research, 1449, 94-116. doi: 10.1016/j.brainres.2012.02.017 pmid: 22421014
[55]	Wozny, D. R., Beierholm, U. R., & Shams, L. (2008). Human trimodal perception follows optimal statistical inference. Journal of Vision, 8(3), 1-11.
[56]	Zhou, H., Li, S., Huang, J., Yang, J., Wang, A., & Zhang, M. (2022). Sound-induced flash illusions at different spatial locations were affected by personality traits. Attention, Perception, & Psychophysics, 85(2), 463-473.