According to the statistically optimal integration theory, our brains weigh various sources of sensory information according to their reliabilities during multisensory integration. Recent behavioral studies have proposed that the prior knowledge about the reliability of different modalities can also play an important role in the weighting of different sensory information. The present study aimed to examine whether the prior knowledge affect multisensory integration in the early perceptual processing stage or in the post-perceptual processes stage. Fourteen participants (19-24 years old) were tested when a single letter (“B” or “E”) was presented simultaneously in both auditory and visual modalities. The letter was displayed in one of the two colors. In one condition the visual information presented was always (probability of 100%) congruent with auditory information and this is the same for either color presented (equal-probability condition). In another condition, the probability in which the visual and auditory information was congruent was varied. The probability was 100% (high probability, HP) when the visual letter was in one color and the probability was 30% (low probability, HP) when the visual letter was in another color. Therefore different colors were associated with different probability (different-probability condition). The participants were instructed to press the button 1 for letter B and button 2 for letter E regardless of the color of the visual stimuli and regardless of the auditory information. Each participant completed 240 trials during equal-probability condition and 1500 trials (consisting of 5 experimental blocks) during different-probability condition. Significantly different responses to the congruent audiovisual stimuli were found between HP and LP and the difference was only observed during block 4 and block 5 of difference-probability condition. These results suggested the different probabilities of auditory-visual congruency modulate the response time to auditory-visual congruent stimuli, which suggested the prior knowledge regarding the reliability of sensory cues affects the multisensory integration in the early perceptual processing stage.

采用线索化范式,操作性地建立线索和靶子间的关联,通过比较不同搜索策略任务下不同线索靶子关联条件间的捕获量考察关联水平及搜索策略对注意捕获的作用,并分析抽象的意义概念对空间注意分配的调节作用.结果发现(1)意义线索的注意捕获符合关联性的注意定向假说,只有符合当前注意控制定势的线索才具有捕获注意的能力;(2)知觉关联在调节无意注意定向上具有主导性,调节能力强,能将其他因素效应掩盖;语义关联单独作用时对无意注意定向有调节作用,但调节程度小,效应易被掩盖;(3)搜索策略虽然能够大大提高反应速度,但只有基于特征独子的搜索模式对无意注意定向有调节作用,且调节能力有限.研究再次认证了空间无意注意转移中注意控制定势的作用,同时指出抽象的概念加工能够根据当前的环境有效指导后续的选择和行为.

There is large amount of evidence showing that the saliency of a stimulus on a dimension (singleton) can involuntarily capture attention. Previous work has also demonstrated that attentional capture induced by feature singleton is modulated by top-down factors. For example, the semantic relationship between target and distractors impacts on the capture of attention; search strategies (Singleton Detection Mode and Feature Search Mode) are another potential factors affecting attentional capture. Here we measured how these factors guided attentional capture in a visual search task by manipulating cue-target relevance and search strategies and attempted to provide systematically evidence on capture of attention. A modified spatial cueing paradigm was employed in the current study. In a trial, a fixation screen was presented for 500 ms, followed by an uninformative red cue which appeared for 100 ms. Then a fixation screen was showed for 100 ms, followed by a target screen with Chinese characters surrounding by squares displayed 1000 ms. Participants had to make a judgment for the gap orientation of a target square while ignoring other distractors. Targets were Chinese characters randomly presented at the cued or non-cued locations, making cues valid or invalid. The effect of attentional capture referred to the fact slower responses to invalid contrasting to those to valid targets. The magnitude of the effect was computed to assess the effect size across different experimental conditions. In the current study, there were ten experimental conditions according to various combinations of perceptional relevance, semantic relevance between cue and target and singleton detection mode and feature search mode. The order of experimental conditions was counterbalanced across participants. As a result, the capture effect was not observed when cue and target were irrelevant whilst no search strategy was adopted, but feature search mode was induced. However, the capture effect was observed when singleton detection mode was used, and when the semantic relevance between the cue and the target was established. The capture effects were also reliable significant when there was the perceptional relevance between the cue and target. In contrast, the capture effects induced by singleton detection mode and by semantic relevance were eliminated when perceptional relevance occurred. It was concluded that (1) the perceptional relevance between cue and target was more pronounced to drive attentional capture than the semantic association between cue and target and search strategies, (2) although the latter was also able to modulate the magnitude of attentional capture, but the effect exclusively occurred when there was no perceptional relevance between cue and target. (3) attentional capture was modulated by the search strategy - singleton detection mode, but not by feature search mode after controlling the perceptional relevance and the semantic relevance between cue and target.

Visual search studies have shown that attention can be top-down biased to a specific target color, so that only items with this color or a similar color can capture attention. According to some theories of attention, colors from different categories (i.e., red, green, blue, yellow) are represented independently. However, other accounts have proposed that these are related ither because color is filtered through broad overlapping channels (4-channel view), or because colors are represented in one continuous feature space (e.g., CIE space) and search is governed by specific principles (e.g., linear separability between colors, or top-down tuning to relative colors). The present study tested these different views using a cueing experiment in which observers had to select one target color (e.g., red) and ignore two or four differently colored distractors that were presented prior to the target (cues). The results showed clear evidence for top-down contingent capture by colors, as a target-colored cue captured attention more strongly than differently colored cues. However, the results failed to support any of the proposed views that different color categories are related to one another by overlapping channels, linear separability, or relational guidance ( N = 96).

Watson and Humphreys (1997) proposed that visual marking is a goal-directed process that enhances visual search through the inhibition of old objects. In addition to the standard marking case with targets at new locations, included in Experiment 1 was a set of trials with targets always at old locations, as well as a set of trials with targets varying between new and old locations. The participants performance when detecting the target at old locations was equivalent to their performance in the full-baseline condition when they knew the target would be at old locations, and was worse when the target appeared at old locations on 50% of the trials. Marking was observed when the target appeared at new locations. In Experiment 2, an offset paradigm was used to eliminate the influence of the salient abrupt-onset feature of the new objects. No significant benefits were found for targets at new locations in the absence of onsets at new locations. The results suggest that visual marking may be an attentional selection mechanism that significantly benefits visual search when (1) the observer has an appropriate search goal, (2) the goal necessitates inhibition of old objects, and (3) the new objects include a salient perceptual feature.

AbstractThis study reports an experiment investigating the relative effects of intramodal, crossmodal and bimodal cues on visual and auditory temporal order judgements. Pairs of visual or auditory targets, separated by varying stimulus onset asynchronies, were presented to either side of a central fixation ( 45 ), and participants were asked to identify the target that had occurred first. In some of the trials, one of the targets was preceded by a short, non-predictive visual, auditory or audiovisual cue stimulus. The cue and target stimuli were presented at the exact same locations in space. The point of subjective simultaneity revealed a consistent spatiotemporal bias towards targets at the cued location. For the visual targets, the intramodal cue elicited the largest, and the crossmodal cue the smallest, bias. The bias elicited by the bimodal cue fell between the intramodal and crossmodal cue biases, with significant differences between all cue types. The pattern for the auditory targets was similar apart from a scaling factor and greater variance, so the differences between the cue conditions did not reach significance. These results provide evidence for multisensory integration in exogenous attentional cueing. The magnitude of the bimodal cueing effect was equivalent to the average of the facilitation elicited by the intramodal and crossmodal cues. Under the assumption that the visual and auditory cues were equally informative, this is consistent with the notion that exogenous attention, like perception, integrates multimodal information in an optimal way.

Dynamically shifting attention between behaviorally relevant stimuli in the environment is a key condition for successful adaptive behavior. Here, we investigated how exogenous (reflexive) and endogenous (voluntary) shifts of visual spatial attention interact to modulate activity of single neurons in extrastriate area MT. We used a double-cueing paradigm, in which the first cue instructed two macaque monkeys to covertly attend to one of three moving random dot patterns until a second cue, whose unpredictable onset exogenously captured attention, either signaled to shift or maintain the current focus of attention. The neuronal activity revealed correlates of both exogenous and endogenous attention, which could be well distinguished by their characteristic temporal dynamics. The earliest effect was a transient interruption of the focus of endogenous attention by the onset of the second cue. The neuronal signature of this exogenous capture of attention was a short-latency decrease of responses to the stimulus attended so far. About 70 ms later, the influence of exogenous attention leveled off, which was reflected in two concurrent processes: responses to the newly cued stimulus continuously increased because of allocation of endogenous attention, while, surprisingly, there was also a gradual rebound of attentional enhancement of the previously relevant stimulus. Only after an additional 110 ms did endogenous disengagement of attention from this previously relevant stimulus become evident. These patterns of attentional modulation can be most parsimoniously explained by assuming two distinct attentional mechanisms drawing on the same capacity-limited system, with exogenous attention having a much faster time course than endogenous attention.

In spatial perception, visual information has higher acuity than auditory information and we often misperceive sound-source locations when spatially disparate visual stimuli are presented simultaneously. Ventriloquists make good use of this auditory illusion. In this study, we investigated neural substrates of the ventriloquism effect to understand the neural mechanism of multimodal integration. This study was performed in 2 steps. First, we investigated how sound locations were represented in the auditory cortex. Secondly, we investigated how simultaneous presentation of spatially disparate visual stimuli affects neural processing of sound locations. Based on the population rate code hypothesis that assumes monotonic sensitivity to sound azimuth across populations of broadly tuned neurons, we expected a monotonic increase of blood oxygenation level-dependent (BOLD) signals for more contralateral sounds. Consistent with this hypothesis, we found that BOLD signals in the posterior superior temporal gyrus increased monotonically as a function of sound azimuth. We also observed attenuation of the monotonic azimuthal sensitivity by spatially disparate visual stimuli. The alteration of the neural pattern was considered to reflect the neural mechanism of the ventriloquism effect. Our findings indicate that conflicting audiovisual spatial information of an event is associated with an attenuation of neural processing of auditory spatial localization.

Understanding the conditions under which the brain integrates the different sensory streams and the mechanisms supporting this phenomenon is now a question at the forefront of neuroscience. In this paper, we discuss the opportunities for investigating these multisensory processes using modern imaging techniques, the nature of the information obtainable from each method and their benefits and limitations. Despite considerable variability in terms of paradigm design and analysis, some consistent findings are beginning to emerge. The detection of brain activity in human neuroimaging studies that resembles multisensory integration responses at the cellular level in other species, suggests similar crossmodal binding mechanisms may be operational in the human brain. These mechanisms appear to be distributed across distinct neuronal networks that vary depending on the nature of the shared information between different sensory cues. For example, differing extents of correspondence in time, space or content seem to reliably bias the involvement of different integrative networks which code for these cues. A combination of data obtained from haemodynamic and electromagnetic methods, which offer high spatial or temporal resolution respectively, are providing converging evidence of multisensory interactions at both “early” and “late” stages of processing––suggesting a cascade of synergistic processes operating in parallel at different levels of the cortex.

中国科学院机构知识库(CAS IR GRID)以发展机构知识能力和知识管理能力为目标，快速实现对本机构知识资产的收集、长期保存、合理传播利用，积极建设对知识内容进行捕获、转化、传播、利用和审计的能力，逐步建设包括知识内容分析、关系分析和能力审计在内的知识服务能力，开展综合知识管理。

Orienting of spatial attention is a family of phylogenetically old mechanisms developed to select information for further processing. Information can be selected via top-down or endogenous mechanisms, depending on the goals of the observers or on the task at hand. Moreover, salient and potentially dangerous events also attract spatial attention via bottom-up or exogenous mechanisms, allowing a rapid and efficient reaction to unexpected but important events. Fronto-parietal brain networks have been demonstrated to play an important role in supporting spatial attentional orienting, although there is no consensus on whether there is a single attentional system supporting both endogenous and exogenous attention, or two anatomical and functionally different attentional systems. In the present paper we review behavioral evidence emphasizing the differential characteristics of both systems, as well as their possible interactions for the control of the final orienting response. Behavioral studies reporting qualitative differences between the effects of both systems as well as double dissociations of the effects of endogenous and exogenous attention on information processing, suggest that they constitute two independent attentional systems, rather than a single one. Recent models of attentional orienting in humans have put forward the hypothesis of a dorsal fronto-parietal network for orienting spatial attention, and a more ventral fronto-parietal network for detecting unexpected but behaviorally relevant events. Non-invasive neurostimulation techniques, as well as neuropsychological data, suggest that endogenous and exogenous attention are implemented in overlapping, although partially segregated, brain circuits. Although more research is needed in order to refine our anatomical and functional knowledge of the brain circuits underlying spatial attention, we conclude that endogenous and exogenous spatial orienting constitute two independent attentional systems, with different behavioral effects, and partially distinct neural substrates.

We investigate early (P1) and late (P3) modulations of event-related potentials produced by endogenous (expected vs. unexpected location trials) and exogenous (cued vs. uncued location trials) orienting of spatial attention. A 75% informative peripheral cue was presented 1000 ms before the target in order to study Inhibition of Return (IOR), a mechanism that produces slower responses to peripherally cued versus uncued locations. Endogenous attention produced its effects more strongly at later stages of processing, while IOR (an index of exogenous orienting) was found to modulate both early and late stages of processing. The amplitude of P1 was reduced for cued versus uncued location trials, even when endogenous attention was maintained at the cued location. This result indicates that the perceptual effects of IOR are not eliminated by endogenous attention, suggesting that the IOR mechanism produces a perceptual decrement on the processing of stimuli at the cued location that cannot be counteracted by endogenous attention.

Top-down control and early multisensory processes: chicken vs. egg

We used 3-T functional magnetic resonance imaging to compare the brain mechanisms underlying selective attention to sound location and pitch. In different tasks, the subjects ( N02=0210) attended to a designated sound location or pitch or to pictures presented on the screen. In the Attend Location conditions, the sound location varied randomly (left or right), while the pitch was kept constant (high or low). In the Attend Pitch conditions, sounds of randomly varying pitch (high or low) were presented at a constant location (left or right). Both attention to location and attention to pitch produced enhanced activity (in comparison with activation caused by the same sounds when attention was focused on the pictures) in widespread areas of the superior temporal cortex. Attention to either sound feature also activated prefrontal and inferior parietal cortical regions. These activations were stronger during attention to location than during attention to pitch. Attention to location but not to pitch produced a significant increase of activation in the premotor/supplementary motor cortices of both hemispheres and in the right prefrontal cortex, while no area showed activity specifically related to attention to pitch. The present results suggest some differences in the attentional selection of sounds on the basis of their location and pitch consistent with the suggested auditory “what” and “where” processing streams.

In unisensory contexts, spatially-focused attention tends to enhance perceptual processing. How attention influences the processing of multisensory stimuli, however, has been of much debate. In some cases, attention has been shown to be important for processes related to the integration of audio-visual stimuli, but in other cases such processes have been reported to occur independently of attention. To address these conflicting results, we performed three experiments to examine how attention interacts with a key facet of multisensory processing: the temporal window of integration (TWI). The first two experiments used a novel cued-spatial-attention version of the bounce/stream illusion, where two moving visual stimuli with intersecting paths tend to be perceived as bouncing off rather than streaming through each other when a brief sound occurs near in time. When the task was to report whether the visual stimuli appeared to bounce or stream, attention served to narrow this measure of the TWI and bias perception toward treaming . When the participants task was to explicitly judge the simultaneity of the sound with the intersection of the moving visual stimuli, however, the results were quite different. Specifically, attention served to mainly widen the TWI, increasing the likelihood of simultaneity perception, while also substantially increasing the simultaneity judgement accuracy when the stimuli were actually physically simultaneous. Finally, in Experiment 3, where the task was to judge the simultaneity of a simple, temporally discrete, flashed, visual stimulus and the same brief tone pip, attention had no effect on the measured TWI. These results highlight the flexibility of attention in enhancing multisensory perception and show that the effect of attention on multisensory processing is highly dependent on the task demands and observer goals.

The role of attention in multisensory integration (MI) is presently uncertain, with some studies supporting an automatic, pre-attentive process and others suggesting possible modulation through selective attention. The goal of this functional magnetic resonance imaging study was to investigate the role of spatial attention on the processing of congruent audiovisual speech stimuli (here indexing MI). Subjects were presented with two simultaneous visual streams (speaking lips in the left and right visual hemifields) plus a single central audio stream (spoken words). In the selective attention conditions, the auditory stream was congruent with one of the two visual streams. Subjects attended to either the congruent or the incongruent visual stream, allowing the comparison of brain activity for attended vs. unattended MI while the amount of multisensory information in the environment and the overall attentional requirements were held constant. Meridian mapping and a lateralized 'speaking-lips' localizer were used to identify early visual areas and to localize regions responding to contralateral visual stimulations. Results showed that attention to the congruent audiovisual stimulus resulted in increased activation in the superior temporal sulcus, striate and extrastriate retinotopic visual cortex, and superior colliculus. These findings demonstrate that audiovisual integration and spatial attention jointly interact to influence activity in an extensive network of brain areas, including associative regions, early sensory-specific visual cortex and subcortical structures that together contribute to the perception of a fused audiovisual percept.

Integration of multiple sensory cues is essential for precise and accurate perception and behavioral performance, yet the reliability of sensory signals can vary across modalities and viewing conditions. Human observers typically employ the optimal strategy of weighting each cue in proportion to its reliability, but the neural basis of this computation remains poorly understood. We trained monkeys to perform a heading discrimination task from visual and vestibular cues, varying cue reliability randomly. The monkeys appropriately placed greater weight on the more reliable cue, and population decoding of neural responses in the dorsal medial superior temporal area closely predicted behavioral cue weighting, including modest deviations from optimality. We found that the mathematical combination of visual and vestibular inputs by single neurons is generally consistent with recent theories of optimal probabilistic computation in neural circuits. These results provide direct evidence for a neural mechanism mediating a simple and widespread form of statistical inference.

Evaluates evidence of the effectiveness of top-down control over attentional capture focusing on the effects of irrelevant featural singletons in visual search. How irrelevant singletons produce distraction effects dissociable from shifts of spatial attention; Information on the mean response times on error rates for the nonsingleton and singleton target conditions; What the results suggested.

Abstract We used event-related potential measures to compare the effects of ipsilateral and bilateral auditory stimuli on audiovisual (AV) integration. Behavioral results showed that the responses to visual stimuli with either type of auditory stimulus were faster than responses to visual stimuli only and that perceptual sensitivity (d') for visual detection was enhanced for visual stimuli with ipsilateral auditory stimuli. Furthermore, event-related potential components related to AV integrations were identified over the occipital areas at 180-200 ms during early-stage sensory processing by the effect of ipsilateral auditory stimuli and over the frontocentral areas at 300-320 ms during late-stage cognitive processing by the effect of ipsilateral and bilateral auditory stimuli. Our results confirmed that AV integration was also elicited, despite the effect of bilateral auditory stimuli, and only occurred at later stages of cognitive processing in response to a visual detection task. Furthermore, integration from early-stage sensory processing was observed by the effect of ipsilateral auditory stimuli, suggesting that the integration of AV information in the human brain might be particularly sensitive to ipsilaterally presented AV stimuli.

We used contingent attentional capture to investigate whether capture in a given trial n was affected by the cue–target position relations in a preceding trial n –1. Typically, attentional capture by a cue facilitates reaction times for targets in valid conditions (with the cue and target at the same position) relative to invalid conditions (with the cue and target at different positions). Also, this validity effect holds for cues with a feature similar to the searched-for target features (i.e., matching cues), but not for cues dissimilar to the searched-for target features (i.e., nonmatching cues), a pattern termed contingent capture because capture is assumed to be contingent on the match between the cue and top-down control settings. Here, we replicated this contingent-capture pattern with cues that were nonpredictive of the target position. In addition, we showed that during search for white onset targets, red nonmatching color cues also created a validity effect if the same nonmatching cue had been used as a valid cue in trial n –1 (Exps. 1 and 2 ). This intertrial contingency of the nonmatching cue’s validity effect was also found if the cues and targets both changed their positions from trial to trial, rendering position priming unlikely (Exp. 2 ). A similar intertrial contingency was found for nonmatching white onset cues, but not for matching red color cues during search for red color targets (Exp. 3 ). These results are discussed in light of explanations of the contingent-capture effect and of intertrial contingencies.

The ability of a stimulus to capture visuospatial attention depends on the interplay between its bottom-up saliency and its relationship to an observer’s top-down control set, such that stimuli capture attention if they match the predefined properties that distinguish a searched-for target from distractors (Folk, Remington, & Johnston, Journal of Experimental Psychology : Human Perception & Performance, 18 , 1030–1044 1992 ). Despite decades of research on this phenomenon, however, the vast majority has focused exclusively on matches based on low-level physical properties. Yet if contingent capture is indeed a “top-down” influence on attention, then semantic content should be accessible and able to determine which physical features capture attention. Here we tested this prediction by examining whether a semantically defined target could create a control set for particular features. To do this, we had participants search to identify a target that was differentiated from distractors by its meaning (e.g., the word “red” among color words all written in black). Before the target array, a cue was presented, and it was varied whether the cue appeared in the physical color implied by the target word. Across three experiments, we found that cues that embodied the meaning of the word produced greater cuing than cues that did not. This suggests that top-down control sets activate content that is semantically associated with the target-defining property, and this content in turn has the ability to exogenously orient attention.

Participants are faster at detecting a visual target when it appears at a cued, as compared with an uncued, location. In general, a reversal of this cost–benefit pattern is observed after exogenous cuing when the cue–target interval exceeds approximately 25002ms (inhibition of return [IOR]), and not after endogenous cuing. We suggest that, usually, no IOR is found with endogenous cues because no bottom-up saliency-based orienting processes are claimed. Therefore, we developed an endogenous feature-based split-cue task to allow for endogenous saliency-based orienting. IOR was observed in the saliency-driven endogenous cuing condition, and not in the control condition that prevented saliency-based orienting. These results suggest that usage of saliency-based orienting processes in either endogenous or exogenous orienting warrants the appearance of IOR.

Abstract We report a study designed to investigate the effectiveness of task-irrelevant unimodal and bimodal audiotactile stimuli in capturing a person's spatial attention away from a highly perceptually demanding central rapid serial visual presentation (RSVP) task. In "Experiment 1", participants made speeded elevation discrimination responses to peripheral visual targets following the presentation of auditory stimuli that were either presented alone or else were paired with centrally presented tactile stimuli. The results showed that the unimodal auditory stimuli only captured spatial attention when participants were not performing the RSVP task, while the bimodal audiotactile stimuli did not result in any performance change in any of the conditions. In "Experiment 2", spatial auditory stimuli were either presented alone or else were paired with a tactile stimulus presented from the same direction. In contrast to the results of "Experiment 1", the bimodal audiotactile stimuli were especially effective in capturing participants' spatial attention from the concurrent RSVP task. These results therefore provide support for the claim that auditory and tactile stimuli should be presented from the same direction if they are to capture attention effectively. Implications for multisensory warning signal design are discussed.

Sensory processing is affected by both endogenous and exogenous mechanisms of attention, although how these mechanisms interact in the brain has remained unclear. In the present study, we recorded event-related potentials (ERPs) to investigate how multiple stages of information processing in the brain are affected when endogenous and exogenous mechanisms are concurrently engaged. We found that the earliest stage of cortical visual processing, the striate-cortex-generated C1, was immune to attentional modulation, even when endogenous and exogenous attention converged on a common location. The earliest stage of processing to be affected in this experiment was the late phase of the extrastriate-cortex-generated P1 component, which was dominated by exogenous attention. Processing at this stage was enhanced by exogenous attention, regardless of where endogenous attention had been oriented. Endogenous attention, however, dominated a later, higher-order stage of processing indexed by an enhancement of the P300 that was unaffected by exogenous attention. Critically, between these early and late stages, an interaction was found wherein endogenous and exogenous attention produced distinct, and overlapping, effects on information processing. At the same time that exogenous attention was producing an extended enhancement of the late-P1, endogenous attention was enhancing the occipital arietal N1 component. These results provide neurophysiological support for theories suggesting that endogenous and exogenous mechanisms represent two attention systems that can affect information processing in the brain in distinct ways. Furthermore, these data provide new evidence regarding the precise stages of neural processing that are, and are not, affected when endogenous and exogenous attentions interact.

Multisensory integration and crossmodal attention have a large impact on how we perceive the world. Therefore, it is important to know under what circumstances these processes take place and how they affect our performance. So far, no consensus has been reached on whether multisensory integration and crossmodal attention operate independently and whether they represent truly automatic processes. This review describes the constraints under which multisensory integration and crossmodal attention occur and in what brain areas these processes take place. Some studies suggest that multisensory integration and crossmodal attention take place in higher heteromodal brain areas, while others show the involvement of early sensory specific areas. Additionally, the current literature suggests that multisensory integration and attention interact depending on what processing level integration takes place. To shed light on this issue, different frameworks regarding the level at which multisensory interactions takes place are discussed. Finally, this review focuses on the question whether audiovisual interactions and crossmodal attention in particular are automatic processes. Recent studies suggest that this is not always the case. Overall, this review provides evidence for a parallel processing framework suggesting that both multisensory integration and attentional processes take place and can interact at multiple stages in the brain.

In perception, prior expectations allow us to quickly deduce plausible interpretations from noisy and ambiguous data. Here, Kok et02al. show that expectation facilitates perception by reducing the overall neural response amplitude, yet increasing the informational content in primary visual cortex.

Attending to a conversation in a crowded scene requires selection of relevant information, while ignoring other distracting sensory input, such as speech signals from surrounding people. The neural mechanisms of how distracting stimuli influence the processing of attended speech are not well understood. In this high-density electroencephalography (EEG) study, we investigated how different types of speech and non-speech stimuli influence the processing of attended audiovisual speech. Participants were presented with three horizontally aligned speakers who produced syllables. The faces of the three speakers flickered at specific frequencies (19 Hz for flanking speakers and 25 Hz for the center speaker), which induced steady-state visual evoked potentials (SSVEP) in the EEG that served as a measure of visual attention. The participants' task was to detect an occasional audiovisual target syllable produced by the center speaker, while ignoring distracting signals originating from the two flanking speakers. In all experimental conditions the center speaker produced a bimodal audiovisual syllable. In three distraction conditions, which were contrasted with a no-distraction control condition, the flanking speakers either produced audiovisual speech, moved their lips, and produced acoustic noise, or moved their lips without producing an auditory signal. We observed behavioral interference in the reaction times (RTs) in particular when the flanking speakers produced naturalistic audiovisual speech. These effects were paralleled by enhanced 19 Hz SSVEP, indicative of a stimulus-driven capture of attention toward the interfering speakers. Our study provides evidence that non-relevant audiovisual speech signals serve as highly salient distractors, which capture attention in a stimulus-driven fashion.

According to most models of selective visual attention, our goals at any given moment and saliency in the visual field determine attentional priority. But selection is not carried out in isolation--we typically track objects through space and time. This is not well captured within the distinction between goal-directed and saliency-based attentional guidance. Recent studies have shown that selection is strongly facilitated when the characteristics of the objects to be attended and of those to be ignored remain constant between consecutive selections. These studies have generated the proposal that goal-directed or top-down effects are best understood as intertrial priming effects. Here, we provide a detailed overview and critical appraisal of the arguments, experimental strategies, and findings that have been used to promote this idea, along with a review of studies providing potential counterarguments. We divide this review according to different types of attentional control settings that observers are thought to adopt during visual search: feature-based settings, dimension-based settings, and singleton detection mode. We conclude that priming accounts for considerable portions of effects attributed to top-down guidance, but that top-down guidance can be independent of intertrial priming.

The ability to remain focused on goal-relevant stimuli in the presence of potentially interfering distractors is crucial for any coherent cognitive function. However, simply instructing people to ignore goal-irrelevant stimuli is not sufficient for preventing their processing. Recent research reveals that distractor processing depends critically on the level and type of load involved in the processing of goal-relevant information. Whereas high perceptual load can eliminate distractor processing, high load on ‘frontal’ cognitive control processes increases distractor processing. These findings provide a resolution to the long-standing early and late selection debate within a load theory of attention that accommodates behavioural and neuroimaging data within a framework that integrates attention research with executive function.

Although many behavioral studies have investigated auditory detection enhancement by crossmodal audiovisual interaction, the results are controversial. In addition, no neuroimaging studies that identify this phenomenon have been conducted. Therefore, we used event-related potential (ERP) measures to investigate this phenomenon by comparing the ERPs elicited by the audiovisual stimuli to the sum of the ERPs elicited by the visual and auditory stimuli, and identified two brain regions that showed significantly different responses: the centro-medial area at 280-300 ms after the presentation of the stimulus and the right fronto-temporal area at 300-320 ms after the presentation of the stimulus. The ERP results suggested that the behavioral enhancement of auditory detection results from late-stage cognitive processes rather than early-stage sensory processes.

Sensory information is multimodal; through audiovisual interaction, task-irrelevant auditory stimuli tend to speed response times and increase visual perception accuracy. However, mechanisms underlying these performance enhancements have remained unclear. We hypothesize that task-irrelevant auditory stimuli might provide reliable temporal and spatial cues for visual target discrimination and behavioral response enhancement. Using signal detection theory, the present study investigated the effects of spatiotemporal relationships on auditory facilitation of visual target discrimination. Three experiments were conducted where an auditory stimulus maintained reliable temporal and/or spatial relationships with visual target stimuli. Results showed that perception sensitivity (d') to visual target stimuli was enhanced only when a task-irrelevant auditory stimulus maintained reliable spatiotemporal relationships with a visual target stimulus. When only reliable spatial or temporal information was contained, perception sensitivity was not enhanced. These results suggest that reliable spatiotemporal relationships between visual and auditory signals are required for audiovisual integration during a visual discrimination task, most likely due to a spread of attention. These results also indicate that auditory facilitation of visual target discrimination follows from late-stage cognitive processes rather than early stage sensory processes.

The integration of multiple sensory inputs is essential for perception of the external world. The spatial factor is a fundamental property of multisensory audiovisual integration. Previous studies of the spatial constraints on bimodal audiovisual integration have mainly focused on the spatial congruity of audiovisual information. However, the effect of spatial reliability within audiovisual information on bimodal audiovisual integration remains unclear. In this study, we used event-related potentials (ERPs) to examine the effect of spatial reliability of task-irrelevant sounds on audiovisual integration. Three relevant ERP components emerged: the first at 140–200ms over a wide central area, the second at 280–320ms over the fronto-central area, and a third at 380–440ms over the parieto-occipital area. Our results demonstrate that ERP amplitudes elicited by audiovisual stimuli with reliable spatial relationships are larger than those elicited by stimuli with inconsistent spatial relationships. In addition, we hypothesized that spatial reliability within an audiovisual stimulus enhances feedback projections to the primary visual cortex from multisensory integration regions. Overall, our findings suggest that the spatial linking of visual and auditory information depends on spatial reliability within an audiovisual stimulus and occurs at a relatively late stage of processing.

The identification of targets in visual search arrays may be improved by suppressing competing information from the surrounding distractor items. The present study provided evidence that this hypothetical filtering process has a neural correlate, the "N2pc" component of the event-related potential waveform. The N2pc was observed when a target item was surrounded by competing distractor items but was absent when the array could be rejected as a nontarget on the basis of simple feature information. In addition, the N2pc was eliminated when filtering was discouraged by removing the distractor items, making the distractors relevant, or making all items within an array identical. Combined with previous topographic analyses, these results suggest that attentional filtering occurs in occipital cortex under the control of feedback from higher cortical regions after a preliminary feature-based analysis of the stimulus array.

78 Multisensory alerting cues were not facilitative for old and young adults. 78 Old and young adults demonstrated significant orienting effects for all multisensory cues. 78 However, orienting benefits varied by age group and multisensory condition. 78 Young adults benefited more from multisensory orienting cues than old adults.

Different electrophysiological components have been associated with behavioural facilitation and inhibition of return (IOR), although there is no consensus about which of these components are essential to the mechanism/s underlying the cueing effects. Different spatial attention hypotheses propound different roles for these components. In this review, we try and describe these inconsistencies by first presenting the electrophysiological component modulations of exogenous spatial attention as predicted by different attentional hypotheses. We then review and quantitatively analyse data from the existing electrophysiological studies trying to accommodate their findings. Variables such as the task at hand, the temporal properties and interactions between cues and targets, the presence/absence of intervening events, or stimuli arrangement in the visual field, might critically explain the discrepancies between the theoretical predictions and the electrophysiological modulations that both facilitation and IOR produce. We conclude that there is no single neural marker for facilitation and IOR because the behavioural effect that is observed depends on the contribution of several components: perceptual (P1), late-perceptual (N1, Nd), spatial selection (N2pc), and decision processes (P3). Many variables determine the electrophysiological modulations of different attentional orienting mechanisms, which jointly define the observed spatial cueing effects.

Numerous studies that have investigated visual selective attention have demonstrated that a salient but task-irrelevant stimulus can involuntarily capture a participant’s attention. Over the years, a lively debate has erupted concerning the impact of contingent top-down control settings on such stimulus-driven attentional capture . In the research reported here, we investigated whether top-down sets would also affect participants’ performance in a multisensory task setting. A nonspatial compatibility task was used, in which the target and the distractor were always presented sequentially from the same spatial location. We manipulated target–distractor similarity by varying the visual and tactile features of the stimuli. Participants always responded to the visual target features (color); the tactile features were incorporated into the participants’ top-down set only when the experimental context allowed for the tactile feature to be used in order to discriminate the target from the distractor. Larger compatibility effects after bimodal distractors were observed only when the participants were searching for a bimodal target and when tactile information was useful. Taken together, these results provide the first demonstration of nonspatial contingent crossmodal capture.

The interplay between top-down and bottom-up factors in attentional selection has been a topic of extensive research and controversy amongst scientists over the past two decades. According to the influential contingent capture hypothesis, a visual stimulus needs to match the feature(s) implemented into the current attentional control sets in order to be automatically selected. Recently, however, evidence has been presented that attentional control sets affect not only visual but also crossmodal selection. The aim of the present study was therefore to establish contingent capture as a general principle of multisensory selection. A non-spatial interference task with bimodal (visual and auditory) distractors and bimodal targets was used. The target and the distractors were presented in close temporal succession. In order to perform the task correctly, the participants only had to process a predefined target feature in either of the two modalities (e.g., colour when vision was the primary modality). Note that the additional crossmodal stimulation (e.g., a specific sound when hearing was the secondary modality) was not relevant for the selection of the correct response. Nevertheless, larger interference effects were observed when the distractor matched both the stimulus of the primary as well as the secondary modality and this pattern was even stronger if vision was the primary modality than if audition was the primary modality. These results are therefore in line with the crossmodal contingent capture hypothesis. Both visual and auditory early processing seem to be affected by top-down control sets even beyond the spatial dimension.

Abstract Multisensory integration increases the salience of sensory events and, therefore, possibly also their ability to capture attention in visual search. This was investigated in two experiments where spatially uninformative color change cues preceded visual search arrays with color-defined targets. Tones were presented synchronously with these cues on half of all trials. Spatial-cuing effects indicative of cue-triggered capture of attention were larger on tone-present than on tone-absent trials, demonstrating multisensory enhancements of attentional capture. Larger capture effects for audiovisual events were found when cues were color singletons, and also when they appeared among heterogeneous color distractors. Tone-induced increases of attentional capture were independent of color-specific top-down task sets, suggesting that this multisensory effect is a stimulus-driven bottom-up phenomenon.

To test whether the attentional selection of targets defined by a combination of visual and auditory features is guided in a modality-specific fashion or by control processes that are integrated across modalities, we measured attentional capture by visual stimuli during unimodal visual and audiovisual search. Search arrays were preceded by spatially uninformative visual singleton cues that matched the current target-defining visual feature. Participants searched for targets defined by a visual feature, or by a combination of visual and auditory features (e.g., red targets accompanied by high-pitch tones). Spatial cueing effects indicative of attentional capture were reduced during audiovisual search, and cue-triggered N2pc components were attenuated and delayed. This reduction of cue-induced attentional capture effects during audiovisual search provides new evidence for the multimodal control of selective attention.

Abstract One of the most impressive features of the central nervous system is its ability to process information from a variety of stimuli to produce an integrated, comprehensive representation of the external world. In the present study, the temporal disparity among combinations of different sensory stimuli was shown to be a critical factor influencing the integration of multisensory stimuli by superior colliculus neurons. Several temporal principles that govern multisensory integration were revealed: (1) maximal levels of response enhancement were generated by overlapping the peak discharge periods evoked by each modality; (2) the magnitude of this enhancement decayed monotonically to zero as the peak discharge periods became progressively more temporally disparate; (3) with further increases in temporal disparity, the same stimulus combinations that previously produced enhancement could often produce depression; and (4) these kinds of interactions could frequently be predicted from the discharge trains initiated by each stimulus alone. Since multisensory superior colliculus neurons project to premotor areas of the brain stem and spinal cord that control the orientation of the receptor organs (eyes, pinnae, head), they are believed to influence attentive and orientation behaviors. Therefore, it is likely that the temporal relationships of different environmental stimuli that control the activity of these neurons are also a powerful determinant of superior colliculus-mediated attentive and orientation behaviors.

As a supplement to Gondan and Minakata’s (2015) tutorial on methods for testing the race model inequality, this theoretical note attempts to clarify further (a) the types of models that obey and viola

Stimuli occurring in multiple sensory modalities that are temporally synchronous or spatially coincident can be integrated together to enhance perception. Additionally, the semantic content or meaning of a stimulus can influence cross-modal interactions, improving task performance when these stimuli convey semantically congruent or matching information, but impairing performance when they contain non-matching or distracting information. Attention is one mechanism that is known to alter processing of sensory stimuli by enhancing perception of task-relevant information and suppressing perception of task-irrelevant stimuli. It is not known, however, to what extent attention to a single sensory modality can minimize the impact of stimuli in the unattended sensory modality and reduce the integration of stimuli across multiple sensory modalities. Our hypothesis was that modality-specific selective attention would limit processing of stimuli in the unattended sensory modality, resulting in a reduction of performance enhancements produced by semantically matching multisensory stimuli, and a reduction in performance decrements produced by semantically non-matching multisensory stimuli. The results from two experiments utilizing a cued discrimination task demonstrate that selective attention to a single sensory modality prevents the integration of matching multisensory stimuli that is normally observed when attention is divided between sensory modalities. Attention did not reliably alter the amount of distraction caused by non-matching multisensory stimuli on this task; however, these findings highlight a critical role for modality-specific selective attention in modulating multisensory integration.

61The human primary visual cortex is multisensory in function.61This evidence emerges from a full pallet of brain imaging/mapping methods.61Haemodynamic methods show convergence and integration in primary visual cortex.61Electromagnetic methods show effects during early post-stimulus stages.61Multisensory effects in primary visual cortex directly impact behaviour/perception.

Abstract; Vroomen and de Gelder in J Exp Psychol Hum 26:1583–1590, ). In the present study, we tested three oddball conditions: a louder tone presented amongst quieter tones, a quieter tone presented amongst louder tones, and the absence of a tone, within an otherwise identical tone sequence. Across three experiments, all three oddball conditions resulted in the crossmodal facilitation of participants’ visual target identification performance. These results therefore suggest that salient oddball stimuli in the form of deviating tones, when synchronized with the target, may be sufficient to capture participants’ attention and facilitate visual target identification. The fact that the absence of a sound in an otherwise-regular sequence of tones also facilitated performance suggests that multisensory integration cannot provide an adequate account for the ‘freezing’ effect. Instead, an attentional capture account is proposed to account for the benefits of oddball cuing in Vroomen and de Gelder’s task.

The combined use of multisensory signals is often beneficial. Based on single cell recordings in the superior colliculius of cats, three basic rules were formulated to describe the effectiveness of multisensory integration: The enhancement of neuronal responses in multi- compared to uni-sensory conditions is largest when signals are presented at the same time (‘temporal rule’), occur at the same location (‘spatial rule’), and when signals are rather weak (‘principle of inverse effectiveness’). These rules are also considered to describe multisensory benefits as observed with behavioral measures, but do they capture these benefits best? To uncover the principles that rule multisensory behavior, we investigated the classical redundant signals effect, i.e., the speed-up of response times in multi- as compared to uni-sensory conditions. In a detection task, we presented both auditory and visual signals at three levels of signal strength and determined the speed-up for all nine combinations of signals. Based on a systematic analysis of empirical response time distributions as well as simulations using probability summation, we propose that two alternative rules apply. First, the ‘principle of equal effectiveness’ states that the benefit with multisensory signals (here the speed-up of reaction times) is largest when performance in the two uni-sensory conditions is similar. Second, the ‘variability rule’ states that the benefit is largest when performance in the uni-sensory conditions is variable. The generality of these rules is discussed with respect to experiments on accuracy and when maximum likelihood estimation instead of probability summation is considered as combination rule.

In environments containing sensory events at competing locations, selecting a target for orienting requires prioritization of stimulus values. Although the superior colliculus (SC) is causally linked to the stimulus selection process, the manner in which SC multisensory integration operates in a competitive stimulus environment is unknown. Here we examined how the activity of visual-auditory SC neurons is affected by placement of a competing target in the opposite hemifield, a stimulus configuration that would, in principle, promote interhemispheric competition for access to downstream motor circuitry. Competitive interactions between the targets were evident in how they altered unisensory and multisensory responses of individual neurons. Responses elicited by a cross-modal stimulus (multisensory responses) proved to be substantially more resistant to competitor-induced than were unisensory responses (evoked by the component modality-specific stimuli). Similarly, when a cross-modal stimulus served as the competitor, it exerted considerably more than did its individual component stimuli, in some cases producing more than predicted by their linear sum. These findings suggest that multisensory integration can help resolve competition among multiple targets by enhancing orientation to the location of cross-modal events while simultaneously suppressing orientation to events at alternate locations.

Traditional views of multisensory integration emphasise the advantage of stimulating or attending to different senses at one single spatial location. We challenge this view demonstrating that in-parallel processing of two sensory modalities can be more efficient when attention is spatially divided rather than focused. We asked subjects to monitor simultaneously vision and audition either at one location (focused attention) or in the two opposite hemifields (divided attention) or to monitor one single modality at one or two locations. Behavioural results demonstrated that the costs of monitoring two modalities, versus one modality, decrease when spatial attention is divided between two separate locations compared with focused attention. Neuroimaging data revealed increased activity in the posterior arietal cortex (PPC) when monitoring two modalities at different locations, while no specific region was recruited in the focused attention conditions. We suggest that supramodal control and the integration of spatial representations hinders the selection of independent sensory streams when attention is spatially focused, while a greater exploitation of modality-specific resources and the engagement of PPC allows in-parallel processing when attention is spatially divided.

Abstract We compared the ability of auditory, visual, and audiovisual (bimodal) exogenous cues to capture visuo-spatial attention under conditions of no load versus high perceptual load. Participants had to discriminate the elevation (up vs. down) of visual targets preceded by either unimodal or bimodal cues under conditions of high perceptual load (in which they had to monitor a rapidly presented central stream of visual letters for occasionally presented target digits) or no perceptual load (in which the central stream was replaced by a fixation point). The results of 3 experiments showed that all 3 cues captured visuo-spatial attention in the no-load condition. By contrast, only the bimodal cues captured visuo-spatial attention in the high-load condition, indicating for the first time that multisensory integration can play a key role in disengaging spatial attention from a concurrent perceptually demanding stimulus.

The aim of this study was to establish whether spatial attention triggered by bimodal exogenous cues acts differently as compared to unimodal and crossmodal exogenous cues due to crossmodal integration. In order to investigate this issue, we examined cuing effects in discrimination tasks and compared these effects in a condition wherein a visual target was preceded by both visual and auditory exogenous cues delivered simultaneously at the same side (bimodal cue), with conditions wherein the visual target was preceded by either a visual (unimodal cue) or an auditory cue (crossmodal cue). The results of two experiments revealed that cuing effects on RTs in these three conditions with an SOA of 200 ms had comparable magnitudes. Differences at a longer SOA of 600 ms (inhibition of return for bimodal cues, Experiment 1) disappeared when catch trials were included (in Experiment 2). The current data do not support an additional influence of crossmodal integration on exogenous orienting, but are well in agreement with the existence of a supramodal spatial attention module that allocates attentional resources towards stimulated locations for different sensory modalities.

Abstract Previous studies have shown that the amplitude of event related brain potentials (ERPs) elicited by a combined audiovisual stimulus is larger than the sum of a single auditory and visual stimulus. This enlargement is thought to reflect multisensory integration. Based on these data, it may be hypothesized that the speeding up of responses, due to exogenous orienting effects induced by bimodal cues, exceeds the sum of single unimodal cues. Behavioral data, however, typically revealed no increased orienting effect following bimodal as compared to unimodal cues, which could be due to a failure of multisensory integration of the cues. To examine this possibility, we computed ERPs elicited by both bimodal (audiovisual) and unimodal (either auditory or visual) cues, and determined their exogenous orienting effects on responses to a to-be-discriminated visual target. Interestingly, the posterior P1 component elicited by bimodal cues was larger than the sum of the P1 components elicited by a single auditory and visual cue (i.e., a superadditive effect), but no enhanced orienting effect was found on response speed. The latter result suggests that multisensory integration elicited by our bimodal cues plays no special role for spatial orienting, at least in the present setting.

In a crowded scene we can effectively focus our attention on a specific speaker while largely ignoring sensory inputs from other speakers. How attended speech inputs are extracted from similar competing information has been primarily studied in the auditory domain. Here we examined the deployment of visuo-spatial attention in multiple speaker scenarios. Steady-state visual evoked potentials (SSVEP) were monitored as a real-time index of visual attention towards three competing speakers. Participants were instructed to detect a target syllable by the center speaker and ignore syllables from two flanking speakers. The study incorporated interference trials (syllables from three speakers), no-interference trials (syllable from center speaker only), and periods without speech stimulation in which static faces were presented. An enhancement of flanking speaker induced SSVEP was found 70 220 ms after sound onset over left temporal scalp during interference trials. This enhancement was negatively correlated with the behavioral performance of participants - those who showed largest enhancements had the worst speech recognition performance. Additionally, poorly performing participants exhibited enhanced flanking speaker induced SSVEP over visual scalp during periods without speech stimulation. The present study provides neurophysiologic evidence that the deployment of visuo-spatial attention to flanking speakers interferes with the recognition of multisensory speech signals under noisy environmental conditions.

78 In animals, largest multisensory interactions occur when inputs evoke low responses. 78 We examine this principle of inverse effectiveness (IE) in humans. 78 Multisensory interactions (MI) in behavior were strongest for low intensity stimuli. 78 Low intensity audiovisual stimuli show strongest MI in early evoked brain activity. 78 The strength of MI in early evoked potentials in humans follows the principle of IE.

61In this EEG study, participants continuously attended to only one hemifield.61With one side ever relevant, attention did not modulate prestimulus alpha activity.61Attention modulated the P1 only ipsilaterally, and the N1 bilaterally.61Prestimulus alpha activity, the P1 and N1 reflect different aspects of attention.61Attention is subserved by multiple inhibitory and facilitatory neural mechanisms.

The McGurk effect is usually presented as an example of fast, automatic, multisensory integration. We report a series of experiments designed to directly assess these claims. We used a syllabic version of the speeded classification paradigm, whereby response latencies to the first (target) syllable of spoken word-like stimuli are slowed down when the second (irrelevant) syllable varies from trial to trial. This interference effect is interpreted as a failure of selective attention to filter out the irrelevant syllable. In Experiment 1 we reproduced the syllabic interference effect with bimodal stimuli containing auditory as well as visual lip movement information, thus confirming the generalizability of the phenomenon. In subsequent experiments we were able to produce (Experiment 2) and to eliminate (Experiment 3) syllabic interference by introducing ‘illusory’ (McGurk) audiovisual stimuli in the irrelevant syllable, suggesting that audiovisual integration occurs prior to attentional selection in this paradigm.

This review addresses the question of when spatial coincidence facilitates multisensory integration in humans. According to the spatial rule (which was first formulated on the basis of neurophysiological data in anesthetized animals), multisensory integration is enhanced when stimuli in different sensory modalities are presented from the same spatial location. While the spatial rule fits with the available data from studies of overt and covert spatial attentional orienting, and from the majority of those studies in which space has been somehow relevant to the participant's task, it is inconsistent with the evidence that has emerged from the majority of multisensory studies of stimulus identification and temporal perception. Such a mixed pattern of behavioral results suggests that the spatial rule does not represent a general constraint on multisensory integration in humans. Instead, it would appear to be a much more task-dependent phenomenon than is often realized. These results, however, are broadly consistent with a distinction between the processing of "where" and "what" (or "how") information processing in the human brain.

The last 30 years have seen numerous studies demonstrating unimodal and crossmodal spatial cuing effects. However, surprisingly few studies have attempted to investigate whether multisensory cues might be any more effective in capturing a person spatial attention than unimodal cues. Indeed, until very recently, the consensus view was that multisensory cues were, in fact, no more effective. However, the results of several recent studies have overturned this conclusion, by showing that multisensory cues retain their attention-capturing ability under conditions of perceptual load (i.e., when participants are simultaneously engaged in a concurrent attention-demanding task) while their constituent signals (when presented unimodally) do not. Here we review the empirical literature on multisensory spatial cuing effects and highlight the implications that this research has for the design of more effective warning signals in applied settings.

For thousands of years science philosophers have been impressed by how effectively the senses work together to enhance the salience of biologically meaningful events. However, they really had no idea how this was accomplished. Recent insights into the underlying physiological mechanisms reveal that, in at least one circuit, this ability depends on an intimate dialogue among neurons at multiple levels of the neuraxis; this dialogue cannot take place until long after birth and might require a specific kind of experience. Understanding the acquisition and usage of multisensory integration in the midbrain and cerebral cortex of mammals has been aided by a multiplicity of approaches. Here we examine some of the fundamental advances that have been made and some of the challenging questions that remain.

AbstractIn natural environments, human sensory systems work in a coordinated and integrated manner to perceive and respond to external events. Previous research has shown that the spatial and temporal relationships of sensory signals are paramount in determining how information is integrated across sensory modalities, but in ecologically plausible settings, these factors are not independent. In the current study, we provide a novel exploration of the impact on behavioral performance for systematic manipulations of the spatial location and temporal synchrony of a visual-auditory stimulus pair. Simple auditory and visual stimuli were presented across a range of spatial locations and stimulus onset asynchronies (SOAs), and participants performed both a spatial localization and simultaneity judgment task. Response times in localizing paired visual-auditory stimuli were slower in the periphery and at larger SOAs, but most importantly, an interaction was found between the two factors, in which the effect of SOA was greater in peripheral as opposed to central locations. Simultaneity judgments also revealed a novel interaction between space and time: individuals were more likely to judge stimuli as synchronous when occurring in the periphery at large SOAs. The results of this study provide novel insights into (a) how the speed of spatial localization of an audiovisual stimulus is affected by location and temporal coincidence and the interaction between these two factors and (b) how the location of a multisensory stimulus impacts judgments concerning the temporal relationship of the paired stimuli. These findings provide strong evidence for a complex interdependency between spatial location and temporal structure in determining the ultimate behavioral and perceptual outcome associated with a paired multisensory (i.e., visual-auditory) stimulus.

Abstract Interactions between multisensory integration and attention were studied using a combined audiovisual streaming design and a rapid serial visual presentation paradigm. Event-related potentials (ERPs) following audiovisual objects (AV) were compared with the sum of the ERPs following auditory (A) and visual objects (V). Integration processes were expressed as the difference between these AV and (A + V) responses and were studied while attention was directed to one or both modalities or directed elsewhere. Results show that multisensory integration effects depend on the multisensory objects being fully attended--that is, when both the visual and auditory senses were attended. In this condition, a superadditive audiovisual integration effect was observed on the P50 component. When unattended, this effect was reversed; the P50 components of multisensory ERPs were smaller than the unisensory sum. Additionally, we found an enhanced late frontal negativity when subjects attended the visual component of a multisensory object. This effect, bearing a strong resemblance to the auditory processing negativity, appeared to reflect late attention-related processing that had spread to encompass the auditory component of the multisensory object. In conclusion, our results shed new light on how the brain processes multisensory auditory and visual information, including how attention modulates multisensory integration processes.

Multisensory integration has often been characterized as an automatic process. Recent findings indicate that multisensory integration can occur across various stages of stimulus processing that are linked to, and can be modulated by, attention. Stimulus-driven, bottom-up mechanisms induced by crossmodal interactions can automatically capture attention towards multisensory events, particularly when competition to focus elsewhere is relatively low. Conversely, top-down attention can facilitate the integration of multisensory inputs and lead to a spread of attention across sensory modalities. These findings point to a more intimate and multifaceted interplay between attention and multisensory integration than was previously thought. We review developments in the current understanding of the interactions between attention and multisensory processing, and propose a framework that unifies previous, apparently discordant, findings.

We used event-related potentials (ERPs) to evaluate the role of attention in the integration of visual and auditory features of multisensory objects. This was done by contrasting the ERPs to multisensory stimuli (AV) to the sum of the ERPs to the corresponding auditory-only (A) and visual-only (V) stimuli [i.e., AV vs. (A + V)]. V, A, and VA stimuli were presented in random order to the left and right hemispaces. Subjects attended to a designated side to detect infrequent target stimuli in either modality there. The focus of this report is on the ERPs to the standard (i.e., nontarget) stimuli. We used rapid variable stimulus onset asynchronies (350-650 msec) to mitigate anticipatory activity and included o-stim trials to estimate and remove ERP overlap from residual anticipatory processes and from adjacent stimuli in the sequence. Spatial attention effects on the processing of the unisensory stimuli consisted of a modulation of visual P1 and N1 components (at 90-130 msec and 160-200 msec, respectively) and of the auditory N1 and processing negativity (100-200 msec). Attended versus unattended multisensory ERPs elicited a combination of these effects. Multisensory integration effects consisted of an initial frontal positivity around 100 msec that was larger for attended stimuli. This was followed by three phases of centro-medially distributed effects of integration and/or attention beginning at around 160 msec, and peaking at 190 (scalp positivity), 250 (negativity), and 300-500 msec (positivity) after stimulus onset. These integration effects were larger in amplitude for attended than for unattended stimuli, providing neural evidence that attention can modulate multisensory-integration processes at multiple stages.

Stimuli from multiple sensory organs can be integrated into a coherent representation through multiple phases of multisensory processing; this phenomenon is called multisensory integration. Multisensory integration can interact with attention. Here, we propose a framework in which attention modulates multisensory processing in both endogenous (goal-driven) and exogenous (stimulus-driven) ways. Moreover, multisensory integration exerts not only bottom-up but also top-down control over attention. Specifically, we propose the following: (1) endogenous attentional selectivity acts on multiple levels of multisensory processing to determine the extent to which simultaneous stimuli from different modalities can be integrated; (2) integrated multisensory events exert top-down control on attentional capture via multisensory search templates that are stored in the brain; (3) integrated multisensory events can capture attention efficiently, even in quite complex circumstances, due to their increased salience compared to unimodal events and can thus improve search accuracy; and (4) within a multisensory object, endogenous attention can spread from one modality to another in an exogenous manner.

Author information: (1)Division of Cognitive Psychology and Neuropsychology, Institute of Behavioural Sciences, University of Helsinki Helsinki, Finland.

Abstract Searching for an object within a cluttered, continuously changing environment can be a very time-consuming process. The authors show that a simple auditory pip drastically decreases search times for a synchronized visual object that is normally very difficult to find. This effect occurs even though the pip contains no information on the location or identity of the visual object. The experiments also show that the effect is not due to general alerting (because it does not occur with visual cues), nor is it due to top-down cuing of the visual change (because it still occurs when the pip is synchronized with distractors on the majority of trials). Instead, we propose that the temporal information of the auditory signal is integrated with the visual signal, generating a relatively salient emergent feature that automatically draws attention. Phenomenally, the synchronous pip makes the visual object pop out from its complex environment, providing a direct demonstration of spatially nonspecific sounds affecting competition in spatial visual processing.

The majority of studies investigating interactions between vision and touch have typically explored single events, presenting one object at a time. The present study investigates how tactile–visual interactions affect competition between multiple visual objects in more dynamic cluttered environments. Participants searched for a horizontal or vertical line segment among distractor line segments of various orientations, all continuously changing color. Search times and search slopes were substantially reduced when the target color change was accompanied by a tactile signal. These benefits were observed even though the tactile signal was uninformative about the location, orientation, or color of the visual target. We conclude that tactile–visual synchrony guides attention in multiple object environments by increasing the saliency of the visual event.

Visual search is markedly improved when a target color change is synchronized with a spatially non-informative auditory signal. This “pip and pop” effect is an automatic process as even a distractor captures attention when accompanied by a tone. Previous studies investigating visual attention have indicated that automatic capture is susceptible to the size of the attentional window. The present study investigated whether the pip and pop effect is modulated by the extent to which participants divide their attention across the visual field We show that participants were better in detecting a synchronized audiovisual event when they divided their attention across the visual field relative to a condition in which they focused their attention. We argue that audiovisual capture is reduced under focused conditions relative to distributed settings.

In dynamic cluttered environments, audition and vision may benefit from each other in determining what deserves further attention and what does not. We investigated the underlying neural mechanisms responsible for attentional guidance by audiovisual stimuli in such an environment. Event-related potentials (ERPs) were measured during visual search through dynamic displays consisting of line elements that randomly changed orientation. Search accuracy improved when a target orientation change was synchronized with an auditory signal as compared to when the auditory signal was absent or synchronized with a distractor orientation change. The ERP data show that behavioral benefits were related to an early multisensory interaction over left parieto-occipital cortex (50–6002ms post-stimulus onset), which was followed by an early positive modulation (80–10002ms) over occipital and temporal areas contralateral to the audiovisual event, an enhanced N2pc (210–25002ms), and a contralateral negative slow wave (CNSW). The early multisensory interaction was correlated with behavioral search benefits, indicating that participants with a strong multisensory interaction benefited the most from the synchronized auditory signal. We suggest that an auditory signal enhances the neural response to a synchronized visual event, which increases the chances of selection in a multiple object environment.Research Highlights? Multisensory stimuli are more noticeable among a background of visual stimuli. ? Spatially uninformative sounds drive a visuo-spatial shifts of attention. ? Multisensory integration processes result in attentional capture. ? Attentional capture of multisensory stimuli is largely automatic. ? The early multisensory interaction was correlated with behavioral search benefits.

61Responses to multisensory targets are often faster than to unimodal targets.61The role of integration and exogenous attention in this effect was investigated.61At 0ms SOA response enhancement was explained by multisensory integration.61Both integration and crossmodal spatial attention were involved at 50ms SOA.61At longer SOAs response enhancement was due to spatial attention, not integration.

Abstract Multisensory integration (MSI) and exogenous spatial attention can both speedup responses to perceptual events. Recently, it has been shown that audiovisual integration at exogenously attended locations is reduced relative to unattended locations. This effect was observed at short cue-target intervals (200-250090009ms). At longer intervals, however, the initial benefits of exogenous shifts of spatial attention at the cued location are often replaced by response time (RT) costs (also known as Inhibition of Return, IOR). Given these opposing cueing effects at shorter versus longer intervals, we decided to investigate whether MSI would also be affected by IOR. Uninformative exogenous visual spatial cues were presented between 350 and 450090009ms prior to the onset of auditory, visual, and audiovisual targets. As expected, IOR was observed for visual targets (invalid cue RT090009<090009valid cue RT). For auditory and audiovisual targets, neither IOR nor any spatial cueing effects were observed. The amount of relative multisensory response enhancement and race model inequality violation was larger for uncued as compared with cued locations indicating that IOR reduces MSI. The results are discussed in the context of changes in unisensory signal strength at cued as compared with uncued locations. 0008 The Author(s) 2016.

We applied behavioral and event-related potential measurements to study human multisensory interactions induced by audiovisual (AV) stimuli presented peripherally in a visual attention task in which an irrelevant auditory stimulus occasionally accompanied the visual stimulus. A stream of visual, auditory, and AV stimuli was randomly presented to the left or right side of the subjects; subjects covertly attended to the visual stimuli on either the left or right side and promptly responded to visual targets on that side. Behavioral results showed that responses to AV stimuli were faster and more accurate than those to visual stimuli only. Three event-related potential components related to AV interactions were identified: (1) over the right temporal area, approximately 200 to 220 milliseconds; (2) over the centromedial area, approximately 290 to 310 milliseconds; and (3) over the left and right ventral temporal area, approximately 290 to 310 milliseconds. We found that these interaction effects occurred slightly later than those reported in previously published AV interaction studies in which AV stimuli were presented centrally. Our results suggest that the retinotopic location of stimuli affects AV interactions occurring at later stages of cognitive processing in response to a visual attention task.

The human brain can anatomically combine task-relevant information from different sensory pathways to form a unified perception; this process is called multisensory integration. The aim of the present study was to test whether the multisensory integration abilities of patients with mild cognitive impairment (MCI) and Alzheimer's disease (AD) differed from those of normal aged controls (NC). A total of 64 subjects were divided into three groups: NC individuals (n = 24), MCI patients (n = 19), and probable AD patients (n = 21). All of the subjects were asked to perform three separate audiovisual integration tasks and were instructed to press the response key associated with the auditory, visual, or audiovisual stimuli in the three tasks. The accuracy and response time (RT) of each task were measured, and the RTs were analyzed using cumulative distribution functions to observe the audiovisual integration. Our results suggest that the mean RT of patients with AD was significantly longer than those of patients with MCI and NC individuals. Interestingly, we found that patients with both MCI and AD exhibited adequate audiovisual integration, and a greater peak (time bin with the highest percentage of benefit) and broader temporal window (time duration of benefit) of multisensory enhancement were observed. However, the onset time and peak benefit of audiovisual integration in MCI and AD patients occurred significantly later than did those of the NC. This finding indicates that the cognitive functional deficits of patients with MCI and AD contribute to the differences in performance enhancements of audiovisual integration compared with NC.