Abstract:

The integration of facial-vocal emotion is an important factor for successful communication that intrigue psychologists and neuroscientists in recent years. Previous studies have elaborated on the behavioral performance and the influence factors for facial-vocal emotion integration, as well as “when” and “where” information from the two modes integrated. However, it remains open questions whether the integration of facial-vocal emotion follows the principles of multisensory integration (eg.the principle of inverse effectiveness), and how the bimodal emotional information merges into a coherence emotional object. Therefore, taking “whether facial-vocal emotion integration obeys the principle of inverse effectiveness” as main line, we designed six experiments which manipulated emotional salience of the dynamic facial-vocal emotional stimuli and task demands systematically. Moreover, using multi-dimensional analysis of behavioral and EEG data, especially time-frequency and coherence analysis of EEG data, we aimed to answer the two proposed questions, to further reveal the neurophysiological mechanism of facial-vocal emotion integration.

Key words: facial emotion, vocal emotion, integration, neural oscillation, ERPs