关键词: 视听整合, 言语感知, 贝叶斯因果推断

Abstract: Audiovisual speech perception is a uniquely human and fundamental multimodal processing capability, constituting a highly interdisciplinary research topic at the intersection of perception and language field. Despite its significance, research on the computational and neural mechanisms underlying audiovisual speech perception remains in its early stages. A key question in this field is: How does the brain construct supramodal semantic representations from modality-specific auditory and visual signals? To address this question, this project comprises two studies involving five experiments. The first study seeks to systematically identify the core brain regions involved in the audiovisual speech perception network from two perspectives: (1) audiovisual operations (Experiment 1) and (2) audiovisual information representations (Experiment 2). Specifically, we will adopt two complementary approaches: on the one hand, we will examine which brain regions participate in specific audiovisual speech processing operations (Experiment 1), such as auditory-specific processing, audiovisual temporal synchrony processing, and audiovisual semantic congruency processing. On the other hand, we will reveal the cortical representation for various complex features embedded in audiovisual speech signals (Experiment 2), such as acoustic features, facial articulation features, and semantic features. The second study, grounded in Bayesian causal inference theory, investigates audiovisual relationships at different levels of abstraction in speech signals. It examines how the human brain flexibly integrates auditory and visual speech signals based on the causal structures underlying these relationships (Experiments 3-5). Specifically, we will focus on three types of audiovisual relationships: (1) the semantic relationship between audiovisual speech signals, i.e., whether the visual and auditory speech signals convey the same semantic concept. This is the most distinctive and abstract audiovisual relationship in speech perception, which requires specialized language learning to acquire. (2) The speaker identity correspondence between audiovisual speech signals, i.e., whether the visual and auditory speech signals originate from the same. This is a distinctive, socially grounded audiovisual relationship in speech perception, acquired through statistical regularities in communicative interactions. (3) The temporal relationship between audiovisual speech signals, i.e., whether the visual and auditory speech signals are temporally synchronized. This is a fundamental audiovisual relationship common to all multimodal processing, biologically universal, and has been one of the major topics in previous computational modeling studies. We will establish a “causal inference-based hierarchical audiovisual speech integration theoretical framework,” delineate the core brain regions of the human audiovisual speech perception network, and reveal how these brain regions dynamically integrate visual and auditory speech signals based on audiovisual relationships at different levels of abstraction to form supramodal semantic representations. At the theoretical level, the findings of this project may advance understanding of the neural mechanisms underlying multimodal information integration in the brain and provide deeper insights into language processing mechanisms. From an applied perspective, the research may informing the development of multimodal artificial intelligence systems that align with human cognitive and behavioral principles.

Key words: audiovisual integration, speech perception, Bayesian causal inference