Abstract:

How semantics are represented in human brains is a central issue in cognitive neuroscience. Previous studies typically detect semantic information by manipulating the properties of stimuli or task demands, or by asking a group of participants to judge the stimuli according to several given dimensions or features. Despite having brought valuable insights into the neurobiology of language, these approaches have some limitations. First, the experimental approach may only provide a coarse depiction of semantic properties, while human judgment is time-consuming and the results may vary substantially across subjects. Second, the conventional approach has difficulty quantifying the effect of context on word meaning. Third, the conventional approach is unable to extract the topic information of discourses, the semantic relations between the different parts of a discourse, or the semantic distance between discourses.
The recently-developed natural language processing (NLP) techniques provide a useful tool that may overcome the above-mentioned limitations. Grounded on the distributional hypothesis of semantics, NLP models represent meanings of words, sentences, or documents in the form of computable vectors, which can be derived from word-word or word-document co-occurrence relationships, and neural networks trained on language tasks.
Recent studies have applied NLP techniques to model the semantics of stimuli and mapped the semantic vectors onto brain activities through representational similarity analyses or linear regression. Those studies have mainly examined how the brain (i) represents word semantics; (ii) integrates context information and represents sentence-level meanings; and (iii) represents the topic information and the semantic structure of discourses. Besides, a few studies have applied NLP to untangle sentences’ syntactic and semantic information and looked for their respective neural representations. A consistent finding across those studies is that, the representation of semantic information of words, sentences and discourses, as well as the syntactic information, seems to recruit a widely distributed network covering the frontal, temporal, parietal and occipital cortices. This observation is in contrast to the results from conventional imaging studies and lesions studies which typically report localized neural correlates for language processing. One possible explanation for this discrepancy is that NLP language models trained on large-scale text corpus may have captured multiple aspects of semantic information, while the conventional experimental approach may selectively activate a (or several) specific aspects of semantics and therefore only a small part of the brain can be detected.
Though NLP techniques provide a powerful tool to quantify semantic information, they still face some limitations when being applied to investigate semantic representations in the brain. Firstly, embeddings from NLP models (especially those from a deep neural network) are uninterpretable. Secondly, models differ from each other in training material, network architecture, amount of parameters, training tasks and so on, which may lead to potential discrepancies among research results. Finally, model training procedures differ from how humans learn language and semantics, and the inner computational and processing mechanism may also be fundamentally different between NLP models and the human brain. Therefore, researchers need to select a proper model based on research questions, test the validity of models with experimental designs, and interpret results carefully. In the future, it is promising to (i) adopt more informational semantic representation methods such as knowledge-graph and multimodal models; (ii) apply NLP models to assess the language ability of patients; (iii) improve the interpretability and performance of models taking advantages of cognitive neuroscience findings about how human process language.

Key words: semantic representation, brain, natural language processing, language model