The brain mechanism of explicit and implicit processing of affective prosodies: An fNIRS study

doi:10.3724/SP.J.1041.2021.00015

Abstract

Abstract:

Emotional expressions of others embedded in speech prosodies are important for social interactions. Affective prosody refers to a way to express and convey emotions through the dynamic changes of various acoustic cues such as pitch, intensity, stress, and intonation in speech, without relying on vocabulary and grammatical structure. Previous studies have shown that STC, IFG, OFC, and other cerebral cortex and subcortical structures are involved in affective prosody processing, and gradually formed a hierarchical model. However, existing studies on the neural mechanism of affective prosody processing mostly focus on the difference between non-neutral affective prosody and neutral prosody, while the comparison between various non-neutral affective prosody is less investigated. Besides, the differences involved in brain regions of affective prosody processing under explicit and implicit tasks are still not clear. Furthermore, it is necessary to further accumulate experimental evidence based on noise-free brain imaging technology such as fNIRS, which has a noise-free feature and is especially suitable for speech processing research.

This study used functional near-infrared spectroscopy to investigate how speech prosodies of different emotional categories are processed in the cortex under different task conditions. A group of 25 college students participated in this study with a 3 (emotion: angry vs. fearful vs. happy) í 2 (task focus: explicit vs. implicit) within-participant factorial design. We manipulated task focus by adopting two different tasks, with emotional discrimination task as explicit condition and sex discrimination task as implicit condition. Ten phonological materials for each of angry, fearful, and happy prosody were selected from the Chinese Speech Emotion Database and consisted of the corresponding affective prosodies and neutral prosodies. The emotional explicit task was to count the affective and neutral sentences contained in each 10-second speech, and the affective implicit task was to count the sentences played by two women in each 10-second speech. A multi-channel fNIRS system was used to record brain activity in a continuous waveform. According to existing literature, the brain regions observed in this study are the bilateral frontal and temporal lobes. Therefore, we used 13 emitters and 15 detectors to form 37 effective observation channels.

We first adopted NirSpark-2442 software to preprocess the data, and then conducted general linear model analyses to calculate the cortical activation related to the task. The results showed that the brain activation was significantly higher when angry prosody was contrasted to fearful and happy ones in left frontal pole / orbitofrontal cortex, when happy prosody was contrasted to fearful and angry ones in left inferior frontal gyrus, and when fearful prosody was contrasted to angry and happy ones in right supramarginal gyrus. Importantly, there was an interaction between emotion and task. In the explicit task, cortex activity in the right supramarginal gyrus was more sensitive to fearful prosody than to angry and happy ones. But no similar results were found under angry and happy prosody. In addition, the brain activation in temporopolar, superior temporal gyrus, and middle temporal gyrus in the explicit task was greater than that in the implicit task.

The present study demonstrated the specific brain regions for processing angry, fearful and happy prosody were left frontal pole / orbitofrontal cortex, right supramarginal gyrus, and left inferior frontal gyrus, respectively, and revealed the important role of right superior temporal gyrus and right supramarginal gyrus in emotional explicit task. These findings partially support the hierarchical model of affective prosody and question the third level of the model.

Key words: emotion, phonetic prosody, superior temporal gyrus, orbitofrontal cortex, inferior frontal gyrus, supramarginal gyrus

LEI Zhen, BI Rong, MO Licheng, YU Wenwen, ZHANG Dandan. (2021). The brain mechanism of explicit and implicit processing of affective prosodies: An fNIRS study. Acta Psychologica Sinica, 53(1), 15-25.

Figures/Tables 6

References 55

[1]	Adolphs, R., Damasio, H., Tranel, D., Cooper, G., & Damasio, A. R. (2000). A role for somatosensory cortices in the visual recognition of emotion as revealed by three-dimensional lesion mapping. The Journal of Neuroscience, 20(7), 2683-2690. URL pmid: 10729349
[2]	Alba-Ferrara, L., Kochen, S., & Hausmann, M. (2018). Affective prosody processing in epilepsy: Some insights on brain reorganization. Frontiers in Human Neuroscience, 12, 92. doi: 10.3389/fnhum.2018.00092 URL pmid: 29593517
[3]	Aryani, A., Hsu, C.-T., & Jacobs, A. M. (2018). The sound of words evokes affective brain responses. Brain Sciences, 8(6), 94. doi: 10.3390/brainsci8060094 URL
[4]	Bach, D. R., Grandjean, D., Sander, D., Herdener, M., Strik, W. K., & Seifritz, E. (2008). The effect of appraisal level on processing of affective prosody in meaningless speech. Neuroimage, 42(2), 919-927. doi: 10.1016/j.neuroimage.2008.05.034 URL pmid: 18586524
[5]	Beaucousin, V., Zago, L., Herve, P.-Y., Strelnikov, K., Crivello, F., Mazoyer, B., & Tzourio-Mazoyer, N. (2011). Sex-dependent modulation of activity in the neural networks engaged during emotional speech comprehension. Brain Research, 1390, 108-117. doi: 10.1016/j.brainres.2011.03.043 URL pmid: 21439268
[6]	Belyk, M., & Brown, S. (2014). Perception of affective and linguistic prosody: An ALE meta-analysis of neuroimaging studies. Social Cognitive and Affective Neuroscience, 9(9), 1395-1403. doi: 10.1093/scan/nst124 URL pmid: 23934416
[7]	Ben-David, B. M., Multani, N., Shakuf, V., Rudzicz, F., & van Lieshout, P. H. H. M. (2016). Prosody and semantics are separate but not separable channels in the perception of emotional speech: test for rating of emotions in speech. Journal of Speech Language and Hearing Research, 59(1), 72-89. doi: 10.1044/2015_JSLHR-H-14-0323 URL
[8]	Beyer, F., Munte, T. F., Gottlich, M., & Kramer, U. M. (2014). Orbitofrontal cortex reactivity to angry facial expression in a social interaction correlates with aggressive behavior. Cerebral Cortex, 25(9), 3057-3063. doi: 10.1093/cercor/bhu101 URL pmid: 24842782
[9]	Brück, C., Kreifelts, B., & Wildgruber, D. (2011). Emotional voices in context: A neurobiological model of multimodal affective information processing. Physics of Life Reviews, 8(4), 383-403. doi: 10.1016/j.plrev.2011.10.002 URL pmid: 22035772
[10]	Calvo, M. G., & Nummenmaa, L. (2016). Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cognition and Emotion, 30(6), 1081-1106. doi: 10.1080/02699931.2015.1049124 URL pmid: 26212348
[11]	Dieler, A. C., Tupak, S. V., & Fallgatter, A. J. (2012). Functional near-infrared spectroscopy for the assessment of speech related tasks. Brain and Language, 121(2), 90-109. doi: 10.1016/j.bandl.2011.03.005 URL pmid: 21507475
[12]	Enea, V., & Iancu, S, (2016). Processing emotional body expressions: state-of-the-art. Social Neuroscience, 11(5), 495-506. doi: 10.1080/17470919.2015.1114020 URL pmid: 26513592
[13]	Ethofer, T., Anders, S., Erb, M., Herbert, C., Wiethoff, S., Kissler, J., … Wildgruber, D. (2006). Cerebral pathways in processing of affective prosody: a dynamic causal modeling study. Neuroimage, 30(2), 580-587. doi: 10.1016/j.neuroimage.2005.09.059 URL pmid: 16275138
[14]	Ethofer, T., Bretscher, J., Gschwind, M., Kreifelts, B., Wildgruber, D., & Vuilleumier, P. (2012). Emotional voice areas: Anatomic location, functional properties, and structural connections revealed by combined fMRI/DTI. Cerebral Cortex, 22(1), 191-200. doi: 10.1093/cercor/bhr113 URL pmid: 21625012
[15]	Ethofer, T., Kreifelts, B., Wiethoff, S., Wolf, J., Grodd, W., Vuilleumier, P., & Wildgruber, D. (2009b). Differential influences of emotion, task, and novelty on brain regions underlying the processing of speech melody. Journal of Cognitive Neuroscience, 21(7), 1255-1268. doi: 10.1162/jocn.2009.21099 URL pmid: 18752404
[16]	Ethofer, T., van de Ville, D., Scherer, K., & Vuilleumier, P. (2009a). Decoding of emotional information in voice-sensitive cortices. Current Biology, 19(12), 1028-1033. doi: 10.1016/j.cub.2009.04.054 URL pmid: 19446457
[17]	Fox, K. C. R., Yih, J., Raccah, O., Pendekanti, S. L., Limbach, L. E., Maydan, D. D., & Parvizi, J. (2018). Changes in subjective experience elicited by direct stimulation of the human orbitofrontal cortex. Neurology, 91(16), e1519-e1527. doi: 10.1212/WNL.0000000000006358 URL pmid: 30232252
[18]	Frühholz, S., Ceravolo, L., & Grandjean, D. (2012). Specific brain networks during explicit and implicit decoding of affective prosody. Cerebral Cortex, 22(5), 1107-1117. doi: 10.1093/cercor/bhr184 URL pmid: 21750247
[19]	Frühholz, S., & Grandjean, D. (2013a). Multiple subregions in superior temporal cortex are differentially sensitive to vocal expressions: A quantitative meta-analysis. Neuroscience and Biobehavioral Reviews, 37(1), 24-35. doi: 10.1016/j.neubiorev.2012.11.002 URL pmid: 23153796
[20]	Frühholz, S., & Grandjean, D. (2013b). Processing of emotional vocalizations in bilateral inferior frontal cortex. Neuroscience and Biobehavioral Reviews, 37(10), 2847-2855. doi: 10.1016/j.neubiorev.2013.10.007 URL
[21]	Frühholz, S., Hofstetter, C., Cristinzio, C., Saj, A., Seeck, M Vuilleumier, P., & Grandjean, D. (2015). Asymmetrical effects of unilateral right or left amygdala damage on auditory cortical processing of vocal emotions. Proceedings of the National Academy of Sciences of the United States of America, 112(5), 1583-1588. doi: 10.1073/pnas.1411315112 URL pmid: 25605886
[22]	Frühholz, S., Trost, W., & Kotz, S. A. (2016). The sound of emotions-Towards a unifying neural network perspective of affective sound processing. Neuroscience and Biobehavioral Reviews, 68, 96-110. doi: 10.1016/j.neubiorev.2016.05.002 URL pmid: 27189782
[23]	Goucha, T., & Friederici, A. D. (2015). The language skeleton after dissecting meaning: A functional segregation within Broca’s Area. Neuroimage, 114, 294-302. doi: 10.1016/j.neuroimage.2015.04.011 URL pmid: 25871627
[24]	Hartwigsen, G., Baumgaertner, A., Price, C. J., Koehnke, M., Ulmer, S., & Siebner, H. R. (2010). Phonological decisions require both the left and right supramarginal gyri. Proceedings of the National Academy of Sciences of the United States of America, 107(38), 16494-16499. doi: 10.1073/pnas.1008121107 URL pmid: 20807747
[25]	Hensel, L., Bzdok, D., Müller, V. I., Zilles, K., & Eickhoff, S. B. (2015). Neural correlates of explicit social judgments on vocal stimuli. Cerebral Cortex, 25(5), 1152-1162. doi: 10.1093/cercor/bht307 URL pmid: 24243619
[26]	Herpertz, S. C., Nagy, K., Ueltzhöffer, K., Schmitt, R., Mancke, F., Schmahl, C., & Bertsch, K. (2017). Brain mechanisms underlying reactive aggression in borderline personality disorder-sex matters. Biological Psychiatry, 82(4), 257-266. doi: 10.1016/j.biopsych.2017.02.1175 URL pmid: 28388995
[27]	Hinojosa, J. A., Mercado, F., & Carretié, L. (2015). N170 sensitivity to facial expression: A meta-analysis. Neuroscience and Biobehavioral Reviews, 55, 498-509. doi: 10.1016/j.neubiorev.2015.06.002 URL pmid: 26067902
[28]	Johnstone, T., van Reekum, C. M., Oakes, T. R., & Davidson, R. J. (2006). The voice of emotion: an FMRI study of neural responses to angry and happy vocal expressions. Social Cognitive and Affective Neuroscience, 1(3), 242-249. doi: 10.1093/scan/nsl027 URL pmid: 17607327
[29]	Kirby, L. A. J., & Robinson, J. L. (2017). Affective mapping: An activation likelihood estimation (ALE) meta-analysis. Brain and Cognition, 118, 137-148. doi: 10.1016/j.bandc.2015.04.006 URL pmid: 26074298
[30]	Knight, M. J., & Baune, B. T. (2019). Social cognitive abilities predict psychosocial dysfunction in major depressive disorder. Depression and Anxiety, 36(1), 54-62. doi: 10.1002/da.22844 URL pmid: 30211966
[31]	Kotz, S. A., Kalberlah, C., Bahlmann, J., Friederici, A. D., & Haynes, J.-D. (2013). Predicting vocal emotion expressions from the human brain. Human Brain Mapping, 34(8), 1971-1981. doi: 10.1002/hbm.22041 URL pmid: 22371367
[32]	Kotz, S. A., Meyer, M., Alter, K., Besson, M., von Cramon, D. Y., & Friederici, A. D. (2003). On the lateralization of affective prosody: an event-related functional MR investigation. Brain and Language, 86(3), 366-376. doi: 10.1016/s0093-934x(02)00532-1 URL pmid: 12972367
[33]	Köchel, A., Schöngassner, F., & Schienle, A. (2013). Cortical activation during auditory elicitation of fear and disgust: a near-infrared spectroscopy (NIRS) study. Neuroscience Letters, 9(549), 197-200.
[34]	Lancaster, J. L., Woldorff, M. G., Parsons, L. M., Liotti, M., Freitas, C. S., Rainey, L., … Fox, P. (2000). Automated Talairach atlas labels for functional brain mapping. Human Brain Mapping, 10(3), 120-131. doi: 10.1002/1097-0193(200007)10:3<120::aid-hbm30>3.0.co;2-8 URL pmid: 10912591
[35]	Liebenthal, E., Silbersweig, D. A., & Stern, E. (2016). The Language, Tone and prosody of emotions: neural substrates and dynamics of spoken-word emotion perception. Frontiers in Aging Neuroscience, 10, 506.
[36]	Lin, Y., Ding, H., & Zhang, Y. (2018). Affective prosody processing in schizophrenic patients: A selective review and meta-analysis. Journal of Clinical Medicine, 7(10), 363. doi: 10.3390/jcm7100363 URL
[37]	Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E., & Barrett, L. F. (2012). The brain basis of emotion: a meta-analytic review. Behavioral and Brain Sciences, 35(3), 121-143. doi: 10.1017/S0140525X11000446 URL
[38]	Liu, P., & Pell, M. D. (2012). Recognizing vocal emotions in Mandarin Chinese: a validated database of Chinese vocal emotional stimuli. Behavior Research Methods, 44, 1042-1051. doi: 10.3758/s13428-012-0203-3 URL pmid: 22539230
[39]	Matsui, T., Nakamura, T., Utsumi, A., Sasaki, A. T., Koike, T., Yoshida, Y., … Sadato, N. (2016). The role of prosody and context in sarcasm comprehension: Behavioral and fMRI evidence. Neuropsychologia, 87, 74-84. doi: 10.1016/j.neuropsychologia.2016.04.031 URL pmid: 27157883
[40]	Mitchell, R. L. C. (2007). fMRI delineation of working memory for affective prosody in the brain: commonalities with the lexico-semantic emotion network. Neuroimage, 36(3), 1015-1025. doi: 10.1016/j.neuroimage.2007.03.016 URL pmid: 17481919
[41]	Mitchell, R. L. C., & Xu, Y. (2015). What is the value of embedding artificial affective prosody in human-computer interactions? Implications for theory and design in psychological science. Frontiers in Psychology, 6, 1750. doi: 10.3389/fpsyg.2015.01750 URL pmid: 26617563
[42]	Molavi, B., & Dumont, G. A. (2012). Wavelet-based motion artifact removal for functional near-infrared spectroscopy. Physiological Measurement, 33(2), 259-270. doi: 10.1088/0967-3334/33/2/259 URL pmid: 22273765
[43]	Mothes-Lasch, M., Mentzel, H.-J., Miltner, W. H. R., & Straube, T. (2011). Visual attention modulates brain activation to angry voices. Journal of Neuroscience, 31(26), 9594-9598. doi: 10.1523/JNEUROSCI.6665-10.2011 URL pmid: 21715624
[44]	Patel, S., Oishi, K., Wright, A., Sutherland-Foggio, H., Saxena, S., Sheppard, S. M., & Hillis, A. E. (2018). Right hemisphere regions critical for expression of emotion through prosody. Frontiers in neurology, 9, 224. doi: 10.3389/fneur.2018.00224 URL pmid: 29681885
[45]	Patterson, R. D., Uppenkamp, S., Johnsrude, I. S., & Griffiths, T. D. (2002). The processing of temporal pitch and melody information in auditory cortex. Neuron, 36(4), 767-776. doi: 10.1016/s0896-6273(02)01060-7 URL pmid: 12441063
[46]	Paulmann, S., Seifert, S., & Kotz, S. A. (2010). Orbito-frontal lesions cause impairment during late but not early emotional prosodic processing. Social Neuroscience, 5(1), 59-75. doi: 10.1080/17470910903135668 URL pmid: 19658025
[47]	Quadflieg, S., Mohr, A., Mentzel, H.-J., Miltner, W. H. R., & Straube, T. (2008). Modulation of the neural network involved in the processing of angry prosody: the role of task-relevance and social phobia. Biological Psychology, 78(2), 129-137. doi: 10.1016/j.biopsycho.2008.01.014 URL pmid: 18353521
[48]	Ross, E. D. (1981). The aprosodias. Functional-anatomic organization of the affective components of language in the right hemisphere. Archives of Neurology, 38(9), 561-569. doi: 10.1001/archneur.1981.00510090055006 URL pmid: 7271534
[49]	Schirmer, A., & Kotz, S. A. (2006). Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. Trends in Cognitive Sciences, 10(1), 24-30. doi: 10.1016/j.tics.2005.11.009 URL pmid: 16321562
[50]	Steber, S., König, N., Stephan, F., & Rossi, S. (2020). Uncovering electrophysiological and vascular signatures of implicit affective prosody. Scientific Reports, 10(1), 5807. doi: 10.1038/s41598-020-62761-x URL pmid: 32242032
[51]	Tong, Y., Hocke, L. M., & Frederick, B. B., (2011). Isolating the sources of widespread physiological fluctuations in functional near-infrared spectroscopy signals. Journal of Biomedical Optics, 16(10), 106005. doi: 10.1117/1.3638128 URL pmid: 22029352
[52]	Witteman, J., van Heuven, V. J., & Schiller, N. O. (2012). Hearing feelings: a quantitative meta-analysis on the neuroimaging literature of affective prosody perception. Neuropsychologia, 50(12), 2752-2763. doi: 10.1016/j.neuropsychologia.2012.07.026 URL pmid: 22841991
[53]	Zhang, D., Chen, Y., Hou, X., & Wu, Y. J. (2019). Near-infrared spectroscopy reveals neural perception of vocal emotions in human neonates. Human Brain Mapping, 40(8), 2434-2448. doi: 10.1002/hbm.24534 URL pmid: 30697881
[54]	Zhang, D., Zhou, Y., Hou, X., Cui, Y., & Zhou, C. (2017). Discrimination of emotional prosodies in human neonates: A pilot fNIRS study. Neuroscience Letters, 658, 62-66. doi: 10.1016/j.neulet.2017.08.047 URL pmid: 28842278
[55]	Zhang, D., Zhou, Y., & Yuan, J. (2018). Speech prosodies of different emotional categories activate different brain regions in adult cortex: an fNIRS study. Scientific Reports, 8(1), 218. doi: 10.1038/s41598-017-18683-2 URL pmid: 29317758

Channel	Brodmann area	F	p *	Angry β value	Fearful β value	Happy β value
3	L Frontopolar/orbitofrontal area	12.51	0.001	0.21 ± 0.20	0.12 ± 0.23	0.06 ± 0.20
9	L pars triangularis/Broca's area	24.24	< 0.001	0.10 ± 0.16	0.10 ± 0.15	0.21 ± 0.15
32	R Supramarginal gyrus	12.48	0.001	0.10 ± 0.56	0.36 ± 0.51	0.11 ± 0.45

Channel	Brodmann area	F	p *	Angry β value	Fearful β value	Happy β value
3	L Frontopolar/orbitofrontal area	12.51	0.001	0.21 ± 0.20	0.12 ± 0.23	0.06 ± 0.20
9	L pars triangularis/Broca's area	24.24	< 0.001	0.10 ± 0.16	0.10 ± 0.15	0.21 ± 0.15
32	R Supramarginal gyrus	12.48	0.001	0.10 ± 0.56	0.36 ± 0.51	0.11 ± 0.45

Channel	Brodmann area	F	p*	Implicit β value	Explicit β value
27	R Temporopolar area	11.62	0.004	0.04 ± 0.36	0.32 ± 0.42
28	R Superior temporal gyrus	26.17	< 0.001	0.05 ± 0.45	0.37 ± 0.43
29	R Middle temporal gyrus	15.84	0.003	-0.03 ± 0.49	0.34 ± 0.53

Channel	Brodmann area	F	p*	Implicit β value	Explicit β value
27	R Temporopolar area	11.62	0.004	0.04 ± 0.36	0.32 ± 0.42
28	R Superior temporal gyrus	26.17	< 0.001	0.05 ± 0.45	0.37 ± 0.43
29	R Middle temporal gyrus	15.84	0.003	-0.03 ± 0.49	0.34 ± 0.53