Abstract: Carpenter and Just (1975) proposed the Constituent Comparison Model (CCM) to translate how a sentence and picture were processed and integrated. They found that semantic incongruency yielded a mismatch effect, and participants experienced greater difficulty in processing and verifying a negation than they did in the case of an affirmation. When the sentence and picture were displayed one after the other, Underwood et al. (2004) did not observe the mismatch effect. However, Knoeferle et al. (2005) found the effect in a sentence comprehension study with serial picture-sentence presentation. The validity of the CCM for other languages has not been confirmed. The following questions remain unanswered: Does syntax interfered with the verification task and mismatch effect? Does the mismatch effect disappear when sentences are presented before the pictures in Chinese?
Forty undergraduates were paid to participate in the experiment. All were native speakers of Mandarin Chinese and had normal or corrected-to-normal vision. There were three different types of simple sentences, and each type consisted of four sentences. Each sentence was paired with one of the following two pictures: an asterisk above the plus sign and the plus sign above an asterisk. The pairings of sentences and pictures yielded 24 different displays. Eye movements were recorded using a SensoMotoric Instruments (SMI) iView Hi-Speed system. Each trial started with the appearance of a cross on the left side of the screen, and terminated when the participant pressed a response key. Subsequently, a sentence appeared in the center of the screen, followed by a picture. The task of the participants was to decide whether or not the sentence and picture corresponded with each other, and to indicate their decision by pressing one of the two response keys.
The results revealed the main effects of sentence type and semantic relation. The difference between affirmation and double negation was not significant in the sentence-picture task. Participants found it more difficult to process and verify a negation than a double negation. The mismatch effect was evident on the total fixation time, fixation times, and pupil diameters. Furthermore, the interaction of sentence type and semantic relation was significant. The negative sentence-picture was believed to be difficult because participants required more fixation time and total fixation time to complete the tasks of verification and integration in the mismatch condition.
Further, the mismatch effect was also found in the case of Chinese. Semantic mismatches tended to increase the participants’ cognitive load, and the syntax influenced the mismatch effect. It is possible that the participants had adopted a special strategy for representing double negative sentences in which double negative verbs canceled each other. Therefore, we should be cautious with regard to the applicability of the CCM as a more general model of sentence-picture tasks in other languages

Key words: sentence-picture, sentence type, semantic relation, semantic mismatch, eye movement