Abstract: Previous research on generative artificial intelligence (AI) has been primarily conducted in the English context, but it remains unclear about linguistic strength and weakness of generative AI in the Chinese context. This study focuses on the accuracy and normativity, affectivity, and creativity of AI in generating language knowledge, and explores its cultural adaptability and ability to generate humanized and personalized content. Evaluating and analyzing these key indicators helps us gain a deeper understanding of the linguistic strengths and weaknesses of AI, as well as cultivating the unique advantages of humans in education.
By combining quantitative and qualitative methods, we evaluated the differences in knowledge accuracy, normativity, affectivity, and creativity between large language models and real students. Specifically, using an explanatory sequential design in the mixed-methods framework, we first tested group differences in each indicator among GPT-4 and ERNIE-4 versus real students on knowledge accuracy, normativity, affectivity, and creativity to test the. Next, through content analyses, we explored the specific performance of large language models on each indicator and the mechanism of their linguistic strengths and weaknesses.
Study 1 found that compared to real students, GPT-4 exhibited higher accuracy in modern text knowledge (especially conceptual knowledge), but lower accuracy in ancient poetry and language usage. The knowledge normativity of GPT-4 were comparable to those of real students, while its affectivity and creativity were lower than those of real students. Moreover, the highest individual scores of GPT-4 in normativity and emotionality were on comparable with the highest scores of real students. Study 2, based on ERNIE-4, confirmed the aforementioned results, and the accuracy in ancient poetry was still lower than that of real students. The results exhibited the advantages of artificial intelligence in the areas of modern knowledge and norms, its shortcomings in ancient poetry knowledge, and its potential in affective and creative expressions.
Taken together, the current findings demonstrate the linguistic strength of generative AI in the knowledge accuracy of modern Chinese literary, and the weakness regarding ancient Chinese poetry and affective and creative writings, as well as generative AI’s potential in normative and affective expressions. This sheds light on the field of the cultural adaptability, affective and creative expressions of generative AI, and has valuable implications for the AI-assistant teaching practice in the Chinese context.

Key words: large language models, language proficiency, accuracy, emotionality, creativity