Abstract:

Rater effects refer to the impact of different raters’ idiosyncrasies in their behaviors on the evaluation results in creativity assessment. Rater effects are due to the difference in raters’ cognitive process of the evaluation, which are externally reflected in the difference of their scorings. This article first summarizes the studies of rater cognition and other influencing factors on creativity assessment, including characteristics of raters, information of creators and socio-cultural factors. It further examines inter-rater reliability indexes and their limitations, as well as the applications of Generalization Theory and Many-Facet Rasch Model in quantifying and controlling of rater effects. Finally, this paper specifies directions of future research based on the existing limitations, including deepening the investigation on rater cognition in creativity assessment, integrating the studies of rater effects on different levels, and developing new methods and techniques of creativity assessment.

Key words: creativity, subjective scoring, rater effects, rater cognition, inter-rater agreement