Please wait a minute...
Advances in Psychological Science    2019, Vol. 27 Issue (1) : 171-180     DOI: 10.3724/SP.J.1042.2019.00171
Research Method |
Rater effects in creativity assessment
HAN Jiantao1,2,3,LIU Wenling1,PANG Weiguo1()
1 School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China
2 School of Educational Science, Anhui Normal University, Wuhu 241000, China
3 School of Literature Media and Educational Science, Chaohu College, Chaohu 238000, China
Download: PDF(551 KB)   HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks     Supporting Info

Rater effects refer to the impact of different raters’ idiosyncrasies in their behaviors on the evaluation results in creativity assessment. Rater effects are due to the difference in raters’ cognitive process of the evaluation, which are externally reflected in the difference of their scorings. This article first summarizes the studies of rater cognition and other influencing factors on creativity assessment, including characteristics of raters, information of creators and socio-cultural factors. It further examines inter-rater reliability indexes and their limitations, as well as the applications of Generalization Theory and Many-Facet Rasch Model in quantifying and controlling of rater effects. Finally, this paper specifies directions of future research based on the existing limitations, including deepening the investigation on rater cognition in creativity assessment, integrating the studies of rater effects on different levels, and developing new methods and techniques of creativity assessment.

Keywords creativity      subjective scoring      rater effects      rater cognition      inter-rater agreement     
ZTFLH:  B841  
Issue Date: 23 November 2018
E-mail this article
E-mail Alert
Articles by authors
Jiantao HAN
Wenling LIU
Weiguo PANG
Cite this article:   
Jiantao HAN,Wenling LIU,Weiguo PANG. Rater effects in creativity assessment[J]. Advances in Psychological Science, 2019, 27(1): 171-180.
URL:     OR
1 贡喆, 刘昌, 沈汪兵 . (2016). 有关创造力测量的一些思考. 心理科学进展,24(1), 31-45.
2 晏子 . (2010). 心理科学领域内的客观测量——Rasch模型之特点及发展趋势. 心理科学进展, 18(8), 1298-1305.
3 Amabile T.M . (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997-1013.
4 Amabile T.M . (1983). The social psychology of creativity. New York, NY:. Springer-Verlag.
5 American Educational Research Association ( AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing (2014 Edition). Washington, DC:. AERA.
6 Baer J., Kaufman J. C., & Riggs M . (2009). Brief report: Rater-domain interactions in the consensual assessment technique. The International Journal of Creativity & Problem Solving, 19(2), 87-92.
7 Beghetto R.A., &Kaufman J.C . (2007). Toward a broader conception of creativity: A case for "mini-c" creativity. Psychology of Aesthetics, Creativity, and the Arts, 1(2), 73-79.
8 Beketayev K., &Runco M.A . (2016). Scoring divergent thinking tests by computer with a semantics-based algorithm. Europe’s Journal of Psychology, 12(2), 210-220.
pmid: 27298632 url:
9 Benedek M., Mühlmann C., Jauk E., & Neubauer A. C . (2013). Assessment of divergent thinking by means of the subjective top-scoring method: Effects of the number of top-ideas and time-on-task on reliability and validity. Psychology of Aesthetics, Creativity, and the Arts, 7(4), 341-349.
pmid: 24790683 url:
10 Benedek M., Nordtvedt N., Jauk E., Koschmieder C., Pretsch J., Krammer G., & Neubauer A. C . (2016). Assessment of creativity evaluation skills: A psychometric investigation in prospective teachers. Thinking Skills and Creativity, 21, 75-84.
11 Birney D. P., Beckmann J. F., & Seah Y. Z . (2016). More than the eye of the beholder: The interplay of person, task, and situation factors in evaluative judgements of creativity. Learning and Individual Differences, 51, 400-408.
12 Blair C.S., &Mumford M.D . (2007). Errors in idea evaluation: Preference for the unoriginal? The Journal of Creative Behavior, 41(3), 197-222.
13 Campbell D.T . (1960). Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological Review, 67(6), 380-400.
pmid: 13690223 url:
14 Cheng K. H.C . (2016). Perceived interpersonal dimensions and its effect on rating bias: How neuroticism as a trait matters in rating creative works. The Journal of Creative Behavior. February 16, 2017, Retrieved from https:// 1002/jocb. 156.
15 Cropley A. . (2006). In praise of convergent thinking , Creativity Research Journal, 18(3), 391-404.
16 Diedrich J., Benedek M., Jauk E., & Neubauer A. C . (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35-40.
17 Ellamil M., Dobson C., Beeman M., & Christoff K . (2012). Evaluative and generative modes of thought during the creative process. NeuroImage, 59(2), 1783-1794.
pmid: 21854855 url:
18 Fink A., Benedek M., Koschutnig K., Pirker E., Berger E., Meister S ., et al. & Elisabeth M. W. ( 2015). Training of verbal creativity modulates brain activity in regions associated with language- and memory-related demands. Human Brain Mapping, 36(10), 4104-4115.
pmid: 4587539 url:
19 Finke R. A., Ward T. B. , & Smith, S. M.(1992). Creative cognition: Theory, research, and applications Cambridge, MA: MIT Press Theory, research, and applications. Cambridge , MA: MIT Press.
20 Forthmann B., Holling H., Zandi N., Gerwig A., ?elik P., Storme M., & Lubart T . (2017). Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores. Thinking Skills and Creativity, 23, 129-139.
21 Galati F. . (2015). Complexity of judgment: What makes possible the convergence of expert and nonexpert ratings in assessing creativity. Creativity Research Journal, 27(1), 24-30.
22 Gilhooly K. J., Fioratou E., Anthony S. H., & Wynn V . (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects. British Journal of Psychology, 98(4), 611-625.
pmid: 17535464 url:
23 Goncalo J.A., &Staw B.M . (2006). Individualism- collectivism and group creativity. Organizational Behavior and Human Decision Processes, 100(1), 96-109.
24 Haller C. S., Courvoisier D. S., & Cropley D. H . (2011). Perhaps there is accounting for taste: Evaluating the creativity of products. Creativity Research Journal, 23(2), 99-109.
25 Han J. T., Long H. Y., & Pang W. G . (2017). Putting raters in ratees' shoes: Perspective taking and assessment of creative products. Creativity Research Journal, 29(3), 270-281.
26 Hao N., Ku Y. X., Liu M. G., Hu Y., Bodner M., Grabner R. H., & Fink A . (2016). Reflection enhances creativity: Beneficial effects of idea evaluation on idea generation. Brain and Cognition, 103, 30-37.
pmid: 26808451 url:
27 Harbison J.I., & Haarmann H. (2014). Automated scoring of originality using semantic representations, Proceedings of the COGSCI, 36, 2327-2332.
28 Hennessey B.A . (1994). The consensual assessment technique: An examination of the relationship between ratings of product and process creativity. Creativity Research Journal, 7(2), 193-208.
29 Hennessey B.A., &Amabile T.M . (2010). Creativity. Annual Review of Psychology, 61, 569-598.
30 Hong S.W., &Lee J.S . (2015). Nonexpert evaluations on architectural design creativity across cultures. Creativity Research Journal, 27(4), 314-321.
31 Hung S. P., Chen P. H., & Chen H. C . (2012). Improving creativity performance assessment: A rater effect examination with many facet Rasch model. Creativity Research Journal, 24(4), 345-357.
32 Kaufman J. C., Baer J., Agars M. D., & Loomis D . (2010). Creativity stereotypes and the consensual assessment technique. Creativity Research Journal, 22(2), 200-205.
33 Kaufman J. C., Baer J., Cole J. C., & Sexton J. D . (2008). A comparison of expert and nonexpert raters using the consensual assessment technique. Creativity Research Journal, 20(2), 171-178.
34 Kaufman J. C., Baer J., Cropley D. H., Reiter-Palmon R., & Sinnett S . (2013). Furious activity vs. understanding: How much expertise is needed to evaluate creative work? Psychology of Aesthetics, Creativity, and the Arts, 7(4), 332-340.
35 Kaufman J. C., Beghetto R. A., & Dilley A . (2016). Understanding creativity in the schools. In Lipnevich, A. A., Preckel, F., & Roberts, R. D.(Eds.), Psychosocial skills and school systems in the 21st century
36 Kozbelt A., & Serafin J. (2009). Dynamic evaluation of high-and low-creativity drawings by artist and nonartist raters. Creativity Research Journal, 21(4), 349-360.
37 Lan L., &Kaufman J.C . (2012). American and Chinese similarities and differences in defining and valuing creative products. The Journal of Creative Behavior, 46(4), 285-306.
38 Lebuda I., & Karwowski M. (2013). Tell me your name and I'll tell you how creative your work is: Author's name and gender as factors influencing assessment of products' creativity in four different domains. Creativity Research Journal, 25(1), 137-142.
39 Licuanan B. F., Dailey L. R., & Mumford M. D . (2007). Idea evaluation: Error in evaluating highly original ideas. The Journal of Creative Behavior, 41(1), 1-27.
40 Linacre J.M . (1994). Many-facet Rasch measurement (2nd Edition). Chicago, IL:. MESA.
41 Long H.Y . (2014 a). An empirical review of research methodologies and methods in creativity studies (2003- 2012). Creativity Research Journal, 26(4), 427-438.
42 Long H.Y . (2014 b). More than appropriateness and novelty: Judges’ criteria of assessing creative products in science tasks. Thinking Skills and Creativity, 13, 183-194.
43 Long H.Y., &Pang W.G . (2015). Rater effects in creativity assessment: A mixed methods investigation. Thinking Skills and Creativity, 15, 13-25.
44 Lu C.C., &Luh D.B . (2012). A comparison of assessment methods and raters in product creativity. Creativity Research Journal, 24(4), 331-337.
45 McGraw K.O., &Wong S.P . (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46.
46 Mueller J. S., Melwani S., & Goncalo J. A . (2012). The bias against creativity: Why people desire but reject creative ideas. Psychological Science, 23(1), 13-17.
pmid: 22127366 url:
47 Mueller J. S., Wakslak C. J., & Krishnan V . (2014). Construing creativity: The how and why of recognizing creative ideas. Journal of Experimental Social Psychology, 51, 81-87.
48 Mumford M. D., Lonergan D. C., & Scott G . (2002). Evaluating creative ideas: Processes, standards, and context. Inquiry: Critical Thinking Across the Disciplines, 22(1), 21-30.
49 Plucker J., Beghetto R. A., & Dow G . (2004). Why isn’t creativity more important to educational psychologists? Potential, pitfalls, and future directions in creativity research. Educational Psychologist, 39(2), 83-96.
50 Plucker J. A. , & Makel, M. C.(2010) . Assessment of creativity. In Kaufman, J. C. & Sternberg, R. J. (Eds.), The Cambridge handbook of creativity (pp. 48-73). New York, NY: Cambridge University Press.
51 Primi R. . (2014). Divergent productions of metaphors: Combining many-facet Rasch measurement and cognitive psychology in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 8(4), 461-474.
52 Runco M.A., &Jaeger G.J . (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92-96.
53 Runco M.A., &Smith W.R . (1992). Interpersonal and intrapersonal evaluations of creative ideas. Personality and Individual Differences, 13(3), 295-302.
54 Silvia P.J . (2008). Discernment and creativity: How well can people identify their most creative ideas? Psychology of Aesthetics, Creativity, and the Arts, 2(3), 139-146.
55 Silvia P.J . (2011). Subjective scoring of divergent thinking: Examining the reliability of unusual uses, instances, and consequences tasks. Thinking Skills and Creativity, 6(1), 24-30.
56 Silvia P. J., Martin C., & Nusbaum E. C . (2009). A snapshot of creativity: Evaluating a quick and simple method for assessing divergent thinking. Thinking Skills and Creativity, 4(2), 79-85.
57 Silvia P. J., Winterstein B. P., Willse J. T., Barona C. M., Cram J. T., Hess K. I., .. Richard C. A . (2008). Assessing creativity with divergent thinking tasks: Exploring the reliability and validity of new subjective scoring methods. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 68-85.
58 Sowden P. T., Pringle A., & Gabora L . (2015). The shifting sands of creative thinking: Connections to dual- process theory. Thinking & Reasoning, 21(1), 40-60.
59 Storme M., Myszkowski N., ?elik P., & Lubart T . (2014). Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges. Learning and Individual Differences, 32(4), 19-25.
60 Tan M., Mourgues C., Hein S., MacCormick J., Barbot B., & Grigorenko E . (2015). Differences in judgments of creativity: How do academic domain, personality, and self-reported creativity influence novice judges’ evaluations of creative productions? Journal of Intelligence, 3(3), 73-90.
61 Wilson R. C., Guilford J. P., & Christensen P. R . (1953). The measurement of individual differences in originality. Psychological Bulletin, 50(5), 362-370.
pmid: 13100527 url:
62 Wolfe E.W . (2004). Identifying rater effects using latent trait models. Psychology Science, 46(1), 35-51.
63 Wolfe E.W., & McVay A. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement: Issues and Practice, 31(3), 31-37.
64 Yang Y. Y., Oosterhof A., & Xia Y . (2015). Reliability of scores on the summative performance assessments. The Journal of Educational Research, 108(6), 465-479.
65 Zhou J., Wang X. M., Song L. J., & Wu J . (2017). Is it new? Personal and contextual influences on perceptions of novelty and creativity. Journal of Applied Psychology, 102(2), 180-202.
pmid: 27893257 url:
66 Zhu, Y. X, Ritter, S. M. MüllerB. C. N., & Dijksterhuis A . (2017). Creativity: Intuitive processing outperforms deliberative processing in creative idea selection. Journal of Experimental Social Psychology, 73, 180-188.
[1] Linjieqiong HUANG,Huiying LIU,Lei AN,Yanan LIU,Shu ZHANG,Caiyu JIN. Multicultural experience fosters creativity[J]. Advances in Psychological Science, 2018, 26(8): 1511-1520.
[2] Jianwei ZHANG,Haihong LI,Yuxin LIU,Hui ZHAO. Effects and mechanisms of paternalistic leadership on multilevel creativity[J]. Advances in Psychological Science, 2018, 26(7): 1319-1330.
[3] ZHANG Yakun, CHEN Lung An, ZHANG Xingli, SHI Jiannong.  An integrated perspective on western creativity theories[J]. Advances in Psychological Science, 2018, 26(5): 810-830.
[4] HUANG Yun; CHENG Liang; QU Yili; ZHAO Qingbai; ZHOU Zhijin. Animal innovation: The simplest flash of intelligence[J]. Advances in Psychological Science, 2017, 25(5): 799-809.
[5] GONG Zhe; LIU Chang; SHEN Wangbing; WANG Xian; SHI Rong. The impact of trust on creativity: Activation, inhibition and inverted U model hypothesis[J]. Advances in Psychological Science, 2017, 25(3): 463-474.
[6] YANG Wen-Jing; JIN Yu-Le; CHEN Qun-Lin; SUN Jiang-Zhou; TONG Dan-Dan; . The creative components and neural mechanism of prototype heuristics in scientific innovation[J]. Advances in Psychological Science, 2016, 24(8): 1139-1146.
[7] GONG Zhe; LIU Chang; SHEN Wangbing. Several thoughts on measuring creativity[J]. Advances in Psychological Science, 2016, 24(1): 31-45.
[8] ZHANG Jinghuan; ZHANG Muzi; ZHANG Shun; REN Feifei. The Genetic Basis of Creativity in Dopamine and 5-hydroxytryptamine Pathway and the Moderating Effect of Family Environment[J]. Advances in Psychological Science, 2015, 23(9): 1489-1498.
[9] SHEN Wangbing; LIU Chang; SHI Chunhua; YUAN Yuan. Gender Differences in Creative Thinking[J]. Advances in Psychological Science, 2015, 23(8): 1380-1389.
[10] ZHAO Qingbai; LI Songqing; CHEN Shi; ZHOU Zhijin; CHENG Liang. Dynamic Neural Processing Mode of Creative Problem Solving[J]. Advances in Psychological Science, 2015, 23(3): 375-384.
[11] LI Yang; BAI Xinwen. Creating for Others: An Experimental Study of the Effects of Intrinsic Motivation and Prosocial Motivation on Creativity[J]. Advances in Psychological Science, 2015, 23(2): 175-181.
[12] HU Weiping; WANG Botao; DUAN Haijun; CHENG Lifang; ZHOU Huan; LI Jingjing. Neural Mechanism of Creative Cognitive Process Influenced by Emotion[J]. Advances in Psychological Science, 2015, 23(11): 1869-1878.
[13] WANG Lei. The Influence of Differential Leadership on Employee and Team Creativity in Chinese Family Businesses: A Multilevel and Longitudinal Study[J]. Advances in Psychological Science, 2015, 23(10): 1688-1700.
[14] TIAN Lin;LI Xinwang;YANG Fan;ZHAO Yudan. The Effect of Novelty Seeking Behavioral Trait on Susceptibility to Drug Addiction and Its Neurobiological Mechanisms[J]. Advances in Psychological Science, 2014, 22(1): 75-85.
[15] YI Xinfa;HU Weiping. Science Creativity and Art Creativity: Priming Effect and Domain Influences YI Xinfa; HU Weiping[J]. Advances in Psychological Science, 2013, 21(1): 22-30.
Full text



Copyright © Advances in Psychological Science
Support by Beijing Magtech