Rater effects in creativity assessment

doi:10.3724/SP.J.1042.2019.00171

Abstract

Abstract:

Rater effects refer to the impact of different raters’ idiosyncrasies in their behaviors on the evaluation results in creativity assessment. Rater effects are due to the difference in raters’ cognitive process of the evaluation, which are externally reflected in the difference of their scorings. This article first summarizes the studies of rater cognition and other influencing factors on creativity assessment, including characteristics of raters, information of creators and socio-cultural factors. It further examines inter-rater reliability indexes and their limitations, as well as the applications of Generalization Theory and Many-Facet Rasch Model in quantifying and controlling of rater effects. Finally, this paper specifies directions of future research based on the existing limitations, including deepening the investigation on rater cognition in creativity assessment, integrating the studies of rater effects on different levels, and developing new methods and techniques of creativity assessment.

Key words: creativity, subjective scoring, rater effects, rater cognition, inter-rater agreement

CLC Number:

B841
G449

HAN Jiantao, LIU Wenling, PANG Weiguo. Rater effects in creativity assessment[J]. Advances in Psychological Science, 2019, 27(1): 171-180.

References 66

1	贡喆, 刘昌, 沈汪兵 . (2016). 有关创造力测量的一些思考. 心理科学进展,24(1), 31-45. doi: 10.3724/SP.J.1042.2016.00031 URL
2	晏子 . (2010). 心理科学领域内的客观测量——Rasch模型之特点及发展趋势. 心理科学进展, 18(8), 1298-1305.
3	Amabile T.M . (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997-1013. doi: 10.1037/0022-3514.43.5.997 URL
4	Amabile T.M . (1983). The social psychology of creativity. New York, NY:. Springer-Verlag.
5	American Educational Research Association ( AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing (2014 Edition). Washington, DC:. AERA.
6	Baer J., Kaufman J. C., & Riggs M . (2009). Brief report: Rater-domain interactions in the consensual assessment technique. The International Journal of Creativity & Problem Solving, 19(2), 87-92.
7	Beghetto R.A., &Kaufman J.C . (2007). Toward a broader conception of creativity: A case for "mini-c" creativity. Psychology of Aesthetics, Creativity, and the Arts, 1(2), 73-79. doi: 10.1037/1931-3896.1.2.73 URL
8	Beketayev K., &Runco M.A . (2016). Scoring divergent thinking tests by computer with a semantics-based algorithm. Europe’s Journal of Psychology, 12(2), 210-220. doi: 10.5964/ejop.v12i2.1127 URL pmid: 27298632
9	Benedek M., Mühlmann C., Jauk E., & Neubauer A. C . (2013). Assessment of divergent thinking by means of the subjective top-scoring method: Effects of the number of top-ideas and time-on-task on reliability and validity. Psychology of Aesthetics, Creativity, and the Arts, 7(4), 341-349. doi: 10.1037/a0033644 URL pmid: 24790683
10	Benedek M., Nordtvedt N., Jauk E., Koschmieder C., Pretsch J., Krammer G., & Neubauer A. C . (2016). Assessment of creativity evaluation skills: A psychometric investigation in prospective teachers. Thinking Skills and Creativity, 21, 75-84. doi: 10.1016/j.tsc.2016.05.007 URL
11	Birney D. P., Beckmann J. F., & Seah Y. Z . (2016). More than the eye of the beholder: The interplay of person, task, and situation factors in evaluative judgements of creativity. Learning and Individual Differences, 51, 400-408. doi: 10.1016/j.lindif.2015.07.007 URL
12	Blair C.S., &Mumford M.D . (2007). Errors in idea evaluation: Preference for the unoriginal? The Journal of Creative Behavior, 41(3), 197-222. doi: 10.1002/j.2162-6057.2007.tb01288.x URL
13	Campbell D.T . (1960). Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological Review, 67(6), 380-400. doi: 10.1037/h0040373 URL pmid: 13690223
14	Cheng K. H.C . (2016). Perceived interpersonal dimensions and its effect on rating bias: How neuroticism as a trait matters in rating creative works. The Journal of Creative Behavior. February 16, 2017, Retrieved from https:// onlinelibrary.wiley.com/doi/full/10. 1002/jocb. 156. doi: 10.1002/jocb.156 URL
15	Cropley A. . (2006). In praise of convergent thinking , Creativity Research Journal, 18(3), 391-404. doi: 10.1207/s15326934crj1803_13 URL
16	Diedrich J., Benedek M., Jauk E., & Neubauer A. C . (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35-40. doi: 10.1037/a0038688 URL
17	Ellamil M., Dobson C., Beeman M., & Christoff K . (2012). Evaluative and generative modes of thought during the creative process. NeuroImage, 59(2), 1783-1794. doi: 10.1016/j.neuroimage.2011.08.008 URL pmid: 21854855
18	Fink A., Benedek M., Koschutnig K., Pirker E., Berger E., Meister S ., et al. & Elisabeth M. W. ( 2015). Training of verbal creativity modulates brain activity in regions associated with language- and memory-related demands. Human Brain Mapping, 36(10), 4104-4115. doi: 10.1002/hbm.22901 URL pmid: 4587539
19	Finke R. A., Ward T. B. , & Smith, S. M.(1992). Creative cognition: Theory, research, and applications Cambridge, MA: MIT Press Theory, research, and applications. Cambridge , MA: MIT Press.
20	Forthmann B., Holling H., Zandi N., Gerwig A., Çelik P., Storme M., & Lubart T . (2017). Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores. Thinking Skills and Creativity, 23, 129-139. doi: 10.1016/j.tsc.2016.12.005 URL
21	Galati F. . (2015). Complexity of judgment: What makes possible the convergence of expert and nonexpert ratings in assessing creativity. Creativity Research Journal, 27(1), 24-30. doi: 10.1080/10400419.2015.992667 URL
22	Gilhooly K. J., Fioratou E., Anthony S. H., & Wynn V . (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects. British Journal of Psychology, 98(4), 611-625. doi: 10.1111/j.2044-8295.2007.tb00467.x URL pmid: 17535464
23	Goncalo J.A., &Staw B.M . (2006). Individualism- collectivism and group creativity. Organizational Behavior and Human Decision Processes, 100(1), 96-109. doi: 10.1016/j.obhdp.2005.11.003 URL
24	Haller C. S., Courvoisier D. S., & Cropley D. H . (2011). Perhaps there is accounting for taste: Evaluating the creativity of products. Creativity Research Journal, 23(2), 99-109. doi: 10.1080/10400419.2011.571182 URL
25	Han J. T., Long H. Y., & Pang W. G . (2017). Putting raters in ratees' shoes: Perspective taking and assessment of creative products. Creativity Research Journal, 29(3), 270-281. doi: 10.1080/10400419.2017.1360062 URL
26	Hao N., Ku Y. X., Liu M. G., Hu Y., Bodner M., Grabner R. H., & Fink A . (2016). Reflection enhances creativity: Beneficial effects of idea evaluation on idea generation. Brain and Cognition, 103, 30-37. doi: 10.1016/j.bandc.2016.01.005 URL pmid: 26808451
27	Harbison J.I., & Haarmann H. (2014). Automated scoring of originality using semantic representations, Proceedings of the COGSCI, 36, 2327-2332.
28	Hennessey B.A . (1994). The consensual assessment technique: An examination of the relationship between ratings of product and process creativity. Creativity Research Journal, 7(2), 193-208. doi: 10.1080/10400419409534524 URL
29	Hennessey B.A., &Amabile T.M . (2010). Creativity. Annual Review of Psychology, 61, 569-598.
30	Hong S.W., &Lee J.S . (2015). Nonexpert evaluations on architectural design creativity across cultures. Creativity Research Journal, 27(4), 314-321. doi: 10.1080/10400419.2015.1087245 URL
31	Hung S. P., Chen P. H., & Chen H. C . (2012). Improving creativity performance assessment: A rater effect examination with many facet Rasch model. Creativity Research Journal, 24(4), 345-357. doi: 10.1080/10400419.2012.730331 URL
32	Kaufman J. C., Baer J., Agars M. D., & Loomis D . (2010). Creativity stereotypes and the consensual assessment technique. Creativity Research Journal, 22(2), 200-205. doi: 10.1080/10400419.2010.481529 URL
33	Kaufman J. C., Baer J., Cole J. C., & Sexton J. D . (2008). A comparison of expert and nonexpert raters using the consensual assessment technique. Creativity Research Journal, 20(2), 171-178. doi: 10.1080/10400410802059929 URL
34	Kaufman J. C., Baer J., Cropley D. H., Reiter-Palmon R., & Sinnett S . (2013). Furious activity vs. understanding: How much expertise is needed to evaluate creative work? Psychology of Aesthetics, Creativity, and the Arts, 7(4), 332-340. doi: 10.1037/a0034809 URL
35	Kaufman J. C., Beghetto R. A., & Dilley A . (2016). Understanding creativity in the schools. In Lipnevich, A. A., Preckel, F., & Roberts, R. D.(Eds.), Psychosocial skills and school systems in the 21st century doi: 10.1007/978-3-319-28606-8_6 URL
36	Kozbelt A., & Serafin J. (2009). Dynamic evaluation of high-and low-creativity drawings by artist and nonartist raters. Creativity Research Journal, 21(4), 349-360. doi: 10.1080/10400410903297634 URL
37	Lan L., &Kaufman J.C . (2012). American and Chinese similarities and differences in defining and valuing creative products. The Journal of Creative Behavior, 46(4), 285-306.
38	Lebuda I., & Karwowski M. (2013). Tell me your name and I'll tell you how creative your work is: Author's name and gender as factors influencing assessment of products' creativity in four different domains. Creativity Research Journal, 25(1), 137-142. doi: 10.1080/10400419.2013.752297 URL
39	Licuanan B. F., Dailey L. R., & Mumford M. D . (2007). Idea evaluation: Error in evaluating highly original ideas. The Journal of Creative Behavior, 41(1), 1-27. doi: 10.1002/j.2162-6057.2007.tb01279.x URL
40	Linacre J.M . (1994). Many-facet Rasch measurement (2nd Edition). Chicago, IL:. MESA.
41	Long H.Y . (2014 a). An empirical review of research methodologies and methods in creativity studies (2003- 2012). Creativity Research Journal, 26(4), 427-438. doi: 10.1080/10400419.2014.961781 URL
42	Long H.Y . (2014 b). More than appropriateness and novelty: Judges’ criteria of assessing creative products in science tasks. Thinking Skills and Creativity, 13, 183-194. doi: 10.1016/j.tsc.2014.05.002 URL
43	Long H.Y., &Pang W.G . (2015). Rater effects in creativity assessment: A mixed methods investigation. Thinking Skills and Creativity, 15, 13-25. doi: 10.1016/j.tsc.2014.10.004 URL
44	Lu C.C., &Luh D.B . (2012). A comparison of assessment methods and raters in product creativity. Creativity Research Journal, 24(4), 331-337. doi: 10.1080/10400419.2012.730327 URL
45	McGraw K.O., &Wong S.P . (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi: 10.1037/1082-989X.1.4.390 URL
46	Mueller J. S., Melwani S., & Goncalo J. A . (2012). The bias against creativity: Why people desire but reject creative ideas. Psychological Science, 23(1), 13-17. doi: 10.1177/0956797611421018 URL pmid: 22127366
47	Mueller J. S., Wakslak C. J., & Krishnan V . (2014). Construing creativity: The how and why of recognizing creative ideas. Journal of Experimental Social Psychology, 51, 81-87. doi: 10.1016/j.jesp.2013.11.007 URL
48	Mumford M. D., Lonergan D. C., & Scott G . (2002). Evaluating creative ideas: Processes, standards, and context. Inquiry: Critical Thinking Across the Disciplines, 22(1), 21-30.
49	Plucker J., Beghetto R. A., & Dow G . (2004). Why isn’t creativity more important to educational psychologists? Potential, pitfalls, and future directions in creativity research. Educational Psychologist, 39(2), 83-96. doi: 10.1207/s15326985ep3902_1 URL
50	Plucker J. A. , & Makel, M. C.(2010) . Assessment of creativity. In Kaufman, J. C. & Sternberg, R. J. (Eds.), The Cambridge handbook of creativity (pp. 48-73). New York, NY: Cambridge University Press.
51	Primi R. . (2014). Divergent productions of metaphors: Combining many-facet Rasch measurement and cognitive psychology in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 8(4), 461-474. doi: 10.1037/a0038055 URL
52	Runco M.A., &Jaeger G.J . (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92-96. doi: 10.1080/10400419.2012.650092 URL
53	Runco M.A., &Smith W.R . (1992). Interpersonal and intrapersonal evaluations of creative ideas. Personality and Individual Differences, 13(3), 295-302. doi: 10.1016/0191-8869(92)90105-X URL
54	Silvia P.J . (2008). Discernment and creativity: How well can people identify their most creative ideas? Psychology of Aesthetics, Creativity, and the Arts, 2(3), 139-146. doi: 10.1037/1931-3896.2.3.139 URL
55	Silvia P.J . (2011). Subjective scoring of divergent thinking: Examining the reliability of unusual uses, instances, and consequences tasks. Thinking Skills and Creativity, 6(1), 24-30. doi: 10.1016/j.tsc.2010.06.001 URL
56	Silvia P. J., Martin C., & Nusbaum E. C . (2009). A snapshot of creativity: Evaluating a quick and simple method for assessing divergent thinking. Thinking Skills and Creativity, 4(2), 79-85. doi: 10.1016/j.tsc.2009.06.005 URL
57	Silvia P. J., Winterstein B. P., Willse J. T., Barona C. M., Cram J. T., Hess K. I., .. Richard C. A . (2008). Assessing creativity with divergent thinking tasks: Exploring the reliability and validity of new subjective scoring methods. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 68-85. doi: 10.1037/1931-3896.2.2.68 URL
58	Sowden P. T., Pringle A., & Gabora L . (2015). The shifting sands of creative thinking: Connections to dual- process theory. Thinking & Reasoning, 21(1), 40-60. doi: 10.1080/13546783.2014.885464 URL
59	Storme M., Myszkowski N., Çelik P., & Lubart T . (2014). Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges. Learning and Individual Differences, 32(4), 19-25. doi: 10.1016/j.lindif.2014.03.002 URL
60	Tan M., Mourgues C., Hein S., MacCormick J., Barbot B., & Grigorenko E . (2015). Differences in judgments of creativity: How do academic domain, personality, and self-reported creativity influence novice judges’ evaluations of creative productions? Journal of Intelligence, 3(3), 73-90.
61	Wilson R. C., Guilford J. P., & Christensen P. R . (1953). The measurement of individual differences in originality. Psychological Bulletin, 50(5), 362-370. doi: 10.1037/h0060857 URL pmid: 13100527
62	Wolfe E.W . (2004). Identifying rater effects using latent trait models. Psychology Science, 46(1), 35-51.
63	Wolfe E.W., & McVay A. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement: Issues and Practice, 31(3), 31-37. doi: 10.1111/j.1745-3992.2012.00241.x URL
64	Yang Y. Y., Oosterhof A., & Xia Y . (2015). Reliability of scores on the summative performance assessments. The Journal of Educational Research, 108(6), 465-479. doi: 10.1080/00220671.2014.917255 URL
65	Zhou J., Wang X. M., Song L. J., & Wu J . (2017). Is it new? Personal and contextual influences on perceptions of novelty and creativity. Journal of Applied Psychology, 102(2), 180-202. doi: 10.1037/apl0000166 URL pmid: 27893257
66	Zhu, Y. X, Ritter, S. M. MüllerB. C. N., & Dijksterhuis A . (2017). Creativity: Intuitive processing outperforms deliberative processing in creative idea selection. Journal of Experimental Social Psychology, 73, 180-188. doi: 10.1016/j.jesp.2017.06.009 URL

[1]	LI Yadan, DU Ying, XIE Cong, LIU Chunyu, YANG Yilong, LI Yangping, QIU Jiang. A meta-analysis of the relationship between semantic distance and creative thinking [J]. Advances in Psychological Science, 2023, 31(4): 519-534.
[2]	LI Qingyang, YIN Junting, LUO Junlong. Legs move, thoughts flow: Physical exercise influences creative thinking [J]. Advances in Psychological Science, 2023, 31(3): 455-466.
[3]	YAO Haijuan, WANG Qi, LI Zhaoqing. Cognitive reappraisal inventiveness in emotion regulation [J]. Advances in Psychological Science, 2022, 30(3): 601-612.
[4]	LI Ziyi, ZHANG Ze, ZHANG Ying, LUO Jing. The incubation effect of creative thinking [J]. Advances in Psychological Science, 2022, 30(2): 291-307.
[5]	YIN Junting, WANG Guan, LUO Junlong. The impact of threats on creativity based on cognitive and emotional processes [J]. Advances in Psychological Science, 2021, 29(5): 815-826.
[6]	YE Chaoqun, LIN Yuhong, LIU Chunlei. Neural oscillation mechanism of creativity [J]. Advances in Psychological Science, 2021, 29(4): 697-706.
[7]	ZHANG Yakun, CHEN Ning, CHEN Lung An, SHI Jiannong. Wisdom minds with creative wings: Igniting creative dynamics focusing on its interest cultivation [J]. Advances in Psychological Science, 2021, 29(4): 707-722.
[8]	WANG Botao, WEI Ping. Moral emotion: A new perspective on the relationship between morality and creativity [J]. Advances in Psychological Science, 2021, 29(2): 268-275.
[9]	ZHANG Shun, YANG Xiaolei, Ren Jiawen, ZHANG Jinghuan. The relationship between methylation of dopamine-related genes, family environment and creativity [J]. Advances in Psychological Science, 2021, 29(11): 1911-1919.
[10]	LI Mingze, YE Huili, ZHANG Guanglei. The influence mechanism of narcissistic leadership on the formation process of team creativity: A multi-perspective study [J]. Advances in Psychological Science, 2020, 28(9): 1437-1453.
[11]	HUANG Chongrong, HU Yu. The relationship between trust and creativity in organizations: Evidence from meta-analysis [J]. Advances in Psychological Science, 2020, 28(7): 1118-1132.
[12]	YANG Jie, ZHANG Lu, HUANG Yong. Cross-level mechanism of playful climate on innovative behaviors in internet companies [J]. Advances in Psychological Science, 2020, 28(4): 523-534.
[13]	BAI Xinwen, QI Shuting, MING Xiaodong, ZHOU Yiyong, HUANG Mingquan. Pearls are everywhere but not the eyes: The mechanism and boundary conditions of the influences of decision maker's mental models on idea recognition [J]. Advances in Psychological Science, 2019, 27(4): 571-586.
[14]	HUANG Linjieqiong, LIU Huiying, AN Lei, LIU Yanan, ZHANG Shu, JIN Caiyu. Multicultural experience fosters creativity [J]. Advances in Psychological Science, 2018, 26(8): 1511-1520.
[15]	ZHANG Jianwei, LI Haihong, LIU Yuxin, ZHAO Hui. Effects and mechanisms of paternalistic leadership on multilevel creativity [J]. Advances in Psychological Science, 2018, 26(7): 1319-1330.