Please wait a minute...
Advances in Psychological Science    2019, Vol. 27 Issue (1) : 171-180     DOI: 10.3724/SP.J.1042.2019.00171
Research Method |
Rater effects in creativity assessment
HAN Jiantao1,2,3,LIU Wenling1,PANG Weiguo1()
1 School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China
2 School of Educational Science, Anhui Normal University, Wuhu 241000, China
3 School of Literature Media and Educational Science, Chaohu College, Chaohu 238000, China
Download: PDF(551 KB)   HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks     Supporting Info
Guide   
Abstract  

Rater effects refer to the impact of different raters’ idiosyncrasies in their behaviors on the evaluation results in creativity assessment. Rater effects are due to the difference in raters’ cognitive process of the evaluation, which are externally reflected in the difference of their scorings. This article first summarizes the studies of rater cognition and other influencing factors on creativity assessment, including characteristics of raters, information of creators and socio-cultural factors. It further examines inter-rater reliability indexes and their limitations, as well as the applications of Generalization Theory and Many-Facet Rasch Model in quantifying and controlling of rater effects. Finally, this paper specifies directions of future research based on the existing limitations, including deepening the investigation on rater cognition in creativity assessment, integrating the studies of rater effects on different levels, and developing new methods and techniques of creativity assessment.

Keywords creativity      subjective scoring      rater effects      rater cognition      inter-rater agreement     
ZTFLH:  B841  
  G449  
Issue Date: 23 November 2018
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
Jiantao HAN
Wenling LIU
Weiguo PANG
Cite this article:   
Jiantao HAN,Wenling LIU,Weiguo PANG. Rater effects in creativity assessment[J]. Advances in Psychological Science, 2019, 27(1): 171-180.
URL:  
http://journal.psych.ac.cn/xlkxjz/EN/10.3724/SP.J.1042.2019.00171     OR     http://journal.psych.ac.cn/xlkxjz/EN/Y2019/V27/I1/171
1 贡喆, 刘昌, 沈汪兵 . (2016). 有关创造力测量的一些思考. 心理科学进展,24(1), 31-45.
url: http://d.wanfangdata.com.cn/Periodical_xlxdt201601004.aspx
2 晏子 . (2010). 心理科学领域内的客观测量——Rasch模型之特点及发展趋势. 心理科学进展, 18(8), 1298-1305.
url: http://d.wanfangdata.com.cn/Periodical/xlxdt201008013
3 Amabile T.M . (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997-1013.
url: http://psycnet.apa.org/journals/psp/43/5/997/
4 Amabile T.M . (1983). The social psychology of creativity. New York, NY:. Springer-Verlag.
5 American Educational Research Association ( AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing (2014 Edition). Washington, DC:. AERA.
6 Baer J., Kaufman J. C., & Riggs M . (2009). Brief report: Rater-domain interactions in the consensual assessment technique. The International Journal of Creativity & Problem Solving, 19(2), 87-92.
url: http://psycnet.apa.org/record/2009-20794-006
7 Beghetto R.A., &Kaufman J.C . (2007). Toward a broader conception of creativity: A case for "mini-c" creativity. Psychology of Aesthetics, Creativity, and the Arts, 1(2), 73-79.
url: http://psycnet.apa.org/journals/aca/1/2/73/
8 Beketayev K., &Runco M.A . (2016). Scoring divergent thinking tests by computer with a semantics-based algorithm. Europe’s Journal of Psychology, 12(2), 210-220.
pmid: 27298632 url: http://www.ncbi.nlm.nih.gov/pubmed/27298632
9 Benedek M., Mühlmann C., Jauk E., & Neubauer A. C . (2013). Assessment of divergent thinking by means of the subjective top-scoring method: Effects of the number of top-ideas and time-on-task on reliability and validity. Psychology of Aesthetics, Creativity, and the Arts, 7(4), 341-349.
pmid: 24790683 url: http://pubmedcentralcanada.ca/pmcc/articles/PMC4001084/
10 Benedek M., Nordtvedt N., Jauk E., Koschmieder C., Pretsch J., Krammer G., & Neubauer A. C . (2016). Assessment of creativity evaluation skills: A psychometric investigation in prospective teachers. Thinking Skills and Creativity, 21, 75-84.
url: http://www.sciencedirect.com/science/article/pii/S187118711630027X
11 Birney D. P., Beckmann J. F., & Seah Y. Z . (2016). More than the eye of the beholder: The interplay of person, task, and situation factors in evaluative judgements of creativity. Learning and Individual Differences, 51, 400-408.
url: http://www.sciencedirect.com/science/article/pii/S1041608015001430
12 Blair C.S., &Mumford M.D . (2007). Errors in idea evaluation: Preference for the unoriginal? The Journal of Creative Behavior, 41(3), 197-222.
url: http://onlinelibrary.wiley.com/doi/10.1002/j.2162-6057.2007.tb01288.x/full
13 Campbell D.T . (1960). Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological Review, 67(6), 380-400.
pmid: 13690223 url: http://psycnet.apa.org/journals/rev/67/6/380/
14 Cheng K. H.C . (2016). Perceived interpersonal dimensions and its effect on rating bias: How neuroticism as a trait matters in rating creative works. The Journal of Creative Behavior. February 16, 2017, Retrieved from https:// onlinelibrary.wiley.com/doi/full/10. 1002/jocb. 156.
url: http://onlinelibrary.wiley.com/doi/10.1002/jocb.156/full
15 Cropley A. . (2006). In praise of convergent thinking , Creativity Research Journal, 18(3), 391-404.
url: http://www.tandfonline.com/doi/abs/10.1207/s15326934crj1803_13
16 Diedrich J., Benedek M., Jauk E., & Neubauer A. C . (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35-40.
url: http://psycnet.apa.org/journals/aca/9/1/35/
17 Ellamil M., Dobson C., Beeman M., & Christoff K . (2012). Evaluative and generative modes of thought during the creative process. NeuroImage, 59(2), 1783-1794.
pmid: 21854855 url: http://www.sciencedirect.com/science/article/pii/S1053811911008950
18 Fink A., Benedek M., Koschutnig K., Pirker E., Berger E., Meister S ., et al. & Elisabeth M. W. ( 2015). Training of verbal creativity modulates brain activity in regions associated with language- and memory-related demands. Human Brain Mapping, 36(10), 4104-4115.
pmid: 4587539 url: http://onlinelibrary.wiley.com/doi/10.1002/hbm.22901/pdf
19 Finke R. A., Ward T. B. , & Smith, S. M.(1992). Creative cognition: Theory, research, and applications Cambridge, MA: MIT Press Theory, research, and applications. Cambridge , MA: MIT Press.
20 Forthmann B., Holling H., Zandi N., Gerwig A., ?elik P., Storme M., & Lubart T . (2017). Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores. Thinking Skills and Creativity, 23, 129-139.
url: http://www.sciencedirect.com/science/article/pii/S1871187116302140
21 Galati F. . (2015). Complexity of judgment: What makes possible the convergence of expert and nonexpert ratings in assessing creativity. Creativity Research Journal, 27(1), 24-30.
url: http://www.tandfonline.com/doi/ref/10.1080/10400419.2015.992667
22 Gilhooly K. J., Fioratou E., Anthony S. H., & Wynn V . (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects. British Journal of Psychology, 98(4), 611-625.
pmid: 17535464 url: http://onlinelibrary.wiley.com/doi/10.1111/j.2044-8295.2007.tb00467.x/full
23 Goncalo J.A., &Staw B.M . (2006). Individualism- collectivism and group creativity. Organizational Behavior and Human Decision Processes, 100(1), 96-109.
url: http://www.sciencedirect.com/science/article/pii/S0749597805001378
24 Haller C. S., Courvoisier D. S., & Cropley D. H . (2011). Perhaps there is accounting for taste: Evaluating the creativity of products. Creativity Research Journal, 23(2), 99-109.
url: http://www.tandfonline.com/doi/full/10.1080/10400419.2011.571182
25 Han J. T., Long H. Y., & Pang W. G . (2017). Putting raters in ratees' shoes: Perspective taking and assessment of creative products. Creativity Research Journal, 29(3), 270-281.
url: http://www.tandfonline.com/doi/ref/10.1080/10400419.2017.1360062
26 Hao N., Ku Y. X., Liu M. G., Hu Y., Bodner M., Grabner R. H., & Fink A . (2016). Reflection enhances creativity: Beneficial effects of idea evaluation on idea generation. Brain and Cognition, 103, 30-37.
pmid: 26808451 url: http://www.sciencedirect.com/science/article/pii/S0278262616300057
27 Harbison J.I., & Haarmann H. (2014). Automated scoring of originality using semantic representations, Proceedings of the COGSCI, 36, 2327-2332.
28 Hennessey B.A . (1994). The consensual assessment technique: An examination of the relationship between ratings of product and process creativity. Creativity Research Journal, 7(2), 193-208.
url: http://www.tandfonline.com/doi/abs/10.1080/10400419409534524
29 Hennessey B.A., &Amabile T.M . (2010). Creativity. Annual Review of Psychology, 61, 569-598.
30 Hong S.W., &Lee J.S . (2015). Nonexpert evaluations on architectural design creativity across cultures. Creativity Research Journal, 27(4), 314-321.
url: http://www.tandfonline.com/doi/full/10.1080/10400419.2015.1087245
31 Hung S. P., Chen P. H., & Chen H. C . (2012). Improving creativity performance assessment: A rater effect examination with many facet Rasch model. Creativity Research Journal, 24(4), 345-357.
url: http://psycnet.apa.org/record/2012-31208-011
32 Kaufman J. C., Baer J., Agars M. D., & Loomis D . (2010). Creativity stereotypes and the consensual assessment technique. Creativity Research Journal, 22(2), 200-205.
url: http://psycnet.apa.org/record/2010-10495-009
33 Kaufman J. C., Baer J., Cole J. C., & Sexton J. D . (2008). A comparison of expert and nonexpert raters using the consensual assessment technique. Creativity Research Journal, 20(2), 171-178.
url: http://psycnet.apa.org/psycinfo/2009-09614-009
34 Kaufman J. C., Baer J., Cropley D. H., Reiter-Palmon R., & Sinnett S . (2013). Furious activity vs. understanding: How much expertise is needed to evaluate creative work? Psychology of Aesthetics, Creativity, and the Arts, 7(4), 332-340.
url: http://psycnet.apa.org/journals/aca/7/4/332/
35 Kaufman J. C., Beghetto R. A., & Dilley A . (2016). Understanding creativity in the schools. In Lipnevich, A. A., Preckel, F., & Roberts, R. D.(Eds.), Psychosocial skills and school systems in the 21st century
url: http://link.springer.com/10.1007/978-3-319-28606-8_6
36 Kozbelt A., & Serafin J. (2009). Dynamic evaluation of high-and low-creativity drawings by artist and nonartist raters. Creativity Research Journal, 21(4), 349-360.
url: http://psycnet.apa.org/record/2010-11277-005
37 Lan L., &Kaufman J.C . (2012). American and Chinese similarities and differences in defining and valuing creative products. The Journal of Creative Behavior, 46(4), 285-306.
38 Lebuda I., & Karwowski M. (2013). Tell me your name and I'll tell you how creative your work is: Author's name and gender as factors influencing assessment of products' creativity in four different domains. Creativity Research Journal, 25(1), 137-142.
url: http://www.tandfonline.com/doi/full/10.1080/10400419.2013.752297
39 Licuanan B. F., Dailey L. R., & Mumford M. D . (2007). Idea evaluation: Error in evaluating highly original ideas. The Journal of Creative Behavior, 41(1), 1-27.
url: http://onlinelibrary.wiley.com/doi/10.1002/j.2162-6057.2007.tb01279.x/full
40 Linacre J.M . (1994). Many-facet Rasch measurement (2nd Edition). Chicago, IL:. MESA.
41 Long H.Y . (2014 a). An empirical review of research methodologies and methods in creativity studies (2003- 2012). Creativity Research Journal, 26(4), 427-438.
url: http://www.tandfonline.com/doi/full/10.1080/10400419.2014.961781
42 Long H.Y . (2014 b). More than appropriateness and novelty: Judges’ criteria of assessing creative products in science tasks. Thinking Skills and Creativity, 13, 183-194.
url: http://www.sciencedirect.com/science/article/pii/S1871187114000303
43 Long H.Y., &Pang W.G . (2015). Rater effects in creativity assessment: A mixed methods investigation. Thinking Skills and Creativity, 15, 13-25.
url: http://www.sciencedirect.com/science/article/pii/S1871187114000595
44 Lu C.C., &Luh D.B . (2012). A comparison of assessment methods and raters in product creativity. Creativity Research Journal, 24(4), 331-337.
url: http://psycnet.apa.org/record/2012-31208-009
45 McGraw K.O., &Wong S.P . (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46.
url: http://psycnet.apa.org/journals/met/1/1/30/
46 Mueller J. S., Melwani S., & Goncalo J. A . (2012). The bias against creativity: Why people desire but reject creative ideas. Psychological Science, 23(1), 13-17.
pmid: 22127366 url: http://www.ncbi.nlm.nih.gov/pubmed/22127366/
47 Mueller J. S., Wakslak C. J., & Krishnan V . (2014). Construing creativity: The how and why of recognizing creative ideas. Journal of Experimental Social Psychology, 51, 81-87.
url: http://www.sciencedirect.com/science/article/pii/S0022103113001935
48 Mumford M. D., Lonergan D. C., & Scott G . (2002). Evaluating creative ideas: Processes, standards, and context. Inquiry: Critical Thinking Across the Disciplines, 22(1), 21-30.
url: http://psycnet.apa.org/record/2004-10090-003
49 Plucker J., Beghetto R. A., & Dow G . (2004). Why isn’t creativity more important to educational psychologists? Potential, pitfalls, and future directions in creativity research. Educational Psychologist, 39(2), 83-96.
url: http://psycnet.apa.org/record/2004-95010-001
50 Plucker J. A. , & Makel, M. C.(2010) . Assessment of creativity. In Kaufman, J. C. & Sternberg, R. J. (Eds.), The Cambridge handbook of creativity (pp. 48-73). New York, NY: Cambridge University Press.
51 Primi R. . (2014). Divergent productions of metaphors: Combining many-facet Rasch measurement and cognitive psychology in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 8(4), 461-474.
url: http://psycnet.apa.org/journals/aca/8/4/461
52 Runco M.A., &Jaeger G.J . (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92-96.
url: http://psycnet.apa.org/psycinfo/2012-04002-011
53 Runco M.A., &Smith W.R . (1992). Interpersonal and intrapersonal evaluations of creative ideas. Personality and Individual Differences, 13(3), 295-302.
url: http://www.sciencedirect.com/science/article/pii/019188699290105X
54 Silvia P.J . (2008). Discernment and creativity: How well can people identify their most creative ideas? Psychology of Aesthetics, Creativity, and the Arts, 2(3), 139-146.
url: http://psycnet.apa.org/journals/aca/2/3/139/
55 Silvia P.J . (2011). Subjective scoring of divergent thinking: Examining the reliability of unusual uses, instances, and consequences tasks. Thinking Skills and Creativity, 6(1), 24-30.
url: http://www.sciencedirect.com/science/article/pii/S1871187110000295
56 Silvia P. J., Martin C., & Nusbaum E. C . (2009). A snapshot of creativity: Evaluating a quick and simple method for assessing divergent thinking. Thinking Skills and Creativity, 4(2), 79-85.
url: http://www.sciencedirect.com/science/article/pii/S1871187109000285
57 Silvia P. J., Winterstein B. P., Willse J. T., Barona C. M., Cram J. T., Hess K. I., .. Richard C. A . (2008). Assessing creativity with divergent thinking tasks: Exploring the reliability and validity of new subjective scoring methods. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 68-85.
url: http://psycnet.apa.org/journals/aca/2/2/68
58 Sowden P. T., Pringle A., & Gabora L . (2015). The shifting sands of creative thinking: Connections to dual- process theory. Thinking & Reasoning, 21(1), 40-60.
url: http://www.tandfonline.com/doi/abs/10.1080/13546783.2014.885464
59 Storme M., Myszkowski N., ?elik P., & Lubart T . (2014). Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges. Learning and Individual Differences, 32(4), 19-25.
url: http://www.sciencedirect.com/science/article/pii/S104160801400048X
60 Tan M., Mourgues C., Hein S., MacCormick J., Barbot B., & Grigorenko E . (2015). Differences in judgments of creativity: How do academic domain, personality, and self-reported creativity influence novice judges’ evaluations of creative productions? Journal of Intelligence, 3(3), 73-90.
61 Wilson R. C., Guilford J. P., & Christensen P. R . (1953). The measurement of individual differences in originality. Psychological Bulletin, 50(5), 362-370.
pmid: 13100527 url: http://www.ncbi.nlm.nih.gov/pubmed/13100527
62 Wolfe E.W . (2004). Identifying rater effects using latent trait models. Psychology Science, 46(1), 35-51.
url: http://psycnet.apa.org/psycinfo/2004-19990-003
63 Wolfe E.W., & McVay A. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement: Issues and Practice, 31(3), 31-37.
url: http://onlinelibrary.wiley.com/doi/10.1111/j.1745-3992.2012.00241.x/full
64 Yang Y. Y., Oosterhof A., & Xia Y . (2015). Reliability of scores on the summative performance assessments. The Journal of Educational Research, 108(6), 465-479.
url: http://www.tandfonline.com/doi/full/10.1080/00220671.2014.917255
65 Zhou J., Wang X. M., Song L. J., & Wu J . (2017). Is it new? Personal and contextual influences on perceptions of novelty and creativity. Journal of Applied Psychology, 102(2), 180-202.
pmid: 27893257 url: http://www.ncbi.nlm.nih.gov/pubmed/27893257
66 Zhu, Y. X, Ritter, S. M. MüllerB. C. N., & Dijksterhuis A . (2017). Creativity: Intuitive processing outperforms deliberative processing in creative idea selection. Journal of Experimental Social Psychology, 73, 180-188.
url: http://www.sciencedirect.com/science/article/pii/S002210311730118X
[1] Linjieqiong HUANG,Huiying LIU,Lei AN,Yanan LIU,Shu ZHANG,Caiyu JIN. Multicultural experience fosters creativity[J]. Advances in Psychological Science, 2018, 26(8): 1511-1520.
[2] Jianwei ZHANG,Haihong LI,Yuxin LIU,Hui ZHAO. Effects and mechanisms of paternalistic leadership on multilevel creativity[J]. Advances in Psychological Science, 2018, 26(7): 1319-1330.
[3] ZHANG Yakun, CHEN Lung An, ZHANG Xingli, SHI Jiannong.  An integrated perspective on western creativity theories[J]. Advances in Psychological Science, 2018, 26(5): 810-830.
[4] HUANG Yun; CHENG Liang; QU Yili; ZHAO Qingbai; ZHOU Zhijin. Animal innovation: The simplest flash of intelligence[J]. Advances in Psychological Science, 2017, 25(5): 799-809.
[5] GONG Zhe; LIU Chang; SHEN Wangbing; WANG Xian; SHI Rong. The impact of trust on creativity: Activation, inhibition and inverted U model hypothesis[J]. Advances in Psychological Science, 2017, 25(3): 463-474.
[6] YANG Wen-Jing; JIN Yu-Le; CHEN Qun-Lin; SUN Jiang-Zhou; TONG Dan-Dan; . The creative components and neural mechanism of prototype heuristics in scientific innovation[J]. Advances in Psychological Science, 2016, 24(8): 1139-1146.
[7] GONG Zhe; LIU Chang; SHEN Wangbing. Several thoughts on measuring creativity[J]. Advances in Psychological Science, 2016, 24(1): 31-45.
[8] ZHANG Jinghuan; ZHANG Muzi; ZHANG Shun; REN Feifei. The Genetic Basis of Creativity in Dopamine and 5-hydroxytryptamine Pathway and the Moderating Effect of Family Environment[J]. Advances in Psychological Science, 2015, 23(9): 1489-1498.
[9] SHEN Wangbing; LIU Chang; SHI Chunhua; YUAN Yuan. Gender Differences in Creative Thinking[J]. Advances in Psychological Science, 2015, 23(8): 1380-1389.
[10] ZHAO Qingbai; LI Songqing; CHEN Shi; ZHOU Zhijin; CHENG Liang. Dynamic Neural Processing Mode of Creative Problem Solving[J]. Advances in Psychological Science, 2015, 23(3): 375-384.
[11] LI Yang; BAI Xinwen. Creating for Others: An Experimental Study of the Effects of Intrinsic Motivation and Prosocial Motivation on Creativity[J]. Advances in Psychological Science, 2015, 23(2): 175-181.
[12] HU Weiping; WANG Botao; DUAN Haijun; CHENG Lifang; ZHOU Huan; LI Jingjing. Neural Mechanism of Creative Cognitive Process Influenced by Emotion[J]. Advances in Psychological Science, 2015, 23(11): 1869-1878.
[13] WANG Lei. The Influence of Differential Leadership on Employee and Team Creativity in Chinese Family Businesses: A Multilevel and Longitudinal Study[J]. Advances in Psychological Science, 2015, 23(10): 1688-1700.
[14] TIAN Lin;LI Xinwang;YANG Fan;ZHAO Yudan. The Effect of Novelty Seeking Behavioral Trait on Susceptibility to Drug Addiction and Its Neurobiological Mechanisms[J]. Advances in Psychological Science, 2014, 22(1): 75-85.
[15] YI Xinfa;HU Weiping. Science Creativity and Art Creativity: Priming Effect and Domain Influences YI Xinfa; HU Weiping[J]. Advances in Psychological Science, 2013, 21(1): 22-30.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Advances in Psychological Science
Support by Beijing Magtech