Preventing and detecting insufficient effort survey responding

doi:10.3724/SP.J.1042.2021.00225

Abstract

Abstract:

Surveys are commonly used in psychological and educational research. Insufficient effort response (IER), as one source of invalid response data, is somewhat prevalent due to the low-stakes nature of the majority of surveys, which often leads to statistically significantly biased estimates and invalid inferences. The current literature shows: (a) IER is commonly believed to be caused by some inner causes, (e.g., low motivation), showing as specific patterns, (e.g., random responding); (b) The most common methods to prevent IER include reducing task difficulty and increasing respondents’ motivation; (c) Current detection methods fall into three main categories, which are proactive approaches/ direct screening methods, response patterns analysis, and response time analysis. Recommendations for future research directions and practitioners are (a) deepening the investigation on IER mechanism and improving the preventing methods, (b) examining the effectiveness of IER identification methods’ applicability of cross-situation and developing new approaches, and (c) delving into the identification and treatment of partial IER.

Key words: insufficient effort responding (IER), data screening, invalid response, survey and questionnaire design & construction

CLC Number:

B841

ZHONG Xiaoyu, LI Mingyao, LI Lingyan. Preventing and detecting insufficient effort survey responding[J]. Advances in Psychological Science, 2021, 29(2): 225-237.

Figures/Tables 4

References 83

[1]	车文博. (2001). 心理咨询大百科全书. 杭州: 浙江科学技术出版社.
[2]	刘蔚华, 陈远. (1991). 方法大辞典. 济南: 山东人民出版社.
[3]	牟智佳. (2017). MOOCs学习参与度影响因素的结构关系与效应研究——自我决定理论的视角. 电化教育研究, 38(10), 37-43.
[4]	王俪嘉, 朱德全. (2009). 中小学教师对待公开课态度的调查研究. 上海教育科研, (8), 28-31.
[5]	卫旭华, 张亮花. (2019). 单题项测量: 质疑、回应及建议. 心理科学进展, 27(7), 1194-1204.
[6]	姚成, 龚毅, 濮光宁, 葛文龙. (2012). 学生评教异常数据的筛选与处理. 牡丹江师范学院学报(自然科学版) (3), 7-8.
[7]	郑云翔, 杨浩, 冯诗晓. (2018). 高校教师信息化教学适应性绩效评价研究. 中国电化教育, (2), 21-28.
[8]	Anduiza, E., & Galais, C. (2017). Answering without reading: IMCs and strong satisficing in online surveys. International Journal of Public Opinion Research, 29(3), 497-519.
[9]	Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68(1), 139-151. URL pmid: 16370774
[10]	Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of survey data. Research in Higher Education, 53(2), 182-200. doi: 10.1007/s11162-011-9251-2 URL
[11]	Beach, D. A. (1989). Identifying the random responder. The Journal of Psychology, 123(1), 101-103.
[12]	Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340.
[13]	Börger, T. (2016). Are fast responses more random? Testing the effect of response time on scale in an online choice experiment. Environmental and Resource Economics, 65(2), 389-413.
[14]	Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218. URL pmid: 26927958
[15]	Burns, G. N., Christiansen, N. D., Morris, M. B., Periard, D. A., & Coaster, J. A. (2014). Effects of applicant personality on resume evaluations. Journal of Business and Psychology, 29(4), 573-591.
[16]	Carrier, L. M., Cheever, N. A., Rosen, L. D., Benitez, S., & Chang, J. (2009). Multitasking across generations: Multitasking choices and difficulty ratings in three generations of Americans. Computers in Human Behavior, 25(2), 483-489.
[17]	Cialdini, R. B. (2001). Harnessing the science of persuasion. Harvard Business Review, 79(9), 72-81.
[18]	Cibelli, K. L. (2017). The effects of respondent commitment and feedback on response quality in online surveys. (Unpublished doctorial dissertation), University of Michigan, Ann Arbor.
[19]	Costa Jr, P. T., & McCrae, R. R. (2008). The Revised NEO Personality Inventory (NEO-PI-R). In G. J. Boyle, G. Matthews, & D. H. Saklofske (Eds.), The SAGE Handbook of Personality Theory and Assessment: Personality Measurement and Testing (pp. 179-198). London: SAGE Publications Ltd.
[20]	Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596-612.
[21]	Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19.
[22]	DeSimone, J. A., DeSimone, A. J., Harms, P. D., & Wood, D. (2018). The differential impacts of two forms of insufficient effort responding. Applied Psychology, 67(2), 309-338.
[23]	DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171-181.
[24]	Drasgow, F., Levine, M. V., & Williams, E. A. (1985) Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology 38, 67-86.
[25]	Dunn, A. M., Heggestad, E. D., Shanock, L. R., & Theilgard, N. (2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33(1), 105-121.
[26]	Emons, W. H. M. (2008). Nonparametric person-fit analysis of polytomous item scores. Applied Psychological Measurement, 32(3), 224-247.
[27]	Evans, J. R., & Mathur, A. (2005). The value of online surveys. Internet Research, 15(2), 195-219.
[28]	Fang, J. M., Prybutok, V., & Wen, C. (2016). Shirking behavior and socially desirable responding in online surveys: A cross-cultural study comparing Chinese and American samples. Computers in Human Behavior, 54, 310-317. doi: 10.1016/j.chb.2015.08.019 URL
[29]	Fang, J. M., Wen, C., & Prybutok, V. (2014). An assessment of equivalence between paper and social media surveys: The role of social desirability and satisficing. Computers in Human Behavior, 30, 335-343. doi: 10.1016/j.chb.2013.09.019 URL
[30]	Francavilla, N. M., Meade, A. W., & Young, A. L. (2019). Social interaction and internet-based surveys: Examining the effects of virtual and in-person proctors on careless response. Applied Psychology, 68(2), 223-249. doi: 10.1111/apps.v68.2 URL
[31]	García, A. A. (2011). Cognitive interviews to test and refine questionnaires. Public Health Nursing, 28(5), 444-450. doi: 10.1111/j.1525-1446.2010.00938.x URL
[32]	Gough, H. G., & Bradley, P. (1996). The California psychological^™ inventory manual: Third edition. Palo Alto, CA: Consulting Psychologists Press.
[33]	Grau, I., Ebbeler, C., & Banse, R. (2019). Cultural differences in careless responding. Journal of Cross-Cultural Psychology, 50(3), 336-357.
[34]	Guttman, L. (1944). A basis for scaling qualitative data. American Sociological Review, 9(2), 139-150.
[35]	Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stouffer, L. Guttman, E. A. Suchman, P. F. Lazarsfeld, S. A. Star, & J. A. Clausen (Eds.), Measurement and prediction (pp. 60-90). Princeton, NJ: Princeton University Press.
[36]	He, J., & van de Vijver, F. J., R. (2013). A general response style factor: Evidence from a multi-ethnic study in the Netherlands. Personality and Individual Differences, 55(7), 794-800.
[37]	He, J., & van de Vijver, F. J., R. (2015a). Effects of a general response style on cross-cultural comparisons: Evidence from the teaching and learning international survey. Public Opinion Quarterly, 79(S1), 267-290.
[38]	He, J., & van de Vijver, F. J., R. (2015b). Self-presentation styles in self-reports: Linking the general factors of response styles, personality traits, and values in a longitudinal study. Personality and Individual Differences, 81, 129-134.
[39]	He, J., & van de Vijver, F. J., R. (2016). Response styles in factual items: Personal, contextual and cultural correlates. International Journal of Psychology, 51(6), 445-452. URL pmid: 26899311
[40]	Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (2000). Understanding robust and exploratory data analysis. New York, NY: John Wiley.
[41]	Höhne, J. K., & Schlosser, S. (2018). Investigating the adequacy of response time outlier definitions in computer-based web surveys using paradata SurveyFocus. Social Science Computer Review, 36(3), 369-378.
[42]	Holtzman, N. S., & Donnellan, M. B. (2017). A simulator of the degree to which random responding leads to biases in the correlations between two individual differences. Personality and Individual Differences, 114, 187-192.
[43]	Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99-114.
[44]	Huang, J. L., Liu, M. Q., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828-845.
[45]	Jackson, D. N. (1976). The appraisal of personal reliability. Paper presented at the meetings of the Society of Multivariate Experimental Psychology, University Park, PA.
[46]	Jackson, D. N. (1977). Jackson vocational interest survey: manual. Port Huron, MI: Research Psychologists Press.
[47]	Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103-129.
[48]	Kam, C. C. S., & Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality. Organizational Research Methods, 18(3), 512-541.
[49]	Karabatsos, G. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16(4), 277-298.
[50]	Kountur, R. (2016). Detecting careless responses to self-reported questionnaires. Eurasian Journal of Educational Research,(64), 307-318.
[51]	Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213-236.
[52]	Lenzner, T., Kaczmirek, L., & Lenzner, A. (2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24(7), 1003-1020.
[53]	Levine, M. V., & Rubin, D. B. (1979) Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics 4, 269-290.
[54]	Lloyd, K., & Devine, P. (2010). Using the internet to give children a voice: An online survey of 10- and 11-year-old children in Northern Ireland. Field Methods, 22(3), 270-289.
[55]	Mahalanobis, P. C. (1936). On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, 2, 49-55.
[56]	Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83.
[57]	Marjanovic, Z., Holden, R., Struthers, W., Cribbie, R., & Greenglass, E. (2015). The inter-item standard deviation (ISD): An index that discriminates between conscientious and random responders. Personality and Individual Differences, 84, 79-83.
[58]	Mayerl, J. (2013). Response latency measurement in surveys: Detecting strong attitudes and response effects. Survey Methods: Insights from the Field. Retrieved from https://surveyinsights.org/?p=1063
[59]	McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136(3), 450-470. doi: 10.1037/a0019216 URL pmid: 20438146
[60]	Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437. URL pmid: 22506584
[61]	Meijer, R. R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25(2), 107-135.
[62]	Melipillán, E. R. (2019). Careless survey respondents: Approaches to identify and reduce their negative impact on survey estimates. (Unpublished doctorial dissertation), University of Michigan, Ann Arbor.
[63]	Nguyen, H. L. T. (2017). Tired of survey fatigue? Insufficient effort responding due to survey fatigue (Unpublished master’s thesis), Middle Tennessee State University, Murfreesboro.
[64]	Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use? Journal of Research in Personality, 63, 1-11.
[65]	Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.
[66]	Revilla, M., & Ochoa, C. (2015). What are the links in a web survey among response time, quality, and auto-evaluation of the efforts done? Social Science Computer Review, 33(1), 97-114.
[67]	Rousseau, B., & Ennis, J. M. (2013). Importance of correct instructions in the tetrad test. Journal of Sensory Studies, 28(4), 264-269.
[68]	Schneider, S., May, M., & Stone, A. A. (2018). Careless responding in internet-based quality of life assessments. Quality of Life Research, 27(4), 1077-1088. URL pmid: 29248996
[69]	Schnell, R. (1994). Graphisch gestützte datenanalyse [Graphically supported data analysis]. München, Germany: Oldenbourg.
[70]	Soland, J., Wise, S. L., & Gao, L. Y. (2019). Identifying disengaged survey responses: New evidence using response time metadata. Applied Measurement in Education, 32(2), 151-165.
[71]	van der Flier, H. (1980). Vergelijkbaarheid van individuele testprestaties [Comparability of individual test performance]. Lisse, Netherlands: Swets & Zeitlinger.
[72]	Velleman, P. F., & Welsch, R. E. (1981). Efficient computing of regression diagnostics. The American Statistician, 35(4), 234-242.
[73]	Wang, C., & Xu, G. J. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456-477.
[74]	Ward, M. K., & Meade, A. W. (2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology, 67(2), 231-263.
[75]	Ward, M. K., & Pond, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554-568.
[76]	Wise, S. L. (2017). Rapid-guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52-61.
[77]	Wise, S. L., & Demars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19-38.
[78]	Wise, S. L., & Kong, X. J. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163-183.
[79]	Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28(3), 186-191.
[80]	Yan, T., & Tourangeau, R. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22(1), 51-68.
[81]	Zhang, C. (2013). Satisficing in web surveys: Implications for data quality and strategies for reduction. (Unpublished doctorial dissertation). University of Michigan, Ann Arbor.
[82]	Zhang, C., & Conrad, F. G. (2018). Intervening to reduce satisficing behaviors in web surveys. Social Science Computer Review, 36(1), 57-81.
[83]	Zijlstra, W. P., van der Ark, L. A., & Sijtsma, K. (2011). Outliers in questionnaire data: Can they be detected and should they be removed? Journal of Educational and Behavioral Statistics, 36(2), 186-212.

文献来源	阈值下限	阈值上限
Mayerl (2013)	Mean - (2 × SD)	Mean + (2 × SD)
Schnell (1994)	Q_.50 - (1.5 × IQR)	Q_.50 + (1.5 × IQR)
Hoaglin et al. (2000)	Q_.50 - (1.5 × (Q_.50 - Q_.25))	Q_.50 + (1.5 × (Q_.75 - Q_.50))
Hoaglin et al. (2000)	Q_.50 - (3 × (Q_.50 - Q_.25))	Q_.50 + (3 × (Q_.75 - Q_.50))
Lenzner et al. (2010)	Q_.01	Q_.99

文献来源	阈值下限	阈值上限
Mayerl (2013)	Mean - (2 × SD)	Mean + (2 × SD)
Schnell (1994)	Q_.50 - (1.5 × IQR)	Q_.50 + (1.5 × IQR)
Hoaglin et al. (2000)	Q_.50 - (1.5 × (Q_.50 - Q_.25))	Q_.50 + (1.5 × (Q_.75 - Q_.50))
Hoaglin et al. (2000)	Q_.50 - (3 × (Q_.50 - Q_.25))	Q_.50 + (3 × (Q_.75 - Q_.50))
Lenzner et al. (2010)	Q_.01	Q_.99