基于作答时间数据的改变点分析在检测加速作答中的探索——已知和未知项目参数

doi:10.3724/SP.J.1041.2022.01277

摘要/Abstract

摘要：

相对于传统的离散作答数据, 作答时间作为连续数据, 可以提供更多信息。改变点分析(change point analysis)技术在心理和教育领域是一个比较新的技术。本文一方面对改变点分析在心理测量领域的应用进行了一个综合的总结和分析; 另一方面, 将基于作答数据的两种改变点分析统计量推广到作答时间数据, 将改变点分析技术应用到测验异常作答模式：加速作答speededness的检测上。采用两种检验方法：似然比检验和Wald检验, 分别在已知和未知项目参数的条件下, 实现异常作答模式的检测。结果表明, 所采用的方法对于加速作答行为的检测具有很高的检验力, 同时能够很好的控制I类错误率。实证数据分析进一步表明本文中所使用的方法具有应用价值。

关键词: 改变点分析法, 异常作答行为, 作答时间, 加速作答, 统计过程控制

Abstract:

In recent years, response time has received a rapidly growing amount of attention in psychometric research, likely due to the increasing availability of (item-level) response time data through computer-based testing and online survey data collection. Compared to the conventional item response data that are often dichotomous or polytomous, the response time is continuous and can provide much more information. Aberrant response behaviors are frequently encountered during testing. It could cause various negative effects. Change point analysis (CPA) is a well-established statistical process control method to detect changes in a sequence, and it has provided testing professionals a new lens through to understand test-taking behavior at both the examinee and item levels.

In this paper, we took test speededness as an example to illustrate how the CPA method can be used to detect aberrant behavior using item response time data. Response time under speededness was simulated using the gradual-change log-normal model for response time. Two CPA-based test statistics, the Likelihood Ratio Test and Wald Test, were used to detect aberrant response behaviors. The critical values were obtained through Monte Carlo simulations and compared with the approximate critical values in a previous study. Based on the chosen critical values, we examined the performance of the likelihood ratio test and Wald test in detecting speeded responses, specifically in terms of power and empirical Type-I error.

On the one hand, the critical values are almost identical for Wald and the likelihood ratio test. They vary substantially at different nominal α levels, but do not differ much across different test lengths. On the other hand, compared to approximate critical values, the critical values are not too far away from them but are different. That may be because the approximate critical values are suitable for situations where the change point appears in the middle of the test. Results indicate that the proposed method is much more powerful based on the critical values than conventional methods that use item response data. The power was close to 1 for most of the conditions while keeping the type-I error rate well-controlled. Real data analysis also demonstrates the performance of the method.

This study uses CPA with response time data and offers a very promising approach to detecting aberrant response behavior. Through the simulation study, we demonstrated that it is possible to use fixed critical values in different test lengths, which makes the application of the method straightforward. It also means that it is unnecessary to reconduct the simulation to update critical values when small changes occur in the test. CPA is very flexible. This study assumed that the log-normal model fits the response time data, but the method is not bounded by that assumption.

Key words: change point analysis, aberrant response behaviors, response time, test speededness, statistical process control

中图分类号:

B841

钟小缘, 喻晓锋, 苗莹, 秦春影, 彭亚风, 童昊. (2022). 基于作答时间数据的改变点分析在检测加速作答中的探索——已知和未知项目参数. 心理学报, 54(10), 1277-1292.

ZHONG Xiaoyuan, YU Xiaofeng, MIAO Ying, QIN Chunying, PENG Yafeng, TONG Hao. (2022). Exploration of change point analysis in detecting speededness based on response time data with known/unknown item parameters. Acta Psychologica Sinica, 54(10), 1277-1292.

图/表 8

参考文献 51

[1]	Andrews, D. W. K. (1993). Tests for parameter instability and structural change with unknown change point. Econometrical, 61(4), 821-856. https://doi.org/10.2307/2951764. doi: 10.2307/2951764 URL
[2]	Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (second edition). Taylor & Francis Group. http://ebookcentral.proquest.com/lib/pqopenlayer/detail.action?docID=5378595.
[3]	Bejar, I. I. (1985). Test speededness under number-right scoring: An analysis of the test of English as a foreign language. (Report No. ETS-RR-85-11). Princeton, NJ: Educational Testing Services. https://doi.org/10.1002/j.2330-8516.1985.tb00096.x.
[4]	Belov, D. I. (2016). Comparing the performance of eight item preknowledge detection statistics. Applied Psychological Measurement, 40(2), 83-97. https://doi.org/10.1177/0146621615603327. doi: 10.1177/0146621615603327 URL pmid: 29881040
[5]	Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical Theories of Mental Test Scores (pp. 397-479). Reading, MA: Addison-Wesley.
[6]	Bolt, D. M., Cohen, A. S., & Wollack, J. A. (2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints. Journal of Educational Measurement, 39(4), 331-348. https://doi.org/10.1111/j.1745-3984.2002.tb01146.x. doi: 10.1111/j.1745-3984.2002.tb01146.x URL
[7]	Bradlow, E. T., Weiss, R. E., & Cho, M. (1998). Bayesian identification of outliers in computerized adaptive tests. Journal of the American Statistical Association, 93(443), 910-919. https://doi.org/10.1080/01621459.1998.1047374. doi: 10.1080/01621459.1998.10473747 URL
[8]	Chen, Q., Ding, S. L., Zhu, L. Y., & Xu, Z. Y. (2010). Three-parameter graded response model and its parameter estimation. Journal of Jiangxi Normal University (Natural Science), 34(2), 117-122.
	[陈青, 丁树良, 朱隆尹, 许志勇. (2010). 3参数等级反应模型及其参数估计. 江西师范大学学报(自然科学版), 34(2), 117-122.]
[9]	Cheng, X. Y., Ding, S. L., Zhu, L. Y., & Wu, H. F. (2012). The stratified item selection strategy with maximal information under graded response model. Journal of Jiangxi Normal University (Natural Science), 36(5), 446-451.
	[程小杨, 丁树良, 朱隆尹, 巫华芳. (2012). 等级评分模型下的最大信息量分层选题策略. 江西师范大学学报(自然科学版), 36(5), 446-451.]
[10]	Choe, E. M., Zhang, J., & Chang, H.-H. (2018). Sequential detection of compromised items using response times in computerized adaptive testing. Psychometrika, 83(3), 650-673. https://doi.org/10.1007/s11336-017-9596-3. doi: 10.1007/s11336-017-9596-3 URL pmid: 29168039
[11]	Evans, F. R., & Reilly, R. R. (1972). A study of speededness as a source of test bias. Journal of Educational Measurement, 9(2), 123-131. https://doi.org/10.1111/j.1745-3984.1972.tb00767.x. doi: 10.1111/j.1745-3984.1972.tb00767.x URL
[12]	Fox, J.-P., Entink, R. K., & van der Linden, W. (2007). Modeling of responses and response times with the package cirt. Journal of Statistical Software, 20(7), 1-14. https://doi.org/10.18637/jss.v020.i07.
[13]	Fox, J.-P., Klotzke, K., & Simsek, A. S. (2021). LNIRT: An R package for joint modeling of response accuracy and times. ArXiv:2106.10144 [Stat]. http://arxiv.org/abs/2106.10144.
[14]	Goegebeur, Y., de Boeck, P., Wollack, J. A., & Cohen, A. S. (2008). A speeded item response model with gradual process change. Psychometrika, 73(1), 65-87. https://doi. org/10.1007/s11336-007-9031-2. doi: 10.1007/s11336-007-9031-2 URL
[15]	Guo, J., Tay, L., & Drasgow, F. (2009). Conspiracies and test compromise: An evaluation of the resistance of test systems to small-scale cheating. International Journal of Testing, 9(4), 283-309. https://doi.org/10.1080/15305050903351901. doi: 10.1080/15305050903351901 URL
[16]	Guo, X. J., & Luo, Z. S. (2019). A psychometric model for speed-accuracy tradeoff and application. Psychological Exploation, 39(5), 451-460.
	[郭小军, 罗照盛. (2019). 基于速度与准确率权衡的心理测量学模型及应用. 心理学探新, 39(5), 451-460.]
[17]	Hawkins, D. M., Qiu, P., & Kang, C. W. (2003). The changepoint model for statistical process control. Journal of Quality Technology, 35(4), 355-366. https://doi.org/10.1080/00224065.2003.11980233. doi: 10.1080/00224065.2003.11980233 URL
[18]	Li, J., & Ding, S. (2018). The several stratified methods of CAT in the presence of calibration error on GRM. Journal of Jiangxi Normal University (Natural Science), 42(4), 374-378.
	[李佳, 丁树良. (2018). 基于GRM模型的CAT分层方法在校准误差中的应用研究. 江西师范大学学报(自然科学版), 42(4), 374-378.]
[19]	Luo, F., Wang, X., Xu, Y., & Feng, W. (2020). Research progress of cheating detection technology in examinations: Detection of group cheating. China Examinations, (11), 37-41.
	[骆方, 王欣夷, 徐永泽, 封慰. (2020). 考试作弊甄别技术的研究进展:团体作弊的甄别. 中国考试, (11), 37-41.]
[20]	Marianti, S., Fox, J.-P., Avetisyan, M., Veldkamp, B. P., & Tijmstra, J. (2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39(6), 426-451. https://doi.org/ 10.3102/1076998614559412. doi: 10.3102/1076998614559412 URL
[21]	McLeod, L., Lewis, C., & Thissen, D. (2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27(2), 121-137. https://doi.org/10.1177/0146621602250534. doi: 10.1177/0146621602250534 URL
[22]	Oshima, T. C. (1994). The effect of speededness on parameter estimation in item response theory. Journal of Educational Measurement, 31(3), 200-219. https://doi.org/10.1111/j.1745-3984.1994.tb00443.x. doi: 10.1111/j.1745-3984.1994.tb00443.x URL
[23]	Page, E. S. (1955). A test for a change in a parameter occurring at an unknown point. Biometrika, 42(3/4), 523-527. https://doi.org/10.2307/2333401. doi: 10.1093/biomet/42.3-4.523 URL
[24]	Pan, Y., & Wollack, J. A. (2021). An unsupervised-learning- based approach to compromised items detection. Journal of Educational Measurement, 58(3), 413-433. https://doi.org/10.1111/jedm.12299. doi: 10.1111/jedm.12299 URL
[25]	Patton, J., Cheng, Y., Hong, M., & Diao, Q. (2019). Detection and treatment of careless responses to improve item parameter estimation. Journal of Educational and Behavioral Statistics, 44(3), 309-341. https://doi.org/10.3102/1076998618825116.
[26]	Patton, J. M. (2015). Some consequences of response time model misspecification in educational measurement (Unpublished doctoral dissertation). University of Notre Dame.
[27]	Shao, C. (2016). Aberrant response detection using change-point analysis (Unpublished doctoral dissertation). University of Notre Dame. https://curate.nd.edu/show/5425k932c5j.
[28]	Shao, C., Li, J., & Cheng, Y. (2016). Detection of test speededness using change-point analysis. Psychometrika, 81(4), 1118-1141. https://doi.org/10.1007/s11336-015-9476-7. URL pmid: 26305400
[29]	Sinharay, S. (2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics, 41(5), 521-549. https://doi.org/10.3102/1076998616658331. doi: 10.3102/1076998616658331 URL
[30]	Sinharay, S. (2017a). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42(1), 46-68. https://doi.org/10.3102/1076998616673872. doi: 10.3102/1076998616673872 URL
[31]	Sinharay, S. (2017b). Some remarks on applications of tests for detecting a change point to psychometric problems. Psychometrika, 82(4), 1149-1161. https://doi.org/10.1007/s11336-016-9531-z. doi: 10.1007/s11336-016-9531-z URL
[32]	Sinharay, S. (2017c). Which statistic should be used to detect item pre-knowledge when the set of compromised items is known? Applied Psychological Measurement, 41(6), 403-421. https://doi.org/10.1177/0146621617698453. doi: 10.1177/0146621617698453 URL
[33]	Stefan, Z., Dietrich, K., & Wolfgang, H. (2016). Are exam questions known in advance? Using local dependence to detect cheating. PLOS ONE, 11(12), e0167545. https://doi.org/10.1371/journal.pone.0167545.
[34]	Suh, Y., Cho, S.-J., & Wollack, J. A. (2012). A comparison of item calibration procedures in the presence of test speededness. Journal of Educational Measurement, 49(3), 285-311. https://doi.org/10.1111/j.1745-3984.2012.00176.x. doi: 10.1111/j.1745-3984.2012.00176.x URL
[35]	van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181-204. https://doi.org/10.3102/10769986031002181. doi: 10.3102/10769986031002181 URL
[36]	van der Linden, W. J. (2011). Test design and speededness. Journal of Educational Measurement, 48(1), 44-60. https://doi.org/10.1111/j.1745-3984.2010.00130.x. doi: 10.1111/j.1745-3984.2010.00130.x URL
[37]	van der Linden, W. J., & Guo, F. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365-384. https://doi.org/10.1007/s11336-007-9046-8. doi: 10.1007/s11336-007-9046-8 URL
[38]	van der Linden, W. J., & van Krimpen-Stoop, E. M. L. A. (2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68(2), 251-265. https://doi.org/10.1007/BF02294800. doi: 10.1007/BF02294800 URL
[39]	Wang, C., & Xu, G. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456-477. https://doi.org/10.1111/bmsp.12054. doi: 10.1111/bmsp.12054 URL
[40]	Wang, C., Xu, G., & Shang, Z. (2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83(1), 223-254. https://doi.org/10.1007/s11336-016-9525-x. doi: 10.1007/s11336-016-9525-x URL pmid: 27796763
[41]	Wang, T., & Hanson, B. A. (2005). Development and calibration of an item response model that incorporates response time. Applied Psychological Measurement, 29(5), 323-339. https://doi.org/10.1177/0146621605275984. doi: 10.1177/0146621605275984 URL
[42]	Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163-183. https://doi.org/10.1207/s15324818ame1802_2. doi: 10.1207/s15324818ame1802_2 URL
[43]	Wollack, J. A., & Cohen, A. S. (2004). A model for simulating speeded test data. Paper presented at annual meeting of the American Educational Research Association, San Diego, CA. https://testing.wisc.edu/research papers/AERA 2004 (Wollack & Cohen).pdf.
[44]	Worsley, K. J. (1979). On the likelihood ratio test for a shift in location of normal populations. Journal of the American Statistical Association, 74(366a), 365-367. https://doi.org/10.1080/01621459.1979.10482519.
[45]	Xiong, J., Luo, H., Wang, X., & Ding, S. (2018). The online calibration based on graded response model. Journal of Jiangxi Normal University (Natural Science), 42(1), 62-66.
	[熊建华, 罗慧, 王晓庆, 丁树良. (2018). 基于GRM的在线校准研究. 江西师范大学学报(自然科学版), 42(1), 62-66.]
[46]	Yu, X. F., & Cheng, Y. (2019). A change-point analysis procedure based on weighted residuals to detect back random responding. Psychological Methods, 24(5), 658-674. https://doi.org/10.1037/met0000212. doi: 10.1037/met0000212 URL pmid: 30762378
[47]	Yu, X. F., & Cheng, Y. (2022). A comprehensive review and comparison of CUSUM and change-point-analysis methods to detect test speededness. Multivariate Behavioral Research, 57(1), 112-133. https://doi.org/10.1080/00273171.2020.1809981. doi: 10.1080/00273171.2020.1809981 URL
[48]	Zhan, P. D. (2019). Joint modeling for response times and response accuracy in computer-based multidimensional assessments. Journal of Psychological Science, 42(1), 170-178.
	[詹沛达. (2019). 计算机化多维测验中作答时间和作答精度数据的联合分析. 心理科学, 42(1), 170-178.]
[49]	Zhan, P. D., Hong, J., & Man, K. W. (2020). The multidimensional log-normal response time model: An exploration of the multidimensionality of latent processing speed. Acta Psychologica Sinica, 52(9), 1132-1142. doi: 10.3724/SP.J.1041.2020.01132 URL
	[詹沛达, Hong Jiao, & Kaiwen Man. (2020). 多维对数正态作答时间模型:对潜在加工速度多维性的探究. 心理学报, 52(9), 1132-1142.]
[50]	Zhang, J. (2014). A sequential procedure for detecting compromised items in the item pool of a CAT system. Applied Psychological Measurement, 38(2), 87-104. https://doi.org/10.1177/0146621613510062 doi: 10.1177/0146621613510062 URL
[51]	Zhang, L., Wang, X., Cai, Y., & Tu, D. (2020). Change point analysis: A new method to detect aberrant responses in psychological and educational testing. Advances in Psychological Science, 28(9), 1462-1477. doi: 10.3724/SP.J.1042.2020.01462 URL
	[张龙飞, 王晓雯, 蔡艳, 涂冬波. (2020). 心理与教育测验中异常反应侦查新技术:变点分析法. 心理科学进展, 28(9), 1462-1477.]

文献	测验类型	研究类型	数据类型	测验长度	题目计分	样本量	模型	统计量	临界值
Zhang (2014)	A	S&E	R	40	2	10,000	3PLM	${{\hat{Z}}_{nm}}$	经验临界值
Li & Cheng (2016)	NA	S&E	R	50,80(S)32(E)	2	500 (S), 5000 (E)	2PLM	$\nabla {{l}_{i}}$	基于置换分布得到临界值
Shao (2016)	NA&A	S&E	R	50(A),40,50, 60, 80(NA)	2	1000 (NA &A)	Rasch &2PLM	$\nabla {{l}_{i}},W_{i}^{(j)}$	经验临界值
Sinharay (2016)	A	S&E	R	20,40,60,100(S) [60-250] (E)	2	100,000 (S) 70,000 (E)	Rasch	${{W}_{\max }},{{L}_{\max }},{{S}_{\max }}$	近似临界值
Sinharay (2017a)	NA	S&E	R	170(E,NA),170 (S, NA),50 (S,A)	2	1636, 1644 (E, NA), 100,000 (S,NA), 1000 (S,A)	Rasch (NA), 3PLM (A)	${{L}_{S}},{{R}_{S}}$	经验临界值
Sinharay (2017b)	A	S&E	R	[60,250],170	2	70,000, 1644	Rasch	${{T}_{\max }},{{L}_{\max }},{{W}_{\max }}$	经验临界值
Sinharay (2017c)	NA&A	S&E	R	170 (E,NA),100 (S, NA),50 (S,A)	2	1636, 1644 (E, NA), 1000 (S,NA&A)	3PLM	${{L}_{S}}$	经验临界值
Yu & Cheng (2019)	NA	S&E	R	20, 40, 60, 80, 100, 120(S),19(E)	5(S), 4(E)	10000 (S), 6457 (E)	GRM	${{W}_{\max }},{{L}_{\max }},{{S}_{\max }},{{R}_{\max }}$	经验临界值
Yu & Cheng (2022)	NA	S&E	R	40,60,80(S)30(E)	2	1000 (S) 3000 (E)	2PLM	${{W}_{\max }},{{L}_{\max }},{{S}_{\max }}$	经验临界值

文献	测验类型	研究类型	数据类型	测验长度	题目计分	样本量	模型	统计量	临界值
Zhang (2014)	A	S&E	R	40	2	10,000	3PLM	${{\hat{Z}}_{nm}}$	经验临界值
Li & Cheng (2016)	NA	S&E	R	50,80(S)32(E)	2	500 (S), 5000 (E)	2PLM	$\nabla {{l}_{i}}$	基于置换分布得到临界值
Shao (2016)	NA&A	S&E	R	50(A),40,50, 60, 80(NA)	2	1000 (NA &A)	Rasch &2PLM	$\nabla {{l}_{i}},W_{i}^{(j)}$	经验临界值
Sinharay (2016)	A	S&E	R	20,40,60,100(S) [60-250] (E)	2	100,000 (S) 70,000 (E)	Rasch	${{W}_{\max }},{{L}_{\max }},{{S}_{\max }}$	近似临界值
Sinharay (2017a)	NA	S&E	R	170(E,NA),170 (S, NA),50 (S,A)	2	1636, 1644 (E, NA), 100,000 (S,NA), 1000 (S,A)	Rasch (NA), 3PLM (A)	${{L}_{S}},{{R}_{S}}$	经验临界值
Sinharay (2017b)	A	S&E	R	[60,250],170	2	70,000, 1644	Rasch	${{T}_{\max }},{{L}_{\max }},{{W}_{\max }}$	经验临界值
Sinharay (2017c)	NA&A	S&E	R	170 (E,NA),100 (S, NA),50 (S,A)	2	1636, 1644 (E, NA), 1000 (S,NA&A)	3PLM	${{L}_{S}}$	经验临界值
Yu & Cheng (2019)	NA	S&E	R	20, 40, 60, 80, 100, 120(S),19(E)	5(S), 4(E)	10000 (S), 6457 (E)	GRM	${{W}_{\max }},{{L}_{\max }},{{S}_{\max }},{{R}_{\max }}$	经验临界值
Yu & Cheng (2022)	NA	S&E	R	40,60,80(S)30(E)	2	1000 (S) 3000 (E)	2PLM	${{W}_{\max }},{{L}_{\max }},{{S}_{\max }}$	经验临界值

因素	水平
测验长度	40, 60, 80
加速作答考生的比例	10%, 20%, 30%
改变点的位置参数${{\eta }_{i}}$	Median (0.6,0.7)×$\sigma _{\eta }^{2}$(0.04,0.001)
项目参数	已知, 未知

因素	水平
测验长度	40, 60, 80
加速作答考生的比例	10%, 20%, 30%
改变点的位置参数${{\eta }_{i}}$	Median (0.6,0.7)×$\sigma _{\eta }^{2}$(0.04,0.001)
项目参数	已知, 未知

项目参数	测验长度	α_0.05	α_0.01	α_0.001
已知	40	8.068 (0.08)	11.214 (0.21)	15.702 (0.58)
	60	8.261 (0.07)	11.470 (0.20)	15.885 (0.58)
	80	8.352 (0.09)	11.732 (0.19)	16.247 (0.61)
未知	40	8.247 (0.12)	11.389 (0.30)	15.824 (0.65)
	60	8.353 (0.10)	11.517 (0.36)	15.889 (0.66)
	80	8.366(0.21)	11.798 (0.34)	16.456 (0.73)