Online calibration based on computerized adaptive testing: Design and method

doi:10.3724/SP.J.1042.2020.01970

Abstract

Abstract:

Item replenishment is essential for item bank development and maintenance, where new items’ parameter calibration plays a significant role. Two core techniques of item replenishment under the circumstances of computerized adaptive testing (CAT) are: 1) online calibration design; 2) online calibration method. The former investigates the administration way of new items, while the later explores parameter estimation methods. This paper aims to clarify the development ideas and contexts of online calibration design and online calibration method. Additionally, their characteristics, relations and performance were illustrated and evaluated in details. At the end, several future research directions were pointed out. It is important to further study online calibration design based on different information indicators and online calibration methods based on joint estimations and error corrections. Moreover, future study could explore the online calibration technique in cognitive diagnostic CAT (CD-CAT) and multidimensional CAT (MCAT), as well as the empirical applications of item replenishment.

Key words: computerized adaptive testing, cognitive diagnostic theory, item replenishment, online calibration design, online calibration method

CLC Number:

B841

ZHANG Xueqin, MAO Xiuzhen, LI Jia. Online calibration based on computerized adaptive testing: Design and method[J]. Advances in Psychological Science, 2020, 28(11): 1970-1978.

Figures/Tables 2

References 38

[1]	陈平 . ( 2016). 两种新的计算机化自适应测验在线标定方法. 心理学报, 48( 9), 1184-1198.
[2]	陈平, 辛涛 . ( 2011a). 认知诊断计算机化自适应测验中在线标定方法的开发. 心理学报, 43( 06), 710-724.
[3]	陈平, 辛涛 . ( 2011b). 认知诊断计算机化自适应测验中的项目增补. 心理学报, 43(07), 836-850.
[4]	陈平, 张佳慧, 辛涛 . ( 2013). 在线标定技术在计算机化自适应测验中的应用. 心理科学进展, 21( 10), 1883-1892.
[5]	谭青蓉 . ( 2019). CD-CAT广义在线标定方法开发研究(硕士学位论文). 江西师范大学, 南昌.
[6]	汪文义, 丁树良, 游晓锋 . ( 2011). 计算机化自适应诊断测验中原始题的属性标定. 心理学报, 43( 08), 964-976.
[7]	熊建华, 罗慧, 王晓庆, 丁树良 . ( 2018). 基于GRM的在线校准研究. 江西师范大学学报(自然科学版), 42( 01), 62-66.
[8]	游晓锋, 丁树良, 刘红云 . ( 2010). 计算机化自适应测验中原始题项目参数的估计. 心理学报, 42( 7), 813-820.
[9]	Ali, U. S., & Chang, H. H . ( 2014). An item-driven adaptive design for calibrating pretest items, ETS Research Report Series, 2014( 2), 1-12.
[10]	Ban, J.-C., Hanson, B. A., Wang, T. Y., Yi, Q., & Harris, D. J . ( 2001). A comparative study of on-line pretest item-calibration/ scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38(3), 191-212. doi: 10.1111/jedm.2001.38.issue-3 URL
[11]	Berger, M. P. F . ( 1992). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57(4), 521-538. doi: 10.1007/BF02294418 URL
[12]	Berger, M. P. F . ( 1994). D-Optimal Sequential Sampling Designs for Item Response Theory Models. Journal of Educational Statistics, 19( 1), 43-56. doi: 10.3102/10769986019001043 URL
[13]	Buyske, S. ( 1998). Optimal design for item calibration in computerized adaptive testing: The 2PL case. In N. Flournoy et al.(Ed.), New developments and applications in experimental design. Lecture Notes—Monograph Series, 34. Haywood, CA: Institute of Mathematical Statistics.
[14]	Chang, Y.-C. I., & Lu, H. Y . ( 2010). Online calibration via variable length computerized adaptive testing. Psychometrika, 75( 1), 140-157. doi: 10.1007/s11336-009-9133-0 URL
[15]	Chen, P. ( 2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42( 5), 559-590. doi: 10.3102/1076998617695098 URL
[16]	Chen, P., & Wang, C. ( 2015). A new online calibration method for multidimensional computerized adaptive testing, Psychometrika, 81( 3), 674-701. doi: 10.1007/s11336-015-9482-9 URL pmid: 26608960
[17]	Chen, Y., Liu, J., & Ying, Z . ( 2015). Online item calibration for Q-matrix in CD-CAT. Applied Psychological Measurement, 39( 1), 5-15. doi: 10.1177/0146621613513065 URL pmid: 29882531
[18]	Cheng, Y. ( 2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74( 4) 619-632. doi: 10.1007/s11336-009-9123-2 URL
[19]	Hassan, M. U., & Miller, F. ( 2019). Optimal item calibration for computerized achievement tests. Psychometrika, 84(4), 1101-1128. doi: 10.1007/s11336-019-09673-6 URL pmid: 31183669
[20]	He, Y. H., & Chen, P. ( 2020). Optimal online calibration designs for item replenishment in adaptive testing. Psychometrika, 85( 1), 35-55. doi: 10.1007/s11336-019-09687-0 URL pmid: 31531789
[21]	He, Y. H., Chen, P., & Li, Y . ( 2019). New efficient and practicable adaptive designs for calibrating items online. Applied Psychological Measurement, 44( 1), 3-16. doi: 10.1177/0146621618824854 URL pmid: 31853155
[22]	He, Y. H., Chen, P., Li, Y., & Zhang, S. M . ( 2017). A new online calibration method based on Lord's Bias-Correction. Applied Psychological Measurement. 41( 6), 456-471. doi: 10.1177/0146621617697958 URL pmid: 29882532
[23]	Jones, D. H., & Jin, Z. Y . ( 1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59( 1), 59-75. doi: 10.1007/BF02294265 URL
[24]	Kang, H. A., Zheng, Y., & Chang, H. H . ( 2020). Online calibration of a joint model of item responses and response times in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 45( 2), 175-208. doi: 10.3102/1076998619879040 URL
[25]	Kingsbury, G. G. ( 2009. Adaptive item calibration: A process for estimating item parameters within a computerized adaptive test. In D. J. Weiss (Ed.), Proceedings of the 2009 GMAC conference on computerized adaptive testing (pp.1-15). Retrieved from http://www.psych.umn.edu/psylabs/CATCentral/
[26]	Makransky, G. ( 2009). An automatic online calibration design in adaptive testing. Paper presented at the 2007 GMAC Conference on Computerized Adaptive Testing, McLean, USA.
[27]	Mulder, J., & van der Linden, W. J . ( 2009, June). Multidimensional adaptive testing with optimal design criteria for Item Selection. Psychometrika, 74( 2), 273-296. doi: 10.1007/s11336-008-9097-5 URL pmid: 20119511
[28]	Ren, H., van der Linden, W. J., & Diao, Q . ( 2017). Continuous online item calibration: Parameter recovery and item utilization. Psychometrika, 82( 2), 498-522. doi: 10.1007/s11336-017-9553-1 URL pmid: 28290109
[29]	Stefanski, L. A., & Carroll, R. J . ( 1985). Covariate measurement error in logistic regression. Annals of Statistics, 13( 4), 1335-1351.
[30]	Stocking, M. L . ( 1988). Scale drift in on-line calibration (Research Rep. 88-28). Princeton, NJ: ETS.
[31]	van der, Linden, W., J., & Ren, H. ( 2015). Optimal bayesian adaptive design for test-item calibration. Psychometrika, 80( 2), 263-288. doi: 10.1007/s11336-013-9391-8 URL pmid: 24407735
[32]	Wainer, H., & Mislevy, R. J . ( 1990). Item response theory, item calibration, and proficiency estimation. In H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, R. J. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (Chap. 4, pp. 65-102). Hillsdale, NJ: Erlbaum.
[33]	Wang, C., & Chang, H. H . ( 2011). Item selection in multidimensional computerized adaptive testing-gaining information from different angles. Psychometrika, 76( 3), 363-384. doi: 10.1007/s11336-011-9215-7 URL
[34]	Xiong, J., Ding, S., Luo, F., & Luo, Z . ( 2020). Online calibration of polytomous items under the graded response model. Frontiers in Psychology, 10( 1), 3085. doi: 10.3389/fpsyg.2019.03085 URL
[35]	Xu, X. L., Chang, H. H., & Douglas, J . ( 2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Canada.
[36]	Zheng, Y. ( 2014). New methods of online calibration for item bank replenishment (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign, Champaign, IL.
[37]	Zheng, Y. ( 2016). Online calibration of polytomous items under the generalized partial credit model. Applied Psychological Measurement, 40( 6), 434-450. doi: 10.1177/0146621616650406 URL pmid: 29881063
[38]	Zheng, Y., & Chang, H. H . ( 2017). A comparison of five methods for pretest item selection in online calibration. International Journal of Quantitative Research in Education, 4( 1), 133-158. doi: 10.1504/IJQRE.2017.086500 URL

分类标准	方法	特点
项目视角:参数信息量	D-优化、序贯D-优化	自适应选取被试
项目视角:参数信息量	D-TP、D-VR、ED 和D-c方法	自适应选取项目
考生视角:能力与样本量	OIRPI、SI指标	自适应选取项目

分类标准	方法	特点
项目视角:参数信息量	D-优化、序贯D-优化	自适应选取被试
项目视角:参数信息量	D-TP、D-VR、ED 和D-c方法	自适应选取项目
考生视角:能力与样本量	OIRPI、SI指标	自适应选取项目

分类标准	方法	特点	适用情景
条件极大似然估计	MethodA、MethodB、FFMLE-A和ECSE-A	简单、易操作, 需要大样本	传统CAT/MCAT
	MLE-LBCI-A		传统CAT
	CD-MethodA、MLE		CD-CAT
MMLE/EM算法	OEM、MEM	计算复杂, 耗时, 不易收敛	传统CAT中二级和多级评分项目/MCAT
MMLE/EM算法	CD-OEM、CD-MEM、MMLE	计算复杂, 耗时, 不易收敛	CD-CAT
贝叶斯算法	贝叶斯版本:方法A, OEM和MEM	精度高、计算复杂, 耗时	传统CAT/MCAT
联合极大似然估计	JEA、SIE、SimIE、SIE-R、JEA-R、SIE-R-BIC、JEA-R-BIC RMSEA-N	联合估计Q矩阵和项目参数	CD-CAT

分类标准	方法	特点	适用情景
条件极大似然估计	MethodA、MethodB、FFMLE-A和ECSE-A	简单、易操作, 需要大样本	传统CAT/MCAT
	MLE-LBCI-A		传统CAT
	CD-MethodA、MLE		CD-CAT
MMLE/EM算法	OEM、MEM	计算复杂, 耗时, 不易收敛	传统CAT中二级和多级评分项目/MCAT
MMLE/EM算法	CD-OEM、CD-MEM、MMLE	计算复杂, 耗时, 不易收敛	CD-CAT
贝叶斯算法	贝叶斯版本:方法A, OEM和MEM	精度高、计算复杂, 耗时	传统CAT/MCAT
联合极大似然估计	JEA、SIE、SimIE、SIE-R、JEA-R、SIE-R-BIC、JEA-R-BIC RMSEA-N	联合估计Q矩阵和项目参数	CD-CAT