Please wait a minute...
   2012, Vol. 44 Issue (3) : 400-412     DOI:
Dynamic and Comprehensive Item Selection Strategies for Computerized Adaptive Testing Based on Graded Response Model
LUO Fen;DING Shu-Liang;WANG Xiao-Qing
(School of Computer and Information Engineering, Jiangxi Normal University, Nanchang 330022, China)
Download: PDF(817 KB)  
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  Item selection strategy (ISS) is a core component in Computerized Adaptive Testing (CAT). Polytomous items can provide more information about examinee compared with dichotomous items, and adopting polytomously scored items in test is a research direction of CAT. As we know, the most widely used ISS is the maximum Fisher information (MFI) criterion, which raises concerns about cost-efficiency of the pool utilization and poses security risks for CAT programs. Chang & Ying (1999) and Chang, Qian, & Ying (2001) proposed two alternative item selection procedures, the a-stratified method (a-STR) and the a-stratified with b blocking method (b-STR) based on dichotomous model, with the goal to remedy the problems of item overexposure and item underexposure produced by MFI. However, the technology of a-STR and b-STR is static because the items are stratified according to the given information at the beginning of test. Based on graded response model (GRM), a technique of the reduction dimensionality of difficulty (or step) parameters was employed to construct some ISSs recently. The limitation of this dimension reduction technique is that it loses a lot of information. Thus, in order to improve MFI, two new item selection methods are proposed based on GRM: (1) modify the technique of the reduction dimensionality of difficulty (or step) parameters by integrating the interval estimation; (2) dynamic a-STR and dynamic b-STR methods are implemented in the testing process. On one hand, these new ISSs can avoid and remedy the limitations of MFI and make good use of the advantages of the Fisher information function (FIF); FIF compresses all item parameters and ability parameters, so it is a comprehensive tool for all parameters in nature.On the other hand, the new ISSs employ the property that FIF could represent the inverse of the variance of the ability estimation, let ε be the square root of the reciprocal of the Fisher information, d be the absolute deviation between the estimate ability and the function of the parameters of an item, which may be chosen and could be changed during the course of CAT, the inequality of d<ε has the form of interval estimation, and its utility could be imaged as a more flexible shadow item pool.
A simulation study based on GRM was conducted. Four item pools of different structures were simulated, and 1000 examinees was generated and their abilities were randomly drawn from the standard normal distribution N (0,1). Each pool consists of 1000 polytomous items and the maximum score of each item was randomly selected from set {3, 4, 5, 6}. In this paper, we assume the prior distribution of ability is standard normal and the Bayesian expected a posteriori (EAP) is employed to estimate the ability parameter. The CAT test stopped when the accumulative information satisfies the pre-determined value M (M=16) or reaches the pre-assigned test length 30.
The results of the simulation study show that the new item selection methods required shorter test lengths and lower average exposure rates than the other methods, while maintaining the accuracy of ability estimation. More specifically, the new ISSs which applied the idea of the interval estimate were better than the correspondent ISS in terms of the Chi-square value. And the same effect appeared when comparing the dynamic a-STR and dynamic b-STR ISS with MFI. Some important results are also found by comparing different structure of item pool. The accuracy of ability estimation and item exposure rate were related to the distribution of the difficult parameters b, that is, the accuracy of ability estimation obtained from the condition in which b was sampled from N (0,1) was better than that when b was sampled from uniform distribution. The conclusion of item exposure rate is on the contrary. Also, the test length was related to the distribution of the discrimination parameter a, the test length required by the condition in which a was sampled from uniform distribution was shorter than that when the logarithm of a was sampled from N (0,1). In a word, in terms of controlling and balancing the item exposure, the new ISSs may gain an advantage over the former correspondent ISS.
Keywords Graded Response Model (GRM)      Computerized Adaptive Testing (CAT)      Item selection      Interval estimation      b-STR based on polytomously scored model     
Corresponding Authors: DING Shu-Liang   
Issue Date: 28 March 2012
E-mail this article
E-mail Alert
Articles by authors
DING Shu-Liang
WANG Xiao-Qing
Cite this article:   
LUO Fen,DING Shu-Liang,WANG Xiao-Qing. Dynamic and Comprehensive Item Selection Strategies for Computerized Adaptive Testing Based on Graded Response Model[J]. , 2012, 44(3): 400-412.
URL:     OR
[1] GUO Lei; ZHENG Chanjin; BIAN Yufang; SONG Naiqing; XIA Lingxiang. New item selection methods in cognitive diagnostic computerized adaptive testing: Combining item discrimination indices[J]. Acta Psychologica Sinica, 2016, 48(7): 903-914.
[2] LUO Zhaosheng; YU Xiaofeng; GAO Chunlei; LI Yujun; PENG Yafeng;WANG Rui; WANG Yutong. Item Selection Strategies Based on Attribute Mastery Probabilities in CD-CAT[J]. Acta Psychologica Sinica, 2015, 47(5): 679-688.
[3] DAI Buyun; ZHANG Minqiang; JIAO Can; LI Guangming; ZHU Huawei; ZHANG Wenyi. Item Selection Using the Multiple-Strategy RRUM Based on CD-CAT[J]. Acta Psychologica Sinica, 2015, 47(12): 1511-1519.
[4] MAO Xiuzhen;XIN Tao. A Comparison of Item Selection Methods for Controlling Exposure Rate in Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2013, 45(6): 694-703.
[5] CHENG Xiao-Yang,DING Shu-Liang,YAN Shen-Hai,ZHU Long-Yin. New Item Selection Criteria of Computerized Adaptive Testing with Exposure-Control Factor[J]. , 2011, 43(02): 203-212.
[6] LIU Zhen,DING Shu-Liang,LIN Hai-Jing. Item Selection Strategies for Computerized Adaptive Testing with the Generalized Partial Credit Model[J]. , 2008, 40(05): 618-625.
[7] Lin Haijing,Ding-Shuliang. An Exploration and Realization of Computerized Adaptive Testing with Cognitive Diagnosis[J]. , 2007, 39(04): 747-753.
[8] Dai Haiqi,Chen Dezhi,Ding Shuliang,Deng Taiping. The Comparison Among Item Selection Strategies of CAT with Multiple-choice Items[J]. , 2006, 38(05): 778-783.
[9] Chen-Ping,Ding-Shuliang,Lin-,Zhou-Jie. Item Selection Strategies of Computerized Adaptive Testing based on Graded Response Model[J]. , 2006, 38(03): 461-467.
Full text



Copyright © Acta Psychologica Sinica
Support by Beijing Magtech