Please wait a minute...
   2011, Vol. 43 Issue (07) : 836-850     DOI:
|
Item Replenishing in Cognitive Diagnostic Computerized Adaptive Testing
CHEN Ping;XIN Tao
Institute of Developmental Psychology, Beijing Normal University, Beijing 100875, China
Download: PDF(461 KB)  
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  Item replenishing is essential for item bank maintenance and development in cognitive diagnostic computerized adaptive testing (CD-CAT). Compared with item replenishing in regular CAT, item replenishing in CD-CAT is more complicated because it requires constructing the Q matrix (Embretson, 1984; Tatsuoka, 1995) corresponding to the new items (denoted as Qnew_item). However, the Qnew_item is often constructed manually by content experts and psychometricians, which brings about two issues: first, it takes experts a lot of time and efforts to discuss and complete the attribute identification task, especially when the number of new items is large; second, the Qnew_item identified by experts is not guaranteed to be totally correct because experts often disagree in the discussion. Therefore, this study borrowed the main idea of joint maximum likelihood estimation (JMLE) method in unidimensional item response theory (IRT) to propose the joint estimation algorithm (JEA), which depended fully on the examinees’ responses on the operational and new items to jointly estimate the Qnew_item and the item parameters of new items automatically in the context of CD-CAT under the Deterministic Inputs, Noisy “and” Gate (DINA) model.
A simulation study was conducted to investigate whether the JEA algorithm could accurately and efficiently estimate the Qnew_item and the item parameters of new items under different sample sizes and different levels of item parameter range, and the new items were randomly seeded in the random positions of examinees’ CD-CAT tests. In this study, four samples (sample sizes were 100, 300, 1000 and 3000 respectively) were simulated and each examinee had 50% probability of mastering each attribute. On the other hand, three item banks of 360 items were simulated and their item parameters were randomly drawn from U (0.05, 0.25), U (0.15, 0.35) and U (0.25, 0.45) respectively, and the three item banks shared the same Q matrix. 20 new items were simulated and the Qnew_item was constructed by randomly selecting 20 rows from the Q matrix, and the item parameters of new items were randomly drawn from U (0.05, 0.25) or U (0.15, 0.35) or U (0.25, 0.45) depending on the item parameter range of the corresponding operational items. The Shannon Entropy method was employed to select the next available item from the item bank, the Maximum A Posterior method was used to update the knowledge state estimates of examinees, and the fixed-length stopping rule was adopted and the test length was 20.
The results indicated that the JEA worked well in terms of the estimation accuracy of the Qnew_item and the item parameters of new items especially when the item parameter sizes were relatively small and the sample sizes were relatively large. And as the sample size increased, the estimation accuracy of attribute vectors were monotone increasing under all conditions, the calibration accuracy of the guessing and slipping parameters were monotone decreasing under most conditions. Also the sample size, item parameter size and initial item parameter all had effects on the performance of JEA.
Though the results from the simulation study are very encouraging, further studies are proposed for the future investigations such as different cognitive diagnostic models and different attribute hierarchical structures.
Keywords cognitive diagnostic computerized adaptive testing      item replenishing      on-line calibration      automatic attribute identification      new item     
Corresponding Authors: XIN Tao   
Issue Date: 30 July 2011
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
CHEN Ping
XIN Tao
Cite this article:   
CHEN Ping,XIN Tao. Item Replenishing in Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(07): 836-850.
URL:  
http://journal.psych.ac.cn/xlxb/EN/      OR     http://journal.psych.ac.cn/xlxb/EN/Y2011/V43/I07/836
[1] GUO Lei; ZHENG Chanjin; BIAN Yufang; SONG Naiqing; XIA Lingxiang. New item selection methods in cognitive diagnostic computerized adaptive testing: Combining item discrimination indices[J]. Acta Psychologica Sinica, 2016, 48(7): 903-914.
[2] DAI Buyun; ZHANG Minqiang; JIAO Can; LI Guangming; ZHU Huawei; ZHANG Wenyi. Item Selection Using the Multiple-Strategy RRUM Based on CD-CAT[J]. Acta Psychologica Sinica, 2015, 47(12): 1511-1519.
[3] GUO Lei; ZHENG Chanjin; BIAN Yufang. Exposure Control Methods and Termination Rules in Variable-Length Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2015, 47(1): 129-140.
[4] MAO Xiuzhen;XIN Tao. A Comparison of Item Selection Methods for Controlling Exposure Rate in Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2013, 45(6): 694-703.
[5] WANG Wen-Yi,DING Shu-Liang,YOU Xiao-Feng. On-Line Item Attribute Identification in Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(08): 964-976.
[6] CHEN Ping,XIN Tao. Developing On-line Calibration Methods for Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(06): 710-724.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech