Please wait a minute...
Acta Psychologica Sinica
|
A Comparison of Item Selection Methods for Cognitive Diagnostic Computerized Adaptive Testing with Nonstatistical Constraints
MAO Xiuzhen1; XIN Tao2
(1 Institute of Education Science, Sichuan normal university, Chengdu 610066, China) (2 Institute of Developmental Psychology, Beijing normal university, Beijing 100875, China)
Download: PDF(476 KB)   Review File (1 KB) 
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  

It is well known that items in the bank of computerized adaptive testing (CAT) are always expected to be used equally. For one thing, a good deal of manpower and financial resources spent on constructing the item bank will surely be wasted if a large proportion of items are seldom exposed or even never be used. For the other, works for ensuring the test security and maintaining the item bank will become serious for test practitioners if items are exposed extremely skewed. In addition to controlling the item exposure, tests which assembled for different examinees are usually required to satisfy many constraints, such as (a) the well-proportional of each content domain; (b) the “enemy items” could not be appeared in the same test, and (c) the appropriate balance of item keys. Supposing some constraints are violated, it will give some unexpected reactions during the test and result in inaccuracy of trait estimates. Therefore, both item exposure control and content constraints are important non-statistical constraints. They have great influence on the test validity, measurement accuracy and comparability among examinees. So, they need to be incorporated into the designing of item selection for CAT in practical settings. When cognitive diagnostic theory is used in CAT, examinees can receive more detailed diagnostic information regarding their mastery of every attribute. Therefore, cognitive diagnostic CAT (CD-CAT) is a promising research area and has gained much attention because it integrates both the cognitive diagnostic method and adaptive testing. The present study compared the performances of five item selection methods in CD-CAT with item exposure control and content constraints. The item selection methods applied are (a) incorporating the Monte Carlo approach into the item eligibility approach (MC-IE); (b) incorporating the maximum priority index method into the Monte Carlo approach (MC-MPI); (c) incorporating the restrictive threshold method into the Monte Carlo approach (MC-RT); (d) incorporating the restrictive progressive method into the Monte Carlo approach (MC-RPG), and (e) incorporating the maximum post probability of knowledge states method into the Monte Carlo approach (MC-PP). The reparameterized unified model was implemented in the simulation experiments to generate item responses with respect to five item banks constructed according to attribute structures of linear, convergent, divergent, unstructured and independent, respectively. Results indicate that (a) the distributions of item exposure produced by the same item selection method in different item banks are similar, (b) the measurement precisions of each item selection method yield in attribute structures of linear, convergent, divergent, unstructured and independent are decreased gradually; (c) the performances of different item selection methods ordered by the measurement accuracy in each test condition are methods of the MC-PP, the MC-IE, the MC-MPI, the MC-RT, and the MC-RPG; their performances in terms of item exposure control are sorted in the opposite order. According to the value of uniformly dimensional, the MC-RPG method yields a best balance between item exposure control and test accuracy while satisfying some content constraints, and then followed by the MC-MPI method.

Keywords cognitive diagnostic theory      computerized adaptive testing      measurement accuracy      item exposure rate      content constraints.     
Corresponding Authors: XIN Tao, E-mail: xintao@bnu.edu.cn    
Issue Date: 25 December 2014
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
MAO Xiuzhen
XIN Tao
Cite this article:   
MAO Xiuzhen,XIN Tao. A Comparison of Item Selection Methods for Cognitive Diagnostic Computerized Adaptive Testing with Nonstatistical Constraints[J]. Acta Psychologica Sinica, 10.3724/SP.J.1041.2014.01910
URL:  
http://journal.psych.ac.cn/xlxb/EN/10.3724/SP.J.1041.2014.01910     OR     http://journal.psych.ac.cn/xlxb/EN/Y2014/V46/I12/1910
[1] CHEN Ping. Two new online calibration methods for computerized adaptive testing[J]. Acta Psychologica Sinica, 2016, 48(9): 1184-1198.
[2] GUO Lei; ZHENG Chanjin; BIAN Yufang; SONG Naiqing; XIA Lingxiang. New item selection methods in cognitive diagnostic computerized adaptive testing: Combining item discrimination indices[J]. Acta Psychologica Sinica, 2016, 48(7): 903-914.
[3] LIN Zhe; CHEN Pin; XIN Tao. The Block Item Pocket Method to Allow Item Review in CAT[J]. Acta Psychologica Sinica, 2015, 47(9): 1188-1198.
[4] DAI Buyun; ZHANG Minqiang; JIAO Can; LI Guangming; ZHU Huawei; ZHANG Wenyi. Item Selection Using the Multiple-Strategy RRUM Based on CD-CAT[J]. Acta Psychologica Sinica, 2015, 47(12): 1511-1519.
[5] GUO Lei; ZHENG Chanjin; BIAN Yufang. Exposure Control Methods and Termination Rules in Variable-Length Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2015, 47(1): 129-140.
[6] GUO Lei;WANG Zhuoran;WANG Feng;BIAN Yufang. a-Stratified Methods Combining Item Exposure Control and General Test Overlap in Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2014, 46(5): 702-713.
[7] MAO Xiuzhen;XIN Tao. A Comparison of Item Selection Methods for Controlling Exposure Rate in Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2013, 45(6): 694-703.
[8] LUO Fen,DING Shu-Liang,WANG Xiao-Qing. Dynamic and Comprehensive Item Selection Strategies for Computerized Adaptive Testing Based on Graded Response Model[J]. , 2012, 44(3): 400-412.
[9] WANG Wen-Yi,DING Shu-Liang,YOU Xiao-Feng. On-Line Item Attribute Identification in Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(08): 964-976.
[10] CHEN Ping,XIN Tao. Item Replenishing in Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(07): 836-850.
[11] CHEN Ping,XIN Tao. Developing On-line Calibration Methods for Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(06): 710-724.
[12] CHENG Xiao-Yang,DING Shu-Liang,YAN Shen-Hai,ZHU Long-Yin. New Item Selection Criteria of Computerized Adaptive Testing with Exposure-Control Factor[J]. , 2011, 43(02): 203-212.
[13]

CHEN Ping,DING Shu-Liang

. Research on Computerized Adaptive Testing that Allows Reviewing and Changing Answers[J]. , 2008, 40(06): 737-747.
[14] Lin Haijing,Ding-Shuliang. An Exploration and Realization of Computerized Adaptive Testing with Cognitive Diagnosis[J]. , 2007, 39(04): 747-753.
[15] Chen-Ping,Ding-Shuliang,Lin-,Zhou-Jie. Item Selection Strategies of Computerized Adaptive Testing based on Graded Response Model[J]. , 2006, 38(03): 461-467.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech