Please wait a minute...
Acta Psychologica Sinica
|
A Comparison of Item Selection Methods for Controlling Exposure Rate in Cognitive Diagnostic Computerized Adaptive Testing
MAO Xiuzhen;XIN Tao
(1 College of Education, Sichuan Normal University, Chengdu, 610066, China) (2 Institute of Developmental Psychology, Beijing Normal University, Beijing, 100875, China)
Download: PDF(310 KB)  
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  Item exposure rate is the utilization frequency of an item. When the exposure rate is high, examinees will likely share item content. If there are too many over-exposed items, test security and hence the validity of the assessment will certainly be compromised. Furthermore, with a lot of under-exposed items having low or zero item-exposure rates, the manpower and financial resources spent on item construction will be wasted and the item pool construction will become more challenging. Item exposure control is, therefore, an important issue in computerized adaptive testing (CAT). Cognitive diagnostic CAT (CD-CAT) combines and makes use of the strengths of cognitive diagnosis theory and CAT. The system will be able to provide information on the knowledge competence of the examinees by administering fewer items than traditional assessment. Based on the a-stratified method and the item eligibility method in regular CAT, the present study proposed and compared the performance of six techniques, namely, (a) the item eligibility (IE) method, (b) the stratified multistage (SM) approach, (c) the stratified multistage-item eligibility (SMIE) method, (d) the restrictive threshold (RT) method, (e) the maximum modified priority index (MMPI) method, and (f) the restrictive progress (RPG) method. With noting it that the SM approach is similar to the a-stratified method in item selection steps. The SM approach, however, different with the a-stratified method firstly in that it stratifies the remaining item pool based on the values of item information at the estimated attributed mastery pattern while the a-stratified method is based on the values of item discrimination parameter a. Secondly, in the SM method, the remainder item bank are stratified into a number of levels before the selection of each item, whereas in the a-stratified method, the item pool is stratified only once before the test and all the examinees have the same item strata. The SMIE method combines the SM and the IE method. MATLAB (R2010a) was used in the simulation experiments to write the CD-CAT code and the deterministic inputs, noisy “and” gate (DINA) model was applied in this study. Results showed that: (a) the SM method used in CD-CAT produced widely distributed item exposure by increasing the exposure rates of most items and fully utilizing the item pool but without greatly diminishing the maximum exposure rate and measurement accuracy; (b) other than a few items, the exposure rates of the IE method were lower than the setting maximum exposure rate, but most items still had extremely low exposure rates and hence resulting in a narrow distribution of item exposure and the highest measurement precision; (c) SMIE and RT methods behaved similarly in that not only could they increase the utilization frequency of the under-exposed items but they could also decrease the maximum exposure rate to a certain extent; (d) the MMPI and the RPG methods performed similarly with almost evenly distributed item exposure but at the great sacrifice of the measurement precision. As a whole, the performances of different methods in the order of their measurement accuracy are IE, SM, SMIE, RT, RPG and MMPI. The order in terms of their performances in exposure control is: MMPI, RPG, SMIE, RT, SM and IE. All in all, the SMIE and RT methods are able to balance measurement accuracy and item exposure well.
Keywords cognitive diagnostic computerized adaptive testing      measurement accuracy      item exposure control      item selection method     
Corresponding Authors: XIN Tao   
Issue Date: 25 June 2013
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
MAO Xiuzhen
XIN Tao
Cite this article:   
MAO Xiuzhen,XIN Tao. A Comparison of Item Selection Methods for Controlling Exposure Rate in Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 10.3724/SP.J.1041.2013.00694
URL:  
http://journal.psych.ac.cn/xlxb/EN/10.3724/SP.J.1041.2013.00694     OR     http://journal.psych.ac.cn/xlxb/EN/Y2013/V45/I6/694
[1] GUO Lei; ZHENG Chanjin; BIAN Yufang; SONG Naiqing; XIA Lingxiang. New item selection methods in cognitive diagnostic computerized adaptive testing: Combining item discrimination indices[J]. Acta Psychologica Sinica, 2016, 48(7): 903-914.
[2] DAI Buyun; ZHANG Minqiang; JIAO Can; LI Guangming; ZHU Huawei; ZHANG Wenyi. Item Selection Using the Multiple-Strategy RRUM Based on CD-CAT[J]. Acta Psychologica Sinica, 2015, 47(12): 1511-1519.
[3] GUO Lei; ZHENG Chanjin; BIAN Yufang. Exposure Control Methods and Termination Rules in Variable-Length Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2015, 47(1): 129-140.
[4] MAO Xiuzhen; XIN Tao. A Comparison of Item Selection Methods for Cognitive Diagnostic Computerized Adaptive Testing with Nonstatistical Constraints[J]. Acta Psychologica Sinica, 2014, 46(12): 1910-1922.
[5] WANG Wen-Yi,DING Shu-Liang,YOU Xiao-Feng. On-Line Item Attribute Identification in Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(08): 964-976.
[6] CHEN Ping,XIN Tao. Item Replenishing in Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(07): 836-850.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech