关键词: 认知诊断计算机化自适应测验, 选题方法, 测量精度, 项目曝光率

Abstract: Item exposure rate is the utilization frequency of an item. When the exposure rate is high, examinees will likely share item content. If there are too many over-exposed items, test security and hence the validity of the assessment will certainly be compromised. Furthermore, with a lot of under-exposed items having low or zero item-exposure rates, the manpower and financial resources spent on item construction will be wasted and the item pool construction will become more challenging. Item exposure control is, therefore, an important issue in computerized adaptive testing (CAT). Cognitive diagnostic CAT (CD-CAT) combines and makes use of the strengths of cognitive diagnosis theory and CAT. The system will be able to provide information on the knowledge competence of the examinees by administering fewer items than traditional assessment. Based on the a-stratified method and the item eligibility method in regular CAT, the present study proposed and compared the performance of six techniques, namely, (a) the item eligibility (IE) method, (b) the stratified multistage (SM) approach, (c) the stratified multistage-item eligibility (SMIE) method, (d) the restrictive threshold (RT) method, (e) the maximum modified priority index (MMPI) method, and (f) the restrictive progress (RPG) method. With noting it that the SM approach is similar to the a-stratified method in item selection steps. The SM approach, however, different with the a-stratified method firstly in that it stratifies the remaining item pool based on the values of item information at the estimated attributed mastery pattern while the a-stratified method is based on the values of item discrimination parameter a. Secondly, in the SM method, the remainder item bank are stratified into a number of levels before the selection of each item, whereas in the a-stratified method, the item pool is stratified only once before the test and all the examinees have the same item strata. The SMIE method combines the SM and the IE method. MATLAB (R2010a) was used in the simulation experiments to write the CD-CAT code and the deterministic inputs, noisy “and” gate (DINA) model was applied in this study. Results showed that: (a) the SM method used in CD-CAT produced widely distributed item exposure by increasing the exposure rates of most items and fully utilizing the item pool but without greatly diminishing the maximum exposure rate and measurement accuracy; (b) other than a few items, the exposure rates of the IE method were lower than the setting maximum exposure rate, but most items still had extremely low exposure rates and hence resulting in a narrow distribution of item exposure and the highest measurement precision; (c) SMIE and RT methods behaved similarly in that not only could they increase the utilization frequency of the under-exposed items but they could also decrease the maximum exposure rate to a certain extent; (d) the MMPI and the RPG methods performed similarly with almost evenly distributed item exposure but at the great sacrifice of the measurement precision. As a whole, the performances of different methods in the order of their measurement accuracy are IE, SM, SMIE, RT, RPG and MMPI. The order in terms of their performances in exposure control is: MMPI, RPG, SMIE, RT, SM and IE. All in all, the SMIE and RT methods are able to balance measurement accuracy and item exposure well.

Key words: cognitive diagnostic computerized adaptive testing, measurement accuracy, item exposure control, item selection method