关键词: 计算机化自适应测验, 认知诊断, 在线标定, 旧题, 新题

Abstract: Like all computerized adaptive testing (CAT) applications, some items in the item bank maybe flawed or obsolete or overexposed and they should be replaced by new items (Wainer & Mislevy, 1990), item replenishing is essential for item bank maintenance and development in cognitive diagnostic CAT (CD-CAT). In regular CAT, on-line calibration method is commonly used to calibrate the item parameters of new items. However, until now no reference is publicly available about on-line calibration for CD-CAT. Thus, this study investigated the possibility to extend some current methods used in CAT to CD-CAT situation. Three representative on-line calibration methods in regular CAT were under investigation: Method A (Stocking, 1988), marginal maximum likelihood estimate with one EM cycle (OEM) method (Wainer & Mislevy, 1990) and marginal maximum likelihood estimate with multiple EM cycles (MEM) method (Ban, Hanson, Wang, Yi, & Harris, 2001). Under certain theoretical justifications based on the Deterministic Inputs, Noisy “and” Gate (DINA) model, these methods were generalized to CD-CAT situation, denoted as CD-Method A, CD-OEM and CD-MEM, respectively.
Two simulation studies were conducted to compare the performance of the three CD-CAT on-line calibration methods in terms of item-parameter recovery. In the first study, the new items were randomly assigned to the examinees and then were calibrated accordingly. 2000 examinees were generated assuming that each examinee has 50% probability of mastering each attribute, 360 operational items were simulated and their guessing and slipping parameters were all randomly drawn from U (0.05, 0.25). 20 new items were simulated and the Q matrix corresponding to the new items was constructed by randomly selecting 20 rows from the Q matrix corresponding to the operational items, and the item parameters of new items were also randomly drawn from U (0.05, 0.25). The Shannon Entropy method was employed to select the next item and the Maximum A Posterior method was used to update the knowledge state (KS) estimates of examinees. As to the second study, the new items were first administered to a sub-group of the examinees and then were pre-calibrated; then for the remaining examinees, the new items were selected adaptively according to their initial parameter estimates to fit the examinee’s current KS estimates; finally, the item parameters of the new items were re-calibrated sequentially. Note that all the simulation conditions in the second study remained the same as those in the first study except the new items were adaptively selected.
The results of Study 1 indicated that CD-Method A outperformed the other two methods in that it yielded the smallest estimation errors, and the simulated CD-CAT test was able to provide relatively accurate KS estimates for the examinees. The results of Study 2 showed that the adaptive calibration design could improve the item-parameter recovery compared with the random calibration design for CD-Method A, CD-Method A and CD-OEM.
Though the results from the two studies are very encouraging, further studies are proposed for the future investigations such as different sample sizes, different cognitive diagnostic models and different attribute hierarchical structures.

Key words: computerized adaptive testing, cognitive diagnosis, on-line calibration, operational item, new item