Abstract:

Cognitive diagnostic assessment (CDA) is aimed at diagnosing which skills or attributes examinees have or do not have as the name expressed. This technique provides more useful feedback to examinees than a simple overall score got from classical test theory or item response theory. In CDA, multiple-choice (MC) is one of popular item types, which have the superiority on high test reliability, being easy to review, and scoring quickly and objectively. Traditionally, several cognitive diagnostic models (CDMs) have been developed to analyze the MC data by including the potential diagnostic information contained in the distractors.
However, the response to MC items can be viewed as the process of extracting signals (correct options) from noises (distractors). Examinees are supposed to have perceptions of the plausibility of each option, and they make the decision based on the most plausible option. Meanwhile, there are two different states when examinee responses to items: knows or does not know each item. Thus, the signal detection theory can be integrated into CDM to deal with MC data in CDA. The cognitive diagnostic model based on signal detection theory (SDT-CDM) is proposed in this paper and has several advantages over traditional CDMs. Firstly, it does not require the coding of q-vector for each option. Secondly, it provides discrimination and difficulty parameters that traditional CDMs cannot provide. Thirdly, it can directly express the relative differences between each option by plausibility parameters, providing a more comprehensive characterization of item quality.
The results of simulation study 1 showed that (1) the marginal maximum likelihood estimation approach via Expectation Maximization (MMLE/EM) algorithm could effectively estimate the model parameters of the SDT-CDM. (2) The SDT-CDM had high classification accuracy and parameter estimation precision and could provide option-level information for item quality diagnosis. (3) Independent variables such as the number of attributes, item quality, and sample size affected the performance of the SDT-CDM, but the overall results were promising. Figures 1 and 2 showed the parameter estimation bias and RMSE of SDT-CDM in simulation study 1, respectively.
In simulation study 2, we compared the performance of the SDT-CDM model with a traditional CDM (nominal response diagnostic model, NRDM) that can handle MC data. We used both the SDT-CDM and the NRDM as the true model to generate response data respectively, for investigating the advantages of the SDT-CDM in terms of classification accuracy compared to the NRDM. Compared with the NRDM, the SDT-CDM was more accurate in classifying examinees under all data conditions (see in Figures 3 and 4 for details).
Further, an empirical study on the TIMSS 2011 mathematics assessment were conducted using both the SDT-CDM and the NRDM to inspect the ecological validity. The results showed that the SDT-CDM had better model-data fit and a smaller number of model parameters than the NRDM did (see in Table 1). The difficulty parameters of the SDT-CDM were significantly correlated with those of the two- (three-) parameter logistic models: r(-e_DK, β_2PL) = 0.63, p = 0.015, r(-e_DK, β_3PL) = 0.71, p = 0.002, r(-e_K, β_2PL) = 0.89, p < 0.001, r(-e_K, β_3PL) = 0.79, p < 0.001. And the same was true of the discrimination parameters for the SDT-CDM: r(d, a_2PL) = 0.66, p = 0.005, r(d, a_3PL) = 0.79, p < 0.001. However, the correlation between the discrimination parameters of the NRDM and those of the two- (three-) parameter logistic models were low and not significant: r(GDI, a_2PL) = 0.20, p = 0.247, r(GDI, a_3PL) =0.15, p = 0.304. Besides, the classification accuracy and classification consistency values of the SDT-CDM were higher than those of the NRDM (see in Table 2). All the results indicated that the SDT-CDM was worth promoting.

Key words: signal detection theory, cognitive diagnostic assessment, multiple-choice items, expectation maximization algorithm