Abstract:

The Q-matrix is a very important component of cognitive diagnostic assessments, and it maps attributes to items. Cognitive diagnostic assessments infer the attribute mastery pattern of respondents based on item responses. In a cognitive diagnostic assessment, item responses are observable, whereas respondents’ attribute mastery pattern is potentially, but not immediately observable. The Q-matrix plays the role of a bridge in cognitive diagnostic assessments. Therefore, Q-matrix impacts the reliability and validity of cognitive diagnostic assessments greatly. Research on how the errors of Q-matrix affect parameter estimation and classification accuracy showed that the Q-matrix from experts’ definition or experience was easily affected by experts’ personal judgment, leading to a misspecified Q-matrix. Thus, it is important to find more objective Q-matrix inference methods. This paper was inspired by Liu, Xu and Ying’s (2012) algorithm and the item-data fit statistic G2 in the item response theory framework. Further research on the Q-matrix inference, an online Q-matrix estimation method based on the statistic D2 was proposed in the present study. Those items which are the base of the online algorithm are called as base items, and it is assumed that the base items are correctly pre-specified. The online estimation algorithm can jointly estimate item parameters and item attribute vectors in an incrementally manner. In the simulation studies, we considered the DINA model with different Q-matrix (attribute-number is 3, 4 and 5), different sample size (400, 500, 800 and 1000), and different number of correct items (8, 9, 10, 11 and 12) in the initial Q-matrix. The attribute mastery pattern of the sample followed a uniform distribution, and the item parameters followed a uniform distribution with interval [0.05, 0.25]. The results indicated that: when the number of base items was not too small, the online estimation algorithm with the D2 statistic could estimate the attribute vectors of rest items one by one, and further improve the estimation by using the joint estimation. When item parameters were unknown, item number was 20, and item attributes was 3, 4 or 5, based on the initial Q-matrix, the online estimation algorithm could recover the true Q-matrix with a high probability even when the number of base items were as small as 8.

Key words: item attribute specification, DINA model, likelihood ratio, online estimation