ISSN 0439-755X
CN 11-1911/B

心理学报 ›› 2023, Vol. 55 ›› Issue (1): 142-158.doi: 10.3724/SP.J.1041.2023.00142

• 研究报告 • 上一篇    


刘彦楼1(), 吴琼琼2   

  1. 1曲阜师范大学教育大数据研究院
    2曲阜师范大学心理学院, 山东 济宁 273165
  • 收稿日期:2022-03-09 发布日期:2022-10-13
  • 基金资助:

An empirical Q-matrix validation method using complete information matrix in cognitive diagnostic models

LIU Yanlou1(), WU Qiongqiong2   

  1. 1Academy of Big Data for Education; Qufu Normal University, Jining 273165, China
    2School of Psychology, Qufu Normal University, Jining 273165, China
  • Received:2022-03-09 Published:2022-10-13


Q矩阵是CDM的核心元素之一, 反映了测验的内部结构和内容设计, 通常由领域专家根据经验进行主观界定, 因此需要对可能存在的错误进行修正。本研究提出了一种新的Q矩阵修正方法——基于完整经验交叉相乘信息矩阵的Wald-XPD方法。采用Monte Carlo模拟检验了新方法的表现, 并与同类方法进行了比较。研究表明:新开发的Wald-XPD方法在Q矩阵恢复率、保留正确标定属性的比例以及修正错误标定属性的比例这3个主要指标上均有较好的表现, 且整体上优于其他方法, 尤其是在修正错误标定的属性方面。通过实证数据展示了Wald-XPD方法在Q矩阵修正中的良好表现。总之, 本研究为Q矩阵修正提供了有效的方法。

关键词: 认知诊断模型, Q矩阵, XPD矩阵, Wald检验


A Q-matrix, which defines the relations between latent attributes and items, is a central building block of the cognitive diagnostic models (CDMs). In practice, a Q-matrix is usually specified subjectively by domain experts, which might contain some misspecifications. The misspecified Q-matrix could cause several serious problems, such as inaccurate model parameters and erroneous attribute profile classifications. Several Q-matrix validation methods have been developed in the literature, such as the G-DINA discrimination index (GDI), Wald test based on an incomplete information matrix (Wald-IC), and Hull methods. Although these methods have shown promising results on Q-matrix recovery rate (QRR) and true positive rate (TPR), a common drawback of these methods is that they obtain poor results on true negative rate (TNR). It is important to note that the worse performance of the Wald-IC method on TNR might be caused by the incorrect computation of the information matrix.

A new Q-matrix validation method is proposed in this paper that constructs a Wald test with a complete empirical cross-product information matrix (XPD). A simulation study was conducted to evaluate the performance of the Wald-XPD method and compare it with GDI, Wald-IC, and Hull methods. Five factors that may influence the performance of Q-matrix validation were manipulated. Attribute patterns were generated following either a uniform distribution or a higher-order distribution. The misspecification rate was set to two levels: QM = 0.15 and QM= 0.3. Two sample sizes were manipulated: 500 and 1000. The three levels of IQ were defined as high IQ, Pj(0) ~ U(0, 0.2) and Pj(1) ~ U(0.8, 1); medium IQ, Pj(0) ~ U(0.1, 0.3) and Pj(1) ~ U(0.7, 0.9); and low IQ, Pj(0) ~ U(0.2, 0.4) and Pj(1) ~ U(0.6, 0.8). The number of attributes was fixed at K = 4. Two ratios of the number of items to attribute were considered in the study: J = 16[(K= 4)×(JK = 4)] and J = 32[(K= 4)×(JK = 8)].

The simulation results showed the following.

(1) The Wald-XPD method always provided the best results or was close to the best-performing method across the different factor levels, especially in the terms of the TNR. The HullP and Wald-IC methods produced larger values of QRR and TPR but smaller values of TNR. A similar pattern was observed between HullP and HullR, with HullP being better than HullR. Among the Q-matrix validation methods considered in this study, the GDI method was the worst performer.

(2) The results from the comparison of the HullP, Wald-IC, and Wald-XPD methods suggested that the Wald-XPD method is more preferred for Q-matrix validation. Even though the HullP and Wald-IC methods could provide higher TPR values when the conditions were particularly unfavorable (e.g., low item quality, short test length, and low sample size), they obtain very low TNR values. The practical application of the Wald-XPD method was illustrated using real data.

In conclusion, the Wald-XPD method has excellent power to detect and correct misspecified q-entry. In addition, it is a generic method that can serve as an important complement to domain experts’ judgement, which could reduce their workload.

Key words: cognitive diagnostic models, Q-matrix, XPD information matrix, Wald test