认知诊断模型Q矩阵修正：完整信息矩阵的作用

doi:10.3724/SP.J.1041.2023.00142

摘要/Abstract

摘要：

Q矩阵是CDM的核心元素之一, 反映了测验的内部结构和内容设计, 通常由领域专家根据经验进行主观界定, 因此需要对可能存在的错误进行修正。本研究提出了一种新的Q矩阵修正方法——基于完整经验交叉相乘信息矩阵的Wald-XPD方法。采用Monte Carlo模拟检验了新方法的表现, 并与同类方法进行了比较。研究表明：新开发的Wald-XPD方法在Q矩阵恢复率、保留正确标定属性的比例以及修正错误标定属性的比例这3个主要指标上均有较好的表现, 且整体上优于其他方法, 尤其是在修正错误标定的属性方面。通过实证数据展示了Wald-XPD方法在Q矩阵修正中的良好表现。总之, 本研究为Q矩阵修正提供了有效的方法。

关键词: 认知诊断模型, Q矩阵, XPD矩阵, Wald检验

Abstract:

A Q-matrix, which defines the relations between latent attributes and items, is a central building block of the cognitive diagnostic models (CDMs). In practice, a Q-matrix is usually specified subjectively by domain experts, which might contain some misspecifications. The misspecified Q-matrix could cause several serious problems, such as inaccurate model parameters and erroneous attribute profile classifications. Several Q-matrix validation methods have been developed in the literature, such as the G-DINA discrimination index (GDI), Wald test based on an incomplete information matrix (Wald-IC), and Hull methods. Although these methods have shown promising results on Q-matrix recovery rate (QRR) and true positive rate (TPR), a common drawback of these methods is that they obtain poor results on true negative rate (TNR). It is important to note that the worse performance of the Wald-IC method on TNR might be caused by the incorrect computation of the information matrix.

A new Q-matrix validation method is proposed in this paper that constructs a Wald test with a complete empirical cross-product information matrix (XPD). A simulation study was conducted to evaluate the performance of the Wald-XPD method and compare it with GDI, Wald-IC, and Hull methods. Five factors that may influence the performance of Q-matrix validation were manipulated. Attribute patterns were generated following either a uniform distribution or a higher-order distribution. The misspecification rate was set to two levels: QM = 0.15 and QM= 0.3. Two sample sizes were manipulated: 500 and 1000. The three levels of IQ were defined as high IQ, P_j(0) ~ U(0, 0.2) and P_j(1) ~ U(0.8, 1); medium IQ, P_j(0) ~ U(0.1, 0.3) and P_j(1) ~ U(0.7, 0.9); and low IQ, P_j(0) ~ U(0.2, 0.4) and P_j(1) ~ U(0.6, 0.8). The number of attributes was fixed at K = 4. Two ratios of the number of items to attribute were considered in the study: J = 16[(K= 4)×(JK = 4)] and J = 32[(K= 4)×(JK = 8)].

The simulation results showed the following.

(1) The Wald-XPD method always provided the best results or was close to the best-performing method across the different factor levels, especially in the terms of the TNR. The HullP and Wald-IC methods produced larger values of QRR and TPR but smaller values of TNR. A similar pattern was observed between HullP and HullR, with HullP being better than HullR. Among the Q-matrix validation methods considered in this study, the GDI method was the worst performer.

(2) The results from the comparison of the HullP, Wald-IC, and Wald-XPD methods suggested that the Wald-XPD method is more preferred for Q-matrix validation. Even though the HullP and Wald-IC methods could provide higher TPR values when the conditions were particularly unfavorable (e.g., low item quality, short test length, and low sample size), they obtain very low TNR values. The practical application of the Wald-XPD method was illustrated using real data.

In conclusion, the Wald-XPD method has excellent power to detect and correct misspecified q-entry. In addition, it is a generic method that can serve as an important complement to domain experts’ judgement, which could reduce their workload.

Key words: cognitive diagnostic models, Q-matrix, XPD information matrix, Wald test

中图分类号:

B841

刘彦楼, 吴琼琼. (2023). 认知诊断模型Q矩阵修正：完整信息矩阵的作用. 心理学报, 55(1), 142-158.

LIU Yanlou, WU Qiongqiong. (2023). An empirical Q-matrix validation method using complete information matrix in cognitive diagnostic models. Acta Psychologica Sinica, 55(1), 142-158.

图/表 10

参考文献 45

[1]	Chen F., Liu Y., Xin T., & Cui Y. (2018). Applying the M₂ statistic to evaluate the fit of diagnostic classification models in the presence of attribute hierarchies. Frontiers in Psychology, 9, Article 1875.
[2]	Chen J. (2017). A residual-based approach to validate Q-matrix specifications. Applied Psychological Measurement, 41(4), 277-293. doi: 10.1177/0146621616686021 pmid: 29881093
[3]	Chiu C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598-618. doi: 10.1177/0146621613488436 URL
[4]	de la Torre J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362. doi: 10.1111/j.1745-3984.2008.00069.x URL
[5]	de la Torre J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115-130.
[6]	de la Torre J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199. doi: 10.1007/s11336-011-9207-7 URL
[7]	de la Torre J., & Chiu C.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253-273. doi: 10.1007/s11336-015-9467-8 pmid: 25943366
[8]	de la Torre J., & Douglas J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353. doi: 10.1007/BF02295640 URL
[9]	Feng Y. (2013). Estimation and Q-matrix validation for diagnostic classification models (Unpublished doctoral dissertation). University of South Carolina, Los Angeles, America.
[10]	Gu Y., Liu J., Xu G., & Ying Z. (2018). Hypothesis testing of the Q-matrix. Psychometrika, 83(3), 515-537. doi: 10.1007/s11336-018-9629-6 URL
[11]	Heller J., & Wickelmaier F. (2013). Minimum discrepancy estimation in probabilistic knowledge structures. Electronic Notes in Discrete Mathematics, 42, 49-56. doi: 10.1016/j.endm.2013.05.145 URL
[12]	Kang C. H., Yang Y. K., & Zeng P. H. (2019). Q-matrix refinement based on item fit statistic RMSEA. Applied Psychological Measurement, 43(7), 527-542. doi: 10.1177/0146621618813104 pmid: 31534288
[13]	Li J., Mao X., & Wei J. (2022). A simple and effective new method of Q-matrix validation. Acta Psychologica Sinica, 54(8), 996-1008. doi: 10.3724/SP.J.1041.2022.00996 URL
	[李佳, 毛秀珍, 韦嘉. (2022). 一种简单有效的Q矩阵修正新方法. 心理学报, 54(8), 996-1008.]
[14]	Li J., Mao X., & Zhang X. (2021). Q-matrix estimation (validation) methods for cognitive diagnosis. Advances in Psychological Science, 29(12), 2272-2280. doi: 10.3724/SP.J.1042.2021.02272 URL
	[李佳, 毛秀珍, 张雪琴. (2021). 认知诊断Q矩阵估计(修正)方法. 心理科学进展, 29(12), 2272-2280.]
[15]	Li X., & Wang W. (2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52(1), 28-54. doi: 10.1111/jedm.12061 URL
[16]	Lim Y., & Drasgow F. (2017). Nonparametric calibration of item-by-attribute matrix in cognitive diagnosis. Multivariate Behavioral Research, 52(5), 562-575. doi: 10.1080/00273171.2017.1341829 pmid: 28715230
[17]	Liu J., Xu G., & Ying Z. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548-564. pmid: 23926363
[18]	Liu Y., Andersson B., Xin T., Zhang H., & Wang L. (2019). Improved Wald statistics for item-level model comparison in diagnostic classification models. Applied Psychological Measurement, 43(5), 402-414. doi: 10.1177/0146621618798664 pmid: 31235985
[19]	Liu Y., Tian W., & Xin T. (2016). An application of M₂ statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41(1), 3-26.
[20]	Liu Y., Xin T., Andersson B., & Tian W. (2019). Information matrix estimation procedures for cognitive diagnostic models. British Journal of Mathematical and Statistical Psychology, 72(1), 18-37. doi: 10.1111/bmsp.12134
[21]	Liu Y., Xin T., & Jiang Y. (2021). Structural parameter standard error estimation method in diagnostic classification models: Estimation and application. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2021.1919048
[22]	Liu Y., Xin T., Li L., Tian W., & Liu X. (2016). An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix. Acta Psychologica Sinica, 48(5), 588-598. doi: 10.3724/SP.J.1041.2016.00588 URL
	[刘彦楼, 辛涛, 李令青, 田伟, 刘笑笑. (2016). 改进的认知诊断模型项目功能差异检验方法——基于观察信息矩阵的Wald统计量. 心理学报, 48(5), 588-598.]
[23]	Ma W., & de la Torre J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275. doi: 10.1111/bmsp.12070 URL
[24]	Ma W., & de la Torre J. (2020). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73(1), 142-163. doi: 10.1111/bmsp.12156 URL
[25]	McFadden D. (1974). Conditional logit analysis of qualitative choice behavior. In P. Zarembka (Ed.), Frontiers in economics (pp. 105-142). New York, NY: Academic Press.
[26]	Nájera P., Sorrel M. A., & Abad F. J. (2019). Reconsidering cutoff points in the general method of empirical Q-matrix validation. Educational and Psychological Measurement, 79(4), 727-753. doi: 10.1177/0013164418822700 pmid: 32655181
[27]	Nájera P., Sorrel M. A., de la Torre J., & Abad F. J. (2020). Improving robustness in Q-Matrix validation using an iterative and dynamic procedure. Applied Psychological Measurement, 44(6), 431-446. doi: 10.1177/0146621620909904 pmid: 32788815
[28]	Nájera P., Sorrel M. A., de la Torre J., & Abad F. J. (2021). Balancing fit and parsimony to improve Q-matrix validation. British Journal of Mathematical and Statistical Psychology, 74( Suppl 1), 110-130.
[29]	Philipp M., Strobl C., de la Torre J., & Zeileis A. (2018). On the estimation of standard errors in cognitive diagnosis models. Journal of Educational and Behavioral Statistics, 43(1), 88-115.
[30]	Rupp A. A., & Templin J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96. doi: 10.1177/0013164407301545 URL
[31]	Rupp A. A., Templin J., & Henson R. A. (2010). Diagnostic measurement: theory, methods, and applications. Guilford.
[32]	Sessoms J., & Henson R. A. (2018). Applications of diagnostic classification models: A literature review and critical commentary. Measurement: Interdisciplinary Research and Perspectives, 16(1), 1-17. doi: 10.1080/15366367.2018.1435104 URL
[33]	Sorrel M. A., Olea J., Abad F. J., de la Torre J., Aguado D., & Lievens F. (2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods, 19(3), 506-532. doi: 10.1177/1094428116630065 URL
[34]	Tatsuoka K. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 453-488). Hillsdale, NJ: Erlbaum.
[35]	Terzi R. (2017). New Q-matrix validation Procedures (Unpublished doctoral dissertation). The State University of New Jersey, New Brunswick, America.
[36]	Terzi R., & de la Torre J. (2018). An iterative method for empirically-based Q-matrix validation. International Journal of Assessment Tools in Education, 5(2), 248-262.
[37]	Tu D., Cai Y., & Dai H. (2012). A new method of Q-Matrix validation based on DINA model. Acta Psychologica Sinica, 44(4), 558-568. doi: 10.3724/SP.J.1041.2012.00558 URL
	[涂冬波, 蔡艳, 戴海琦. (2012). 基于DINA模型的Q矩阵修正方法. 心理学报, 44(4), 558-568.]
[38]	von Davier M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287-307. doi: 10.1348/000711007X193957 URL
[39]	Wang D., Cai Y., & Tu D. (2020). Q-matrix estimation methods for cognitive diagnosis models: Based on partial known Q-matrix. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2020.1746901
[40]	Wang D., Gao X., Cai Y., & Tu D. (2018). A new Q-matrix estimation method: ICC based on ideal response. Journal of Psychological Science, 41(2), 466-474.
	[汪大勋, 高旭亮, 蔡艳, 涂冬波. (2018). 一种非参数化的Q矩阵估计方法: ICC-IR方法开发. 心理科学, 41(2), 466-474.]
[41]	Wang D., Gao X., Cai Y., & Tu D. (2020). A method of Q-matrix validation for polytomous response cognitive diagnosis model based on relative fit statistics. Acta Psychologica Sinica, 52(1), 93-106. doi: 10.3724/SP.J.1041.2020.00093 URL
	[汪大勋, 高旭亮, 蔡艳, 涂冬波. (2020). 基于类别水平的多级计分认知诊断Q矩阵修正:相对拟合统计量视角. 心理学报, 52(1), 93-106.]
[42]	Wang D., Gao X., Han Y., & Tu D. (2018). A simple and effective Q-matrix estimation method: From non-parametric perspective. Journal of Psychological Science, 41(1), 180-188.
	[汪大勋, 高旭亮, 韩雨婷, 涂冬波. (2018). 一种简单有效的Q矩阵估计方法开发:基于非参数化方法视角. 心理科学, 41(1), 180-188.]
[43]	Wang W., Song L., Ding S., Meng Y., Cao C., & Jie Y. (2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42(6), 446-459. doi: 10.1177/0146621617752991 pmid: 30787487
[44]	Yu X. F., & Cheng Y. (2020). Data-driven Q-matrix validation using a residual-based statistic in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology, 73(Suppl 1), 145-179. doi: 10.1111/bmsp.12191 URL
[45]	Yu X., Luo Z., Qin C., Gao C., & Li J. (2015). Joint estimation of model parameters and Q-matrix based on response data. Acta Psychologica Sinica, 47(2), 273-282.
	[喻晓锋, 罗照盛, 秦春影, 高椿雷, 李喻骏. (2015). 基于作答数据的模型参数和Q矩阵联合估计. 心理学报, 47(2), 273-282.]

因素	因素水平
样本量N	500、1000
项目数和属性数的比例JK	4、8
属性数K	4
平均项目质量IQ	0.4、0.6、0.8
属性分布AD	均匀分布、高阶分布
错误设定的比例QM	0.15、0.3
链接函数	G-DINA模型
Q矩阵修正方法	GDI、Wald-IC、Hull (HullP、HullR)、Wald-XPD

因素	因素水平
样本量N	500、1000
项目数和属性数的比例JK	4、8
属性数K	4
平均项目质量IQ	0.4、0.6、0.8
属性分布AD	均匀分布、高阶分布
错误设定的比例QM	0.15、0.3
链接函数	G-DINA模型
Q矩阵修正方法	GDI、Wald-IC、Hull (HullP、HullR)、Wald-XPD

指标	方法	QM		IQ			N		JK		AD
指标	方法	0.15	0.3	0.4	0.6	0.8	500	1000	4	8	均匀分布	高阶分布
QRR	GDI	0.906	0.828	0.859	0.922	0.945	0.922	0.922	0.906	0.930	0.938	0.906
	Wald-IC	0.945	0.813	0.844	0.922	0.969	0.906	0.938	0.891	0.930	0.938	0.906
	HullP	0.930	0.852	0.875	0.945	0.953	0.938	0.953	0.938	0.945	0.953	0.930
	HullR	0.891	0.797	0.844	0.891	0.922	0.898	0.906	0.906	0.906	0.914	0.891
	Wald-XPD	0.937	0.867	0.820	0.938	0.969	0.906	0.953	0.906	0.945	0.953	0.906
TPR	GDI	0.944	0.922	0.933	0.936	0.953	0.936	0.945	0.944	0.936	0.954	0.926
	Wald-IC	0.945	0.933	0.908	0.954	0.969	0.933	0.956	0.944	0.945	0.956	0.938
	HullP	0.963	0.936	0.963	0.961	0.956	0.953	0.969	0.963	0.956	0.967	0.953
	HullR	0.936	0.911	0.953	0.927	0.930	0.927	0.944	0.956	0.922	0.944	0.926
	Wald-XPD	0.944	0.900	0.835	0.944	0.969	0.917	0.953	0.920	0.944	0.953	0.927
TNR	GDI	0.800	0.684	0.421	0.789	0.900	0.711	0.737	0.579	0.842	0.800	0.684
	Wald-IC	0.789	0.579	0.405	0.700	0.900	0.632	0.684	0.526	0.789	0.700	0.632
	HullP	0.800	0.684	0.368	0.833	0.947	0.737	0.800	0.600	0.895	0.816	0.700
	HullR	0.684	0.579	0.263	0.676	0.895	0.600	0.632	0.421	0.763	0.684	0.579
	Wald-XPD	0.900	0.816	0.684	0.900	0.947	0.840	0.894	0.700	0.920	0.900	0.830
OS	GDI	0	3	3	0	0	0	0	0	0	0	0
	Wald-IC	1	5	3	1	0	1	0	1	0	0	1
	HullP	1	5	5	0	0	0	0	0	0	0	0
	HullR	8	11	9	9	6	8	8	5	11	7	8
	Wald-XPD	1	3	4	1	0	2	1	1	1	1	1
US	GDI	7	10	9	7	5	7	6	5	9	5	8
	Wald-IC	6	10	11	6	3	8	5	5	8	5	7
	HullP	5	8	6	5	4	5	4	3	6	4	6
	HullR	2	5	5	2	1	2	1	1	2	1	2
	Wald-XPD	5	8	12	4	3	7	5	5	6	4	7

指标	方法	QM		IQ			N		JK		AD
指标	方法	0.15	0.3	0.4	0.6	0.8	500	1000	4	8	均匀分布	高阶分布
QRR	GDI	0.906	0.828	0.859	0.922	0.945	0.922	0.922	0.906	0.930	0.938	0.906
	Wald-IC	0.945	0.813	0.844	0.922	0.969	0.906	0.938	0.891	0.930	0.938	0.906
	HullP	0.930	0.852	0.875	0.945	0.953	0.938	0.953	0.938	0.945	0.953	0.930
	HullR	0.891	0.797	0.844	0.891	0.922	0.898	0.906	0.906	0.906	0.914	0.891
	Wald-XPD	0.937	0.867	0.820	0.938	0.969	0.906	0.953	0.906	0.945	0.953	0.906
TPR	GDI	0.944	0.922	0.933	0.936	0.953	0.936	0.945	0.944	0.936	0.954	0.926
	Wald-IC	0.945	0.933	0.908	0.954	0.969	0.933	0.956	0.944	0.945	0.956	0.938
	HullP	0.963	0.936	0.963	0.961	0.956	0.953	0.969	0.963	0.956	0.967	0.953
	HullR	0.936	0.911	0.953	0.927	0.930	0.927	0.944	0.956	0.922	0.944	0.926
	Wald-XPD	0.944	0.900	0.835	0.944	0.969	0.917	0.953	0.920	0.944	0.953	0.927
TNR	GDI	0.800	0.684	0.421	0.789	0.900	0.711	0.737	0.579	0.842	0.800	0.684
	Wald-IC	0.789	0.579	0.405	0.700	0.900	0.632	0.684	0.526	0.789	0.700	0.632
	HullP	0.800	0.684	0.368	0.833	0.947	0.737	0.800	0.600	0.895	0.816	0.700
	HullR	0.684	0.579	0.263	0.676	0.895	0.600	0.632	0.421	0.763	0.684	0.579
	Wald-XPD	0.900	0.816	0.684	0.900	0.947	0.840	0.894	0.700	0.920	0.900	0.830
OS	GDI	0	3	3	0	0	0	0	0	0	0	0
	Wald-IC	1	5	3	1	0	1	0	1	0	0	1
	HullP	1	5	5	0	0	0	0	0	0	0	0
	HullR	8	11	9	9	6	8	8	5	11	7	8
	Wald-XPD	1	3	4	1	0	2	1	1	1	1	1
US	GDI	7	10	9	7	5	7	6	5	9	5	8
	Wald-IC	6	10	11	6	3	8	5	5	8	5	7
	HullP	5	8	6	5	4	5	4	3	6	4	6
	HullR	2	5	5	2	1	2	1	1	2	1	2
	Wald-XPD	5	8	12	4	3	7	5	5	6	4	7

项目	原始Q矩阵
项目	${{\alpha }_{1}}$	${{\alpha }_{2}}$	${{\alpha }_{3}}$	${{\alpha }_{4}}$
1	1	0	0	0
2	0	1*	0*	0
3	0	0	1	0
4	0	0	0	1
5	1*	1	0	0^
6	1*	1	0	0
7	1*	0*	1*	0
8	1*	0*	1	0*
9	1	0	0	1*#^
10	0	1*#^	0	1
11	1*#^	1*#^	0	1
12	1*	0	1*#^	1