多级属性Q矩阵的验证与估计

doi:10.3724/SP.J.1041.2022.01403

摘要/Abstract

摘要：

多级属性是将诊断测验中传统的二值(即两种水平, 通常定义为0和1)属性定义为多值(多个水平可以为0, 1, …), 它不但可以描述学生对于知识属性是否掌握, 而且可以描述学生在属性上的掌握程度, 这样使得诊断测验能提供给被试更丰富的知识掌握详情。本文将适用于二级属性Q矩阵的统计量(S统计量)拓展到多级属性下的Q矩阵验证和估计, 在两种常见的条件下, 设计了两种估计算法：联合估计算法和在线估计算法。模拟实验结果表明：联合估计算法适用于对专家界定的初始Q矩阵进行验证, 当初始Q矩阵中包含较少的错误时, 通过联合估计算法有很大可能恢复正确的Q矩阵; 在线估计算法适用于对“新项目”进行属性向量和项目参数的在线标定, 基于一定数量的“基础项目”, 在线估计算法对于新项目的估计也能达到较满意的成功率。实证数据分析则进一步展示了该方法的使用。

关键词: 多级属性, Q矩阵, p-DINA模型, S统计量

Abstract:

Cognitive diagnosis has recently gained prominence in educational assessment, psychiatric evaluation, and many other disciplines. Generally, entries in the Q-matrix of traditional cognitive diagnostic tests are binary (two levels, defined as 0 and 1). Polytomous attributes (multi-levels, defined as 0, 1, …), particularly those defined as part of the test development process, can provide additional diagnostic information. Compared to binary attributes, polytomous attributes can not only describe the student's knowledge profile, but can provide more extensive details.

As we all know, Q-matrix impacts the accuracy of cognitive diagnostic assessment greatly. Research on the effect of parameter estimation and classification accuracy caused by the error in Q-matrix already existed, and it turned out that Q-matrix gotten from expert definition or experience was more easily subject to be affected by subjective factors, lead to a misspecified Q-matrix. Under this circumstance, it’s urgently needed to find more objective polytomous-attribute Q-matrix verification and inference methods.

The present research proposes the verification and estimation of expert-defined polytomous attribute Q-matrix based on the polytomous deterministic inputs, noisy, ‘‘and’’ gate (p-DINA) model. We intend to extend the methods adapted to binary Q-matrix verification and estimation to polytomous attribute Q-matrix, and the proposed methods which can be used in different conditions are joint estimation and online estimation. Simulation results show that: the joint estimation algorithm can be applied to the Q-matrix validation which needs an initial Q-matrix defined by experts, the online estimation algorithm can be applied to online estimate the “new items” based on a certain number of “based items”. Under the various settings in the simulations, the two estimation algorithms can recover the correct polytomous-attribute Q-matrix at a high probability. Empirical study also indicates that the two proposed algorithms can be applied in Q-matrix validation or estimation for CDA with polytomous attributes.

Key words: polytomous attribute, Q-matrix, p-DINA model, S statistics

中图分类号:

B841

秦春影, 喻晓锋. (2022). 多级属性Q矩阵的验证与估计. 心理学报, 54(11), 1403-1415.

QIN Chunying, YU Xiaofeng. (2022). Validation and estimation of expert-defined Q-matrix with polytomous attribute. Acta Psychologica Sinica, 54(11), 1403-1415.

图/表 19

参考文献 49

[1]	Cai, Y., & Tu, D. B. (2015). Extension of cognitive diagnosis models based on the polytomous attributes framework and their Q-matrices designs. Acta Psychologica Sinica, 47(10), 1300-1310. doi: 10.3724/SP.J.1041.2015.01300 URL
	[蔡艳, 涂冬波. (2015). 属性多级化的认知诊断模型拓展及其Q矩阵设计. 心理学报, 47(10), 1300-1310.]
[2]	Chen, J. S., & de la Torre, J. (2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37(6), 419-437. doi: 10.1177/0146621613479818 URL
[3]	Chen, Y. X., Liu, J. C., Xu, G. J., & Ying, Z. L. (2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850-866. doi: 10.1080/01621459.2014.934827 URL
[4]	Chung, M.-T. (2014). Estimating the Q-matrix for cognitive diagnosis models in a Bayesian framework. (Unpublished doctoral dissertation), Columbia University, New York.
[5]	DeCarlo, L. T. (2012). Recognizing Uncertainty in the Q-Matrix via a Bayesian Extension of the DINA Model. Applied Psychological Measurement, 36(6), 447-468. doi: 10.1177/0146621612449069 URL
[6]	de La Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362. doi: 10.1111/j.1745-3984.2008.00069.x URL
[7]	de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115-130.
[8]	de la Torre, J. (2011). The generalized dina model framework. Psychometrika, 76(2), 179-199. doi: 10.1007/s11336-011-9207-7 URL
[9]	de la Torre, J., & Chiu, C. Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253-273. doi: 10.1007/s11336-015-9467-8 pmid: 25943366
[10]	Ding, S. L., Luo, F., Wang, W. Y., & Xiong, J. H. (2019). The designing cognitive diagnostic test with dichotomous scoring. Journal of Jiangxi Normal University (Natural Science), 43(5), 441-447.
	[丁树良, 罗芬, 汪文义, 熊建华. (2019). 0-1 评分认知诊断测验设计. 江西师范大学学报(自然科学版), 43(5), 441-447.]
[11]	Fung, W.-K. (1993). Unmasking outliers and leverage points: A confirmation. Journal of the American Statistical Association, 88(422), 515-519. doi: 10.1080/01621459.1993.10476302 URL
[12]	Gu, Y. Q., Liu, J. C., Xu, G. J., & Ying, Z. L. (2018). Hypothesis testing of the Q-matrix, Psychometrika, 83(3), 515-537. doi: 10.1007/s11336-018-9629-6 URL
[13]	Gu, Y. Q., & Xu, G. J. (2021). Sufficient and Necessary Conditions for the Identifiability of the Q-matrix. Statistica Sinica, 31, 449-472.
[14]	Haberman, S. J., von Davier, M., & Lee, Y.-H. (2008). Comparison of multidimensional item response models: Multivariate normal ability distributions versus multivariate polytomous ability distributions (ETS Research Report no. RR-08-45). Princeton, NJ: Educational Testing Service.
[15]	Huang, Y., Luo, F., Xiong, J. H., Ding, S. L., & Gan, D. W. (2019). The multiple-strategy cognitive diagnosis method with polytomous scoring. Journal of Jiangxi Normal University (Natural Science), 43(4), 376-381.
	[黄玉, 罗芬, 熊建华, 丁树良, 甘登文. (2019). 多级评分多策略认知诊断方法. 江西师范大学学报(自然科学版), 43(4), 376-381.]
[16]	Karelitz, T. M. (2004). Ordered category attribute coding framework for cognitive assessments. (Unpublished doctoral dissertation), University of Illinois at Urbana-Champaign.
[17]	Leighton, J. P., & Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.
[18]	Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3), 205-237. doi: 10.1111/j.1745-3984.2004.tb01163.x URL
[19]	Li, C. C., Ma, C. C., & Xu, G. J. (2022). Learning large Q-matrix by restricted Boltzmann machines. Psychometrika. https://doi.org/10.1007/s11336-021-09828-4.
[20]	Liu, J. C., Xu, G. J., & Ying, Z. L. (2012). Data driven learning of Q matrix. Applied Psychological Measurement, 36(7), 548-564. doi: 10.1177/0146621612456591 URL
[21]	Liu, J. C., Xu, G. J., & Ying, Z. L. (2013). Theory of self-learning Q-matrix. Bernoulli, 19(5A), 1790-1817.
[22]	Liu, N., Liu, X. L., Li, J. J., Zeng, P. F., Yu, X. J., & Kang, C. H. (2021). Constructing a non-parametric Q-matrix correction method based on Manhattan distance. Journal of Jiangxi Normal University (Natural Science), 45(6), 634-641.
	[刘娜, 刘芯伶, 李俊杰, 曾平飞, 俞向军, 康春花. (2021). 基于曼哈顿距离构建非参数Q矩阵修正方法. 江西师范大学学报(自然科学版), 45(6), 634-641.]
[23]	Luo, Z. S. (2019). Fundamentals of cognitive diagnostic assessment. Beijing Normal University publishing group.
[24]	罗照盛. (2019). 认知诊断评价理论基础. 北京师范大学出版集团.
	Ma, W., & de la Torre, J. (2019). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73(1), 142-163. doi: 10.1111/bmsp.12156 URL
[25]	Peng, Y. F., Luo, Z. S., Li, Y. J., Gao, C. L. (2018). Optimization of test design for examinees with different cognitive structures. Acta Psychologica Sinica, 50(1), 130-140. doi: 10.3724/SP.J.1041.2018.00130 URL
	[彭亚风, 罗照盛, 李喻骏, 高椿雷. (2018). 不同认知结构被试的测验设计模式. 心理学报, 50(1), 130-140.]
[26]	Peng, Y. F., Luo, Z. S., Yu, X. F., Gao, C. L., Li, Y, J. (2016). The optimization of test design in Cognitive Diagnostic Assessment. Acta Psychologica Sinica, 48(12), 1600-1611. doi: 10.3724/SP.J.1041.2016.01600 URL
	[彭亚风, 罗照盛, 喻晓锋, 高椿雷, 李喻骏. (2016). 认知诊断评价中测验结构的优化设计. 心理学报, 48(12), 1600-1611.]
[27]	Qin, C. Y., Jia, S., Fang, X. W., & Yu, X. F. (2020). Relationship validation among items and attributes, Journal of Statistical Computation and Simulation, 90(18), 3360-3375 doi: 10.1080/00949655.2020.1802592 URL
[28]	Qin, C. Y., Zhang, L., Qiu, D., Huang, L., Geng, T., Jiang, H.,... Zhou, J. (2015). Model identification and Q-matrix incremental inference in cognitive diagnosis. Knowledge- Based Systems, 86, 66-76. doi: 10.1016/j.knosys.2015.05.024 URL
[29]	Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.
[30]	Shang, Z. R., Erosheva, E. A., Xu, G. J. (2021). Partial-mastery cognitive diagnosis models. The Annals of Applied Statistics, 15 (3), 1529-1555.
[31]	Sun, J. N., Xin, T., Zhang, S. M., & de la Torre, J. (2013). A polytomous extension of the generalized distance discriminating method. Applied Psychological Measurement, 37(7), 503-521. doi: 10.1177/0146621613487254 URL
[32]	Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method. Routledge.
[33]	Templin, J. L. (2004). Generalized linear mixed proficiency models for cognitive diagnosis. (Unpublished doctoral dissertation), University of Illinois at Urbana-Champaign.
[34]	Templin, J. L., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251-275. doi: 10.1007/s00357-013-9129-4 URL
[35]	Templin, J. L., Bradshaw, L. (2014). The use and misuse of psychometric models. Psychometrika, 79 (2), 347-354. doi: 10.1007/s11336-013-9364-y pmid: 24478023
[36]	Tu, D. B., & Cai, Y. (2015). The development of CD-CAT with polytomous attributes. Acta Psychologica Sinica, 47(11), 1405-1414. doi: 10.3724/SP.J.1041.2015.01405 URL
	[涂冬波, 蔡艳. (2015). 基于属性多级化的认知诊断计算机化自适应测验设计与实现. 心理学报, 47(11), 1405-1414.]
[37]	von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287-307. doi: 10.1348/000711007X193957 URL
[38]	von Davier, M., & Lee, Y.-S. (2019). Handbook of diagnostic classification models. Cham: Springer International Publishing.
[39]	Wang, D. X., Cai, Y, & Tu, D. B. (2020). Q-matrix estimation methods for cognitive diagnosis models: Based on partial known Q-matrix, Multivariate Behavioral Research, 1-13. https://doi.org/10.1080/00273171.2020.1746901.
[40]	Xiang, R. (2013). Nonlinear penalized estimation of true Q-Matrix in cognitive diagnostic models. (Unpublished doctoral dissertation), Columbia University, New York.
[41]	Xu, G.-J. (2013). Statistical inference for diagnostic classification models. (Unpublished doctoral dissertation), Columbia University, New York.
[42]	Yu, X. F., & Cheng, Y. (2020). Data-driven Q-matrix validation using a residual‐based statistic in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology, 73(1), 145-179. doi: 10.1111/bmsp.12191 URL
[43]	Yu, X. F., Luo, Z. S., Gao, C. L., Li, Y. J., Wang, R., & Wang, Y. T. (2015a). An item attribute specification method based on the likelihood D2 statistic. Acta Psychologica Sinica, 47(3), 417-426. doi: 10.3724/SP.J.1041.2015.00417 URL
	[喻晓锋, 罗照盛, 高椿雷, 李喻骏, 王睿, 王钰彤. (2015a). 使用似然比D2统计量的题目属性定义方法. 心理学报, 47(3), 417-426.]
[44]	Yu, X. F., Luo, Z. S., Qin, C. Y., Gao, C. L., & Li, Y. J. (2015b). Joint estimation of model parameters and Q-matrix based on response data. Acta Psychologica Sinica, 47(2), 273-282.
	[喻晓锋, 罗照盛, 秦春影, 高椿雷, 李喻骏. (2015b). 基于作答数据的模型参数和Q矩阵联合估计. 心理学报, 47(2), 273-282.]
[45]	Yu, X. F., Ma, Y. F., Luo, Z. S., & Qin, C. Y. (2021). The attribute hierarchical structure learning based on K2 algorithm. Journal of Jiangxi Normal University (Natural Science), 45(4), 376-383.
	[喻晓锋, 马奕帆, 罗照盛, 秦春影. (2021). 基于K2算法的属性层级结构学习研究. 江西师范大学学报(自然科学版), 45(4), 376-383.]
[46]	Yuan, K.-H., & Zhong, X. (2008). Outliers, leverage observations, and influential cases in factor analysis: Using robust procedures to minimize their effect. Sociological Methodology, 38(1), 329-368. doi: 10.1111/j.1467-9531.2008.00198.x URL
[47]	Zhan, P. D., Bian, Y. F., Wang, L. J. (2016). Factors affecting the classification accuracy of reparametrized diagnostic classification models for expert-defined polytomous attributes. Acta Psychologica Sinica, 48(3), 318-330. doi: 10.3724/SP.J.1041.2016.00318 URL
	[詹沛达, 边玉芳, 王立君. (2016). 重参数化的多分属性诊断分类模型及其判准率影响因素. 心理学报, 48(3), 318-330.]
[48]	Zhan, P. D., Wang, W., Li, X. M. (2020). A partial mastery, higher-order latent structural model for polytomous attributes in cognitive diagnostic assessments. Journal of Classification, 37, 328-351. doi: 10.1007/s00357-019-09323-7 URL
[49]	Zhang, Y. L., Zhao, B., & Tao, J. H. (2021). The study on students' cognitive state based on fuzzy cognitive diagnostic framework. Journal of Jiangxi Normal University (Natural Science), 45(5), 452-459.
	[张玉柳, 赵波, 陶金洪. (2021). 基于模糊认知诊断模型的学生认知状态研究. 江西师范大学学报(自然科学版), 45(5),452-459.]

包含的错误项目数	被试人数
	1000			2000			4000
	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)
3	94	2.05	197397.50	98	2.04	205128.46	98	2.00	211650.67
4	92	2.14	210386.75	95	2.12	208827.46	96	2.14	213588.22
5	81	2.30	234271.81	94	2.19	211649.67	94	2.21	215590.22

包含的错误项目数	被试人数
	1000			2000			4000
	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)
3	94	2.05	197397.50	98	2.04	205128.46	98	2.00	211650.67
4	92	2.14	210386.75	95	2.12	208827.46	96	2.14	213588.22
5	81	2.30	234271.81	94	2.19	211649.67	94	2.21	215590.22

包含的错误项目数	被试人数
	1000			2000			4000
	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)
3	36	3.13	89182.33	46	3.02	101401.32	54	2.92	109542.61
4	21	3.63	90511.47	27	3.44	111399.52	38	3.33	115674.36
5	18	3.89	135365.82	22	3.62	138115.65	25	3.47	144921.76

包含的错误项目数	被试人数
	1000			2000			4000
	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)
3	36	3.13	89182.33	46	3.02	101401.32	54	2.92	109542.61
4	21	3.63	90511.47	27	3.44	111399.52	38	3.33	115674.36
5	18	3.89	135365.82	22	3.62	138115.65	25	3.47	144921.76

包含的错误项目数	被试人数
	1000			2000			4000
	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)	成功率(%)	平均迭代次数	平均执行时间(s)
3	91	2.94	217999.39	97	2.38	207354.22	98	2.43	212017.28
4	90	3.17	221085.75	95	2.58	209615.68	96	2.68	214643.81
5	80	3.75	254841.29	89	3.32	242336.01	93	3.58	287900.52