Q-matrix estimation (validation) methods for cognitive diagnosis

doi:10.3724/SP.J.1042.2021.02272

Abstract

Abstract:

The Q-matrix, which represents important item characteristics by mapping attributes to items has been proved to be the core factor affecting the accuracy of cognitive diagnostic classification. It is of great value to study the methods of Q-matrix estimation (validation). First, the existing methods of Q-matrix estimation and validation are classified into 1) parameterized methods in the CDM perspective, including item differentiation, model-data fit index and parameter estimation; and 2) non-parametric methods in the statistical perspective, including the distance between observed and expected response vector, abnormal responses index and factor analysis. Then, these methods are introduced in terms of differences and relations, characteristics and performance. The advantages and disadvantages of each method are commented. At last, several future research directions are proposed. It is necessary to compare the Q-matrix estimation (validation) methods systematically under complex test conditions. It is also of vital importance to propose Q-matrix estimation (validation) methods by combining multiple thoughts and ways based on the calibration of knowledge state and parameter estimation error. It is meaningful to further study the Q-matrix estimation (validation) methods for polytomous scoring items, mixed test models, polytomous scoring attributes, unknown number of attributes and even continuous Q-matrix.

Key words: cognitive diagnosis models, Q-matrix, Q-matrix estimation (validation) methods, model-data fit, parameter estimation

CLC Number:

B841

LI Jia, MAO Xiuzhen, ZHANG Xueqin. Q-matrix estimation (validation) methods for cognitive diagnosis[J]. Advances in Psychological Science, 2021, 29(12): 2272-2280.

Figures/Tables 1

References 43

[1]	杭丹丹. (2020). 多级计分认知诊断评估中的Q矩阵验证方法与应用研究 (硕士学位论文). 江西师范大学, 南昌,
[2]	刘芯伶. (2020). 多级评分情境下Q矩阵修正的非参数方法 (硕士学位论文). 浙江师范大学, 金华,
[3]	刘彦楼, 辛涛, 李令青, 田伟, 刘笑笑. (2016). 改进的认知诊断模型项目功能差异检验方法--基于观察信息矩阵的Wald统计量. 心理学报, 48(5), 588-598.
[4]	刘彦楼, 张倩萌, 郑宗军, 尹昊. (2019). 认知诊断模型中项目水平模型比较统计量的健壮性. 心理科学, 42, 1251-1259.
[5]	涂冬波, 蔡艳, 戴海琦. (2012). 基于DINA模型的Q矩阵修正方法. 心理学报, 44(4), 558-568.
[6]	汪大勋, 高旭亮, 蔡艳, 涂冬波. (2018). 一种非参数化的Q矩阵估计方法: ICC-IR方法开发. 心理科学, 41(2), 466-474.
[7]	汪大勋, 高旭亮, 蔡艳, 涂冬波. (2019). 一种广义的认知诊断Q矩阵修正新方法. 心理科学, 42(4), 988-996.
[8]	汪大勋, 高旭亮, 蔡艳, 涂冬波. (2020). 基于类别水平的多级计分认知诊断Q矩阵修正: 相对拟合统计量视角. 心理学报, 52(1), 93-106.
[9]	汪大勋, 高旭亮, 韩雨婷, 涂冬波. (2018). 一种简单有效的Q矩阵估计方法开发: 基于非参数化方法视角. 心理科学, 41(1), 180-188.
[10]	汪文义, 高朋, 宋丽红, 汪腾. (2020). 带噪音预处理的改进探索性Q矩阵标定方法. 江西师范大学学报(自然科学版), 44(2), 136-141.
[11]	汪文义, 宋丽红, 丁树良. (2015). 基于探索性因素分析的Q矩阵标定方法. 江西师范大学学报(自然科学版), 39(2), 138-144+170.
[12]	汪文义, 宋丽红, 丁树良. (2018). 基于可达阵的一种Q矩阵标定方法. 心理科学, 41(4), 968-975.
[13]	喻晓锋, 罗照盛, 高椿雷, 李喻骏, 王睿, 王钰彤. (2015). 使用似然比D-2统计量的题目属性定义方法. 心理学报, 47(3), 417-426.
[14]	喻晓锋, 罗照盛, 秦春影, 高椿雷, 李喻骏. (2015). 基于作答数据的模型参数和Q矩阵联合估计. 心理学报, 47(2), 273-282.
[15]	杨亚坤, 朱仕浩, 刘芯伶. (2020). 基于项目拟合统计量RMSEA的Q矩阵估计方法. 心理技术与应用, 8(1), 51-59.
[16]	Barnes, T.(2010). Novel derivation and application of skill matrices:The Q-matrix method. In C. Ramero, S. Vemtora, M. Pechemizkiy, & R. S. J. de Baker (Eds.), Handbook of educational data mining (pp.159-172). Boca Raton, FL: Chapman & Hall.
[17]	Chen, J. S.(2017). A residual-based approach to validate Q-matrix specifications. Applied Psychological Measurement, 41(4), 277-293. doi: 10.1177/0146621616686021 URL
[18]	Chen, J. S., de la Torre, J., & Zhang, Z.(2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123-140. doi: 10.1111/jedm.2013.50.issue-2 URL
[19]	Chen, Y. H., Culpepper, S. A., Chen, Y. G., & Douglas, J.(2018). Bayesian estimation of the DINA Q-matrix. Psychometrika, 83(1), 89-108. doi: 10.1007/s11336-017-9579-4 URL
[20]	Chen, Y, Statistical analysis of Q-matrix based diagnostic classificatio, X. Liu, J. C., Xu, G. J., & Ying, Z. L.(2015). n models. Journal of the American Statistical Association, 110(510), 850-866. doi: 10.1080/01621459.2014.934827 URL
[21]	Chiu, C.-Y.(2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598-618. doi: 10.1177/0146621613488436 URL
[22]	Chung, M. T.(2019). A Gibbs sampling algorithm that estimates the Q-matrix for the DINA model. Journal of Mathematical Psychology, 93, 102275. doi: 10.1016/j.jmp.2019.07.002 URL
[23]	Close, C. N.(2012). An exploratory technique for finding the Q-matrix for the DINA model in cognitive diagnostic assessment: Combining theory with data (Unpublished doctorial dissertation). University of Minnesota.
[24]	Cui, Y.(2007). The hierarchy consistency index: A person -fit statistic for the attribute hierarchy method (Unpublished doctorial dissertation). University of Alberta, Edmonton.
[25]	DeCarlo, L. T.(2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36(6), 447-468. doi: 10.1177/0146621612449069 URL
[26]	de la Torre, J.(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343-362. doi: 10.1111/jedm.2008.45.issue-4 URL
[27]	de la Torre, J.(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115-130.
[28]	de la Torre, J., & Chiu, C.-Y.(2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253- 273. doi: 10.1007/s11336-015-9467-8 URL
[29]	Kang, C. H., Yang, Y. K., & Zeng, P. H.(2019). Q-matrix refinement based on item fit statistic RMSEA. Applied Psychological Measurement, 43(7), 527-542. doi: 10.1177/0146621618813104 URL
[30]	Liu, J. C., Xu, G. J., & Ying, Z. L.(2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548-564. doi: 10.1177/0146621612456591 URL
[31]	Liu, Y. L., Andersson, B., Xin, T., Zhang, H. Y., & Wang, L. L.(2019). Improved Wald statistics for item-level model comparison in diagnostic classification models. Applied Psychological Measurement, 43, 402-414. doi: 10.1177/0146621618798664 URL
[32]	Liu, Y. L., Tian, W., & Xin, T.(2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41, 3-26.
[33]	Liu, Y. L., Xin, T., Andersson, B., & Tian, W.(2019). Information matrix estimation procedures for cognitive diagnostic models. British Journal of Mathematical and Statistical Psychology, 72, 18-37. doi: 10.1111/bmsp.2019.72.issue-1 URL
[34]	Liu, Y. L., Yin, H., Xin, T., Shao, L. C., & Yuan, L.(2019). A comparison of differential item functioning detection methods in cognitive diagnostic models. Frontiers in Psychology, 10, 1137. doi: 10.3389/fpsyg.2019.01137 URL
[35]	Ma, W. C., & de la Torre, J.(2020). An empirical Q-matrix validation method for the sequential generalized DINA model. The British Journal of Mathematical and Statistical Psychology, 73(1), 142-163. doi: 10.1111/bmsp.v73.1 URL
[36]	McKinley, R. L., & Mills, C. N.(1985). A comparison of several goodness-of-fit statistics. Applied Psychological Measurement, 9(1), 49-57. doi: 10.1177/014662168500900105 URL
[37]	Rupp, A. A., & Templin, J.(2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78-96. doi: 10.1177/0013164407301545 URL
[38]	Templin, J. L., & Henson, R. A.(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305. doi: 10.1037/1082-989X.11.3.287 URL
[39]	Wang, D. X., Cai, Y., & Tu, D. B.(2020). Q-matrix estimation methods for cognitive diagnosis models: Based on partial known Q-matrix.
[40]	Wang, W. Y., Song, L. H., Ding, S. L., Meng, Y. R., Cao, C. X., & Jie, Y. J.(2018). An EM-based method for Q-matrix validation. Applied Psychological Measurement, 42(6), 446-459. doi: 10.1177/0146621617752991 URL
[41]	Xiang, R.(2013). Columbia University, New York.
[42]	Xu, G. J., & Shang, Z. R.(2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284-1295. doi: 10.1080/01621459.2017.1340889 URL
[43]	Yu, X. F., & Cheng, Y.(2020). Data-driven Q-matrix validation using a residual-based statistic in cognitive diagnostic assessment. British Journal of Mathematical and Statistical Psychology. 73, 145-179. doi: 10.1111/bmsp.v73.s1 URL

	分类标准	特点	方法	实验目的
参数化方法	最优项目特征	项目区分度	$\delta $法、${{\varsigma }^{2}}$法、stepwise法	修正
	最优项目特征	属性区分度	$\gamma $法	修正
	最优模型数据拟合	绝对拟合指标	S统计量、多级计分的S统计量方法	修正
			非线性惩罚估计法	估计
			RMSEA法、加权残差R法、残差方法	修正
			似然D2统计量方法	估计
		相对拟合指标	-2LL、AIC、BIC方法	修正
			正则化极大似然估计方法	估计
			TS法、LR-S法、LR-E法	估计
	参数估计	基于EM算法	MLE和MMLE方法	修正
		基于贝叶斯的方法	EAP方法	修正
		基于贝叶斯的方法	MCMC方法	估计
非参数方法	最小观察反应与理想反应的距离	依据欧氏距离、海明距离、曼哈顿距离	欧氏距离法、多级计分的欧氏距离法	修正
			海明距离方法	修正
			曼哈顿距离方法	估计
	最小异常反应指标	依据被试在父级、子级和同级项目上的反应	ICC法	估计
		依据被试在父级、子级和同级项目上的反应	多级计分的ICC法	修正
		依据KS与项目反应关系	ID法	修正
	因素分析	将Q矩阵元素视作项目与潜在属性间的因子结构	主成分分析法	估计
	因素分析	将Q矩阵元素视作项目与潜在属性间的因子结构	四分相关矩阵法	估计

	分类标准	特点	方法	实验目的
参数化方法	最优项目特征	项目区分度	$\delta $法、${{\varsigma }^{2}}$法、stepwise法	修正
	最优项目特征	属性区分度	$\gamma $法	修正
	最优模型数据拟合	绝对拟合指标	S统计量、多级计分的S统计量方法	修正
			非线性惩罚估计法	估计
			RMSEA法、加权残差R法、残差方法	修正
			似然D2统计量方法	估计
		相对拟合指标	-2LL、AIC、BIC方法	修正
			正则化极大似然估计方法	估计
			TS法、LR-S法、LR-E法	估计
	参数估计	基于EM算法	MLE和MMLE方法	修正
		基于贝叶斯的方法	EAP方法	修正
		基于贝叶斯的方法	MCMC方法	估计
非参数方法	最小观察反应与理想反应的距离	依据欧氏距离、海明距离、曼哈顿距离	欧氏距离法、多级计分的欧氏距离法	修正
			海明距离方法	修正
			曼哈顿距离方法	估计
	最小异常反应指标	依据被试在父级、子级和同级项目上的反应	ICC法	估计
		依据被试在父级、子级和同级项目上的反应	多级计分的ICC法	修正
		依据KS与项目反应关系	ID法	修正
	因素分析	将Q矩阵元素视作项目与潜在属性间的因子结构	主成分分析法	估计
	因素分析	将Q矩阵元素视作项目与潜在属性间的因子结构	四分相关矩阵法	估计