基于类别水平的多级计分认知诊断Q矩阵修正：相对拟合统计量视角

doi:10.3724/SP.J.1041.2020.00093

摘要/Abstract

摘要：

多级计分认知诊断模型的开发对认知诊断的发展具有重要作用, 但对于多级计分模型下的Q矩阵修正还有待研究。本研究尝试对多级计分认知诊断Q矩阵修正进行研究, 并聚焦更具诊断价值的基于项目类别水平的Q矩阵修正。将相对拟合统计量应用于多级计分认知诊断Q矩阵修正, 并与已有方法Stepwise方法( Ma & de la Torre, 2019)进行比较。研究表明：BIC方法对多级计分认知诊断模型的Q矩阵修正具有较高的模式判准率和属性判准率, 其对Q矩阵的恢复率也高于Stepwise方法, BIC方法修正后的Q矩阵与数据更加拟合; 在复杂模型中, 相对拟合指标BIC比AIC和-2LL表现更好, 在实践中, 使用者可以选择BIC法进行测验Q矩阵修正; Q矩阵修正效果受到被试人数的影响, 增加被试人数可以提高Q矩阵修正的正确率。总之, 本研究为多级计分认知诊断Q矩阵修正提供了重要的方法支持。

关键词: 认知诊断, Q矩阵, seq-GDINA, BIC

Abstract:

Cognitive diagnostic assessments (CDAs) can provide fine-grained diagnostic information about students' knowledge states, so as to help to teach in accordance with the students’ aptitude. The development of cognitive diagnosis model for polytomous response data expands the application scope of cognitive diagnostic assessment. As the basis of CDAs, Q-matrix has aroused more and more attention for the subjective tendency in Q-matrix construction that is typically performed by domain experts. Due to the subjective process of Q-matrix construction, there inevitably have some misspecifications in the Q-matrix, if left unchecked, can result in a serious negative impact on CDAs. To avoid the subjective tendency from experts and to improve the correctness of the Q-matrix, several objective Q-matrix validation methods have been proposed. Many Q-matrix validation methods have been proposed in dichotomous CDMs, however, the research of the Q-matrix validation method under polytomous CDMs is stalling lacking. To address this concern, several relative fit statistics (i.e., -2LL, AIC, BIC) were applied to the Q-matrix validation for polytomous cognitive diagnosis model in this research. The process of Q-matrix validation is as follows:
First, the reduced Q-matrix is represented by${{Q}_{r}}$, which represents a set of potential q-vectors and contains ${{2}^{K}}-1$ possible q-vectors when attributes are independent. When validating the q-vector of the first category of item j, all possible q-vectors in${{Q}_{r}}$can be used as the q-vector of the first category of item j, and the Q-matrix of remaining items remains intact. From this, the item parameters and the attribute patterns of students can be estimated, and the -2LL, AIC, and BIC can be calculated accordingly. The q-vector with the largest likelihood (or smallest AIC/BIC) is regarded as the q-vector of the first category of item j. The q-vector of the next category of the item j can also be obtained in the same way. The algorithm stops when the validated Q-matrix is same as the previous Q-matrix, or every item has been reached. In order to improve the efficiency of the method, a sequential search algorithm was proposed.
Several simulation studies were conducted to evaluate the effectiveness and practicality of these methods, and the performance of the methods in this paper was compared with the stepwise method ( Ma & de la Torre, 2019). Three experimental factors were considered in simulation studies, including sample size, Q-matrix error types and CDMs. The results show that (1) BIC method can be used for Q-matrix validation under polytomous response CDMs, and the performance of the BIC method is better than the stepwise method. (2) In general, the performance of the three methods from good to bad is the BIC method, AIC method, and -2LL method. (3) The performance of Q-matrix validation methods is affected by the sample size, and increasing the number of sample size can improve the accuracy of the Q-matrix validation.
In this study, Q-matrix validation methods for polytomous response CDMs were studied. It was found that the BIC method can be used for the Q-matrix validation under polytomous response CDMs. The method proposed in this paper can not only improve the accuracy of Q-matrix specification but also increase the model-data fit level. Besides, the data-based Q-matrix validation method can also reduce the workload of experts in Q-matrix construction and improve the classification accuracy of cognitive diagnosis.

Key words: cognitive diagnostic assessment, Q-matrix, seq-GDINA, BIC

中图分类号:

B841

汪大勋, 高旭亮, 蔡艳, 涂冬波. (2020). 基于类别水平的多级计分认知诊断Q矩阵修正：相对拟合统计量视角. 心理学报, 52(1), 93-106.

WANG Daxun, GAO Xuliang, CAI Yan, TU Dongbo. (2020). A method of Q-matrix validation for polytomous response cognitive diagnosis model based on relative fit statistics. Acta Psychologica Sinica, 52(1), 93-106.

图/表 9

参考文献 27

[1]	Akaike H . ( 1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19( 6), 716-723. doi: 10.5665/sleep.5436 URL pmid: 26446118
[2]	Chang H.-H . ( 2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80( 1), 1-20. doi: 10.1007/s11336-014-9401-5 URL pmid: 24499939
[3]	Chang H.-H., & Wang W. Y . ( 2016). ‘‘Internet plus’’ measurement and evaluation: A new way for adaptive learning. Journal of Jiangxi Normal University (Natural Science), 40( 5), 441-455.
	[ 张华华, 汪文义 . ( 2016). “互联网+”测评: 自适应学习之路. 江西师范大学学报(自然科学版), 40( 5), 441-455.]
[4]	Chen J. S .( 2017). A residual-based approach to validate Q-matrix specifications. Applied Psychological Measurement, 41( 4), 277-293. doi: 10.1177/0146621616686021 URL pmid: 29881093
[5]	Chen J., de la Torre J., & Zhang Z . ( 2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50( 2), 123-140. doi: 10.1177/0146621617707510 URL pmid: 29882533
[6]	Chiu C.-Y . ( 2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37( 8), 598-618. doi: 10.1177/0146621613488436 URL
[7]	de la Torre J . ( 2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45( 4), 343-362. doi: 10.1111/jedm.2008.45.issue-4 URL
[8]	de la Torre J . ( 2011). The generalized DINA model framework. Psychometrika, 76( 2), 179-199. doi: 10.1007/s11336-011-9207-7 URL
[9]	de la Torre J., & Chiu C.-Y . ( 2016). A general method of empirical Q-matrix validation. Psychometrika, 81( 2), 253-273. doi: 10.1007/s11336-015-9467-8 URL pmid: 25943366
[10]	Haertel E . ( 1984). An application of latent class models to assessment data. Applied Psychological Measurement, 8( 3), 333-346. doi: 10.1177/0163278719871090 URL pmid: 31462073
[11]	Hansen M . ( 2013). Hierarchical item response models for cognitive diagnosis (Unpublished doctoral dissertation). University of California at Los Angeles. doi: 10.11646/zootaxa.4273.2.7 URL pmid: 28610254
[12]	Hartz, S., Hartz S., & Roussos L . ( 2008). The fusion model for skills diagnosis: Blending theory with practicality. Educational Testing Service, Research Report, RR-08-71. Princeton, NJ: Educational Testing Service.
[13]	Lee Y.-S., Park Y. S., & Taylan D . ( 2011). A cognitive diagnostic modeling of attribute mastery in massachusetts, minnesota, and the U.S. national sample using the TIMSS 2007. International Journal of Testing, 11( 2), 144-177. doi: 10.1080/15305058.2010.534571 URL
[14]	Liu Y., Tian W., & Xin T . ( 2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41( 1), 3-26. doi: 10.3102/1076998615621293 URL
[15]	Liu Y., Xin T., Andersson B., & Tian W . ( 2019). Information matrix estimation procedures for cognitive diagnostic models. British Journal of Mathematical and Statistical Psychology, 72( 1), 18-37. doi: 10.1111/bmsp.12134 URL pmid: 29508383
[16]	Ma W., & de la Torre J . ( 2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69( 3), 253-275. doi: 10.1111/bmsp.12070 URL pmid: 27317397
[17]	Ma W., & de la Torre J . ( 2019). An empirical Q‐matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology. . doi: 10.1111/bmsp.12194 URL pmid: 31853965
[18]	Maris E . ( 1999). Estimating multiple classification latent class models. Psychometrika, 64( 2), 187-212. doi: 10.1016/j.jcrc.2018.06.012 URL pmid: 29933169
[19]	Park J. Y., Lee Y.-S., & Johnson M. S . ( 2017). An efficient standard error estimator of the DINA model parameters when analysing clustered data. International Journal of Quantitative Research in Education, 4( 1/2), 159-190. doi: 10.1504/IJQRE.2017.086507 URL
[20]	Schwarz G . ( 1978). Estimating the dimension of a model. Annals of Statistics, 6( 2), 461-464. doi: 10.1111/biom.13195 URL pmid: 31797368
[21]	Templin J. L., & Henson R. A . ( 2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11( 3), 287-305. doi: 10.1037/1082-989X.11.3.287 URL pmid: 16953706
[22]	Tu D.-B., Cai Y., & Dai H.-Q . ( 2012). A new method of Q-Matrix validation based on DINA model. Acta Psychologica Sinica, 44( 4), 558-568.
	[ 涂冬波, 蔡艳, 戴海琦 . ( 2012). 基于DINA模型的Q矩阵修正方法. 心理学报, 44( 4), 558-568.]
[23]	Tu D.-B., Cai Y., Dai H.-Q., & Ding S.-L . ( 2010). A polytomous cognitive diagnosis model: P- DINA model. Acta Psychologica Sinica, 42( 10), 1011-1020.
	[ 涂冬波, 蔡艳, 戴海琦, 丁树良 . ( 2010). 一种多级评分的认知诊断模型: P-DINA模型的开发. 心理学报, 42( 10), 1011-1020.]
[24]	von Davier M . ( 2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61( 2), 287-307. doi: 10.1348/000711007X193957 URL pmid: 17535481
[25]	Wang D.-X., Gao X.-L., Cai Y., & Tu D.-B . ( 2018). A new Q-matrix estimation method: ICC based on ideal response. Journal of Psychological Science, 41( 2), 466-474.
	[ 汪大勋, 高旭亮, 蔡艳, 涂冬波 . ( 2018). 一种非参数化的Q矩阵估计方法: ICC-IR方法开发. 心理科学, 41( 2), 466-474.]
[26]	Wang D.-X., Gao X.-L., Han Y.-T., & Tu D.-B . ( 2018). A simple and effective Q-matrix estimation method: From non-parametric perspective. Journal of Psychological Science, 41( 1), 180-188.
	[ 汪大勋, 高旭亮, 韩雨婷, 涂冬波 . ( 2018). 一种简单有效的Q矩阵估计方法开发:基于非参数化方法视角. 心理科学, 41( 1), 180-188.]
[27]	Xu G., & Shang Z .( 2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, . doi: 10.1080/01621459.2018.1476239 URL pmid: 31777410

Q	Q矩阵错误模拟规则	调整的类别	调整的属性个数	备注
Q1	${{q}_{jk}}=0\to {{q}_{jk}}=1$	$K_{jh}^{*}=1$的类别	5	属性冗余
Q2	${{q}_{jk}}=1\to {{q}_{jk}}=0$	$K_{jh}^{*}>2$的类别	5	属性缺失
Q3	${{q}_{jk}}=0\to {{q}_{jk}}=1,{{q}_{j{k}'}}=1\to {{q}_{j{k}'}}=0$	$K_{jh}^{*}>2$的类别	10	属性既冗余又缺失
Q4	${{q}_{jk}}=0\to {{q}_{jk}}=1$ ${{q}_{jk}}=1\to {{q}_{jk}}=0$${{q}_{jk}}=0\to {{q}_{jk}}=1\text{ }{{q}_{j{k}'}}=1\to {{q}_{j{k}'}}=0$	分别为Q1、Q2和Q3的类别	20	Q1、Q2和Q3的组合
Q5	10%随机调整	随机	20	调整后$1<K_{jh}^{*}<3$
Q6	20%随机调整	随机	40	调整后$1<K_{jh}^{*}<3$

Q	Q矩阵错误模拟规则	调整的类别	调整的属性个数	备注
Q1	${{q}_{jk}}=0\to {{q}_{jk}}=1$	$K_{jh}^{*}=1$的类别	5	属性冗余
Q2	${{q}_{jk}}=1\to {{q}_{jk}}=0$	$K_{jh}^{*}>2$的类别	5	属性缺失
Q3	${{q}_{jk}}=0\to {{q}_{jk}}=1,{{q}_{j{k}'}}=1\to {{q}_{j{k}'}}=0$	$K_{jh}^{*}>2$的类别	10	属性既冗余又缺失
Q4	${{q}_{jk}}=0\to {{q}_{jk}}=1$ ${{q}_{jk}}=1\to {{q}_{jk}}=0$${{q}_{jk}}=0\to {{q}_{jk}}=1\text{ }{{q}_{j{k}'}}=1\to {{q}_{j{k}'}}=0$	分别为Q1、Q2和Q3的类别	20	Q1、Q2和Q3的组合
Q5	10%随机调整	随机	20	调整后$1<K_{jh}^{*}<3$
Q6	20%随机调整	随机	40	调整后$1<K_{jh}^{*}<3$

Q-matrix	N	PMR		AMR		FPR		TPR		RMSEA
Q-matrix	N	Stepwise	BIC	Stepwise	BIC	Stepwise	BIC	Stepwise	BIC	Q_W	Q_Stepwise	Q_BIC
Q1	500	0.795	0.788	0.957	0.963	0.118	0.157	0.958	0.965	0.017	0.015	0.007
	1000	0.879	0.863	0.975	0.977	0.065	0.074	0.975	0.978	0.018	0.009	0.005
	2000	0.918	0.911	0.984	0.986	0.048	0.049	0.985	0.986	0.019	0.005	0.003
Q2	500	0.763	0.790	0.953	0.962	0.367	0.021	0.958	0.962	0.017	0.016	0.007
	1000	0.826	0.856	0.967	0.975	0.257	0.004	0.971	0.975	0.016	0.011	0.005
	2000	0.865	0.903	0.976	0.984	0.219	0.002	0.980	0.984	0.017	0.008	0.003
Q3	500	0.758	0.786	0.952	0.962	0.339	0.126	0.963	0.966	0.033	0.016	0.006
	1000	0.815	0.861	0.964	0.976	0.251	0.089	0.972	0.979	0.034	0.010	0.005
	2000	0.856	0.910	0.974	0.985	0.180	0.065	0.980	0.987	0.035	0.009	0.004
Q4	500	0.680	0.776	0.938	0.961	0.363	0.110	0.962	0.966	0.041	0.020	0.007
	1000	0.721	0.853	0.950	0.975	0.288	0.064	0.968	0.978	0.041	0.015	0.005
	2000	0.745	0.905	0.956	0.984	0.251	0.040	0.972	0.986	0.042	0.013	0.003
Q5	500	0.760	0.777	0.951	0.959	0.112	0.068	0.956	0.961	0.075	0.020	0.008
	1000	0.835	0.851	0.968	0.975	0.082	0.041	0.972	0.976	0.073	0.011	0.004
	2000	0.874	0.903	0.975	0.984	0.065	0.022	0.978	0.984	0.076	0.013	0.004
Q6	500	0.629	0.687	0.924	0.933	0.184	0.105	0.943	0.940	0.100	0.035	0.015
	1000	0.656	0.744	0.931	0.942	0.173	0.097	0.949	0.949	0.102	0.038	0.017
	2000	0.687	0.793	0.935	0.953	0.163	0.081	0.951	0.959	0.102	0.037	0.010

Q-matrix	N	PMR		AMR		FPR		TPR		RMSEA
Q-matrix	N	Stepwise	BIC	Stepwise	BIC	Stepwise	BIC	Stepwise	BIC	Q_W	Q_Stepwise	Q_BIC
Q1	500	0.795	0.788	0.957	0.963	0.118	0.157	0.958	0.965	0.017	0.015	0.007
	1000	0.879	0.863	0.975	0.977	0.065	0.074	0.975	0.978	0.018	0.009	0.005
	2000	0.918	0.911	0.984	0.986	0.048	0.049	0.985	0.986	0.019	0.005	0.003
Q2	500	0.763	0.790	0.953	0.962	0.367	0.021	0.958	0.962	0.017	0.016	0.007
	1000	0.826	0.856	0.967	0.975	0.257	0.004	0.971	0.975	0.016	0.011	0.005
	2000	0.865	0.903	0.976	0.984	0.219	0.002	0.980	0.984	0.017	0.008	0.003
Q3	500	0.758	0.786	0.952	0.962	0.339	0.126	0.963	0.966	0.033	0.016	0.006
	1000	0.815	0.861	0.964	0.976	0.251	0.089	0.972	0.979	0.034	0.010	0.005
	2000	0.856	0.910	0.974	0.985	0.180	0.065	0.980	0.987	0.035	0.009	0.004
Q4	500	0.680	0.776	0.938	0.961	0.363	0.110	0.962	0.966	0.041	0.020	0.007
	1000	0.721	0.853	0.950	0.975	0.288	0.064	0.968	0.978	0.041	0.015	0.005
	2000	0.745	0.905	0.956	0.984	0.251	0.040	0.972	0.986	0.042	0.013	0.003
Q5	500	0.760	0.777	0.951	0.959	0.112	0.068	0.956	0.961	0.075	0.020	0.008
	1000	0.835	0.851	0.968	0.975	0.082	0.041	0.972	0.976	0.073	0.011	0.004
	2000	0.874	0.903	0.975	0.984	0.065	0.022	0.978	0.984	0.076	0.013	0.004
Q6	500	0.629	0.687	0.924	0.933	0.184	0.105	0.943	0.940	0.100	0.035	0.015
	1000	0.656	0.744	0.931	0.942	0.173	0.097	0.949	0.949	0.102	0.038	0.017
	2000	0.687	0.793	0.935	0.953	0.163	0.081	0.951	0.959	0.102	0.037	0.010

Q-matrix	N	PMR		AMR		FPR		TPR		RMSEA
Q-matrix	N	Stepwise	BIC	Stepwise	BIC	Stepwise	BIC	Stepwise	BIC	Q_W	Q_Stepwise	Q_BIC
Q1	500	0.750	0.841	0.952	0.975	0.083	0.022	0.952	0.975	0.006	0.007	0.006
	1000	0.823	0.884	0.968	0.982	0.037	0.041	0.968	0.983	0.005	0.005	0.005
	2000	0.864	0.915	0.976	0.987	0.029	0.020	0.977	0.987	0.004	0.005	0.004
Q2	500	0.746	0.839	0.953	0.975	0.332	0.199	0.958	0.978	0.027	0.008	0.007
	1000	0.819	0.890	0.968	0.983	0.264	0.153	0.972	0.986	0.026	0.006	0.005
	2000	0.843	0.919	0.974	0.988	0.252	0.117	0.978	0.990	0.026	0.005	0.003
Q3	500	0.734	0.847	0.949	0.976	0.300	0.121	0.959	0.980	0.022	0.008	0.006
	1000	0.794	0.877	0.963	0.981	0.241	0.086	0.971	0.983	0.023	0.006	0.005
	2000	0.843	0.914	0.973	0.987	0.171	0.057	0.979	0.989	0.023	0.005	0.003
Q4	500	0.714	0.832	0.946	0.974	0.275	0.123	0.963	0.981	0.030	0.008	0.007
	1000	0.770	0.881	0.959	0.982	0.215	0.085	0.973	0.987	0.031	0.006	0.005
	2000	0.796	0.917	0.966	0.987	0.195	0.058	0.978	0.991	0.032	0.005	0.003
Q5	500	0.751	0.841	0.952	0.975	0.098	0.047	0.956	0.976	0.039	0.008	0.006
	1000	0.807	0.880	0.965	0.982	0.073	0.038	0.968	0.983	0.035	0.005	0.005
	2000	0.849	0.914	0.974	0.987	0.053	0.021	0.976	0.987	0.039	0.005	0.004
Q6	500	0.686	0.817	0.941	0.968	0.134	0.063	0.953	0.973	0.063	0.014	0.008
	1000	0.726	0.848	0.948	0.973	0.127	0.058	0.960	0.978	0.070	0.012	0.007
	2000	0.748	0.896	0.953	0.982	0.120	0.032	0.966	0.984	0.063	0.009	0.004