基于分部评分模型思路的多级评分认知诊断模型开发

doi:10.3724/SP.J.1041.2019.01386

摘要/Abstract

摘要：

基于分部评分模型的思路, 本文提出了一般化的分部评分认知诊断模型(General Partial Credit Diagnostic Model, GPCDM), 与国际上已有的基于分部评分模型思路的多级评分模型GDM (von Davier, 2008)和PC-DINA (de la Torre, 2012)相比, GPCDM的Q矩阵定义更加灵活, 项目参数的约束条件更少。Monte Carlo实验研究表明, GPCDM模型的参数估计精度指标RMSE介于[0.015, 0.043], 表明估计精度尚可; TIMSS (2007)实证数据应用研究表明, 与GDM和PC-DINA模型相比, GPCDM与该数据的拟合度更好, 并且使用GPCDM分析该数据的诊断效果也更优。总之, 本研究提供了一种约束条件更少、功能更为强大的多级评分认知诊断模型。

关键词: 认知诊断, 多级评分认知诊断模型, GDM, PC-DINA

Abstract:

Currently, a large number of cognitive diagnosis models (CDMs) have been proposed to satisfy the demands of the cognitively diagnostic assessment. However, most existing CDMs are only suitable for dichotomously scored items. In practice, there are lager polytomously-score items/data in educational and psychological tests. Therefore, it is very necessary to develop CDMs for polytomous data.
Under the item response theory (IRT) framework, the polytomous models can be divided into three categories: (i) the cumulative probability (or graded-response) models, (ii) continuation ratios (or sequential) models, and (iii) the adjacent-category (or partial-credit) models.
At present, several efforts have been made to develop polytomous partial-credit CDMs, including the general diagnostic model (GDM; von Davier, 2008) and the partial credit DINA (PC-DINA; de la Torre, 2012) model. However, the existing polytomous partial-credit CDMs need to be improved in the following aspects: (1) These CDMs do not consider the relationship between attributes and response categories by assuming that all response categories of an item measure the same attributes. This may result in loss of diagnostic information, because different response categories could measure different attributes; (2) More importantly, the PC-DINA is based on reduced DINA model. Therefore, the current polytomous CDMs are established under strong assumptions and do not have the advantages of general cognitive diagnosis model.
The current article proposes a general partial credit diagnostic model (GPCDM) for polytomous responses with less restrictive assumptions. Item parameters of the proposed models can be estimated using the marginal maximum likelihood estimation approach via Expectation Maximization (MMLE/EM) algorithm.
Study 1 aims to examine (1) whether the EM algorithm can accurately estimate the parameters of the proposed models, and (2) whether using item level Q-matrix (referred to as the Item-Q) to analyze data generated by category level Q-matrix (referred to as the Cat-Q) will reduce the accuracy of parameter estimation. Results showed that when using Cat-Q fitting data, the maximum RMSE was less than 0.05. When the number of attributes was equal to 5 or 7, the minimum pattern match rate (PMR) was 0.9 and 0.8, respectively. These results indicated that item and person parameters could be recovered accurately based on the proposed estimation algorithm. In addition, the results also showed that when Item-Q is used to fit the data generated by Cat-Q, the estimation accuracy of both the item and person parameters could be reduced. Therefore, it is suggested that when constructing the polytomously-scored items for cognitively diagnostic assessment, the item writer should try to identify the association between attributes and categories. In the process, more diagnostic information may be extracted, which in turn helps improve the diagnostic accuracy.
The purpose of Study 2 is to apply the proposed model to the TIMSS (2007) fourth-grade mathematics assessment test to demonstrate its application and feasibility and compare with the exiting GDM and PC-DINA model. The results showed that compared with GDM and PC-DINA models, the new model had a better model fit of test-level, higher attribute reliability and better diagnostic effect.

Key words: cognitive diagnosis, polytomous CDMs, GDM model, PC-DINA model

中图分类号:

B841

高旭亮, 汪大勋, 王芳, 蔡艳, 涂冬波. (2019). 基于分部评分模型思路的多级评分认知诊断模型开发. 心理学报, 51(12), 1386-1397.

GAO Xuliang, WANG Daxun, WANG Fang, CAI Yan, TU Dongbo. (2019). Development of a Generalized Cognitive Diagnosis Model for polytomous responses based on Partial Credit Model. Acta Psychologica Sinica, 51(12), 1386-1397.

图/表 10

参考文献 19

[1]	Akaike H . (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723.
[2]	Cai Y., Miao Y., & Tu D. B . (2016). The polytomously scored cognitive diagnosis computerized adaptive testing. Journal of Psychological Science, 48(10), 1338-1346.
	[ 蔡艳, 苗莹, 涂冬波 . (2016). 多级评分的认知诊断计算机化适应测验. 心理学报, 48(10), 1338-1346.]
[3]	de la Torre J . (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.
[4]	de la Torre J . (2012). Application of the DINA model framework to enhance assessment and learning. In Self- directed learning oriented assessments in the Asia-Pacific(pp. 87-103). Springer, Dordrecht.
[5]	de la J Torre., & Minchen N . (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89-97.
[6]	Hansen M . (2013). Hierarchical item response models for cognitive diagnosis. Unpublished doctoral dissertation. University of California at Los Angeles.
[7]	Leighton J. P., & Gierl M. J . (2007). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 27(2), 3-16.
[8]	Ma W., & de la Torre J . (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275.
[9]	Mellenbergh G. J . (1995). Conceptual notes on models for discrete polytomous item responses. Applied Psychological Measurement, 19(1), 91-100.
[10]	Samejima F . (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, 34(S1), 1-97.
[11]	Schwarz G . (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464.
[12]	Templin J. L., & Bradshaw L . (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251-275.
[13]	Templin J. L., & Henson R. A . (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.
[14]	Tu D. B., Cai Y., Dai H. Q., & Ding S. L . (2010). A polytomous cognitive diagnosis model: P-DINA model. Acta Psychologica Sinica, 42(10), 1011-1020.
	[ 涂冬波, 蔡艳, 戴海琦, 丁树良 . (2010). 一种多级评分的认知诊断模型: P-DINA 模型的开发. 心理学报, 42(10), 1011-1020.]
[15]	Tu D. B., Cai Y., Dai H. Q., & Ding S. L . (2012). A new multiple-strategies cognitive diagnosis model: The MSCD method. Acta Psychologica Sinica, 44(11), 1547-1553.
	[ 涂冬波, 蔡艳, 戴海琦, 丁树良 . (2012). 一种多策略认知诊断方法: MSCD方法的开发. 心理学报, 44(11), 1547-1553.]
[16]	Tu D., Zheng C., Cai Y., Gao X., & Wang D . (2017). A polytomous model of cognitive diagnostic assessment for graded data. International Journal of Testing, 18(3), 231-252.
[17]	Tutz G . (1997). Sequential models for ordered responses. In Handbook of modern item response theory(pp. 139-152). Springer, New York, NY.
[18]	van Der Ark L. A . (2001). Relationships and properties of polytomous item response theory models. Applied Psychological Measurement, 25(3), 273-282.
[19]	von Davier M . (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287-307.

步骤	得分类别	Cat-Q			Item-Q
		A1	A2	A3	A1	A2	A3
		减法	除法	开方	减法	除法	开方
$\sqrt{8.5/0.5-8}$					1	1	1
步骤1: $8.5/0.5=17$	1	0	1	0
步骤2: $17-8=9$	2	1	0	0
步骤3: $\sqrt{9}=3$	3	0	0	1

步骤	得分类别	Cat-Q			Item-Q
		A1	A2	A3	A1	A2	A3
		减法	除法	开方	减法	除法	开方
$\sqrt{8.5/0.5-8}$					1	1	1
步骤1: $8.5/0.5=17$	1	0	1	0
步骤2: $17-8=9$	2	1	0	0
步骤3: $\sqrt{9}=3$	3	0	0	1

属性个数	测验长度	Q矩阵的类型	被试样本容量
属性个数	测验长度	Q矩阵的类型	500	1000	2000	4000
5	20	Item-Q	0.931	0.939	0.943	0.951
		Cat-Q	0.942	0.948	0.949	0.954
	40	Item-Q	0.991	0.993	0.995	0.996
		Cat-Q	0.995	0.996	0.998	0.998
7	25	Item-Q	0.818	0.827	0.852	0.858
		Cat-Q	0.864	0.866	0.868	0.872
	50	Item-Q	0.977	0.979	0.981	0.986
		Cat-Q	0.985	0.987	0.989	0.991

属性个数	测验长度	Q矩阵的类型	被试样本容量
属性个数	测验长度	Q矩阵的类型	500	1000	2000	4000
5	20	Item-Q	0.931	0.939	0.943	0.951
		Cat-Q	0.942	0.948	0.949	0.954
	40	Item-Q	0.991	0.993	0.995	0.996
		Cat-Q	0.995	0.996	0.998	0.998
7	25	Item-Q	0.818	0.827	0.852	0.858
		Cat-Q	0.864	0.866	0.868	0.872
	50	Item-Q	0.977	0.979	0.981	0.986
		Cat-Q	0.985	0.987	0.989	0.991

属性个数	测验长度	Q矩阵的类型	被试样本容量
属性个数	测验长度	Q矩阵的类型	500	1000	2000	4000
5	20	Item-Q	0.103	0.087	0.067		0.053
		Cat-Q	0.043	0.028	0.022		0.015
	40	Item-Q	0.101	0.086	0.065		0.052
		Cat-Q	0.038	0.028	0.019		0.015
7	25	Item-Q	0.104	0.092	0.079		0.049
		Cat-Q	0.042	0.032	0.020		0.014
	50	Item-Q	0.108	0.089	0.070		0.047
		Cat-Q	0.038	0.026	0.019		0.014