Please wait a minute...
Acta Psychologica Sinica    2019, Vol. 51 Issue (12) : 1386-1397     DOI: 10.3724/SP.J.1041.2019.01386
Reports of Empirical Studies |
Development of a Generalized Cognitive Diagnosis Model for polytomous responses based on Partial Credit Model
GAO Xuliang1,2,WANG Daxun1,WANG Fang2,CAI Yan1,TU Dongbo1()
1 School of Psychology Jiangxi normal university, Nanchang 330022, China
2 School of Psychology Guizhou normal university, Guiyang 550000, China
Download: PDF(558 KB)   HTML Review File (1 KB) 
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks     Supporting Info
Guide   
Abstract  

Currently, a large number of cognitive diagnosis models (CDMs) have been proposed to satisfy the demands of the cognitively diagnostic assessment. However, most existing CDMs are only suitable for dichotomously scored items. In practice, there are lager polytomously-score items/data in educational and psychological tests. Therefore, it is very necessary to develop CDMs for polytomous data.
Under the item response theory (IRT) framework, the polytomous models can be divided into three categories: (i) the cumulative probability (or graded-response) models, (ii) continuation ratios (or sequential) models, and (iii) the adjacent-category (or partial-credit) models.
At present, several efforts have been made to develop polytomous partial-credit CDMs, including the general diagnostic model (GDM; von Davier, 2008) and the partial credit DINA (PC-DINA; de la Torre, 2012) model. However, the existing polytomous partial-credit CDMs need to be improved in the following aspects: (1) These CDMs do not consider the relationship between attributes and response categories by assuming that all response categories of an item measure the same attributes. This may result in loss of diagnostic information, because different response categories could measure different attributes; (2) More importantly, the PC-DINA is based on reduced DINA model. Therefore, the current polytomous CDMs are established under strong assumptions and do not have the advantages of general cognitive diagnosis model.
The current article proposes a general partial credit diagnostic model (GPCDM) for polytomous responses with less restrictive assumptions. Item parameters of the proposed models can be estimated using the marginal maximum likelihood estimation approach via Expectation Maximization (MMLE/EM) algorithm.
Study 1 aims to examine (1) whether the EM algorithm can accurately estimate the parameters of the proposed models, and (2) whether using item level Q-matrix (referred to as the Item-Q) to analyze data generated by category level Q-matrix (referred to as the Cat-Q) will reduce the accuracy of parameter estimation. Results showed that when using Cat-Q fitting data, the maximum RMSE was less than 0.05. When the number of attributes was equal to 5 or 7, the minimum pattern match rate (PMR) was 0.9 and 0.8, respectively. These results indicated that item and person parameters could be recovered accurately based on the proposed estimation algorithm. In addition, the results also showed that when Item-Q is used to fit the data generated by Cat-Q, the estimation accuracy of both the item and person parameters could be reduced. Therefore, it is suggested that when constructing the polytomously-scored items for cognitively diagnostic assessment, the item writer should try to identify the association between attributes and categories. In the process, more diagnostic information may be extracted, which in turn helps improve the diagnostic accuracy.
The purpose of Study 2 is to apply the proposed model to the TIMSS (2007) fourth-grade mathematics assessment test to demonstrate its application and feasibility and compare with the exiting GDM and PC-DINA model. The results showed that compared with GDM and PC-DINA models, the new model had a better model fit of test-level, higher attribute reliability and better diagnostic effect.

Keywords cognitive diagnosis      polytomous CDMs      GDM model      PC-DINA model     
ZTFLH:  B841  
Corresponding Authors: Dongbo TU     E-mail: tudongbo@aliyun.com
Issue Date: 21 October 2019
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
Xuliang GAO
Daxun WANG
Fang WANG
Yan CAI
Dongbo TU
Cite this article:   
Xuliang GAO,Daxun WANG,Fang WANG, et al. Development of a Generalized Cognitive Diagnosis Model for polytomous responses based on Partial Credit Model[J]. Acta Psychologica Sinica, 2019, 51(12): 1386-1397.
URL:  
http://journal.psych.ac.cn/xlxb/EN/10.3724/SP.J.1041.2019.01386     OR     http://journal.psych.ac.cn/xlxb/EN/Y2019/V51/I12/1386
步骤 得分类别 Cat-Q Item-Q
A1 A2 A3 A1 A2 A3
减法 除法 开方 减法 除法 开方
$\sqrt{8.5/0.5-8}$ 1 1 1
步骤1: $8.5/0.5=17$ 1 0 1 0
步骤2: $17-8=9$ 2 1 0 0
步骤3: $\sqrt{9}=3$ 3 0 0 1
  
题目 得分 A1 A2 A3 A4 A5 题目 得分 A1 A2 A3 A4 A5
1 1 1 0 0 0 0 11 1 1 1 0 0 0
1 2 0 1 0 0 0 11 2 0 0 0 0 1
2 1 0 0 1 0 0 12 1 0 1 0 0 0
2 2 0 0 1 1 0 12 2 0 0 0 1 0
3 1 1 0 0 0 1 12 3 0 0 0 0 1
3 2 1 0 0 0 0 13 1 0 0 0 0 1
4 1 0 0 0 0 1 13 2 0 0 0 1 0
4 2 0 0 0 1 1 13 3 0 0 1 0 0
5 1 0 0 1 0 0 14 1 1 0 0 0 0
5 2 0 1 0 1 0 14 2 0 1 0 0 0
6 1 1 1 0 0 0 14 3 0 0 1 0 0
6 2 0 0 1 0 0 15 1 0 0 0 1 0
7 1 0 1 0 0 0 15 2 0 0 0 0 1
7 2 0 1 0 1 0 15 3 1 0 0 0 0
8 1 0 0 0 1 0 16 1 1 0 0 0 0
8 2 1 0 1 0 0 17 1 0 1 0 0 0
9 1 0 0 0 1 1 18 1 0 0 1 0 0
9 2 0 0 1 0 1 19 1 0 0 0 1 0
10 1 0 1 1 0 0 20 1 0 0 0 0 1
10 2 1 0 0 0 0
  
题目 得分 A1 A2 A3 A4 A5 A6 A7 题目 得分 A1 A2 A3 A4 A5 A6 A7
1 1 1 0 0 0 0 0 0 17 1 0 1 1 0 0 0 0
2 1 0 1 0 0 0 0 0 17 2 1 0 0 0 0 0 0
3 1 0 0 1 0 0 0 0 18 1 1 1 0 0 0 0 0
4 1 0 0 0 1 0 0 0 18 2 0 0 0 0 1 0 0
5 1 0 0 0 0 1 0 0 19 1 1 0 0 0 0 0 0
6 1 0 0 0 0 0 1 0 19 2 0 1 0 0 0 0 0
7 1 0 0 0 0 0 0 1 19 3 0 0 1 0 0 0 1
8 1 1 0 0 0 0 0 0 20 1 0 0 0 0 1 0 0
8 2 0 1 0 0 0 0 0 20 2 0 0 0 0 0 0 1
9 1 0 1 1 0 0 0 0 20 3 0 0 0 1 0 0 1
9 2 0 0 1 1 0 0 0 21 1 0 0 1 0 0 0 0
10 1 1 0 0 0 1 0 0 21 2 0 0 0 1 0 0 0
10 2 1 0 0 1 0 0 1 21 3 0 0 0 0 1 1 0
11 1 0 0 0 0 1 0 0 22 1 0 0 0 1 0 0 0
11 2 1 0 0 0 1 0 0 22 2 0 0 0 0 0 0 1
12 1 0 0 0 0 0 1 0 22 3 0 0 0 0 0 1 1
12 2 0 0 0 0 1 0 1 23 1 0 0 0 0 1 0 0
13 1 0 1 0 0 0 0 0 23 2 0 0 0 0 0 1 0
13 2 0 0 1 0 0 1 0 23 3 0 0 0 0 0 1 1
14 1 0 1 0 0 0 0 0 24 1 1 0 0 0 0 1 1
14 2 0 1 0 1 0 0 0 24 2 0 1 0 0 0 0 0
15 1 0 0 0 1 0 0 0 24 3 0 0 0 0 0 1 0
15 2 1 0 1 0 0 0 0 25 1 0 0 1 0 0 0 0
16 1 0 0 0 1 0 1 0 25 2 0 0 0 0 1 0 0
16 2 0 0 1 0 0 1 0 25 3 0 0 0 0 0 0 1
  
属性个数 测验长度 Q矩阵的类型 被试样本容量
500 1000 2000 4000
5 20 Item-Q 0.931 0.939 0.943 0.951
Cat-Q 0.942 0.948 0.949 0.954
40 Item-Q 0.991 0.993 0.995 0.996
Cat-Q 0.995 0.996 0.998 0.998
7 25 Item-Q 0.818 0.827 0.852 0.858
Cat-Q 0.864 0.866 0.868 0.872
50 Item-Q 0.977 0.979 0.981 0.986
Cat-Q 0.985 0.987 0.989 0.991
  
属性个数 测验长度 Q矩阵的类型 被试样本容量
500 1000 2000 4000
5 20 Item-Q 0.103 0.087 0.067 0.053
Cat-Q 0.043 0.028 0.022 0.015
40 Item-Q 0.101 0.086 0.065 0.052
Cat-Q 0.038 0.028 0.019 0.015
7 25 Item-Q 0.104 0.092 0.079 0.049
Cat-Q 0.042 0.032 0.020 0.014
50 Item-Q 0.108 0.089 0.070 0.047
Cat-Q 0.038 0.026 0.019 0.014
  
题目 Q矩阵的类型 题目 Q矩阵的类型
Cat-Q Item-Q Cat-Q Item-Q
1 0.025 0.095 11 0.025 0.082
2 0.032 0.092 12 0.026 0.088
3 0.033 0.069 13 0.027 0.091
4 0.036 0.081 14 0.029 0.086
5 0.024 0.086 15 0.028 0.088
6 0.034 0.082 16 0.018 0.019
7 0.033 0.083 17 0.021 0.020
8 0.023 0.079 18 0.019 0.019
9 0.034 0.069 19 0.020 0.019
10 0.024 0.084 20 0.020 0.021
  
Item Cat A1 A2 A3 A4 A5 A6 A7 A8
1 1 1 1 0 0 0 0 0 0
2 1 0 1 1 0 1 0 0 0
3 1 1 0 0 0 0 1 0 1
3 2 1 0 0 0 0 1 0 1
4 1 0 1 1 0 0 0 0 0
5 1 0 1 1 0 0 0 0 0
6 1 0 1 0 1 0 0 0 0
7 1 0 1 1 0 1 0 0 0
7 2 0 0 0 0 0 0 1 0
8 1 0 1 1 0 1 0 1 0
9 1 0 1 1 1 0 0 0 0
9 2 0 1 1 1 0 0 0 0
10 1 0 1 1 0 0 0 0 0
11 1 1 1 0 0 0 1 0 1
  
模型 拟合指标
-2LL AIC BIC
GDM 10964 11576 13017
PC-DINA 11191 11757 13089
GPCDM 10598 11312 12993
  
分数 模型 A1 A2 A3 A4 A5 A6 A7 A8 Mean
0 GDM 0.024 0.000 0.001 0.076 0.062 0.150 0.278 0.150 0.093
PC-DINA 0.548 0.108 0.387 0.204 0.432 0.470 0.382 0.470 0.375
GPCDM 0.000 0.000 0.000 0.000 0.005 0.000 0.000 0.000 0.001
14 GDM 0.786 1.000 0.999 0.980 0.971 0.671 0.975 0.671 0.881
PC-DINA 0.647 0.988 0.934 0.698 0.601 0.609 0.905 0.609 0.749
GPCDM 0.984 0.981 1.000 1.000 0.839 0.998 1.000 0.998 0.975
  
模型 A1 A2 A3 A4 A5 A6 A7 A8 Mean
GDM 0.844 0.887 0.899 0.946 0.906 0.997 0.914 0.711 0.888
PC-DINA 0.644 0.716 0.827 0.721 0.507 0.529 0.779 0.529 0.656
GPCDM 0.966 0.907 0.881 0.951 0.873 0.973 0.985 0.841 0.922
  
[1] Akaike H . (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723.
[2] Cai Y., Miao Y., & Tu D. B . (2016). The polytomously scored cognitive diagnosis computerized adaptive testing. Journal of Psychological Science, 48(10), 1338-1346.
[2] [ 蔡艳, 苗莹, 涂冬波 . (2016). 多级评分的认知诊断计算机化适应测验. 心理学报, 48(10), 1338-1346.]
[3] de la Torre J . (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.
[4] de la Torre J . (2012). Application of the DINA model framework to enhance assessment and learning. In Self- directed learning oriented assessments in the Asia-Pacific(pp. 87-103). Springer, Dordrecht.
[5] de la J Torre., & Minchen N . (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89-97.
[6] Hansen M . (2013). Hierarchical item response models for cognitive diagnosis. Unpublished doctoral dissertation. University of California at Los Angeles.
[7] Leighton J. P., & Gierl M. J . (2007). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 27(2), 3-16.
[8] Ma W., & de la Torre J . (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275.
[9] Mellenbergh G. J . (1995). Conceptual notes on models for discrete polytomous item responses. Applied Psychological Measurement, 19(1), 91-100.
[10] Samejima F . (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, 34(S1), 1-97.
[11] Schwarz G . (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464.
[12] Templin J. L., & Bradshaw L . (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251-275.
[13] Templin J. L., & Henson R. A . (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.
[14] Tu D. B., Cai Y., Dai H. Q., & Ding S. L . (2010). A polytomous cognitive diagnosis model: P-DINA model. Acta Psychologica Sinica, 42(10), 1011-1020.
[14] [ 涂冬波, 蔡艳, 戴海琦, 丁树良 . (2010). 一种多级评分的认知诊断模型: P-DINA 模型的开发. 心理学报, 42(10), 1011-1020.]
[15] Tu D. B., Cai Y., Dai H. Q., & Ding S. L . (2012). A new multiple-strategies cognitive diagnosis model: The MSCD method. Acta Psychologica Sinica, 44(11), 1547-1553.
[15] [ 涂冬波, 蔡艳, 戴海琦, 丁树良 . (2012). 一种多策略认知诊断方法: MSCD方法的开发. 心理学报, 44(11), 1547-1553.]
[16] Tu D., Zheng C., Cai Y., Gao X., & Wang D . (2017). A polytomous model of cognitive diagnostic assessment for graded data. International Journal of Testing, 18(3), 231-252.
[17] Tutz G . (1997). Sequential models for ordered responses. In Handbook of modern item response theory(pp. 139-152). Springer, New York, NY.
[18] van Der Ark L. A . (2001). Relationships and properties of polytomous item response theory models. Applied Psychological Measurement, 25(3), 273-282.
[19] von Davier M . (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287-307.
[1] ZHAN Peida,YU Zhaohui,LI Feiming,WANG Lijun. Using a multi-order cognitive diagnosis model to assess scientific literacy[J]. Acta Psychologica Sinica, 2019, 51(6): 734-746.
[2] LIU Yanlou; XIN Tao; LI Lingqing; TIAN Wei; LIU Xiaoxiao. An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix[J]. Acta Psychologica Sinica, 2016, 48(5): 588-598.
[3] ZHAN Peida; BIAN Yufang; WANG Lijun. Factors affecting the classification accuracy of reparametrized diagnostic classification models for expert-defined polytomous attributes[J]. Acta Psychologica Sinica, 2016, 48(3): 318-330.
[4] ZHAN Peida; CHEN Ping; BIAN Yufang. Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis[J]. Acta Psychologica Sinica, 2016, 48(10): 1347-1356.
[5] CAI Yan; MIAO Ying; TU Dongbo. The polytomously scored cognitive diagnosis computerized adaptive testing[J]. Acta Psychologica Sinica, 2016, 48(10): 1338-1346.
[6] KANG Chunhua; REN Ping; ZENG Pingfei. Nonparametric Cognitive Diagnosis: A Cluster Diagnostic Method  Based on Grade Response Items[J]. Acta Psychologica Sinica, 2015, 47(8): 1077-1088.
[7] TANG Xiaojuan; DING Shuliang; YU Zonghuo. Application of Rough Set Theory in Item Cognitive Attribute Identification[J]. Acta Psychologica Sinica, 2015, 47(7): 950-962.
[8] ZHAN Peida; LI Xiaomin; WANG Wen-Chung; BIAN Yufang; WANG Lijun. The Multidimensional Testlet-Effect Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2015, 47(5): 689-701.
[9] CAI Yan; TU Dongbo. Extension of Cognitive Diagnosis Models Based on the Polytomous Attributes Framework and Their Q-matrices Designs[J]. Acta Psychologica Sinica, 2015, 47(10): 1300-1308.
[10] WANG Zhuoran; GUO Lei; BIAN Yufang. Comparison of DIF Detecting Methods in Cognitive Diagnostic Test[J]. Acta Psychologica Sinica, 2014, 46(12): 1923-1932.
[11] TU Dongbo;CAI Yan;DAI Haiqi. Comparison and Selection of Five Noncompensatory Cognitive Diagnosis Models Based on Attribute Hierarchy Structure[J]. Acta Psychologica Sinica, 2013, 45(2): 243-252.
[12] CAI Yan;TU Dongbo;DING Shuliang. A Simulation Study to Compare Five Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2013, 45(11): 1295-1304.
[13] TU Dong-Bo,CAI Yan,DAI Hai-Qi. A New Method of Q-matrix Validation Based on DINA Model[J]. , 2012, 44(4): 558-568.
[14] TU Dong-Bo;CAI Yan;DAI Hai-Qi;DING Shu-Liang. A New Multiple-Strategies Cognitive Diagnosis Model: the MSCD Method[J]. Acta Psychologica Sinica, 2012, 44(11): 1547-1553.
[15] DING Shu-Liang;MAO Meng-Meng;WANG Wen-Yi;LUO Fen;CUI Ying. Evaluating the Consistency of Test Items Relative to the Cognitive Model for Educational Cognitive Diagnosis[J]. Acta Psychologica Sinica, 2012, 44(11): 1535-1546.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech