Please wait a minute...
Acta Psychologica Sinica
|
Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis
ZHAN Peida; CHEN Ping; BIAN Yufang
(Collaborative Innovation Center of Assessment toward Basic Education Quality, Beijing Normal University, Beijing 100875, China)
Download: PDF(531 KB)   Review File (1 KB) 
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  

Traditional testing methods, such as classical testing theory or unidimensional item response theory models (UIRMs), typically provide a single sum score or overall ability. Advances in psychometrics have focused on measuring multiple dimensions of ability to provide more detailed and refined feedback for students. In recent years, cognitive diagnostic models (CDMs) have received great attention, particularly in the areas of educational and psychological measurement. The outcome of a DCM analysis is a profile of a set of attributes, α, also called a latent class, for each person; this provides cognitive diagnostic information about distinct skills underlying a test that students mastery or non-mastery. During the same period, another kind of models, multidimensional IRT models (MIRTMs), which also can provide fine-grained information about students’ strengths and weaknesses in the learning process were neglected. MIRTMs are different from CDMs in that latent variables in MIRTMs are continuous (namely, latent traits; θ) rather than categorical (typically binary). However, categorical variables in CDMs may be too rough to describe students’ skills when compared with the continuous latent traits in MIRTMs. Diagnostic measurement is the process of analyzing data from a diagnostic assessment for the purpose of making classification-based decisions. Currently, all testing method that have cognitive diagnostic function require substantive information about the attributes involved in specific items. Especially for CDMs, a confirmatory matrix that indicating which latent variables are required for an item, often referred to as Q matrix , is a essential term to analysis response data. Actually, such confirmatory matrices also exist in some multidimensional IRT models (MIRTMs), such as the scoring matrix in multidimensional random coefficients multinomial logit model. Therefore, it can be deduced that when MIRTMs are formulated in a confirmatory model defined by Q matrix, may also have diagnostic potential. Although some articles have noticed that viewpoint (e.g., Embretson & Yang, 2013; Stout, 2007; Wang & Nydick, 2015), no one really explored the diagnostic potential of confirmatory MIRTMs (C-MIRTMs). The main reason can be deduced that latent traits in MIRTMs are continuous, which can not be directly used to make classification-based diagnostic decisions. No matter MIRTMs or CDMs, multidimensional models normally can be specified into compensatory and non-compensatory models due to the relationship among dimensions. In compensatory models, students with high level on one dimension can compensate for lower levels on the other dimensions. Conversely, non-compensatory models assume that every dimensions are independent or partially independent with each others. Comparatively speaking, compensatory models are more general than non-compensatory models. Thus, only two compensatory models were concerned in this study, multidimensional 2-parameter logistic model (M2PLM) and linear logistic model (LLM) respectively, due to space limited. To explore the cognitive diagnostic function of MIRTMs, a confirmatory compensatory M2PLM (CC-M2PLM) were presented by introducing Q matrix in the item response function of M2PLM firstly. Then a cutoff point (CP) was used to transform estimated latent traits in CC-M2PLM to categorical variables (namely, trans-border attributes). This transformation step can be done after data analysis, thus two kinds of analysis results can be reported simultaneously: continuous latent traits and categorical trans-border attributes. Therefore, a suitable CP is very important, because of different CP will lead to different classification results. A simple pilot study was done to found the suitable CP: a test created with the CC-M2PLM but estimated with the LLM revealed that the LLM approximately divided the latent traits distribution in half, with a value of zero in IRT scale being the location of where masters (α = 1 if θ > 0) and non-masters (α = 0 if θ ≤ 0) were set. According to the result of pilot study, the CP was set equal to 0 for all dimensions (i.e., CPk = 0). Parameters in CC-M2PLM and LLM can be estimated by the mirt and CDM packages in R respectively. In simulation study, a series of simulations were conducted to evaluate cognitive diagnostic function of CC-M2PLM. The response data was generated by LLM, which can be treated as a diagnostic measurement dataset. CC-M2PLM and LLM were all used to fit that dataset, and results showed that the pattern (profile) correct classification ratio (PCCR) and the attribute correct classification ratio (ACCR) of trans-border attributes (from CC-M2PLM) and estimated attributes (from LLM) are almost same, the extent of most differences are smaller than 1%. Results of simulation study indicated that CC-M2PLM can be used to diagnostic measurement and its cognitive diagnostic function was as good as that of LLM. Finally, two empirical examples of diagnostic measurement were given to demonstrate applications and implications of the CC-M2PLM.

Keywords item response theory      multidimensional item response theory      cognitive diagnostic models      cognitive diagnosis      Q matrix      confirmatory factor analysis     
Corresponding Authors: CHEN Ping, E-mail: pchen@bnu.edu.cn; BIAN Yufang, E-mail: bianyufang66@126.com    
Issue Date: 25 October 2016
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
ZHAN Peida
CHEN Ping
BIAN Yufang
Cite this article:   
ZHAN Peida,CHEN Ping,BIAN Yufang. Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis[J]. Acta Psychologica Sinica, 10.3724/SP.J.1041.2016.01347
URL:  
http://journal.psych.ac.cn/xlxb/EN/10.3724/SP.J.1041.2016.01347     OR     http://journal.psych.ac.cn/xlxb/EN/Y2016/V48/I10/1347
[1] CHEN Ping. Two new online calibration methods for computerized adaptive testing[J]. Acta Psychologica Sinica, 2016, 48(9): 1184-1198.
[2] MENG Xiangbin; TAO Jian; CHEN Shali. Warm’sweighted maximum likelihood estimation of latent trait in the four-parameter logistic model[J]. Acta Psychologica Sinica, 2016, 48(8): 1047-1056.
[3] KANG Chunhua; REN Ping; ZENG Pingfei. The influence factors of grade response cluster diagnostic method[J]. Acta Psychologica Sinica, 2016, 48(7): 891-902.
[4] LIU Yanlou; XIN Tao; LI Lingqing; TIAN Wei; LIU Xiaoxiao. An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix[J]. Acta Psychologica Sinica, 2016, 48(5): 588-598.
[5] ZHAN Peida; BIAN Yufang; WANG Lijun. Factors affecting the classification accuracy of reparametrized diagnostic classification models for expert-defined polytomous attributes[J]. Acta Psychologica Sinica, 2016, 48(3): 318-330.
[6] WANG Wenyi;SONG Lihong;DING Shuliang. Classification accuracy and consistency indices for complex decision rules in multidimensional item response theory[J]. Acta Psychologica Sinica, 2016, 48(12): 1612-1624.
[7] PENG Yafeng; LUO Zhaosheng; YU Xiaofeng; GAO Chunlei; LI Yujun. The optimization of test design in Cognitive Diagnostic Assessment[J]. Acta Psychologica Sinica, 2016, 48(12): 1600-1611.
[8] CAI Yan; MIAO Ying; TU Dongbo. The polytomously scored cognitive diagnosis computerized adaptive testing[J]. Acta Psychologica Sinica, 2016, 48(10): 1338-1346.
[9] KANG Chunhua; REN Ping; ZENG Pingfei. Nonparametric Cognitive Diagnosis: A Cluster Diagnostic Method  Based on Grade Response Items[J]. Acta Psychologica Sinica, 2015, 47(8): 1077-1088.
[10] TANG Xiaojuan; DING Shuliang; YU Zonghuo. Application of Rough Set Theory in Item Cognitive Attribute Identification[J]. Acta Psychologica Sinica, 2015, 47(7): 950-962.
[11] ZHAN Peida; LI Xiaomin; WANG Wen-Chung; BIAN Yufang; WANG Lijun. The Multidimensional Testlet-Effect Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2015, 47(5): 689-701.
[12] CAI Yan; TU Dongbo. Extension of Cognitive Diagnosis Models Based on the Polytomous Attributes Framework and Their Q-matrices Designs[J]. Acta Psychologica Sinica, 2015, 47(10): 1300-1308.
[13] WANG Zhuoran; GUO Lei; BIAN Yufang. Comparison of DIF Detecting Methods in Cognitive Diagnostic Test[J]. Acta Psychologica Sinica, 2014, 46(12): 1923-1932.
[14] YAO Ruosong;ZHAO Baonan;LIU Ze;MIAO Qunying. The Application of Many-Facet Rasch Model in Leaderless Group Discussion[J]. Acta Psychologica Sinica, 2013, 45(9): 1039-1049.
[15] LIU Yue;LIU Hongyun. Comparison of MIRT Linking Methods for Different Common Item Designs[J]. Acta Psychologica Sinica, 2013, 45(4): 466-480 .
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech