Please wait a minute...
Acta Psychologica Sinica
|
Comparison of DIF Detecting Methods in Cognitive Diagnostic Test
WANG Zhuoran1; GUO Lei1; BIAN Yufang1,2
(1 National Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing 100875, China) (2 National Cooperative Innovation Center for Assessment and Improvement of Basic Education Quality, Beijing Normal University, Beijing 100875, China)
Download: PDF(490 KB)   Review File (1 KB) 
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  

DIF detecting is an important issue when using cognitive diagnostic tests in practice. MH method, SIBTEST method and Wald test have been introduced into cognitive diagnostic test DIF detection. However, all of them have some limitations. As Logistic Regression is not based on certain model, has a pretty good performance in detecting DIF in IRT test, and could distinguish uniform DIF from non-uniform ones, it is predictable that Logistic Regression (LR) could make up some of the flaws the methods used in cognitive diagnostic test have. The performance of LR was compared with that of MH method and Wald test. Matching criteria, DIF type, DIF size and sample size were also considered. In this simulation study, data was generated using HO-DINA model. When detecting DIF using MH method and LR, 3 kinds of matching criteria were used. Sum score was computed by summing up right answers of each examinee; q was calculated with 2PL model; KS was calculated with 3 different cognitive diagnostic methods, which are DINA model, RSM method and AHM method. Wald test could be directly applied to DINA model. The 4 kinds of DIF are s increases, g increases, s and g increase simultaneously, and s increases while g decreases. Two levels of DIF size are 0.05 and 0.1. Two levels of sample size are 500 examinees per group and 1000 examinees per group. Here are the results: (1) LR did a great job in cognitive diagnostic test DIF detection with a pretty high power and low type I error; (2) LR was not constrained by cognitive diagnostic models, thus it can use KS estimated by whichever cognitive diagnostic methods; (3) LR can distinguish uniform DIF from non-uniform DIF, and the power and type I error are fairly good; (4) Using KS as matching criteria in cognitive diagnostic test DIF detection can provide ideal power and type I error; (5) With the increase of DIF size and sample size, power grew significantly while type I error rate did not change. LR has a satisfying performance in cognitive diagnostic test DIF detection with a high power and a low and stable type I error rate. KS should be the ideal matching criteria in cognitive diagnostic test DIF detection. In the long run, the unique characters of DIF in cognitive diagnostic test should be explored, and pertinence DIF detecting methods should be developed.

Keywords cognitive diagnosis      differential item functioning      logistic regression      uniform DIF      non-uniform DIF     
Corresponding Authors: BIAN Yufang, E-mail: bianyufang66@126.com    
Issue Date: 25 December 2014
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
WANG Zhuoran
GUO Lei
BIAN Yufang
Cite this article:   
WANG Zhuoran,GUO Lei,BIAN Yufang. Comparison of DIF Detecting Methods in Cognitive Diagnostic Test[J]. Acta Psychologica Sinica, 10.3724/SP.J.1041.2014.01923
URL:  
http://journal.psych.ac.cn/xlxb/EN/10.3724/SP.J.1041.2014.01923     OR     http://journal.psych.ac.cn/xlxb/EN/Y2014/V46/I12/1923
[1] LIU Yanlou; XIN Tao; LI Lingqing; TIAN Wei; LIU Xiaoxiao. An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix[J]. Acta Psychologica Sinica, 2016, 48(5): 588-598.
[2] ZHAN Peida; BIAN Yufang; WANG Lijun. Factors affecting the classification accuracy of reparametrized diagnostic classification models for expert-defined polytomous attributes[J]. Acta Psychologica Sinica, 2016, 48(3): 318-330.
[3] ZHAN Peida; CHEN Ping; BIAN Yufang. Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis[J]. Acta Psychologica Sinica, 2016, 48(10): 1347-1356.
[4] CAI Yan; MIAO Ying; TU Dongbo. The polytomously scored cognitive diagnosis computerized adaptive testing[J]. Acta Psychologica Sinica, 2016, 48(10): 1338-1346.
[5] KANG Chunhua; REN Ping; ZENG Pingfei. Nonparametric Cognitive Diagnosis: A Cluster Diagnostic Method  Based on Grade Response Items[J]. Acta Psychologica Sinica, 2015, 47(8): 1077-1088.
[6] TANG Xiaojuan; DING Shuliang; YU Zonghuo. Application of Rough Set Theory in Item Cognitive Attribute Identification[J]. Acta Psychologica Sinica, 2015, 47(7): 950-962.
[7] ZHAN Peida; LI Xiaomin; WANG Wen-Chung; BIAN Yufang; WANG Lijun. The Multidimensional Testlet-Effect Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2015, 47(5): 689-701.
[8] CAI Yan; TU Dongbo. Extension of Cognitive Diagnosis Models Based on the Polytomous Attributes Framework and Their Q-matrices Designs[J]. Acta Psychologica Sinica, 2015, 47(10): 1300-1308.
[9] ZHANG Xun;LI Lingyan;LIU Hongyun;SUN Yan. Applying IRT_ΔB Procedure and Adapted LR Procedure to Detect DIF in Tests with Matrix Sampling[J]. Acta Psychologica Sinica, 2013, 45(8): 921-934.
[10] TU Dongbo;CAI Yan;DAI Haiqi. Comparison and Selection of Five Noncompensatory Cognitive Diagnosis Models Based on Attribute Hierarchy Structure[J]. Acta Psychologica Sinica, 2013, 45(2): 243-252.
[11] CAI Yan;TU Dongbo;DING Shuliang. A Simulation Study to Compare Five Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2013, 45(11): 1295-1304.
[12] LIU Hong-Yun,LI Chong,ZHANG Ping-Ping,LUO Fang. Testing Measurement Equivalence of Categorical Items’ Threshold/Difficulty Parameters: A Comparison of CCFA and (M)IRT Approaches[J]. Acta Psychologica Sinica, 2012, 44(8): 1124-1136.
[13] TU Dong-Bo,CAI Yan,DAI Hai-Qi. A New Method of Q-matrix Validation Based on DINA Model[J]. , 2012, 44(4): 558-568.
[14] TU Dong-Bo;CAI Yan;DAI Hai-Qi;DING Shu-Liang. A New Multiple-Strategies Cognitive Diagnosis Model: the MSCD Method[J]. Acta Psychologica Sinica, 2012, 44(11): 1547-1553.
[15] DING Shu-Liang;MAO Meng-Meng;WANG Wen-Yi;LUO Fen;CUI Ying. Evaluating the Consistency of Test Items Relative to the Cognitive Model for Educational Cognitive Diagnosis[J]. Acta Psychologica Sinica, 2012, 44(11): 1535-1546.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech