Please wait a minute...
   2011, Vol. 43 Issue (08) : 964-976     DOI:
|
On-Line Item Attribute Identification in Cognitive Diagnostic Computerized Adaptive Testing
WANG Wen-Yi;DING Shu-Liang;YOU Xiao-Feng
(1 School of Psychology, Jiangxi Normal University, Nanchang 330027, China)
(2 School of Computer and Information Engineering, Jiangxi Normal University, Nanchang 330027, China)
(3 Foreign Language Teaching and Research Press, Beijing 100089, China)
Download: PDF(476 KB)  
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  Cognitive Diagnostic Assessment (CDA) combining psychometrics and cognitive science has received increased attention recently, but it is still in its infancy (Leighton and Gierl, 2007). The CDA based on the incidence Q-matrix (Tatsuoka, 1990) is quite different from the traditional Item Response Theory. The entries in each column of the incidence Q-matrix indicate which skills and knowledge are involved in the solution of each item. So the Q-matrix plays an important role in establishing the relation between the latent knowledge states and the ideal response patterns so as to provide information about students’ cognitive strengths and weaknesses. On the other hand, CDA requires the specifications which latent attributes are measured by the test items and how these characteristics are related to one another. Leighton, Gierl and Hunka (2004) indicated the logic of Attribute Hierarchy Method (AHM) as following. Firstly, the hierarchy of attributes must be specified through protocol techniques before test item construction. Secondly, test items are developed by specialists according to the attribute hierarchy and finally, the hierarchy of attributes and item attributes are necessary to be validated. In real situations, whether the items have or have not been identified attributes before its construction, it will cost a lot of money, require more efforts to identify attributes through specialists according the above described procedure and yet can’t completely assume the correctness due to the subjectivity. As a result, invalid inferences about the student performance will be made if the attributes of some items are specified incorrectly. Chang (2010) pointed out that the on-line calibration for regular computerized adaptive testing may be one of the most effective processes. Although the great significance of Q-matrix in CDA has been widely recognized, few, if any, on-line item attribute identification has been found in the literature. So this study discussed how to implement the on-line item attribute identification in cognitive diagnostic computerized adaptive testing (CD-CAT).
The study introduced three methods of on-line item attribute identification, Maximum Likelihood Estimation (MLE), Marginal Maximum Likelihood Estimation (MMLE) and a novel method named as Inter & Diff based on intersection and difference. The new method is Cognitive Diagnostic Model-free (CDM-free). In other words, when model-data fit is not so good, the Inter-Diff method could be employed to identify attributes in the raw items on-line. Intersection and difference are set operations in Set Theory.
The simulation results showed that MMLE worked better than MLE and Inter & Diff, but MMLE was slightly sensitive to the fixed item parameters. Adaptively seeding raw items worked better than randomly seeding raw items when the correct classification rate of the entire pattern was relatively high. Especially when the attribute hierarchies are linear type, convergent type and syllogistic reasoning hierarchy, the result of Inter & Diff also could be comparable to MMLE or MLE as the number of response and the accuracy of knowledge states classification increased. However, Inter & Diff could work without assuming item parameters.
The significance of this study is essential for item bank maintenance in CD-CAT and can ease the specialists’ burden for giving the feedback information on the item attributes.
Keywords cognitive diagnostic computerized adaptive testing      on-line item attribute identification      MMLE      DINA     
Corresponding Authors: DING Shu-Liang   
Issue Date: 30 August 2011
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
WANG Wen-Yi
DING Shu-Liang
YOU Xiao-Feng
Cite this article:   
WANG Wen-Yi,DING Shu-Liang,YOU Xiao-Feng. On-Line Item Attribute Identification in Cognitive Diagnostic Computerized Adaptive Testing[J]. , 2011, 43(08): 964-976.
URL:  
http://journal.psych.ac.cn/xlxb/EN/      OR     http://journal.psych.ac.cn/xlxb/EN/Y2011/V43/I08/964
[1] GUO Haiying; CHEN Lihua; YE Zhi; PAN Jin; LIN Danhua. Characteristics of peer victimization and the bidirectional relationship between peer victimization and internalizing problems among rural-to-urban migrant children in China: A longitudinal study[J]. Acta Psychologica Sinica, 2017, 49(3): 336-348.
[2] GUO Lei; ZHENG Chanjin; BIAN Yufang; SONG Naiqing; XIA Lingxiang. New item selection methods in cognitive diagnostic computerized adaptive testing: Combining item discrimination indices[J]. Acta Psychologica Sinica, 2016, 48(7): 903-914.
[3] CHEN Leni; WANG Zhen; LUO Ben Nanfeng; LUO Zhengxue. Leader-subordinate extraversion fit and subordinate work engagement: Based on dominance complementarity theory[J]. Acta Psychologica Sinica, 2016, 48(6): 710-721.
[4] HUANG Bijuan; FENG Hongmin; SI Jiwei; ZHANG Jie; WANG Xiangyan. Dual-task coordination and task presentation mode influence arithmetic strategy execution in adults: Evidence from computational estimation[J]. Acta Psychologica Sinica, 2016, 48(6): 671-683.
[5] ZHAN Peida; BIAN Yufang; WANG Lijun. Factors affecting the classification accuracy of reparametrized diagnostic classification models for expert-defined polytomous attributes[J]. Acta Psychologica Sinica, 2016, 48(3): 318-330.
[6] ZHAO Ying; CHENG Yahua; WU Xinchun; NGUYEN Thi Phuong. The reciprocal relationship between morphological awareness and vocabulary knowledge among Chinese children: A longitudinal study[J]. Acta Psychologica Sinica, 2016, 48(11): 1434-1444.
[7] CAI Yan; MIAO Ying; TU Dongbo. The polytomously scored cognitive diagnosis computerized adaptive testing[J]. Acta Psychologica Sinica, 2016, 48(10): 1338-1346.
[8] AI Jiru; ZHANG Hongduan; SI Jiwei; LU Chun; ZHANG Tangzheng. The effects of presenting mode, reaction order of dual task on adults’ arithmetic strategy choice and execution[J]. Acta Psychologica Sinica, 2016, 48(10): 1248-1257.
[9] ZHAN Peida; LI Xiaomin; WANG Wen-Chung; BIAN Yufang; WANG Lijun. The Multidimensional Testlet-Effect Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2015, 47(5): 689-701.
[10] LV Jie; ZHANG Gang. The Mechanisms of Knowledge Heterogeneity on Creativity of Knowledge Teams: An Interactive Cognition Perspective[J]. Acta Psychologica Sinica, 2015, 47(4): 533-544.
[11] YU Xiaofeng; LUO Zhaosheng; GAO Chunlei; LI Yujun; WANG Rui; WANG Yutong. An Item Attribute Specification Method Based On the Likelihood D2 Statistic[J]. Acta Psychologica Sinica, 2015, 47(3): 417-426.
[12] YU Xiaofeng; LUO Zhaosheng; QIN Chunying; GAO Chunlei; LI Yujun. Joint Estimation of Model Parameters and Q-Matrix Based on Response Data[J]. Acta Psychologica Sinica, 2015, 47(2): 273-282.
[13] XU Peng; QI Lu; XIONG Jian; YE Haosheng. A Regression Analysis Model of Ordinal Variable to Psychological Data[J]. Acta Psychologica Sinica, 2015, 47(12): 1520-1528.
[14] DAI Buyun; ZHANG Minqiang; JIAO Can; LI Guangming; ZHU Huawei; ZHANG Wenyi. Item Selection Using the Multiple-Strategy RRUM Based on CD-CAT[J]. Acta Psychologica Sinica, 2015, 47(12): 1511-1519.
[15] CAI Yan; TU Dongbo. Extension of Cognitive Diagnosis Models Based on the Polytomous Attributes Framework and Their Q-matrices Designs[J]. Acta Psychologica Sinica, 2015, 47(10): 1300-1308.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech