关键词: 计算机化自适应诊断测验, 在线属性向量标定, MMLE, DINA模型

Abstract: Cognitive Diagnostic Assessment (CDA) combining psychometrics and cognitive science has received increased attention recently, but it is still in its infancy (Leighton and Gierl, 2007). The CDA based on the incidence Q-matrix (Tatsuoka, 1990) is quite different from the traditional Item Response Theory. The entries in each column of the incidence Q-matrix indicate which skills and knowledge are involved in the solution of each item. So the Q-matrix plays an important role in establishing the relation between the latent knowledge states and the ideal response patterns so as to provide information about students’ cognitive strengths and weaknesses. On the other hand, CDA requires the specifications which latent attributes are measured by the test items and how these characteristics are related to one another. Leighton, Gierl and Hunka (2004) indicated the logic of Attribute Hierarchy Method (AHM) as following. Firstly, the hierarchy of attributes must be specified through protocol techniques before test item construction. Secondly, test items are developed by specialists according to the attribute hierarchy and finally, the hierarchy of attributes and item attributes are necessary to be validated. In real situations, whether the items have or have not been identified attributes before its construction, it will cost a lot of money, require more efforts to identify attributes through specialists according the above described procedure and yet can’t completely assume the correctness due to the subjectivity. As a result, invalid inferences about the student performance will be made if the attributes of some items are specified incorrectly. Chang (2010) pointed out that the on-line calibration for regular computerized adaptive testing may be one of the most effective processes. Although the great significance of Q-matrix in CDA has been widely recognized, few, if any, on-line item attribute identification has been found in the literature. So this study discussed how to implement the on-line item attribute identification in cognitive diagnostic computerized adaptive testing (CD-CAT).
The study introduced three methods of on-line item attribute identification, Maximum Likelihood Estimation (MLE), Marginal Maximum Likelihood Estimation (MMLE) and a novel method named as Inter & Diff based on intersection and difference. The new method is Cognitive Diagnostic Model-free (CDM-free). In other words, when model-data fit is not so good, the Inter-Diff method could be employed to identify attributes in the raw items on-line. Intersection and difference are set operations in Set Theory.
The simulation results showed that MMLE worked better than MLE and Inter & Diff, but MMLE was slightly sensitive to the fixed item parameters. Adaptively seeding raw items worked better than randomly seeding raw items when the correct classification rate of the entire pattern was relatively high. Especially when the attribute hierarchies are linear type, convergent type and syllogistic reasoning hierarchy, the result of Inter & Diff also could be comparable to MMLE or MLE as the number of response and the accuracy of knowledge states classification increased. However, Inter & Diff could work without assuming item parameters.
The significance of this study is essential for item bank maintenance in CD-CAT and can ease the specialists’ burden for giving the feedback information on the item attributes.

Key words: cognitive diagnostic computerized adaptive testing, on-line item attribute identification, MMLE, DINA