Abstract:

Cognitive diagnosis is an important topic in modern psychometric area. Now more than 70 cognitive diagnosis models (CDMs) are developed. There are some questions among these models: (1) When the attribute hierarchy structure is known, how to choose the most suitable model? (2) When the attribute hierarchy structure is unknown, and cognitive diagnosis is required, how to do it? These problems seems especially more puzzled for the practice workers. This paper only paid main attention on three international popular models. Therefore, five cognitive diagnosis models (RSM, AHM_A, GDD, DINA and DINA_HC) were compared corresponding to the above questions from psychometric opinion. In this paper, Monte Carlo simulation study was used. Although the number of slips and the hierarchy structure are two important factors that affect the performance on corrected match ratio of cognitive diagnosis, this study would pay attention on other three factors: the distribution of cognitive pattern, the sample size, the number of attributes. The findings identified: (1) When the characteristic of data was known, focusing on specific factor, the five methods had different advantages. a) For the distributions of cognitive pattern, although they have different effects on different methods, the same conclusion could find that the performance on negative bias distribution was the best, and that of DINA_HC and DINA were better than the rest methods on any discussed distributions. b) Considering the sample size, the performance of GDD with small scale assessment (100/20, persons/items)was the best one; with medium and large scale assessment (1000/60, 5000/100, persons/items), the performance of DINA_HC and DINA were better than the rest c) For the number of attributes, the more the attributes are the worse the performance will be. But for the methods, the performance the performance of DINA_HC and DINA were also better than the rest. All these reflected that the most suitable method could be adapt from the three methods: GDD, DINA_HC and DINA, corresponding to the real scenario. And the RSM was the worst cognitive diagnosis method. (2) When the characteristic of data was unknown, an unstructured attribute hierarchy is treated as a coarsened version of a structured one, and the DINA method has a similar performance under unstructured hierarchy with the GDD, DINA_HC methods under structured hierarchy. Thus if the hierarchy structure could not be identified clearly, and the test Q matrix was clear, then the DINA could be adapt.

Key words: cognitive attributes, cognitive diagnosis, cognitive diagnosis models, attributes Match Ratio