关键词: 多级评分, 非参数认知诊断, 聚类分析, K-means算法

Abstract:

Examinations help students learn more efficiently by filling their learning gaps. To achieve this goal, we have to differentiate students who have from those who have not mastered a set of attributes as measured by the test through cognitive diagnostic assessment. K-means cluster analysis, being a nonparametric cognitive diagnosis method requires the Q-matrix only, which reflects the relationship between attributes and items. This does not require the estimation of the parameters, so is independent of sample size, simple to operate, and easy to understand. Previous research use the sum score vectors or capability scores vector as the clustering objects. These methods are only adaptive for dichotomous data. Structural response items are, however, the main type used in examinations, particularly as required in recent reforms. On the basis of previous research, this paper puts forward a method to calculate a capability matrix reflecting the mastery level on skills and is applicable to grade response items. Our study included four parts. First, we introduced the K-means cluster diagnosis method which has been adapted for dichotomous data. Second, we expanded the K-means cluster diagnosis method for grade response data (GRCDM). Third, in Part Two, we investigated the performance of the method introduced using a simulation study. Fourth, we investigated the performance of the method in an empirical study. The simulation study focused on three factors. First, the sample size was set to be 100, 500, and 1000. Second, the percentage of random errors was manipulated to be 5%, 10%, and 20%. Third, it had four hierarchies, as proposed by Leighton. All experimental conditions composed of seven attributes, different items according to hierarchies. Simulation results showed that: (1) GRCDM had a high pattern match ratio (PMR) and high marginal match ratio (MMR). This method was shown to be feasible in cognitive diagnostic assessment. (2) The classification accuracy (MMR and PMR) of this method was not dependent on the size of sample. This method had high classification accuracy even in a sample of 100, so it can be used in small assessment and classroom assessment. (3) The classification accuracy under divergent or unstructured attribute hierarchy structure were better than other attribute hierarchy structure in having a high classification accuracy, especially under divergent attribute hierarchy structure. We used this method as in the simulation studies to analyze empirical data from cognitive diagnostic assessment on primary school student’s arithmetic word problem solving, and compared the results to that obtained using the rule space method which was based on the graded response model. Results showed that the attributes difficulty with the classification method was suitable for attributes nature. In different types of schools, the proportion of mastery in each attribute can be different. The better ones have more students mastering each attribute than the worse ones. So, GRCDM has high internal and external validity for use in these situations. In conclusion, this paper provides a new method and a new direction for the development of cognitive diagnostic assessment.

Key words: grade response items, nonparametric cognitive diagnosis, cluster analysis, K-means method