关键词: 多级评分, 测验设计, 最简完备Q矩阵, 结构化Q矩阵, 非结构化Q矩阵, 算法

Abstract: Most test designs primarily focused on dichotomously-scored items, neglecting the need for polytomous cognitive diagnostic test design. This limitation hinders the development of cognitive diagnosis. To address this gap, this paper introduces designs and algorithms for the polytomous structured and unstructured simplest complete Q matrix (SSCQM/USCQM). The proposed approach considered all ideal response patterns (IRPs) of knowledge states (KSs) on the reachability matrix as research objects, with the objective of minimizing the number of columns selected from the reachability matrix. This ensured a one-to-one correspondence between the set of KSs and the set of IRPs. The completeness of the SSCQM was verified. Furthermore, a polytomous USCQM was derived by considering between the SSCQM and the sub-matrix of the corresponding identity matrix, while guaranteeing at least one "1" on each row. Interestingly, the construction process revealed that there were more USCQMs than SSCQMs. This innovative approach expanded the possibilities for polytomous cognitive diagnostic test design. The study aimed to investigate how factors such as the number of attributes, attribute hierarchies, and item parameters influenced the precision of the SSCQM, the USCQM, and the reachability matrix. Two studies were conducted to examine these factors comprehensively. In the first study, this investigation allowed for a detailed understanding of how varying attribute structures and item parameter values affect the accuracy of the Q matrices. The second study delved into the effects of attribute hierarchies and the number of attributes ranging from 5 to 8 on the precision of the SSCQM, USCQM, and reachability matrix. The findings from this comprehensive methodology contribute significantly to the design and development of cognitive diagnostic tests, enhancing our understanding and improving future research and practice in this field. The simulation results revealed several key findings. Firstly, it was observed that having more Q matrices positively impacted the accuracy of the results. Additionally, when considering the same number of items, it was found that the precision of multiple SSCQM was higher than that of the reachability matrix. On the other hand, the precision of multiple USCQM was slightly lower than that of the reachability matrix, except when there were 5 attributes involved. Lastly, the study examined the input-output ratio, which represents the ratio of precision (measured by MMR and PMR) to the number of input items. It was observed that the input-output ratio of the simplest complete Q matrix was generally higher than that of the reachability matrix across all four attribute hierarchies. These findings provide valuable insights into the factors influencing the precision of Q matrices, shedding light on the benefits of increasing the number of matrices, the impact of item parameters, and the performance of different matrix types. Understanding these relationships is crucial for optimizing the design and implementation of cognitive diagnostic testing. In summary, when comparing different Q matrices under similar conditions, the Q matrix that considered attribute hierarchy performed the best. Specifically, the SSCQM demonstrated the highest precision and was deemed the optimal choice for Q matrix design. Following closely behind was the reachability matrix, which exhibited satisfactory performance. While the USCQM had slightly lower precision, it still remained a viable alternative due to its versatility and acceptable level of accuracy. Overall, these findings highlight the importance of considering attribute hierarchy and suggest that the SSCQM should be the primary choice, with the reachability matrix and USCQM serving as viable alternatives for Q matrix selection.

Key words: polytomous, test design, the simplest complete Q matrix, structured Q matrix, unstructured Q matrix, algorithm