基于可达阵的多级评分最简完备Q矩阵设计

doi:10.3724/SP.J.1041.2024.01634

摘要/Abstract

摘要： Q矩阵的完备性是认知诊断模型具有可识别性的关键。多级评分含有比0-1评分更丰富的诊断信息, 却鲜见多级评分完备Q矩阵的设计研究。用最少的题量获得最高判准率是测验设计者追求的目标, 借鉴0-1评分完备Q矩阵的设计方法, 本文提出从可达阵中获取多级评分结构化/非结构化最简完备Q矩阵(SSCQM/USCQM)的方法和算法。模拟实验得出以下结论：(1)测验含SSCQM/USCQM越多, 判准率越高; (2)当列数相同时, 含多个SSCQM或多个USCQM测验的判准率与含可达阵测验的判准率非常接近; (3)对于一些结构, 纵使多个SSCQM/USCQM的列数少于可达阵列数, 其判准率仍不低于可达阵。总之, 短测验设计优先选择SSCQM; 长测验设计优先选择USCQM。

关键词: 多级评分, 测验设计, 结构化最简完备Q矩阵, 非结构化最简完备Q矩阵, 算法

Abstract: The identifiability of cognitive diagnosis models relies heavily on the completeness of the Q matrix. However, existing test designs primarily focus on dichotomously-scored items, neglecting the importance of polytomous cognitive diagnostic test design. Moreover, this limitation poses a significant obstacle to the advancement of cognitive diagnosis. To bridge this gap, this paper aimed to introduce novel designs for the construction of polytomous structured and unstructured simplest complete Q matrices (SSCQM/USCQM). Our proposed approach considered all ideal response patterns (IRPs) of knowledge states (KSs) on the reachability matrix as research objects, with the objective of minimizing the number of columns selected from the reachability matrix. This ensured one-to-one correspondence between the set of KSs and the set of IRPs, thereby enhancing the completeness of the SSCQM. Additionally, we derived a polytomous USCQM by considering the relationship between the SSCQM and the sub-matrix of the corresponding identity matrix while ensuring that each row contains at least one “1”. Interestingly, the construction process revealed that there were more USCQMs than SSCQMs. This innovative approach expanded the possibilities for polytomous cognitive diagnostic test design.
This study focused on the design and evaluation of cognitive diagnostic tests using polytomous structured and unstructured Q matrices (SSCQM/USCQM). We conducted two studies to comprehensively examine the influence of factors such as the number of attributes, attribute hierarchies, and item parameters on the precision of the SSCQM, USCQM, and reachability matrix. In the first study, variations in attribute structures and item parameter values were investigated to understand their impact on Q matrix accuracy. On the other hand, the second study explored the effects of attribute hierarchies and the number of attributes on the precision of the SSCQM, USCQM, and reachability matrix.
Both simulation studies and actual measurement data were utilized to assess the robustness and efficacy of the two methods. Firstly, the simulation results revealed several key findings. Notably, increasing the number of SSCQMs or USCQMs positively influenced the accuracy of the results. In the context of long tests, the USCQM demonstrated higher Pattern Match Ratio (PMR) and Marginal Match Ratio (MMR) compared to the SSCQM and the reachability matrix. This trend was particularly evident when there was an increase in item parameters, attribute numbers, or a change in attribute hierarchy. However, it is noteworthy that, regardless of these various factors, the PMR and MMR of the three tests exhibited minimal differences. On the other hand, in short tests with good item quality, the SSCQM achieved the best performance compared to other methods. This highlights the importance of considering specific test characteristics and item quality when selecting the appropriate Q matrix type. These findings provide valuable insights into the factors that influence the precision of Q matrices. They emphasize the benefits of increasing the number of matrices, understanding the impact of item parameters, and recognizing the performance disparities among different matrix types. Obtaining a comprehensive understanding of these relationships is vital for optimizing the design and implementation of cognitive diagnostic testing, ultimately guaranteeing accurate assessments of individual knowledge states. Secondly, analysis of the actual measurement data showed high identification repetition rates for the SSCQM and the reachability matrix, with a minimal difference in attribute mastery ratio.
In summary, both the SSCQM and the USCQM demonstrate adequate performance when compared to other Q matrices under similar conditions. These findings emphasize the significance of prioritizing completeness in cognitive diagnostic testing. This research seeks to contribute to the advancement of cognitive diagnosis by addressing the limitations of existing test designs and introducing new techniques for constructing polytomous Q matrices. In addition, the findings presented in this paper offer valuable insights for researchers and practitioners seeking to design high-quality cognitive diagnostic tests that accurately assess individual knowledge states.

Key words: polytomous, test design, SSCQM, USCQM, algorithm

中图分类号:

B841

唐小娟, 彭志霞, 秦珊珊, 丁树良, 毛萌萌, 李瑜. (2024). 基于可达阵的多级评分最简完备Q矩阵设计. 心理学报, 56(11), 1634-1650.

TANG Xiaojuan, PENG Zhixia, QIN Shanshan, DING Shuliang, MAO Mengmeng, LI Yu. (2024). Design of the polytomous simplest complete Q matrix based on the reachability matrix. Acta Psychologica Sinica, 56(11), 1634-1650.

参考文献

[1] Cai, Y., & Tu, D. B. (2015). Extension of cognitive diagnosis models based on the polytomous attributes framework and their Q-matrices designs. Acta Psychologica Sinica, 47(10), 1300-1308.
[蔡艳, 涂冬波. (2015). 属性多级化的认知诊断模型拓展及其Q矩阵设计.心理学报, 47(10), 1300-1308.]
[2] Cai Y., Zhao Y., Liu S. C., Zhang S. F., & Tu D. B. (2017). An extended polytomous cognitive diagnostic model.Journal of Psychological Science, 40(6), 1491-1497.
[蔡艳, 赵洋, 刘舒畅, 张淑芳, 涂冬波. (2017). 一种优化的多级评分认知诊断模型.心理科学, 40(6), 1491-1497.]
[3] Chen J.,& de la Torre, J.(2018). Introducing the general polytomous diagnosis modeling framework. Frontiers in Psychology, 9. https://doi.org/10.3389/fpsyg.2018.01474
[4] Chiu, C. Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis.Applied Psychological Measurement, 37(8), 598-618.
[5] Chiu C. Y., Douglas J., & Li X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications.Psychometrika, 74, 633-665.
[6] Chiu, C. Y., & Köhn, H. F. (2015). Consistency of cluster analysis for cognitive diagnosis: The DINO model and the DINA model revisited. Applied Psychological Measurement, 39(6), 465-479.
[7] De Carlo, L. T. (2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the q-matrix.Applied Psychological Measurement, 35(1), 8-26.
[8] De Carlo, L. T. (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36(6), 447-468.
[9] de la Torre, J. (2011). The Generalized DINA Model Framework.Psychometrika, 76(2), 179-199.
[10] Ding S. L., Luo F., & Wang W. Y. (2014). Design of polytomous cognitively diagnostic test blueprint——For the independent and the rhombus attribute hierarchies.Journal of Jiangxi Normal University (Natural Sciences Edition), 38(3), 265-269.
[丁树良, 罗芬, 汪文义. (2014). 多级评分认知诊断测验蓝图的设计——独立型和收敛型结构.江西师范大学学报(自然科学版), 38(3), 265-269.]
[11] Ding S. L., Luo F., Wang W. Y., LI J., & Xiong J. H. (2022). The structure of unstructured complete Q matrices and their identification.Journal of Jiangxi Normal University (Natural Sciences Edition), 46(5), 441-446.
[丁树良, 罗芬, 汪文义, 李佳, 熊建华. (2022). 非结构化完备Q阵的构造与判定.江西师范大学学报(自然科学版), 46(5), 441-446.]
[12] Ding S. L., Luo F., Wang W. Y., & Xiong J. H. (2015). The properties of 0-1 and polytomous Reach ability matrices and their applications.Journal of Jiangxi Normal University (Natural Sciences Edition), 39(1), 64-68.
[丁树良, 罗芬, 汪文义, 熊建华. (2015). 0-1和多值可达矩阵的性质及应用.江西师范大学学报:自然科学版, 39(1), 64-68.]
[13] Ding S. L., Mao M. M., Wang W. Y., Luo F., & Cui Y. (2012). Evaluating the consistency of test items relative to the cognitive model for educational cognitive diagnosis.Acta Psychologica Sinica, 44(11), 1535-1546.
[丁树良, 毛萌萌, 汪文义, 罗芬, Cui, Y. (2012). 教育认知诊断测验与认知模型一致性的评估.心理学报, 44(11), 1535-1546.]
[14] Ding S. L., Wang W. Y., & Luo F. (2014). Design of polytomous cognitively diagnostic test blueprint——For the rooted tree type.Journal of Jiangxi Normal University (Natural Sciences Edition), 38(2), 111-118.
[丁树良, 汪文义, 罗芬. (2014). 多级评分认知诊断测验蓝图的设计——根树型结构.江西师范大学学报(自然科学版), 38(2), 111-118.]
[15] Ding S. L., Wang W. Y., & Yang S. Q. (2011). The design of cognitive diagnostic test blueprints.Journal of Psychological Science, 34(2), 258-265.
[丁树良, 汪文义, 杨淑群. (2011). 认知诊断测验蓝图的设计.心理科学, 34(2), 258-265.]
[16] Ding S. L., Yang S. Q., & Wang W. Y. (2010). The importance of reachability matrix in constructing cognitively diagnostic testing.Journal of Jiangxi Normal University (Natural Sciences Edition), 34(5), 490-494.
[丁树良, 杨淑群, 汪文义. (2010). 可达矩阵在认知诊断测验编制中的重要作用.江西师范大学学报(自然科学版), 34(5), 490-494.]
[17] Fang G., Liu J., & Ying Z. (2019). On the identifiability of diagnostic classification models.Psychometrika, 84(1), 19-40.
[18] Gao X. L., Gong Y., & Wang F. (2021). Research progress in polytomous cognitive diagnosis model.Journal of Psychological Science, 44(2), 457-464.
[高旭亮, 龚毅, 王芳. (2021). 多级评分认知诊断模型述评.心理科学, 44(2), 457-464.]
[19] Groß, J., & George, A. C. (2014). On permissible attribute classes in noncompensatory cognitive diagnosis models.Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 10(3), 100-107.
[20] Gu, Y., & Xu, G. (2019). The sufficient and necessary condition for the identifiability and estimability of the DINA model.Psychometrika, 84(2), 468-483.
[21] Gu, Y., & Xu, G. (2021a). Sufficient and necessary conditions for the identifiability of the Q-matrix.Statistica Sinica, 31(1), 449-472.
[22] Gu Y.,& Xu, G. (2021b). Identifiability of hierarchical latent attribute models. preprintarXiv: 1906.07869v4.
[23] Kang C. H., Xin T., & Tian W. (2013). Development and validation of diagnostic test for primary school arithmetic word problems.Examinations Research, 41(6), 24-43.
[康春花, 辛涛, 田伟. (2013). 小学数学应用题认知诊断测验编制及效度验证.考试研究, 41(6), 24-43.]
[24] Köhn, H. F., & Chiu, C. Y. (2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests.Psychometrika, 82, 112-132.
[25] Köhn, H. F., & Chiu, C. Y. (2021). A unified theory of the completeness of Q-Matrices for the DINA model.Journal of Classification, 38(3), 500-518.
[26] Leighton J. P.,& Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and applications Cambridge University Press Theory and applications. Cambridge University Press.
[27] Lin, M., & Xu, G. (2023). Sufficient and Necessary Conditions for the Identifiability of DINA Models with Polytomous Responses. preprintarXiv: 2304.01363.
[28] Liu J., Xu G., & Ying Z. (2012). Data-driven learning of Q-matrix.Applied Psychological Measurement, 36(7), 548-564.
[29] Liu J., Xu G., & Ying Z. (2013). Theory of the self-learning Q-matrix.Bernoulli, 19, 1790-1817.
[30] Ma, W. C., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses.British Journal of Mathematical and Statistical Psychology, 69(3), 253-275.
[31] Madison, M. J., & Bradshaw, L. P. (2015). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model.Educational and Psychological Measurement, 75(3), 491-511.
[32] Ouyang, J., & Xu, G. (2022). Identifiability of latent class models with covariates.Psychometrika, 87(4), 1343-1360.
[33] Sun J. N., Xin T., Zhang S. M., & de la Torre, J. (2013). A polytomous extension of the generalized distance discriminating method.Applied Psychological Measurement, 37(7), 503-521.
[34] Tatsuoka, K. K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. Erlbaum: Hillsdale, 327-359.
[35] Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method. Routledge/Taylor & Francis Group.
[36] Xu, G., & Zhang, S. (2016). Identifiability of diagnostic classification models.Psychometrika, 81(3), 625-649.