基于信号检测论的认知诊断评估：构建与应用

doi:10.3724/SP.J.1041.2024.00339

摘要/Abstract

摘要：

作答选择题可被看作从噪音中提取信号的过程, 研究提出了一种基于信号检测论的认知诊断模型(SDT-CDM)。新模型的优势在于：(1)无需对选项进行属性层面的编码。(2)能获得传统诊断模型无法提供的题目区分度和难度参数。(3)可以直接表达每个选项之间的合理性差异, 对题目性能刻画更加细微全面。两个模拟研究结果表明：(1)EM算法可以实现对新模型的参数估计过程, 便捷有效。(2) SDT-CDM具备良好性能, 分类准确性和参数估计精度较高以外, 还能提供选项层面的估计信息, 用于题目质量诊断与修订。(3)属性数量、题目质量与样本量等因素会影响SDT-CDM的表现。(4)与称名诊断模型NRDM相比, SDT-CDM在所有实验条件下对被试的分类准确性更高。实证研究表明：SDT-CDM比NRDM具有更好的模型数据拟合结果, 其分类准确性和一致性更高, 尤其当属性考察次数较少时具有很强的稳定性, 难度和区分度参数与IRT模型估计结果的相关性也更高, 值得推广。

关键词: 信号检测论, 认知诊断, 选择题, EM算法

Abstract:

Cognitive diagnostic assessment (CDA) is aimed at diagnose which skills or attributes examinees have or do not have as the name expressed. This technique provides more useful feedback to examinees than a simple overall score got from classical test theory or item response theory. In CDA, multiple-choice (MC) is one of popular item types, which have the superiority on high test reliability, being easy to review, and scoring quickly and objectively. Traditionally, several cognitive diagnostic models (CDMs) have been developed to analyze the MC data by including the potential diagnostic information contained in the distractors.

However, the response to MC items can be viewed as the process of extracting signals (correct options) from noises (distractors). Examinees are supposed to have perceptions of the plausibility of each options, and they make the decision based on the most plausible option. Meanwhile, there are two different states when examinee response to items: knows or does not know each item. Thus, the signal detection theory can be integrated into CDM to deal with MC data in CDA. The cognitive diagnostic model based on signal detection theory (SDT-CDM) is proposed in this paper and has several advantages over traditional CDMs. Firstly, it does not require the coding of q-vector for each option. Secondly, it provides discrimination and difficulty parameters that traditional CDMs cannot provide. Thirdly, it can directly express the relative differences between each options by plausibility parameters, providing a more comprehensive characterization of item quality.

The results of two simulation studies showed that (1) the marginal maximum likelihood estimation approach via Expectation Maximization (MMLE/EM) algorithm could effectively estimate the model parameters of the SDT-CDM. (2) the SDT-CDM had high classification accuracy and parameter estimation precision, and could provide option-level information for item quality diagnosis. (3) independent variables such as the number of attributes, item quality, and sample size affected the performance of the SDT-CDM, but the overall results were promising. (4) compared with the nominal response diagnostic model (NRDM), the SDT-CDM was more accurate in classifying examinees under all data conditions.

Further, an empirical study on the TIMSS 2011 mathematics assessment were conducted using both the SDT-CDM and the NRDM to inspect the ecological validity for the new model. The results showed that the SDT-CDM had better fitting and a smaller number of model parameters than the NRDM. The difficulty parameters of the SDT-CDM were significantly correlated with those of the two- (three-) parameter logical models. And the same was true of the discrimination parameters for the SDT-CDM. However, the correlation between the discrimination parameters of the NRDM and those of the two- (three-) parameter logical models was low and not significant. Besides, the classification accuracy and classification consistency of the SDT-CDM were higher than those of the NRDM. All the results indicated that the SDT-CDM was worth promoting.

Key words: signal detection theory, cognitive diagnostic assessment, multiple-choice items, expectation maximization algorithmtext

中图分类号:

B841

郭磊, 秦海江. (2024). 基于信号检测论的认知诊断评估：构建与应用. 心理学报, 56(3), 339-351.

GUO Lei, QIN Haijiang. (2024). Cognitive diagnostic assessment based on signal detection theory: Modeling and application. Acta Psychologica Sinica, 56(3), 339-351.

图/表 14

参考文献 38

[1]	Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37 (1, Pt. 1), 29-51. doi: 10.1007/BF02291411 URL
[2]	Bradshaw, L., & Templin, J. (2014). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403-425. doi: 10.1007/s11336-013-9350-4 pmid: 25205005
[3]	Chalmers, R,P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29.
[4]	Chen, Y., Zhang, J., Yang, Y., & Lee, Y.-S. (2022). Latent space model for process data. Journal of Educational Measurement, 59(4), 517-535. doi: 10.1111/jedm.v59.4 URL
[5]	Chiu, C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598-618. doi: 10.1177/0146621613488436 URL
[6]	Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2^nd ed.). New York, NY: Erlbaum.
[7]	DiBello, L. V., Henson, R. A., & Stout, W. F. (2015). A family of generalized diagnostic classification models for multiple choice option-based scoring. Applied Psychological Measurement, 39(1), 62-79. doi: 10.1177/0146621614561315 pmid: 29880994
[8]	DeCarlo, L, T. (2021). A signal detection model for multiple- choice exams. Applied Psychological Measurement, 45(6), 423-440. doi: 10.1177/01466216211014599 URL
[9]	de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115-130. doi: 10.3102/1076998607309474 URL
[10]	de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199. doi: 10.1007/s11336-011-9207-7 URL
[11]	Fang, G., Liu, J., & Ying, Z. (2019). On the identifiability of diagnostic classification models. Psychometrika, 84, 19-40. doi: 10.1007/s11336-018-09658-x pmid: 30673967
[12]	Guo, L., Yuan, C. Y., & Bian, Y. F. (2013). Discussing the development tendency of cognitive diagnosis from the perspective of new models. Advances in Psychological Science, 21(12), 2256-2264. doi: 10.3724/SP.J.1042.2013.02256
	[ 郭磊, 苑春永, 边玉芳. (2013). 从新模型视角探讨认知诊断的发展趋势. 心理科学进展, 21(12), 2256-2264.] doi: 10.3724/SP.J.1042.2013.02256
[13]	Guo, L., Zheng, C., Bian, Y., Song, N., & Xia, L., (2016). New item selection methods in cognitive diagnostic computerized adaptive testing: Combining item discrimination indices. Acta Psychologica Sinica, 48(7), 903-914. doi: 10.3724/SP.J.1041.2016.00903
	[ 郭磊, 郑蝉金, 边玉芳, 宋乃庆, 夏凌翔. (2016). 认知诊断计算机化自适应测验中新的选题策略:结合项目区分度指标. 心理学报, 48(7), 903-914.]
[14]	Guo, L., & Zhou, W. J. (2021). Nonparametric methods for cognitive diagnosis to multiple-choice test items. Acta Psychologica Sinica, 53(9), 1032-1043. doi: 10.3724/SP.J.1041.2021.01032
	[ 郭磊, 周文杰. (2021). 基于选项层面的认知诊断非参数方法. 心理学报, 53(9), 1032-1043.] doi: 10.3724/SP.J.1041.2021.01032
[15]	He, Q., Borgonovi, F., & Paccagnella, M. (2021). Leveraging process data to assess adults’ problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166: 104170. doi: 10.1016/j.compedu.2021.104170 URL
[16]	Kelly, F. J. (1916). The kansas silent reading tests. Journal of Educational Psychology, 7(2), 63-80. doi: 10.1037/h0073542 URL
[17]	Lee, S. Y. (2017). Growth curve cognitive diagnosis models for longitudinal assessment (Unpublished doctorial dissertation). University of California, Berkeley.
[18]	Liu, Y. (2022). Standard errors and confidence intervals for cognitive diagnostic models: Parallel bootstrap methods. Acta Psychologica Sinica, 54(6), 703-724. doi: 10.3724/SP.J.1041.2022.00703
	[ 刘彦楼. (2022). 认知诊断模型的标准误与置信区间估计:并行自助法. 心理学报, 54(6), 703-724.] doi: 10.3724/SP.J.1041.2022.00703
[19]	Ma, W., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275. doi: 10.1111/bmsp.2016.69.issue-3 URL
[20]	Ma, W., & de la Torre, J. (2020). An empirical Q-matrix validation method for the sequential generalized DINA model. British Journal of Mathematical and Statistical Psychology, 73(1), 142-163. doi: 10.1111/bmsp.v73.1 URL
[21]	Ozaki, K. (2015). DINA models for multiple-choice items with few parameters: Considering incorrect answers. Applied Psychological Measurement, 39(6), 431-447. doi: 10.1177/0146621615574693 pmid: 29881017
[22]	Tang, X., Wang, Z., He, Q., Liu, J., & Ying, Z. (2020). Latent feature extraction for process data via multidimensional scaling. Psychometrika, 85, 378-397. doi: 10.1007/s11336-020-09708-3 pmid: 32572672
[23]	Tang, X., Wang, Z., Liu, J., & Ying, Z. (2021). An exploratory analysis of the latent structure of process data via action sequence autoencoder. British Journal of Mathematical and Statistical Psychology, 74(1), 1-33.
[24]	Templin, J., Henson, R., Rupp, A., Jang, E., & Ahmed, M. (2008). Cognitive diagnosis models for nominal response data. Annual Meeting of the National Council on Measurement in Education, New Brunswick, New Jersey.
[25]	Thissen, D., & Steinberg, L. (1997). A response model for multiple-choice items. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 51-65). Springer.
[26]	Wang, S., Yang, Y., Culpepper, S. A., & Douglas, J. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57-87.
[27]	Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52, 457-476. doi: 10.1111/jedm.2015.52.issue-4 URL
[28]	Wang, Y., Chiu, C.-Y., & Kohn, H. F. (2023). Nonparametric classification method for multiple-choice items in cognitive diagnosis. Journal of Educational and Behavioral Statistics, 48(2), 189-219.
[29]	Xu, G. (2017). Identifiability of restricted latent class models with binary responses. The Annals of Statistics, 45(2), 675-707.
[30]	Xu, X., Chang, H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Montreal, Quebec, Canada.
[31]	Zeng, Z., Gu, Y., & Xu, G. (2023). A tensor-EM method for large-scale latent class analysis with binary responses. Psychometrika, 88, 580-612. doi: 10.1007/s11336-022-09887-1
[32]	Zhan, P. D. (2022). Joint-cross-loading multimodal cognitive diagnostic modeling incorporating visual fixation counts. Acta Psychologica Sinica, 54(11), 1416-1432. doi: 10.3724/SP.J.1041.2022.01416
	[ 詹沛达. (2022). 引入眼动注视点的联合-交叉负载多模态认知诊断建模. 心理学报, 54(11), 1416-1432.] doi: 10.3724/SP.J.1041.2022.01416
[33]	Zhan, P. D., Jiao, H., Liao, D. D., & Li, F. M. (2019). A longitudinal higher-order diagnostic classification model. Journal of Educational and Behavioral Statistics, 44(3), 251-281.
[34]	Zhan, P. D., & Qiao, X. (2022). Diagnostic classification analysis of problem-solving competence using process data: An item expansion method. Psychometrika, 87(4), 1529-547. doi: 10.1007/s11336-022-09855-9 pmid: 35389193
[35]	Zhang, H. C., & Xu, J. P. (2015). Modern psychology and educational statistics (4^th ed.). Beijing Normal University Press.
	[ 张厚粲, 徐建平. (2015). 现代心理与教育统计学(第4版). 北京师范大学出版社.]
[36]	Zhang, S. S., & Chang, H. H. (2020). A multilevel logistic hidden markov model for learning under cognitive diagnosis. Behavior Research Methods, 52, 408-421. doi: 10.3758/s13428-019-01238-w pmid: 31065938
[37]	Zheng, T. P., Zhou, W. J., & Guo, L. (2023). Cognitive diagnosis modelling based on response times. Journal of Psychological Science, 46(2), 478-490.
	[ 郑天鹏, 周文杰, 郭磊. (2023). 基于题目作答时间信息的认知诊断模型. 心理科学, 46(2), 478-490.]
[38]	Zhu, W., Ding, S., & Chen, X. (2006). Minimum chi-square/EM estimation under IRT. Acta Psychologica Sinica, 38(3), 453-460.
	[ 朱玮, 丁树良, 陈小攀. (2006). IRT中最小化χ²/EM参数估计方法. 心理学报, 38(3), 453-460.]

序号/题目编号	A1	A2	A3	A4	A5	A6
1/M032679	0	0	0	1	1	0
2/M042024	0	1	0	0	0	0
3/M042016	1	0	0	0	0	0
4/M042077	1	0	1	0	0	0
5/M042235	0	0	1	0	0	0
6/M042150	0	0	0	1	0	0
7/M032352	1	0	0	0	0	1
8/M032738	0	0	1	0	0	0
9/M032295	0	0	1	0	0	0
10/M032331	0	0	0	1	1	0
11/M042041	0	1	0	0	0	0
12/M032047	1	0	0	0	0	0
13/M032398	0	0	0	1	0	0
14/M032424	0	1	1	0	0	0

序号/题目编号	A1	A2	A3	A4	A5	A6
1/M032679	0	0	0	1	1	0
2/M042024	0	1	0	0	0	0
3/M042016	1	0	0	0	0	0
4/M042077	1	0	1	0	0	0
5/M042235	0	0	1	0	0	0
6/M042150	0	0	0	1	0	0
7/M032352	1	0	0	0	0	1
8/M032738	0	0	1	0	0	0
9/M032295	0	0	1	0	0	0
10/M032331	0	0	0	1	1	0
11/M042041	0	1	0	0	0	0
12/M032047	1	0	0	0	0	0
13/M032398	0	0	0	1	0	0
14/M032424	0	1	1	0	0	0

Model	模型参数数量	−2LL	AIC	BIC
SDT-CDM	71	19965.49	20107.49	20169.54
NRDM	87	20007.68	20181.68	20257.71

Model	模型参数数量	−2LL	AIC	BIC
SDT-CDM	71	19965.49	20107.49	20169.54
NRDM	87	20007.68	20181.68	20257.71

评价指标	模型	模式	属性
评价指标	模型	模式	A1	A2	A3	A4	A5	A6
分类准确性	SDT-CDM	0.608	0.864	0.918	0.932	0.884	0.819	0.953
	NRDM	0.437	0.895	0.907	0.930	0.875	0.780	0.770
	提升率	39.13%	−3.46%	1.21%	0.22%	1.03%	5.00%	23.77%
分类一致性	SDT-CDM	0.650	0.809	0.880	0.901	0.833	0.757	0.921
	NRDM	0.647	0.850	0.866	0.895	0.823	0.720	0.716
	提升率	0.46%	−4.82%	1.62%	0.67%	1.22%	5.14%	28.63%