基于选项层面的认知诊断非参数方法

doi:10.3724/SP.J.1041.2021.01032

摘要/Abstract

摘要：

充分挖掘选择题(Multiple-Choice, MC)的诊断信息受到了较多关注, 将干扰项信息考虑在内可以提升诊断精度。为了弥补参数模型基于大样本才能获得可靠估计的不足, 以及适用于班级水平的小样本诊断测验情境, 本研究提出了非参数的多选题诊断方法。模拟和实证研结果表明：(1)当MC测验中题目参数不存在较大差异时, ${{d}_{\text{ph}-\text{MC}}}$法在多数情况下表现优于参数类诊断模型。(2)当MC测验中题目参数存在较大差异时, ${{d}_{ph-MC}}$法的表现最优。(3)实证研究中非参数方法和参数类模型的分类一致性程度较高, ${{d}_{\text{ph}-\text{MC}}}$距离法估计得到的考生属性总体掌握程度与总分相关最高。最后, 基于MC诊断测验的特点提出了若干研究方向。

关键词: 认知诊断评估, 选择题, 干扰项信息, 非参数诊断方法, 汉明距离

Abstract:

Cognitive diagnostic assessment (CDA) focuses on evaluating students' advantages and disadvantages in knowledge mastering, providing an opportunity for individualized teaching. Therefore, CDA has attracted attention of many scholars, teachers, and students at domestic and overseas. In CDA and a large number of standardized tests, multiple-choice (MC) are typical item types, which have the advantages of not being affected by subjective errors, improving test reliability, being easy to review, scoring quickly, and meeting the needs of content balance. To fulfil the potential of MC items for CDA, researchers proposed the MC-cognitive diagnosis models (MC-CDMs). However, these MC-CDMs pertain to parameter methods, which need a large sample size to obtain accurate parameter estimation. They are not suitable for small samples at class level, and the MCMC algorithm is very time-consuming. In this study, three nonparametric MC cognitive diagnosis methods based on hamming-distance are proposed, aiming at maximizing the diagnostic efficacy of MC items and being suitable for the diagnosis target of a small sample.

Simulation study 1 considered four factors: sample size (30, 50, 100), test length (10, 20, 30), item quality (high and low), and the true model (MC-S-DINA1, MC-S-DINA2). Three nonparametric MC methods and two parametric models were compared. The results showed that in most conditions, the pattern accuracy rates and average attribute accuracy rates of the nonparametric MC method(${{d}_{\text{h}-\text{MC}}}$) were higher than those of parametric models, especially when the test length was short or item quality was low.

In a real test situation, the quality of different items in a test may vary greatly. Based on this, simulation study 2 set the first half of the items at high quality and the remaining items at low quality. The results showed that the pattern accuracy rates and average attribute accuracy rates of the nonparametric MC method (${{d}_{\text{ph}-\text{MC}}}$) were higher than those of the parametric models in all conditions.

In an empirical study, the nonparametric MC methods and the parametric models were used to analyze a set of real data simultaneously. The results showed that nonparametric MC methods and parametric models presented high classification consistency rates. Furthermore, the ${{d}_{\text{ph}-\text{MC}}}$ method had satisfactory estimations.

In sum, ${{d}_{\text{h}-\text{MC}}}$ was suitable in most conditions, especially when the test length was short or the item quality was low When the quality of different items was quite diverse, ${{d}_{\text{ph}-\text{MC}}}$ was a better choice compared with parameteric approaches.

Key words: cognitive diagnostic assessment, multiple-choice item, distractor information, nonparametric diagnostic method, hamming distance

中图分类号:

B841

郭磊, 周文杰. (2021). 基于选项层面的认知诊断非参数方法. 心理学报, 53(9), 1032-1043.

GUO Lei, ZHOU Wenjie. (2021). Nonparametric methods for cognitive diagnosis to multiple-choice test items. Acta Psychologica Sinica, 53(9), 1032-1043.

图/表 7

参考文献 27

[1]	Bock R.D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51. doi: 10.1007/BF02291411 URL
[2]	Bradshaw L., & Templin J. (2014). Combining item response theory and diagnostic classification models: a psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403-425. doi: 10.1007/s11336-013-9350-4 pmid: 25205005
[3]	Chang Y.-P., Chiu C.-Y., & Tsai R.-C. (2019). Nonparametric CAT for CD in educational settings with small samples. Applied Psychological Measurement, 43(7), 543-561. doi: 10.1177/0146621618813113 URL
[4]	Chiu C.-Y., & Douglas J.A. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30(2), 225- 250. doi: 10.1007/s00357-013-9132-9 URL
[5]	Chiu C.-Y., Douglas J.A., & Li X.D. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633-665. doi: 10.1007/s11336-009-9125-0 URL
[6]	Chiu C.-Y., Sun Y., & Bian Y.H. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355-375. doi: 10.1007/s11336-017-9595-4 URL
[7]	de la Torre J.(2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33(3), 163-183. doi: 10.1177/0146621608320523 URL
[8]	de la Torre J.(2011). The generalized DINA model framework. Psychometrika, 76, 179-199. doi: 10.1007/s11336-011-9207-7 URL
[9]	DiBello L.V., Henson R.A., & Stout W.F. (2015). A family of generalized diagnostic classification models for multiple choice option-based scoring. Applied Psychological Measurement, 39(1), 62-79. doi: 10.1177/0146621614561315 pmid: 29880994
[10]	Guo L., Yang J., & Song N.Q. (2018). Application of spectral clustering algorithm under various attribute hierarchical structures for cognitive diagnostic assessment. Journal of Psychological Science, 41(3), 735-742.
	[ 郭磊, 杨静, 宋乃庆. (2018). 谱聚类算法在不同属性层级结构诊断评估中的应用. 心理科学, 41(3), 735-742.]
[11]	Guo L., Yang J., & Song N.Q. (2020). Spectral clustering algorithm for cognitive diagnostic assessment. Fronties in Psychology, 11, 944. doi: 10.3389/fpsyg.2020.00944 doi: 10.3389/fpsyg.2020.00944
[12]	Henson R.A., Templin J.L., & Willse J.T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191-210. doi: 10.1007/s11336-008-9089-5 URL
[13]	Kang C.H., Ren P., & Zeng P.F. (2015). Nonparametric cognitive diagnosis: A cluster diagnostic method based on grade response items. Acta Psychologica Sinica, 47(8), 1077-1088. doi: 10.3724/SP.J.1041.2015.01077 URL
	[ 康春花, 任平, 曾平飞. (2015). 非参数认知诊断方法:多级评分的聚类分析. 心理学报, 47(8), 1077-1088.]
[14]	Kang C.H., Yang Y.K., & Zeng P.F. (2019). Approach to cognitive diagnosis: The manhattan distance discriminating method. Journal of Psychological Science, 42(2), 455-462.
	[ 康春花, 杨亚坤, 曾平飞. (2019). 一种混合计分的非参数认知诊断方法:曼哈顿距离判别法. 心理科学, 42(2), 455-462.]
[15]	Levine M.V., & Drasgow F. (1983). The relation between incorrect option choice and estimated ability. Educational and Psychological Measurement, 43(3), 675-685. doi: 10.1177/001316448304300301 URL
[16]	Li S.Z. (2019). Application of back propagation neural network based teaching cognitive diagnosis (Unpublished master’s thesis). Henan Normal University, China.
	[ 李世珍. (2019). 基于BP神经网络的教学认知诊断方法及应用 (硕士学位论文). 河南师范大学.]
[17]	Li Y. (2014). The construction for cognitive diagnosis tests of multiple-choice items and the development of multiple- choice cognitive diagnosis model for multiple strategies. (Unpublished doctoral dissertation). Jiangxi Normal University, China.
	[ 李瑜. (2014). 多选题认知诊断测验编制及多策略的多选题认知诊断模型的开发 (博士学位论文). 江西师范大学.]
[18]	Liu T. (2016). Using distractor information in computerized adaptive testing (Unpublished doctoral dissertation). Beijing Normal University.
	[ 刘拓. (2016). 干扰项信息在计算机化自适应测验中的利用 (博士学位论文). 北京师范大学.]
[19]	Ma W.C., Iaconangelo C., de la Torre J. & (2016). Model similarity, Model selection, and attribute classification. Applied Psychological Measurement, 40(3), 200-271. doi: 10.1177/0146621615621717 URL
[20]	Osterlind S.J. (1998). Constructing test items: Multiple- choice, constructed-response, performance and other formats (2nd ed.). Boston: Kluwer Academic.
[21]	Ozaki K. (2015). DINA Models for multiple-choice items with few parameters: Considering incorrect answers. Applied Psychological Measurement, 39(6), 431-447. doi: 10.1177/0146621615574693 URL
[22]	Steven M.D. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38(9), 1006-1012. doi: 10.1111/med.2004.38.issue-9 URL
[23]	Thissen D.M. (1976). Information in wrong responses to the raven progressive matrices. Journal of Educational Measurement, 13(3), 201-214. doi: 10.1111/jedm.1976.13.issue-3 URL
[24]	Thissen D.M., & Steinberg L. (1984). A response model for multiple-choice items. Psychometrika, 49(4), 501-519. doi: 10.1007/BF02302588 URL
[25]	Thissen D.M., & Wainer H. (1993). Combining multiple- choice and constructed-response test scores: Toward a Marxist theory of test construction. Applied Measurement in Education, 6(2), 103-118. doi: 10.1207/s15324818ame0602_1 URL
[26]	von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287-307. doi: 10.1348/000711007X193957 URL
[27]	Yigit H.D., Sorrel M.A., & de la Torre J.(2019). Computerized adaptive testing for cognitively based multiple-choice data. Applied Psychological Measurement, 43(5), 388-401. doi: 10.1177/0146621618798665 URL

$2\frac{4}{12}-\frac{7}{12}$		属性
$2\frac{4}{12}-\frac{7}{12}$		S1	S2	S3
A	$2\frac{3}{12}$		√
B	$2\frac{1}{4}$		√	√
C	$1\frac{9}{12}$	√	√
D	$1\frac{3}{4}$	√	√	√

$2\frac{4}{12}-\frac{7}{12}$		属性
$2\frac{4}{12}-\frac{7}{12}$		S1	S2	S3
A	$2\frac{3}{12}$		√
B	$2\frac{1}{4}$		√	√
C	$1\frac{9}{12}$	√	√
D	$1\frac{3}{4}$	√	√	√

题目质量	题目数量	样本量	PCCR				AACCR
题目质量	题目数量	样本量	${{d}_{\text{h}-\text{MC}}}$	${{d}_{\text{wh}-\text{MC}}}$	MC1	MC2	${{d}_{\text{h}-\text{MC}}}$	${{d}_{\text{wh}-\text{MC}}}$	MC1	MC2
高质量	10	30	0.784	0.710	0.763	0.703	0.918	0.884	0.906	0.896
		50	0.783	0.701	0.749	0.690	0.916	0.883	0.900	0.889
		100	0.789	0.703	0.757	0.704	0.922	0.888	0.902	0.896
	20	30	0.911	0.893	0.896	0.888	0.968	0.962	0.930	0.928
		50	0.911	0.895	0.879	0.863	0.976	0.962	0.918	0.970
		100	0.912	0.895	0.905	0.896	0.973	0.963	0.921	0.968
	30	30	0.957	0.947	0.979	0.964	0.987	0.984	0.992	0.991
		50	0.951	0.934	0.973	0.966	0.986	0.980	0.992	0.989
		100	0.954	0.940	0.976	0.970	0.986	0.982	0.993	0.983
低质量	10	30	0.575	0.495	0.498	0.450	0.843	0.798	0.814	0.799
		50	0.588	0.501	0.505	0.428	0.843	0.801	0.820	0.788
		100	0.590	0.501	0.518	0.420	0.849	0.806	0.828	0.784
	20	30	0.802	0.768	0.742	0.655	0.933	0.919	0.917	0.888
		50	0.798	0.762	0.742	0.651	0.935	0.921	0.919	0.889
		100	0.793	0.760	0.752	0.671	0.930	0.917	0.922	0.892
	30	30	0.865	0.849	0.820	0.757	0.964	0.959	0.952	0.935
		50	0.868	0.845	0.837	0.777	0.965	0.957	0.957	0.940
		100	0.874	0.853	0.848	0.801	0.967	0.959	0.961	0.947

题目质量	题目数量	样本量	PCCR				AACCR
题目质量	题目数量	样本量	${{d}_{\text{h}-\text{MC}}}$	${{d}_{\text{wh}-\text{MC}}}$	MC1	MC2	${{d}_{\text{h}-\text{MC}}}$	${{d}_{\text{wh}-\text{MC}}}$	MC1	MC2
高质量	10	30	0.784	0.710	0.763	0.703	0.918	0.884	0.906	0.896
		50	0.783	0.701	0.749	0.690	0.916	0.883	0.900	0.889
		100	0.789	0.703	0.757	0.704	0.922	0.888	0.902	0.896
	20	30	0.911	0.893	0.896	0.888	0.968	0.962	0.930	0.928
		50	0.911	0.895	0.879	0.863	0.976	0.962	0.918	0.970
		100	0.912	0.895	0.905	0.896	0.973	0.963	0.921	0.968
	30	30	0.957	0.947	0.979	0.964	0.987	0.984	0.992	0.991
		50	0.951	0.934	0.973	0.966	0.986	0.980	0.992	0.989
		100	0.954	0.940	0.976	0.970	0.986	0.982	0.993	0.983
低质量	10	30	0.575	0.495	0.498	0.450	0.843	0.798	0.814	0.799
		50	0.588	0.501	0.505	0.428	0.843	0.801	0.820	0.788
		100	0.590	0.501	0.518	0.420	0.849	0.806	0.828	0.784
	20	30	0.802	0.768	0.742	0.655	0.933	0.919	0.917	0.888
		50	0.798	0.762	0.742	0.651	0.935	0.921	0.919	0.889
		100	0.793	0.760	0.752	0.671	0.930	0.917	0.922	0.892
	30	30	0.865	0.849	0.820	0.757	0.964	0.959	0.952	0.935
		50	0.868	0.845	0.837	0.777	0.965	0.957	0.957	0.940
		100	0.874	0.853	0.848	0.801	0.967	0.959	0.961	0.947

题目质量	题目数量	样本量	PCCR				AACCR
题目质量	题目数量	样本量	${{d}_{\text{h}-\text{MC}}}$	${{d}_{\text{wh}-\text{MC}}}$	MC1	MC2	${{d}_{\text{h}-\text{MC}}}$	${{d}_{\text{wh}-\text{MC}}}$	MC1	MC2.
高质量	10	30	0.772	0.700	0.746	0.697	0.915	0.884	0.904	0.896
		50	0.781	0.700	0.747	0.701	0.917	0.880	0.900	0.893
		100	0.788	0.705	0.753	0.705	0.921	0.889	0.903	0.897
	20	30	0.907	0.888	0.887	0.888	0.966	0.961	0.935	0.967
		50	0.909	0.892	0.884	0.905	0.965	0.959	0.923	0.972
		100	0.911	0.896	0.886	0.916	0.967	0.961	0.923	0.971
	30	30	0.953	0.938	0.960	0.976	0.985	0.980	0.991	0.991
		50	0.949	0.938	0.966	0.973	0.985	0.981	0.989	0.992
		100	0.952	0.936	0.972	0.973	0.986	0.981	0.987	0.993
低质量	10	30	0.566	0.501	0.490	0.424	0.835	0.798	0.807	0.787
		50	0.580	0.493	0.497	0.424	0.841	0.797	0.815	0.786
		100	0.593	0.501	0.516	0.422	0.847	0.803	0.823	0.786
	20	30	0.787	0.752	0.723	0.642	0.931	0.917	0.915	0.886
		50	0.793	0.761	0.744	0.656	0.930	0.917	0.917	0.889
		100	0.792	0.762	0.754	0.666	0.931	0.918	0.921	0.892
	30	30	0.872	0.849	0.830	0.759	0.964	0.957	0.954	0.935
		50	0.873	0.846	0.844	0.777	0.965	0.956	0.959	0.940
		100	0.873	0.848	0.849	0.797	0.965	0.956	0.959	0.945