一种基于多阶认知诊断模型测评科学素养的方法

doi:10.3724/SP.J.1041.2019.00734

摘要/Abstract

摘要：

科学素养是指作为一名有反思意识的公民所具有的解决科学问题和运用科学理念的能力。为实现在认知诊断中对科学素养的测评, 本文基于PISA 2015科学素养测评框架首次提出科学素养包含的三阶潜在结构, 使用新提出的多阶认知诊断模型对PISA 2015科学测评数据进行分析, 并通过模拟研究探究新模型的心理测量学性能。结果表明：(1)新模型能够较好地分析包含三阶潜在结构的科学素养; (2)科学知识对科学素养的影响最大, 科学背景次之, 科学能力的影响最小; (3)全贝叶斯MCMC算法能够为新模型提供较精准的参数估计。

关键词: 科学素养, 认知诊断, PISA, DINA模型

Abstract:

In PISA 2015, scientific literacy is defined as “the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen”. There are four interdependent dimensions are specified in the scientific literacy assessment framework for PISA 2015: Competencies, Knowledge, Contexts, and Attitudes. Given that knowledge of scientific literacy contributes significantly to individuals’ personal, social, and professional lives, it is of vital importance to find an objectively and accurately assessment method for scientific literacy. However, only unidimensional IRT models were used in the analysis in PISA 2015. Which means that the analysis model does not match with such a multidimensional assessment framework. It is desired to develop a new analysis model. This study attempts to measure scientific literacy in cognitive diagnostic assessment for the first time.

According to the scientific literacy assessment framework for PISA 2015, a third-order latent structure for scientific literacy is first pointed out. Specifically, the scientific literacy is treated as the third-order latent trait; Competencies, Knowledge, Contexts, and Attitudes are all treated as second-order latent traits; And nine subdomains, e.g., explain phenomena scientifically and content knowledge, were treated as first-order traits (or attributes). Unfortunately, however, there is still a lack of cognitive diagnosis models that can deal with such a third-order latent structure. To this end, a multi-order DINA (MO-DINA) model was developed in this study. The new model is an extension of the higher-order (HO-DINA) model, which is similar to the third-order IRT models. To illustrate the application and advantages of the MO-DINA model, a sub-data of PISA 2015 science assessment data were analyzed. Items were chosen from the S01 cluster, and participants were chosen from China. After data cleaning, 1076 participants with 18 items were retained. Three models were fitted to this sub-data and compared, the MO-DINA model, in which the third-order latent structure of scientific literacy was considered; the HO-DINA model, in which the scientific literacy was treated as a second-order latent trait and contacted with attributes directly; and the DINA model.

All three models appear to provide a reasonably good fit to data according to the posterior predictive model checking. According to the -2LL, AIC, BIC, and DIC, the DINA model fits the data worst, and the MO-DINA model fits the data best, the results of MO-DINA model are used to make further interpretations. The results indicated that (1) the quality of 18 items are not good enough; (2) The correlations among second-order latent traits are high (0.8, approximately); (3) Knowledge has the greatest influence on scientific literacy, Contexts second, and Competencies least; (4) Explain phenomena scientifically, procedural knowledge, and local/national has the greatest influence on Competencies, Knowledge, and Contexts, respectively. In addition, a simulation study was conducted to evaluate the psychometric properties of the proposed model. The results showed that the proposed Bayesian MCMC estimation algorithm can provide accurate model parameter estimation.

Overall, the proposed MO-DINA model works well in real data analysis and simulation study and meets the needs of assessment for PISA 2015 scientific literacy which included a third-order latent structure.

Key words: scientific literacy, cognitive diagnosis, PISA, DINA model

中图分类号:

B841

詹沛达, 于照辉, 李菲茗, 王立君. (2019). 一种基于多阶认知诊断模型测评科学素养的方法. 心理学报, 51(6), 734-746.

ZHAN Peida, YU Zhaohui, LI Feiming, WANG Lijun. (2019). Using a multi-order cognitive diagnosis model to assess scientific literacy. Acta Psychologica Sinica, 51(6), 734-746.

图/表 15

参考文献 43

[1]	Birnbaum A., ( 1968). Some latent trait models and their use in inferring a student’s ability. In F. M. Lord & M. R. Novick (Eds.). Statistical theories of mental test scores. Addison- Wesley, Reading, MA.
[2]	Brooks S. P., & Gelman A. , ( 1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics. 7(4), 434-455.
[3]	Chen F., Zhan P., Wang L., Chen C., & Cai M . ( 2015). The development and application of higher-order item response models. Advances in Psychological Science, 23, 150-157.
	[ 陈飞鹏, 詹沛达, 王立君, 陈春晓, 蔡毛 . ( 2015). 高阶项目反应模型的发展与应用. 心理科学进展, 23, 150-157.]
[4]	de la Torre J. , ( 2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45( 4), 343-362. doi: 10.1111/jedm.2008.45.issue-4 URL
[5]	de la Torre J. , ( 2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34( 1), 115-130 doi: 10.3102/1076998607309474 URL
[6]	de la Torre J., & Douglas J. A . ( 2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69( 3), 333-353. doi: 10.1007/BF02295640 URL
[7]	de la Torre J., & Song H. , ( 2009). Simultaneously estimation of overall and domain abilities: A higher-order IRT model approach. Applied Psychological Measurement, 33( 8), 620-639. doi: 10.1177/0146621608326423 URL
[8]	Gao H. B . ( 2011). Results of the eighth survey on Chinese citizens' scientific literacy were released. Bulletin of National Natural Science Foundation of China, 25, 63-64.
	[ 高宏斌 . ( 2011). 第八次中国公民科学素养调查结果发布. 中国科学基金, 25, 63-64.]
[9]	Hu Y., Yang S., & Lu K . ( 2012). The research of assessment tools of adolescents’ scientific literacy and its quality analysis. Education Research Monthly, 3, 16-21.
	[ 胡咏梅, 杨素红, 卢珂 . ( 2012). 青少年科学素养测评工具研发及质量分析. 教育学术月刊, 3, 16-21.]
[10]	Huang H.-Y., Wang W.-C., Chen P.-H., & Su C.-M . ( 2013). Higher-order item response models for hierarchical latent traits. Applied Psychological Measurement, 37( 8), 619-637. doi: 10.1177/0146621613488819 URL
[11]	Junker B. W., & Sijtsma K. , ( 2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272. doi: 10.1177/01466210122032064 URL
[12]	Karelitz T. M . ( 2004). Ordered category attribute coding framework for cognitive assessments (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign
[13]	Kang C., & Xin T. , ( 2010). New development in test theory: multidimensional item response theory. Advances in Psychological Science, 18( 3), 530-536
	[ 康春花, 辛涛 . ( 2010). 测验理论的新发展:多维项目反应理论. 心理科学进展, 18( 3), 530-536.]
[14]	Köhn H.-F. & Chiu C.-Y. , ( 2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests. Psychometrika, 82( 1), 112-132 doi: 10.1007/s11336-016-9536-7 URL
[15]	Leighton J. & Gierl, M. (Eds.). , ( 2007) . Cognitive diagnostic assessment for education: Theory and applications Cambridge University Press Theory and applications. Cambridge University Press.
[16]	Leighton J. P., Gierl M. J., & Hunka S. M . ( 2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41( 1), 205-237. doi: 10.1111/jedm.2004.41.issue-3 URL
[17]	Li F., Cohen A Bottge, B, & Templin, J. ., ( 2016). A latent transition analysis model for assessing change in cognitive skills. Educational and Psychological Measurement, 76( 2), 181-204. doi: 10.1177/0013164415588946 URL
[18]	Liu H., Liu Y., & Li M . ( 2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372.
[19]	Liu K., Li C . ( 2015). The content and characteristic of PISA 2015 draft science framework. Comparative Education Review, 37( 7), 98-105.
	[ 刘克文, 李川 . ( 2015). PISA 2015科学素养测试内容及特点. 比较教育研究, 37( 7), 98-105.]
[20]	Macready G. B., & Dayton C. M . ( 1977). The use of probabilistic models in the assessment of mastery. Journal of Educational and Behavioral Statistics, 2( 2), 99-120. doi: 10.3102/10769986002002099 URL
[21]	Miller J. D . ( 1983). Scientific literacy: A conceptual and empirical review. Daedalus, 112( 2), 29-48.
[22]	OECD. ( 2006). Assessing Scientific, Reading and Mathematical Literacy: A Framework for PISA 2006. Paris: PISA, OECD Publishing
[23]	OECD. ( 2016). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy. Paris: PISA, OECD Publishing
[24]	OECD. ( 2017). PISA 2015 Technical Report. Paris: PISA, OECD Publishing
[25]	Qin H., & Qian Y. , ( 2008). A survey report on Shanghai adolescents’ scientific literacy. Research in Educational Development,( 24), 31-35.
	[ 秦浩正, 钱源伟 . ( 2008). 上海青少年科学素养调查报告. 教育发展研究, ( 24), 31-35.]
[26]	Reckase M. D . ( 2009). Multidimensional item response theory. New York: Springer.
[27]	Ren L., Zhang C., & He W . ( 2013). Constructing and anallysis of the model of how the factors affect the scientific literacy of Chinese citizens and a comparative investigation. Studies in Science of Science, 31, 983-990.
	[ 任磊, 张超, 何薇 . ( 2013). 中国公民科学素养及其影响因素模型的建构与分析. 科学学研究, 31( 7), 983-990.]
[28]	Rijmen F., Jeon M., von Davier M., & Rabe-Hesketh S . ( 2014). A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. Journal of Educational and Behavioral Statistics, 39( 4), 235-256. doi: 10.3102/1076998614531045 URL
[29]	Roos J. M . ( 2014). Measuring science or religion? A measurement analysis of the National Science Foundation sponsored science literacy scale 2006-2010. Public Understanding of Science, 23( 7), 797-813. doi: 10.1177/0963662512464318 URL
[30]	Rupp A. A., Templin J., & Henson R. A . ( 2010). Diagnostic Measurement: Theory, Methods, and Applications. New York, NY: Guilford Press
[31]	Ryan J. J., & Schnakenberg-Ott S. D . ( 2003). Scoring reliability on the Wechsler Adult Intelligence Scale-Third Edition (WAIS-III). Assessment, 10( 2), 151-159. doi: 10.1177/1073191103010002006 URL
[32]	Templin J. L., Henson R. A., Templin S. E., & Roussos L . ( 2008). Robustness of hierarchical modeling of skill association in cognitive diagnosis models. Applied Psychological Measurement, 32( 7), 559-574. doi: 10.1177/0146621607300286 URL
[33]	The Ministry of Education of the People's Republic of China. ( 2017). Compulsory education primary school curriculum standards. Retrieved June 2, 2017, from
	[中华人民共和国教育部. ( 2017. 义务教育小学科学课程标准. ]
[34]	Tu D, Cai Y & Ding S. , ( 2012). Cognitive diagnosis: Theory, Methods, and Applications. Beijing: Beijing Normal University Publishing Group.
	[ 涂冬波, 蔡艳, 丁树良 . ( 2012). 认知诊断理论、方法与应用. 北京: 北京师范大学出版社.
[35]	Wang W.-C., & Chen P.-H. , ( 2004). Implementation and measurement efficiency of multidimensional computerized adaptive testing. Applied Psychological Measurement, 28( 5), 295-316. doi: 10.1177/0146621604265938 URL
[36]	Zhan P., Chen P., & Bian Y . ( 2016). Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis. Acta Psychologica Sinica, 48( 10), 1347-1356.
	[ 詹沛达, 陈平, 边玉芳 . ( 2016). 使用验证性补偿多维IRT模型进行认知诊断评估. 心理学报, 48( 10), 1347-1356.]
[37]	Zhan P., Jiao H., & Liao D . ( 2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71( 2), 262-286. doi: 10.1111/bmsp.2018.71.issue-2 URL
[38]	Zhan P., Jiao H., Liao D & Li F. , (in press). A longitudinal higher-order diagnostic classification model.Journal of Educational and Behavioral Statistics.
[39]	Zhan P., Jiao H., Liao M., & Bian Y . ( 2018). Bayesian DINA modeling incorporating within-item characteristic dependency. Applied Psychological Measurement. Advanced online publication. URL
[40]	Zhan P., Jiao H., Man K & Wang L. ,(in press). Using JAGS for Bayesian cognitive diagnosis modeling: A tutorial.Journal of Educational and Behavioral Statistics.
[41]	Zhan P., Ma W., Jiao H & Ding S. , (in press). A sequential higher-order latent structural model for hierarchical attributes in cognitive diagnostic assessments. Applied Psychological Measurement.
[42]	Zhan P., Wang W.-C & Li X. ,(in press). A partial mastery, higher-order latent structural model for polytomous attributes in cognitive diagnostic assessments.Journal of Classification.
[43]	Zhan P., Wang W.-C., & Wang L . ( 2013). Testlet response theory: an introduction and new developments. Advances in Psychological Science, 21( 12), 2265-2280.
	[ 詹沛达, 王文中, 王立君 . ( 2013). 项目反应理论新进展之题组反应理论. 心理科学进展, 21( 12), 2265-2280.]

题目	θ⁽³⁾
	θ₁⁽²⁾			θ₂⁽²⁾			θ₃⁽²⁾
	A1	A2	A3	A4	A5	A6	A7	A8	A9
DS269Q01	1			1					1
DS269Q03	1			1					1
CS269Q04	1			1					1
CS408Q01	1			1				1
DS408Q03	1			1				1
CS408Q04	1			1				1
CS408Q05		1			1			1
CS521Q02	1			1			1
CS521Q06	1			1			1
DS519Q01		1				1	1
CS519Q02	1			1			1
DS519Q03			1	1			1
CS527Q01			1			1			1
CS527Q03	1			1					1
CS527Q04	1					1			1
CS466Q01		1			1			1
CS466Q07		1				1		1
CS466Q05			1		1			1

题目	θ⁽³⁾
	θ₁⁽²⁾			θ₂⁽²⁾			θ₃⁽²⁾
	A1	A2	A3	A4	A5	A6	A7	A8	A9
DS269Q01	1			1					1
DS269Q03	1			1					1
CS269Q04	1			1					1
CS408Q01	1			1				1
DS408Q03	1			1				1
CS408Q04	1			1				1
CS408Q05		1			1			1
CS521Q02	1			1			1
CS521Q06	1			1			1
DS519Q01		1				1	1
CS519Q02	1			1			1
DS519Q03			1	1			1
CS527Q01			1			1			1
CS527Q03	1			1					1
CS527Q04	1					1			1
CS466Q01		1			1			1
CS466Q07		1				1		1
CS466Q05			1		1			1

模型	-2LL	AIC	BIC	DIC	ppp
MO-DINA	19332	19389	19673	24775	0.738
HO-DINA	19345	19399	19668	24644	0.716
DINA	19415	19962	22687	24856	0.692

模型	-2LL	AIC	BIC	DIC	ppp
MO-DINA	19332	19389	19673	24775	0.738
HO-DINA	19345	19399	19668	24644	0.716
DINA	19415	19962	22687	24856	0.692

题目	g_i		s_i	95% CI (g_i)	95% CI (s_i)		IDI_i
DS269Q01	0.325		0.119	[0.263, 0.386]	[0.082, 0.158]		0.556
DS269Q03	0.459		0.070	[0.397, 0.521]	[0.042, 0.102]		0.471
CS269Q04	0.237		0.351	[0.190, 0.289]	[0.304, 0.398]		0.412
CS408Q01	0.434		0.181	[0.373, 0.489]	[0.142, 0.222]		0.385
DS408Q03	0.033		0.810	[0.015, 0.058]	[0.776, 0.843]		0.157
CS408Q04	0.429		0.261	[0.374, 0.487]	[0.219, 0.300]		0.310
CS408Q05	0.295		0.213	[0.220, 0.357]	[0.160, 0.266]		0.492
CS521Q02	0.548		0.133	[0.494, 0.602]	[0.097, 0.170]		0.319
CS521Q06	0.849		0.008	[0.809, 0.883]	[0.002, 0.017]		0.143
DS519Q01	0.106		0.524	[0.047, 0.163]	[0.457, 0.582]		0.370
CS519Q02	0.281		0.304	[0.231, 0.332]	[0.256, 0.353]		0.415
DS519Q03	0.323		0.228	[0.212, 0.404]	[0.174, 0.282]		0.449
CS527Q01	0.033		0.788	[0.012, 0.055]	[0.742, 0.831]		0.179
CS527Q03	0.393		0.330	[0.343, 0.442]	[0.289, 0.371]		0.277
CS527Q04	0.281		0.373	[0.203, 0.343]	[0.316, 0.423]		0.346
CS466Q01	0.448		0.182	[0.378, 0.514]	[0.140, 0.226]		0.370
CS466Q07	0.649	0.050		[0.543, 0.726]		[0.026, 0.080]	0.301
CS466Q05	0.342	0.243		[0.284, 0.398]		[0.184, 0.300]	0.415