Please wait a minute...
Acta Psychologica Sinica    2019, Vol. 51 Issue (6) : 734-746     DOI: 10.3724/SP.J.1041.2019.00734
Reports of Empirical Studies |
Using a multi-order cognitive diagnosis model to assess scientific literacy
ZHAN Peida(),YU Zhaohui,LI Feiming,WANG Lijun
College of Teacher Education, Zhejiang Normal University, Jinhua 321004, China
Download: PDF(756 KB)   HTML Review File (1 KB) 
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks     Supporting Info
Guide   
Abstract  

In PISA 2015, scientific literacy is defined as “the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen”. There are four interdependent dimensions are specified in the scientific literacy assessment framework for PISA 2015: Competencies, Knowledge, Contexts, and Attitudes. Given that knowledge of scientific literacy contributes significantly to individuals’ personal, social, and professional lives, it is of vital importance to find an objectively and accurately assessment method for scientific literacy. However, only unidimensional IRT models were used in the analysis in PISA 2015. Which means that the analysis model does not match with such a multidimensional assessment framework. It is desired to develop a new analysis model. This study attempts to measure scientific literacy in cognitive diagnostic assessment for the first time.

According to the scientific literacy assessment framework for PISA 2015, a third-order latent structure for scientific literacy is first pointed out. Specifically, the scientific literacy is treated as the third-order latent trait; Competencies, Knowledge, Contexts, and Attitudes are all treated as second-order latent traits; And nine subdomains, e.g., explain phenomena scientifically and content knowledge, were treated as first-order traits (or attributes). Unfortunately, however, there is still a lack of cognitive diagnosis models that can deal with such a third-order latent structure. To this end, a multi-order DINA (MO-DINA) model was developed in this study. The new model is an extension of the higher-order (HO-DINA) model, which is similar to the third-order IRT models. To illustrate the application and advantages of the MO-DINA model, a sub-data of PISA 2015 science assessment data were analyzed. Items were chosen from the S01 cluster, and participants were chosen from China. After data cleaning, 1076 participants with 18 items were retained. Three models were fitted to this sub-data and compared, the MO-DINA model, in which the third-order latent structure of scientific literacy was considered; the HO-DINA model, in which the scientific literacy was treated as a second-order latent trait and contacted with attributes directly; and the DINA model.

All three models appear to provide a reasonably good fit to data according to the posterior predictive model checking. According to the -2LL, AIC, BIC, and DIC, the DINA model fits the data worst, and the MO-DINA model fits the data best, the results of MO-DINA model are used to make further interpretations. The results indicated that (1) the quality of 18 items are not good enough; (2) The correlations among second-order latent traits are high (0.8, approximately); (3) Knowledge has the greatest influence on scientific literacy, Contexts second, and Competencies least; (4) Explain phenomena scientifically, procedural knowledge, and local/national has the greatest influence on Competencies, Knowledge, and Contexts, respectively. In addition, a simulation study was conducted to evaluate the psychometric properties of the proposed model. The results showed that the proposed Bayesian MCMC estimation algorithm can provide accurate model parameter estimation.

Overall, the proposed MO-DINA model works well in real data analysis and simulation study and meets the needs of assessment for PISA 2015 scientific literacy which included a third-order latent structure.

Keywords scientific literacy      cognitive diagnosis      PISA      DINA model     
ZTFLH:  B841  
Corresponding Authors: Peida ZHAN     E-mail: pdzhan@gmail.com
Issue Date: 25 April 2019
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
Peida ZHAN
Zhaohui YU
Feiming LI
Lijun WANG
Cite this article:   
Peida ZHAN,Zhaohui YU,Feiming LI, et al. Using a multi-order cognitive diagnosis model to assess scientific literacy[J]. Acta Psychologica Sinica, 2019, 51(6): 734-746.
URL:  
http://journal.psych.ac.cn/xlxb/EN/10.3724/SP.J.1041.2019.00734     OR     http://journal.psych.ac.cn/xlxb/EN/Y2019/V51/I6/734
  
  
  
  
题目 θ(3)
θ1(2) θ2(2) θ3(2)
A1 A2 A3 A4 A5 A6 A7 A8 A9
DS269Q01 1 1 1
DS269Q03 1 1 1
CS269Q04 1 1 1
CS408Q01 1 1 1
DS408Q03 1 1 1
CS408Q04 1 1 1
CS408Q05 1 1 1
CS521Q02 1 1 1
CS521Q06 1 1 1
DS519Q01 1 1 1
CS519Q02 1 1 1
DS519Q03 1 1 1
CS527Q01 1 1 1
CS527Q03 1 1 1
CS527Q04 1 1 1
CS466Q01 1 1 1
CS466Q07 1 1 1
CS466Q05 1 1 1
  
模型 -2LL AIC BIC DIC ppp
MO-DINA 19332 19389 19673 24775 0.738
HO-DINA 19345 19399 19668 24644 0.716
DINA 19415 19962 22687 24856 0.692
  
题目 gi si 95% CI (gi) 95% CI (si) IDIi
DS269Q01 0.325 0.119 [0.263, 0.386] [0.082, 0.158] 0.556
DS269Q03 0.459 0.070 [0.397, 0.521] [0.042, 0.102] 0.471
CS269Q04 0.237 0.351 [0.190, 0.289] [0.304, 0.398] 0.412
CS408Q01 0.434 0.181 [0.373, 0.489] [0.142, 0.222] 0.385
DS408Q03 0.033 0.810 [0.015, 0.058] [0.776, 0.843] 0.157
CS408Q04 0.429 0.261 [0.374, 0.487] [0.219, 0.300] 0.310
CS408Q05 0.295 0.213 [0.220, 0.357] [0.160, 0.266] 0.492
CS521Q02 0.548 0.133 [0.494, 0.602] [0.097, 0.170] 0.319
CS521Q06 0.849 0.008 [0.809, 0.883] [0.002, 0.017] 0.143
DS519Q01 0.106 0.524 [0.047, 0.163] [0.457, 0.582] 0.370
CS519Q02 0.281 0.304 [0.231, 0.332] [0.256, 0.353] 0.415
DS519Q03 0.323 0.228 [0.212, 0.404] [0.174, 0.282] 0.449
CS527Q01 0.033 0.788 [0.012, 0.055] [0.742, 0.831] 0.179
CS527Q03 0.393 0.330 [0.343, 0.442] [0.289, 0.371] 0.277
CS527Q04 0.281 0.373 [0.203, 0.343] [0.316, 0.423] 0.346
CS466Q01 0.448 0.182 [0.378, 0.514] [0.140, 0.226] 0.370
CS466Q07 0.649 0.050 [0.543, 0.726] [0.026, 0.080] 0.301
CS466Q05 0.342 0.243 [0.284, 0.398] [0.184, 0.300] 0.415
  
参数 后验均值 95% CI 相关系数
Σ σβ2 1.773 [0.873, 3.571] 1.000
ρβδσβσδ -1.833 [-3.719, -0.856] -0.890
σδ2 2.394 [1.145, 4.778] 1.000
μ μβ -0.783 [-1.408, -0.154]
μδ -1.212 [-1.924, -0.493]
  
  
被试 α θ1(2) θ2(2) θ3(2) θ(3)
2 111111111 0.582 [-0.863, 2.194] 0.661 [-0.586, 2.174] 0.656 [-0.572, 2.175] 0.664 [-0.581, 2.194]
5 010001000 -0.873 [-2.317, 0.537] -0.940 [-2.290, 0.276] -0.910 [-2.307, 0.357] -0.939 [-2.302, 0.263]
7 010000000 -0.919 [-2.429, 0.541] -1.022 [-2.432, 0.198] -1.028 [-2.445, 0.211] -1.027 [-2.453, 0.183]
23 111111111 0.202 [-1.182, 1.950] 0.283 [-1.057, 1.961] 0.338 [-0.999, 1.959] 0.294 [-1.035, 1.968]
54 010101000 -0.831 [-2.414, 0.620] -0.880 [-2.319, 0.461] -0.870 [-2.368, 0.525] -0.886 [-2.341, 0.426]
86 111101110 -0.404 [-2.082, 1.368] -0.462 [-2.054, 1.314] -0.468 [-2.034, 1.293] -0.467 [-2.062, 1.300]
  
  
  
  
参数 tbias RMSE Cor
平均绝对值 标准差 最小值 最大值 平均值 标准差 最小值 最大值
θ(3) 0.100 0.124 -0.380 0.368 0.686 0.090 0.408 0.983 0.721
θ1(2) 0.100 0.125 -0.378 0.352 0.689 0.092 0.385 0.983 0.719
θ2(2) 0.104 0.126 -0.372 0.351 0.683 0.089 0.416 0.947 0.726
θ3(2) 0.104 0.130 -0.481 0.381 0.690 0.095 0.358 1.050 0.715
  
参数 bias RMSE Cor
平均绝对值 标准差 最小值 最大值 平均值 标准差 最小值 最大值
λ0k 0.042 0.048 -0.066 0.072 0.189 0.062 0.129 0.305 0.982
λ1km 0.116 0.051 0.015 0.172 0.346 0.057 0.245 0.429 0.982
γ1(2) -0.031 0.053
γ2(2) -0.012 0.076
γ3(2) -0.012 0.076
  
[1] Birnbaum A., ( 1968). Some latent trait models and their use in inferring a student’s ability. In F. M. Lord & M. R. Novick (Eds.). Statistical theories of mental test scores. Addison- Wesley, Reading, MA.
[2] Brooks S. P., & Gelman A. , ( 1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics. 7(4), 434-455.
[3] Chen F., Zhan P., Wang L., Chen C., & Cai M . ( 2015). The development and application of higher-order item response models. Advances in Psychological Science, 23, 150-157.
[3] [ 陈飞鹏, 詹沛达, 王立君, 陈春晓, 蔡毛 . ( 2015). 高阶项目反应模型的发展与应用. 心理科学进展, 23, 150-157.]
[4] de la Torre J. , ( 2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45( 4), 343-362.
url: http://blackwell-synergy.com/doi/abs/10.1111/jedm.2008.45.issue-4
[5] de la Torre J. , ( 2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34( 1), 115-130
url: http://journals.sagepub.com/doi/10.3102/1076998607309474
[6] de la Torre J., & Douglas J. A . ( 2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69( 3), 333-353.
url: http://link.springer.com/10.1007/BF02295640
[7] de la Torre J., & Song H. , ( 2009). Simultaneously estimation of overall and domain abilities: A higher-order IRT model approach. Applied Psychological Measurement, 33( 8), 620-639.
url: http://journals.sagepub.com/doi/10.1177/0146621608326423
[8] Gao H. B . ( 2011). Results of the eighth survey on Chinese citizens' scientific literacy were released. Bulletin of National Natural Science Foundation of China, 25, 63-64.
[8] [ 高宏斌 . ( 2011). 第八次中国公民科学素养调查结果发布. 中国科学基金, 25, 63-64.]
[9] Hu Y., Yang S., & Lu K . ( 2012). The research of assessment tools of adolescents’ scientific literacy and its quality analysis. Education Research Monthly, 3, 16-21.
[9] [ 胡咏梅, 杨素红, 卢珂 . ( 2012). 青少年科学素养测评工具研发及质量分析. 教育学术月刊, 3, 16-21.]
[10] Huang H.-Y., Wang W.-C., Chen P.-H., & Su C.-M . ( 2013). Higher-order item response models for hierarchical latent traits. Applied Psychological Measurement, 37( 8), 619-637.
url: http://journals.sagepub.com/doi/10.1177/0146621613488819
[11] Junker B. W., & Sijtsma K. , ( 2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272.
url: http://journals.sagepub.com/doi/10.1177/01466210122032064
[12] Karelitz T. M . ( 2004). Ordered category attribute coding framework for cognitive assessments (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign
[13] Kang C., & Xin T. , ( 2010). New development in test theory: multidimensional item response theory. Advances in Psychological Science, 18( 3), 530-536
[13] [ 康春花, 辛涛 . ( 2010). 测验理论的新发展:多维项目反应理论. 心理科学进展, 18( 3), 530-536.]
[14] Köhn H.-F. & Chiu C.-Y. , ( 2017). A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests. Psychometrika, 82( 1), 112-132
url: http://link.springer.com/10.1007/s11336-016-9536-7
[15] Leighton J. & Gierl, M. (Eds.). , ( 2007) . Cognitive diagnostic assessment for education: Theory and applications Cambridge University Press Theory and applications. Cambridge University Press.
[16] Leighton J. P., Gierl M. J., & Hunka S. M . ( 2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41( 1), 205-237.
url: http://www.blackwell-synergy.com/toc/jedm/41/3
[17] Li F., Cohen A Bottge, B, & Templin, J. ., ( 2016). A latent transition analysis model for assessing change in cognitive skills. Educational and Psychological Measurement, 76( 2), 181-204.
url: http://journals.sagepub.com/doi/10.1177/0013164415588946
[18] Liu H., Liu Y., & Li M . ( 2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372.
[19] Liu K., Li C . ( 2015). The content and characteristic of PISA 2015 draft science framework. Comparative Education Review, 37( 7), 98-105.
[19] [ 刘克文, 李川 . ( 2015). PISA 2015科学素养测试内容及特点. 比较教育研究, 37( 7), 98-105.]
[20] Macready G. B., & Dayton C. M . ( 1977). The use of probabilistic models in the assessment of mastery. Journal of Educational and Behavioral Statistics, 2( 2), 99-120.
url: http://journals.sagepub.com/doi/10.3102/10769986002002099
[21] Miller J. D . ( 1983). Scientific literacy: A conceptual and empirical review. Daedalus, 112( 2), 29-48.
[22] OECD. ( 2006). Assessing Scientific, Reading and Mathematical Literacy: A Framework for PISA 2006. Paris: PISA, OECD Publishing
[23] OECD. ( 2016). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematic and Financial Literacy. Paris: PISA, OECD Publishing
[24] OECD. ( 2017). PISA 2015 Technical Report. Paris: PISA, OECD Publishing
[25] Qin H., & Qian Y. , ( 2008). A survey report on Shanghai adolescents’ scientific literacy. Research in Educational Development,( 24), 31-35.
[25] [ 秦浩正, 钱源伟 . ( 2008). 上海青少年科学素养调查报告. 教育发展研究, ( 24), 31-35.]
[26] Reckase M. D . ( 2009). Multidimensional item response theory. New York: Springer.
[27] Ren L., Zhang C., & He W . ( 2013). Constructing and anallysis of the model of how the factors affect the scientific literacy of Chinese citizens and a comparative investigation. Studies in Science of Science, 31, 983-990.
[27] [ 任磊, 张超, 何薇 . ( 2013). 中国公民科学素养及其影响因素模型的建构与分析. 科学学研究, 31( 7), 983-990.]
[28] Rijmen F., Jeon M., von Davier M., & Rabe-Hesketh S . ( 2014). A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. Journal of Educational and Behavioral Statistics, 39( 4), 235-256.
url: http://journals.sagepub.com/doi/10.3102/1076998614531045
[29] Roos J. M . ( 2014). Measuring science or religion? A measurement analysis of the National Science Foundation sponsored science literacy scale 2006-2010. Public Understanding of Science, 23( 7), 797-813.
url: http://journals.sagepub.com/doi/10.1177/0963662512464318
[30] Rupp A. A., Templin J., & Henson R. A . ( 2010). Diagnostic Measurement: Theory, Methods, and Applications. New York, NY: Guilford Press
[31] Ryan J. J., & Schnakenberg-Ott S. D . ( 2003). Scoring reliability on the Wechsler Adult Intelligence Scale-Third Edition (WAIS-III). Assessment, 10( 2), 151-159.
url: http://journals.sagepub.com/doi/10.1177/1073191103010002006
[32] Templin J. L., Henson R. A., Templin S. E., & Roussos L . ( 2008). Robustness of hierarchical modeling of skill association in cognitive diagnosis models. Applied Psychological Measurement, 32( 7), 559-574.
url: http://journals.sagepub.com/doi/10.1177/0146621607300286
[33] The Ministry of Education of the People's Republic of China. ( 2017). Compulsory education primary school curriculum standards. Retrieved June 2, 2017, from
url: http://www.moe.edu.cn/srcsite/A26/ s8001/201702/t20170215_296305.html
[33] [中华人民共和国教育部. ( 2017. 义务教育小学科学课程标准. ]
url: http://www.moe.edu.cn/srcsite/A26/ s8001/201702/t20170215_296305.html
[34] Tu D, Cai Y & Ding S. , ( 2012). Cognitive diagnosis: Theory, Methods, and Applications. Beijing: Beijing Normal University Publishing Group.
[34] [ 涂冬波, 蔡艳, 丁树良 . ( 2012). 认知诊断理论、方法与应用. 北京: 北京师范大学出版社.
[35] Wang W.-C., & Chen P.-H. , ( 2004). Implementation and measurement efficiency of multidimensional computerized adaptive testing. Applied Psychological Measurement, 28( 5), 295-316.
url: http://journals.sagepub.com/doi/10.1177/0146621604265938
[36] Zhan P., Chen P., & Bian Y . ( 2016). Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis. Acta Psychologica Sinica, 48( 10), 1347-1356.
[36] [ 詹沛达, 陈平, 边玉芳 . ( 2016). 使用验证性补偿多维IRT模型进行认知诊断评估. 心理学报, 48( 10), 1347-1356.]
[37] Zhan P., Jiao H., & Liao D . ( 2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71( 2), 262-286.
url: http://doi.wiley.com/10.1111/bmsp.2018.71.issue-2
[38] Zhan P., Jiao H., Liao D & Li F. , (in press). A longitudinal higher-order diagnostic classification model.Journal of Educational and Behavioral Statistics.
[39] Zhan P., Jiao H., Liao M., & Bian Y . ( 2018). Bayesian DINA modeling incorporating within-item characteristic dependency. Applied Psychological Measurement. Advanced online publication. URL
url: https://doi.org/10.1177/0146621618781594
[40] Zhan P., Jiao H., Man K & Wang L. ,(in press). Using JAGS for Bayesian cognitive diagnosis modeling: A tutorial.Journal of Educational and Behavioral Statistics.
[41] Zhan P., Ma W., Jiao H & Ding S. , (in press). A sequential higher-order latent structural model for hierarchical attributes in cognitive diagnostic assessments. Applied Psychological Measurement.
[42] Zhan P., Wang W.-C & Li X. ,(in press). A partial mastery, higher-order latent structural model for polytomous attributes in cognitive diagnostic assessments.Journal of Classification.
[43] Zhan P., Wang W.-C., & Wang L . ( 2013). Testlet response theory: an introduction and new developments. Advances in Psychological Science, 21( 12), 2265-2280.
[43] [ 詹沛达, 王文中, 王立君 . ( 2013). 项目反应理论新进展之题组反应理论. 心理科学进展, 21( 12), 2265-2280.]
[1] LIU Yanlou; XIN Tao; LI Lingqing; TIAN Wei; LIU Xiaoxiao. An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix[J]. Acta Psychologica Sinica, 2016, 48(5): 588-598.
[2] ZHAN Peida; BIAN Yufang; WANG Lijun. Factors affecting the classification accuracy of reparametrized diagnostic classification models for expert-defined polytomous attributes[J]. Acta Psychologica Sinica, 2016, 48(3): 318-330.
[3] ZHAN Peida; CHEN Ping; BIAN Yufang. Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis[J]. Acta Psychologica Sinica, 2016, 48(10): 1347-1356.
[4] CAI Yan; MIAO Ying; TU Dongbo. The polytomously scored cognitive diagnosis computerized adaptive testing[J]. Acta Psychologica Sinica, 2016, 48(10): 1338-1346.
[5] KANG Chunhua; REN Ping; ZENG Pingfei. Nonparametric Cognitive Diagnosis: A Cluster Diagnostic Method  Based on Grade Response Items[J]. Acta Psychologica Sinica, 2015, 47(8): 1077-1088.
[6] TANG Xiaojuan; DING Shuliang; YU Zonghuo. Application of Rough Set Theory in Item Cognitive Attribute Identification[J]. Acta Psychologica Sinica, 2015, 47(7): 950-962.
[7] ZHAN Peida; LI Xiaomin; WANG Wen-Chung; BIAN Yufang; WANG Lijun. The Multidimensional Testlet-Effect Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2015, 47(5): 689-701.
[8] YU Xiaofeng; LUO Zhaosheng; GAO Chunlei; LI Yujun; WANG Rui; WANG Yutong. An Item Attribute Specification Method Based On the Likelihood D2 Statistic[J]. Acta Psychologica Sinica, 2015, 47(3): 417-426.
[9] YU Xiaofeng; LUO Zhaosheng; QIN Chunying; GAO Chunlei; LI Yujun. Joint Estimation of Model Parameters and Q-Matrix Based on Response Data[J]. Acta Psychologica Sinica, 2015, 47(2): 273-282.
[10] CAI Yan; TU Dongbo. Extension of Cognitive Diagnosis Models Based on the Polytomous Attributes Framework and Their Q-matrices Designs[J]. Acta Psychologica Sinica, 2015, 47(10): 1300-1308.
[11] GUO Lei; ZHENG Chanjin; BIAN Yufang. Exposure Control Methods and Termination Rules in Variable-Length Cognitive Diagnostic Computerized Adaptive Testing[J]. Acta Psychologica Sinica, 2015, 47(1): 129-140.
[12] WANG Zhuoran; GUO Lei; BIAN Yufang. Comparison of DIF Detecting Methods in Cognitive Diagnostic Test[J]. Acta Psychologica Sinica, 2014, 46(12): 1923-1932.
[13] TU Dongbo;CAI Yan;DAI Haiqi. Comparison and Selection of Five Noncompensatory Cognitive Diagnosis Models Based on Attribute Hierarchy Structure[J]. Acta Psychologica Sinica, 2013, 45(2): 243-252.
[14] CAI Yan;TU Dongbo;DING Shuliang. A Simulation Study to Compare Five Cognitive Diagnostic Models[J]. Acta Psychologica Sinica, 2013, 45(11): 1295-1304.
[15] TU Dong-Bo,CAI Yan,DAI Hai-Qi. A New Method of Q-matrix Validation Based on DINA Model[J]. , 2012, 44(4): 558-568.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Acta Psychologica Sinica
Support by Beijing Magtech