ISSN 0439-755X
CN 11-1911/B
主办:中国心理学会
   中国科学院心理研究所
出版:科学出版社

心理学报 ›› 2020, Vol. 52 ›› Issue (12): 1452-1465.doi: 10.3724/SP.J.1041.2020.01452

• 研究报告 • 上一篇    

基于基尼指数的双目标CD-CAT选题策略

罗芬1,2, 王晓庆2, 蔡艳1, 涂冬波1()   

  1. 1 江西师范大学心理学院, 南昌 330022
    2 江西师范大学计算机信息工程学院, 南昌 330022
  • 收稿日期:2019-10-14 出版日期:2020-12-25 发布日期:2020-10-26
  • 通讯作者: 涂冬波 E-mail:tudongbo@aliyun.com
  • 基金资助:
    * 国家自然科学基金(61967009);国家自然科学基金(31660278);国家自然科学基金(31760288);国家自然科学基金(31960186);江西省教育厅科学技术研究项目(GJJ150356);江西省教育厅科学技术研究项目(GJJ160282)

A new dual-objective CD-CAT item selection method based on the Gini index

LUO Fen1,2, WANG Xiaoqing2, CAI Yan1, TU Dongbo1()   

  1. 1 School of Psychology, Jiangxi Normal University, Nanchang 330022, China
    2 College of Computer Information Engineering, Jiangxi Normal University, Nanchang 330022, China
  • Received:2019-10-14 Online:2020-12-25 Published:2020-10-26
  • Contact: TU Dongbo E-mail:tudongbo@aliyun.com

摘要:

双目标CD-CAT的测验结果既可用于形成性评估也可用于终结性评估。基尼指数可度量随机变量的不确定性程度, 值越小则随机变量的不确定程度越低。本文用基尼指数度量被试知识状态类别以及能力估计置信区间后验概率的变化, 提出基于基尼指数的选题策略。Monte Carlo实验表明与已有的选题策略相比, 新策略的知识状态分类精度和能力估计精度都较高, 同时能有效兼顾题库利用均匀性, 并能快速实时响应, 且受认知诊断模型和被试知识状态分布的影响较小, 可用于实际测验中含多种认知诊断模型的混合题库。

关键词: 认知诊断, 项目反应理论, 基尼指数, 双目标CD-CAT, 选题策略

Abstract:

Existing literature has shown that dual-objective CD-CAT testing can facilitate the achievement of measurement objectives for both formative and summative assessments. And the Gini Index can be used as a measurement for the degree of uncertainty of random variables since a smaller Gini value indicates a lower degree of uncertainty. Hence, this paper proposed a Gini-Index-based selection method for dual-objective CD-CAT, and it measured the changes in the posterior probability of knowledge state and confidence interval for latent traits estimation. By adopting the Bayesian Decision Theory, the potential information of participants could be detected based on participants’ responses and changes in posterior probability distribution of two the random variables.
Monte Carlo Simulation was used to test the performances of the selection method based on Gini, ASI, IPA and JSD, respectively. The item banks measured 5 attributes consisting of 250 items in total, and each item measured 3 attributes at most. The true knowledge state of each participant was generated by HO-CDM and Multivariate Normal Models (both means were 0 and covariance coefficient was 0.8 and 0.2, respectively). G-DINA, DINA and R-RUM were adopted as the cognitive diagnostic models and the item bank of each of these three models included both CDM and 2PL parameters. Specifically, CDM parameters were generated by a G-DINA package in R software with the slipping and guessing parameters randomly selected from uniform distribution in a range from 0.05 to 0.25. The 2PL parameters were estimated by factoring in the responses elicited from 3, 000 participants’ responses to all items in item banks using the mirt package. Four indexes, namely the pattern match ratio, root mean square error of latent trait, chi-square value and time needed for item selection, were adopted in comparing the efficiency of different item selection methods. The value for each index was the mean of 10 repeated simulations of 1, 000 participants’ responses to all item bank.
The results showed that (1) The Gini and IPA selection methods had similar performance in terms of pattern match ratio, root mean square error of latent trait and chi-square value. Both methods were high in precision measurement and low in sensitivity to CDM and the distribution of participants’ cognitive patterns, making both methods applicable to the item banks featuring a mixture of cognitive diagnosis models. By comparison, the Gini method outperformed slightly the IPA method in pattern match ratio and time needed for item selection in which the Gini method was only one-tenth that of the IPA method; (2) Both the Gini and ASI selection methods were weighted linear combination approaches. The performances of the two methods were very close in the short test. In the long test, however, although time needed for item selection using the ASI method was only one-third that of the Gini method, the latter was superior to the former in terms of measurement accuracy and chi-square value; (3) Although the JSD method outperformed the Gini method in terms of uniformity of item bank usage and time needed for item selection, its measurement accuracy was far less than the latter.
To summarize, the Gini, IPA and ASI selection methods all have good measurement accuracy and hence are all recommended for short tests. For medium and long tests with a limited number of attributes and a smaller item bank, the Gini and IPA selection methods are recommended. As the number of attributes and item bank size grow, the Gini method is recommended. When there are high correlations among different attributes, as well as a large number of attributes and big item bank size, the ASI and JSD selection methods are recommended with the ASI method slightly outperforming the JSD method in measurement accuracy.

Key words: cognitive diagnostic, items response theory, Gini index, dual objective CD-CAT, selection method

中图分类号: