Development of main effect DIF and interactive DIF detection method in cognitive diagnosis assessments: A recursive partitioning-based perspective

doi:10.3724/SP.J.1041.2026.0995

Abstract

Abstract: With the growing recognition of the advantages of cognitive diagnosis (CD) in psychological and educational measurement, applying the CD framework to test development has become an important research direction in the field of psychology. In the development of cognitive diagnostic assessments, detecting differential item functioning (DIF) remains a crucial quality control procedure to ensure test fairness and validity. However, existing CD-based DIF detection methods typically focus on a single covariate at a time. While these approaches are effective for identifying main effect DIF induced by a single covariate, they are limited in detecting interactive DIF caused by the interaction among multiple covariates. Such limitations may compromise the fairness and interpretability of assessment outcomes. To address this issue, the present study integrates CD modeling with recursive partitioning techniques by proposing a novel DIF detection method, namely the Item-based Sequential Recursive Partitioning Method (ISRPM). Building on the core principles of recursive partitioning, the ISRPM allows the simultaneous consideration of multiple covariates within a single DIF detection procedure and facilitates the identification of both main effect DIF and interactive DIF in cognitive diagnostic assessments.
To evaluate the performance of the proposed method, a series of Monte Carlo simulation studies were conducted focusing on two key objectives: (1) examining how factors such as sample size per group, DIF magnitude, DIF type, item quality, correlations among attributes, and the influence of demographic covariates on attribute mastery distribution affect the performance of ISRPM; and (2) comparing ISRPM with several existing DIF detection methods across varied experimental conditions. In addition, to illustrate its practical utility, ISRPM was applied to a cognitive diagnostic version of the Schizotypal Personality Questionnaire (DC-SPQ) and compared with five established DIF detection methods.
The results showed that (1) sample size, DIF magnitude, and item quality substantially influenced the performance of all methods; and (2) when items exhibited interactive DIF, ISRPM achieved higher detection accuracy than the Wald, LR, FS-Wald, FS-LR, and Mantel-Haenszel (MH) approaches. When only the main effect DIF was present, the overall performance of ISRPM was comparable to that of the existing methods.
These findings suggest that ISRPM provides a flexible and effective framework for identifying both main effect DIF and interactive DIF in cognitive diagnostic assessments, thereby contributing to methodological advancements in fairness evaluation and the broader application of CD-based measurement in psychological and educational measurement.

Key words: cognitive diagnosis assessments, differential item functioning, main effect DIF, interactive DIF, recursive partitioning

LIU Kai, GUO Zhichen, WANG Qin, WANG Daxun, CAI Yan, TU Dongbo. (2026). Development of main effect DIF and interactive DIF detection method in cognitive diagnosis assessments: A recursive partitioning-based perspective. Acta Psychologica Sinica, 58(5), 995-1014.

References

[1] American Education Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). The standards for educational and psychological testing. Washington, DC: AERA Publications.
[2] Barnett, V., & Lewis, T. (1994). Outliers in statistical data. Hoboken: Wiley.
[3] Bauer, D. J. (2017). A more general model for testing measurement invariance and differential item functioning.Psychological Methods, 22(3), 507-526.
[4] Belzak, W. C. (2023). The multidimensionality of measurement bias in high-stakes testing: Using machine learning to evaluate complex sources of differential item functioning.Educational Measurement: Issues and Practice, 42(1), 24-33.
[5] Belzak, W. C. M., & Bauer, D. J. (2020). Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning.Psychological Methods, 25(6), 673-690.
[6] Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289-300.
[7] Bollmann S., Berger M., & Tutz G. (2018). Item-focused trees for the detection of differential item functioning in partial credit models.Educational and Psychological Measurement, 78(5), 781-804.
[8] Collins, P. H. (1990). Black feminist thought: Knowledge, consciousness, and the politics of empowerment. UnwinHyman.
[9] de la Torre, J. (2011). The generalized DINA model framework.Psychometrika, 76(2), 179-199.
[10] de la Torre J., van der Ark, L. A., & Rossi G. (2018). Analysis of clinical data from a cognitive diagnosis modeling framework.Measurement and Evaluation in Counseling and Development, 51(4), 281-296.
[11] DiBello L. V., Roussos L. A., & Stout W. (2006). 31a review of cognitively diagnostic assessment and a summary of psychometric models.Handbook of Statistics, 26, 979-1030.
[12] Finch W. H., Hernández Finch M. E., & French B. F. (2015). Recursive partitioning to identify potential causes of differential item functioning in cross-national data. International Journal of Testing, 16(1), 21-53.
[13] Holland, P. W., & Wainer, H. (1993). Differential item functioning. Hillsdale, NJ: Erlbaum.
[14] Hothorn T., Hornik K., & Zeileis A. (2006). Unbiased recursive partitioning: A conditional inference framework.Journal of Computational and Graphical Statistics, 15(3), 651-674.
[15] Hou, L. (2013). Differential item functioning assessment in cognitive diagnostic modeling: Applying the Wald test to investigate DIF in the generalized DINA model framework (Unpublished doctoral dissertation). University of Delaware.
[16] Hou L., de la Torre J., & Nandakumar R. (2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the Wald test to investigate DIF in the DINA model.Journal of Educational Measurement, 51(1), 98-125.
[17] Komboz B., Strobl C., & Zeileis A. (2016). Tree-based global model tests for polytomous Rasch models.Educational and Psychological Measurement, 78(1), 128-166.
[18] Leighton J. P.,& Gierl, M. (2007). Cognitive diagnostic assessment for education: Theory and Applications Cambridge, UK: Cambridge University Press Theory and Applications. Cambridge, UK: Cambridge University Press.
[19] Li, F. (2008). A modified higher-order DINA model for detecting differential item functioning and differential attribute functioning (Unpublished doctoral dissertation). University of Georgia.
[20] Li L., Zhou X., Huang J., Tu D., Gao X., Yang Z., & Li M. (2020). Assessing kindergarteners’ mathematics problem solving: The development of a cognitive diagnostic test.Studies in Educational Evaluation, 66, 100879.
[21] Li, X., & Wang, W.-C. (2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52(1), 28-54.
[22] Liu Y., Xin T., Li L., Tian W., & Liu X. (2016). An improved method for differential item functioning detection in cognitive diagnosis models: An application of Wald statistic based on observed information matrix.Acta Psychologica Sinica, 48(5), 588-598.
[刘彦楼, 辛涛, 李令青, 田伟, 刘笑笑. (2016). 改进的认知诊断模型项目功能差异检验方法——基于观察信息矩阵的Wald统计量.心理学报, 48(5), 588-598.]
[23] Ma, W., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling.Journal of Statistical Software, 93(14), 1-26.
[24] Ma W., Terzi R., & de la Torre, J. (2021). Detecting differential item functioning using multiple-group cognitive diagnosis models.Applied Psychological Measurement, 45(1), 37-53.
[25] Magis D., Béland S., Tuerlinckx F., & de Boeck P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning.Behavior Research Methods, 42(3), 847-862.
[26] Meade, A. W., & Wright, N. A. (2012). Solving the measurement invariance anchor item problem in item response theory.Journal of Applied Psychology, 97(5), 1016-1031.
[27] Mehrazmay R., Ghonsooly B., & de la Torre, J. (2021). Detecting differential item functioning using cognitive diagnosis models: Applications of the Wald test and likelihood ratio test in a university entrance examination.Applied Measurement in Education, 34(4), 262-284.
[28] Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance.Psychometrika, 58(4), 525-543.
[29] Nichols P. D., Chipman S. F., & Brennan R. L. (1995). Cognitively diagnostic assessment. Routledge.
[30] Paulsen J., Svetina D., Feng Y., & Valdivia M. (2020). Examining the impact of differential item functioning on classification accuracy in cognitive diagnostic models.Applied Psychological Measurement, 44(4), 267-281.
[31] R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.Rproject.org/
[32] Rupp A. A., Templin J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications Guilford Press Theory, methods, and applications. Guilford Press.
[33] Strobl C., Kopf J., & Zeileis A. (2015). Rasch trees: A new method for detecting differential item functioning in the Rasch model.Psychometrika, 80(2), 289-316.
[34] Strobl C., Malley J., & Tutz G. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests.Psychological Methods, 14(4), 323-348.
[35] Sun X., Liu Y., Wang S., Xin T., Song N., & Zhou M. (2022). Using information matrix-based method to detect differential item functioning with multiple groups in cognitive diagnostic test.Journal of Psychological Science, 45(3), 710-717.
[孙小坚, 刘彦楼, 王诗梦, 辛涛, 宋乃庆, 周蔓. (2022). 认知诊断测验中基于信息矩阵的多群组DIF检验.心理科学, 45(3), 710-717.]
[36] Tan Z., de La Torre J., Ma W., Huh D., Larimer M. E., & Mun E.-Y. (2023). A tutorial on cognitive diagnosis modeling for characterizing mental health symptom profiles using existing item responses.Prevention Science, 24(3), 480-492.
[37] Tay L., Huang Q., & Vermunt J. K. (2015). Item response theory with covariates (IRT-C): Assessing item recovery and differential item functioning for the three-parameter logistic model.Educational and Psychological Measurement, 76(1), 22-42.
[38] Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models.Psychological Methods, 11(3), 287-305.
[39] Tu D., Cai Y., Gao X., & Wang, D. (2019). Advanced cognitive diagnosis. Beijing: Beijing Normal University Publishing Group.
[涂冬波, 蔡艳, 高旭亮, 汪大勋. (2019). 高级认知诊断. 北京: 北京师范大学出版社.]
[40] Tutz, G., & Berger, M. (2016). Item-focussed trees for the identification of items in differential item functioning.Psychometrika, 81(3), 727-750.
[41] Wang D., Gao X., Cai Y., & Tu D. (2019). Development of a new instrument for depression with cognitive diagnosis models. Frontiers in Psychology, 10, 1306.
[42] Wang, X. (2019). Development and verification of cognitive diagnostic test for cross-grade pupils’ mathematics learning ability,Chinese Exam, 8, 71-78.
[王欣瑜. (2019). 跨年级小学数学学力认知诊断测验的开发与验证.中国考试, 8, 71-78.]
[43] Wang Z., Guo L., & Bian Y. (2014). Comparison of DIF detecting methods in cognitive diagnostic test.Acta Psychologica Sinica, 46(12), 1923-1932.
[王卓然, 郭磊, 边玉芳. (2014). 认知诊断测验中的项目功能差异检测方法比较.心理学报, 46(12), 1923-1932.]
[44] Xi C., Cai Y., Peng S., Lian J., & Tu D. (2020). A diagnostic classification version of schizotypal personality questionnaire using diagnostic classification models.International Journal of Methods in Psychiatric Research, 29(1), e1807.
[45] Yuan K. H., Liu H., & Han Y. (2021). Differential item functioning analysis without a priori information on anchor items: QQ plots and graphical test.Psychometrika, 86(2), 345-377.
[46] Zhang, W. (2006). Detecting differential item functioning using the DINA model (Unpublished doctoral dissertation). The University of North Carolina at Greensboro.