Joint-cross-loading multimodal cognitive diagnostic modeling incorporating visual fixation counts

doi:10.3724/SP.J.1041.2022.01416

Abstract

Abstract:

Multimodal data enables accurate cognitive structure diagnosis as well as comprehensive feedback on various cognitive characteristics (e.g., cognitive style). This work presents three multimodal cognitive diagnosis models based on joint-cross-loading modeling to achieve joint analysis of response accuracy, response time (RT), and visual fixation counts (FC). The results of the empirical and simulation studies show that: (1) joint analysis is more suitable for multimodal data than separated analysis; (2) the proposed models can directly use the information in RT and FC to improve the estimation accuracy of latent ability and latent attributes; (3) the proposed models' parameters can be well recovered; and (4) negative results caused by ignoring cross-loading are more serious than those caused by redundant consideration.

Key words: cognitive diagnosis, multimodal data, item response times, fixation counts, cognitive style, eye-tracking

ZHAN Peida. (2022). Joint-cross-loading multimodal cognitive diagnostic modeling incorporating visual fixation counts. Acta Psychologica Sinica, 54(11), 1416-1432.

Figures/Tables 23

References 56

[1]	An L., Wang Y., & Sun Y. (2017). Reading words or pictures: Eye movement patterns in adults and children differ by age group and receptive language ability. Frontiers in Psychology, 8, 791. https://doi.org/10.3389/fpsyg.2017.00791
[2]	Bezirhan U., von Davier M., & Grabovsky I. (2021). Modeling item revisit behavior: The hierarchical speed-accuracy-revisits model. Educational and Psychological Measurement, 81(2), 363-387.
[3]	Biancarosa G., & Shanley L. (2015). What is fluency? In K. D. Cummings & Y. Petscher (Eds.),The fluency construct: Curriculum- based measurement concepts and applications (pp. 1-18). Springer.
[4]	Bolsinova M., de Boeck P., & Tijmstra J. (2017). Modelling conditional dependence between response time and accuracy. Psychometrika, 82, 112-1148. https://doi.org/10.1007/s11336-016-9537-6
[5]	Bolsinova M., & Tijmstra J. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71(1), 13-38.
[6]	Bos W., Lankes E.-M., Prenzel M., Schwippert K., Valtin R., & Walther G. (Eds). ( 2005). IGLU: Vertiefende Analysen zu Leseverständnis, Rahmenbedingungen und Zusatzstudien [IGLU: Supplementary in-depth analyses of reading comprehension, context effects, and additional studies]. Münster: Waxmann.
[7]	Brooks S. P., & Gelman A. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7(4), 434-455. https://doi.org/10.2307/1390675
[8]	de Boeck P., & Jeon M. (2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102.
[9]	de la Torre J. (2011). The generalized DINA model framework. Psychometrika, 76, 179-199.
[10]	de la Torre J., & Douglas J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353. https://doi.org/10.1007/BF02295640
[11]	Gardner R. W., Holzman P. S., Klein G. S., Linton H. B., & Spence D. (1959). Cognitive control: A study of individual consistencies in cognitive behavior. Psychological Issues, 1, Monograph 4.
[12]	Gelman A., Carlin J. B., Stern H. S., Dunson D. B., Vehtari A., & Rubin D. B. (2014). Bayesian data analysis. Boca Raton: CRC Press.
[13]	Gu Y., & Xu G. (2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31, 449-472.
[14]	Guo L. Shang P., & Xia L. (2017). Advantages and illustrations of application of response time model in psychological and educational testing. Advances in Psychological Science, 25(4), 701-712.
[15]	Holzman P. S. (1966). Scanning: A principle of reality contact. Perceptual and Motor Skills, 23, 835-844.
[16]	Jeon M., de Boeck P., Luo J., Li X., & Lu Z.-L. (2021). Modeling within-item dependencies in parallel data on test responses and brain activation. Psychometrika, 86(1), 239-271. https://doi.org/10.1007/s11336-020-09741-2
[17]	Junker B. W., & Sijtsma K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272.
[18]	Kagan J. (1965). Reflection-impulsivity and reading ability in primary grade children. Child Development, 36(3), 609-628.
[19]	Lahat D., Adali T., & Jutten C. (2015). Multimodal data fusion: An overview of methods, challenges, and prospects. Proceedings of the IEEE, 103(9), 1449-1477.
[20]	Levy R., & Mislevy R. J. (2016). Bayesian psychometric modeling. Boca Raton, FL: CRC Press.
[21]	Li M., Liu Y., & Liu H. (2020). Analysis of the problem-solving strategies in computer-based dynamic assessment: The extension and application of multilevel mixture IRT model. Acta Psychologica Sinica, 52(4), 528-540.
[22]	Liu H., Liu Y., & Li M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372.
[23]	Liu Y., Xu H., Chen Q., & Zhan P. (2022). The measurement of problem-solving competence using process data. Advances in Psychological Science, 30(3), 522-525.
[24]	Ma W., & de la Torre J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253-275.
[25]	Ma W., & Guo W. (2019). Cognitive diagnosis models for multiple strategies. British Journal of Mathematical and Statistical Psychology, 72(2), 370-392.
[26]	Man K., & Harring J. R. (2019). Negative binomial models for visual fixation counts on test items. Educational and Psychological Measurement, 79(4), 617-635. http://doi.org/0.1177/0013164418824148
[27]	Man K., & Harring J. R. (2020). Assessing preknowledge cheating via innovative measures: A multiple-group analysis of jointly modeling item responses, response times, and visual fixation counts. Educational and Psychological Measurement, 81(3), 441-465. https://doi.org/10.1177/0013164420968630
[28]	Messick S. (1989). Cognitive style and personality: Scanning and orientation toward affect. ETS Research Report Series, RR-89-16. https://doi.org/10.1002/j.2330-8516.1989.tb00342.x
[29]	Molenaar D., Tuerlinckx F., & van der Maas H. L. (2015). A bivariate generalized linear item response theory modeling Framework to the Analysis of Responses and Response Times. Multivariate Behavioral Research, 50(1), 56-74.
[30]	Plummer M. (2015). Jags: Just another Gibbs sampler (version 4.0.0).
[31]	Poole A., Ball L. J., & Phillips P. (2004). In search of salience:A response-time and eye-movement analysis of bookmark recognition. In S. Fincher, P. Markopoulos, D. Moore, & R. Ruddle (Eds.), People and computers xviii: Design for life (pp. 363-378). London, England: Springer.
[32]	Ranger J. (2013). A note on the hierarchical model for responses and response times in tests of van der Linden (2007). Psychometrika, 78(3), 538-544.
[33]	Ren H., Xu N., Lin Y., Zhang S., & Yang T. (2021). Remedial teaching and learning from a cognitive diagnostic model perspective: Taking the data distribution characteristics as an example. Frontiers in Psychology, 12, 628607. https://doi.org/10.3389/fpsyg.2021.628607
[34]	Riding R. J. (1997). On the nature of cognitive style. Educational Psychology, 17(1-2), 29-49.
[35]	Rimawi O., Al-Halabiyah F., & Hussein O. (2020). The cognitive style (focusing-scanning) among Al-Quds University students. International Journal of Humanities and Cultural Studies, 7(1), 143-154.
[36]	Rupp A. A., Templin J. L., & Henson R. (2010). Diagnostic measurement: Theory, methods, and applications. New York: Guilford Press.
[37]	Schaeffer G. A., Reese C. M., Steffen M., McKinley R. L., & Mills C. N. (1993). Field test of a computer-based GRE general test (ETS Research report No. 93-07). Princeton, NJ: Educational Testing Service.
[38]	Tang F., & Zhan P. (2021). Does diagnostic feedback promote learning? Evidence from a longitudinal cognitive diagnostic assessment. AERA Open, 7. https://doi.org/10.1177/23328584211060804
[39]	Unkelbach C. (2006). The learned interpretation of cognitive fluency. Psychological Science, 17(4), 339-345.
[40]	van der Linden W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181-204.
[41]	van der Linden W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287-308.
[42]	von Davier M., & Lee Y.-S. (2019). Handbook of diagnostic classification models: Models and model extensions, applications, software packages. New York, NY: Springer.
[43]	Wang L., Tang F., & Zhan P. (2020). Effect analysis of individualized remedial teaching based on cognitive diagnostic assessment: Taking “linear equation with one unknown” as an example. Journal of Psychological Science, 43(6), 1490-1497.
[44]	Wang S., & Chen Y. (2020). Using response times and response accuracy to measure fluency within cognitive diagnosis models. Psychometrika, 85(2), 600-629.
[45]	Wang W. C., & Qiu X. L. (2019). Multilevel modeling of cognitive diagnostic assessment: The multilevel DINA example. Applied Psychological Measurement, 43(1), 34-50.
[46]	Wise S. L., & Kong X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163-183
[47]	Zhan P. (2018). Bayesian cognitive diagnosis modeling incorporating time information: joint analysis of response times and response accuracy data (Unpublished doctoral dissertation). Beijing Normal University.
[48]	Zhan P. (2019). Joint modeling for response times and response accuracy in computer-based multidimensional assessments. Journal of Psychological Science, 42(1), 170-178.
[49]	Zhan P. (2020). A Markov estimation strategy for longitudinal learning diagnosis: Providing timely diagnostic feedback. Educational and Psychological Measurement, 80(6), 1145-1167. https://doi.org/10.1177/0013164420912318
[50]	Zhan P., Jiao H., & Liao D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262-286.
[51]	Zhan P., Jiao H., Man K, & Wang L. (2019). Using JAGS for Bayesian cognitive diagnosis modeling: A tutorial. Journal of Educational and Behavioral Statistics, 44(4), 473-503.
[52]	Zhan P., Liao M., & Bian Y. (2018). Joint testlet cognitive diagnosis modeling for paired local item dependence in response times and response accuracy. Frontiers in Psychology, 9, 607.
[53]	Zhan P., Man K., Wind S. A., & Malone J. (2022). Cognitive diagnosis modeling incorporating response times and fixation counts: Providing comprehensive feedback and accurate diagnosis. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/10769986221111085
[54]	Zhan P., & Qiao X. (2022). Diagnostic Classification analysis of problem-solving competence using process data: An item expansion method. Psychometrika. https://doi.org/10.1007/s11336-022-09855-9
[55]	Zheng T., Zhou W., & Guo L. (in press). Cognitive diagnosis modelling based on response times. Journal of Psychological Science.
[56]	Zoanetti N. (2010). Interactive computer based assessment tasks: How problem-solving process data can inform instruction. Australasian Journal of Educational Technology, 26(5), 585-606.

φ_i	λ_i
φ_i	> 0	< 0
> 0	Much key information and low cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↓, FC↑; If f(θ_n, α_n, q_i) ↓, then RT↑, FC↓;	Less key information and low cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↓, FC↓; If f(θ_n, α_n, q_i) ↓, then RT↑, FC↑;
< 0	Much key information and high cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↑, FC↑; If f(θ_n, α_n, q_i) ↓, then RT↓, FC↓;	Less key information and high cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↑, FC↓; If f(θ_n, α_n, q_i) ↓, then RT↓, FC↑;

φ_i	λ_i
φ_i	> 0	< 0
> 0	Much key information and low cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↓, FC↑; If f(θ_n, α_n, q_i) ↓, then RT↑, FC↓;	Less key information and low cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↓, FC↓; If f(θ_n, α_n, q_i) ↓, then RT↑, FC↑;
< 0	Much key information and high cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↑, FC↑; If f(θ_n, α_n, q_i) ↓, then RT↓, FC↓;	Less key information and high cognitive load requirements: If f(θ_n, α_n, q_i) ↑, then RT↑, FC↓; If f(θ_n, α_n, q_i) ↓, then RT↓, FC↑;

θ	τ	ε	Corresponding Inference	Possible Cause or Behavior
+	+	+	Fluency + Focuser	Able to extract key information from items comprehensively
+	+	-	Fluency + Non-focuser	Able to eliminate distractions and extract rare key information from items
+	-	+	Reflective + Focuser	Deliberate on possible solutions based on the key information extracted
+	-	-	Reflective + Non-focuser	Able to eliminate distractions and deliberate based on a few key information
-	+	+	Impulsive + Focuser	Quickly extract some information and quickly validate the solution
-	+	-	Impulsive + Non-focuser	Under-extracting information and tending to make rapid-guessing
-	-	+	Non-fluency + Focuser	Motivated to respond but has the low cognitive ability
-	-	-	Non-fluency + Non-focuser	Unmotivated

θ	τ	ε	Corresponding Inference	Possible Cause or Behavior
+	+	+	Fluency + Focuser	Able to extract key information from items comprehensively
+	+	-	Fluency + Non-focuser	Able to eliminate distractions and extract rare key information from items
+	-	+	Reflective + Focuser	Deliberate on possible solutions based on the key information extracted
+	-	-	Reflective + Non-focuser	Able to eliminate distractions and deliberate based on a few key information
-	+	+	Impulsive + Focuser	Quickly extract some information and quickly validate the solution
-	+	-	Impulsive + Non-focuser	Under-extracting information and tending to make rapid-guessing
-	-	+	Non-fluency + Focuser	Motivated to respond but has the low cognitive ability
-	-	-	Non-fluency + Non-focuser	Unmotivated

Model	-2LL	DIC	ppp_RA	ppp_RT	ppp_FC
S-MCDM	11104.13	11890.70	0.42	0.52	0.78
H-MCDM	11040.62	11804.00	0.42	0.51	0.57
C-MCDM-θ	10275.27	11542.78	0.43	0.51	0.68
C-MCDM-D	10127.21	10549.14	0.40	0.52	0.64
C-MCDM-C	10009.65	10434.95	0.36	0.51	0.67