Binary modeling of action sequences in problem-solving tasks: One- and two-parameter action sequence model

doi:10.3724/SP.J.1041.2023.01383

Abstract

Abstract:

Process data refers to the human-computer or human-human interaction data recorded in computerized learning and assessment systems that reflect respondents’ problem-solving processes. Among the process data, action sequences are the most typical data because they reflect how respondents solve the problem step by step. However, the non-standardized format of action sequences (i.e., different data lengths for different participants) also poses difficulties for the direct application of traditional psychometric models. Han, Liu, and Ji (2022) proposed the sequential response model (SRM) by combining dynamic Bayesian networks with the nominal response model (NRM) to address the shortcomings of existing methods. Similar to the NRM, the SRM uses multinomial logistic modeling, which in turn assigns different parameters to each possible action or state transition in the task, leading to high model complexity. Given that actions or state transitions in problem-solving tasks have correct and incorrect outcomes rather than equivalence relations without quantitative order, this study proposed two action sequence models based on binary logistic modeling with relatively low model complexity: the one- and two-parameter action sequence models (1P and 2P-ASM). Unlike the SRM, which applies the NRM to action sequence analysis, the 1P-ASM and 2P-ASM adapt the simpler one- and two-parameter IRT models to action sequence analysis, respectively. An empirical example was provided to compare the performance of SRM and two ASMs with a real-world computer-based interactive item, “Tickets,” in the PISA 2012, and a simulation study was further conducted to explore the psychometric performance of the proposed models in different test scenarios.

Key words: process data, action sequence, problem state transition, action sequence model, item response theory

FU Yanbin, CHEN Qipeng, ZHAN Peida. (2023). Binary modeling of action sequences in problem-solving tasks: One- and two-parameter action sequence model. Acta Psychologica Sinica, 55(8), 1383-1396.

Figures/Tables 30

References 48

[1]	Arieli-Attali M., Ou L., & Simmering V. R. (2019). Understanding test takers' choices in a self-adapted test: A hidden Markov modeling of process data. Frontiers in Psychology, 10, 83. doi: 10.3389/fpsyg.2019.00083 pmid: 30787889
[2]	Beck L. W. (1943). The principle of parsimony in empirical science. The Journal of Philosophy, 40(23), 617-633. https://doi.org/10.2307/2019692 doi: 10.2307/2019692 URL
[3]	Bergner Y., Walker E., & Ogan A. (2017). Dynamic Bayesian network models for peer tutoring interactions. In A. A. von Davier, M. Zhu, & P. C. Kyllonen (Eds), Innovative assessment of collaboration (pp. 249-268). Cham: Springer.
[4]	Birnbaum A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp.397-124). Reading, MA: Addison-Wesley.
[5]	Bock R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29-51. https://doi.org/10.1007/BF02291411 doi: 10.1007/BF02291411 URL
[6]	Buchner A., & Funke J. (1993). Finite-state automata: Dynamic task environments in problem-solving research. The Quarterly Journal of Experimental Psychology, 46(1), 83-118. doi: 10.1080/14640749308401068 URL
[7]	Chen Y. (2020). A continuous-time dynamic choice measurement model for problem-solving process data. Psychometrika, 85(4), 1052-1075. doi: 10.1007/s11336-020-09734-1 pmid: 33346883
[8]	Fu Y., Zhan P., Chen Q., & Jiao H. (2022). Joint modeling of action sequences and action times in problem-solving tasks. PsyArXiv. Retrieved from psyarxiv.com/e3nbc
[9]	Gelman A., Meng X.-L., & Stern H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6, 733-760.
[10]	Gelman A., & Rubin D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457-511.
[11]	Han Y., Liu H., & Ji F. (2022). A sequential response model for analyzing process data on technology-based problem-solving tasks. Multivariate Behavioral Research, 57(6), 960-977. doi: 10.1080/00273171.2021.1932403 URL
[12]	Han Y., & Wilson M. (2022). Analyzing student response processes to evaluate success on a technology-based problem-solving task. Applied Measurement in Education, 35(1), 33-45. doi: 10.1080/08957347.2022.2034821 URL
[13]	Han Y., Xiao Y., & Liu H. (2022). Feature extraction and ability estimation of process data in the problem-solving test. Advances in Psychological Science, 30(6), 1393-1409. doi: 10.3724/SP.J.1042.2022.01393
[14]	Hao J., Shu Z., & von Davier A. (2015). Analyzing process data from game/scenario-based tasks: An edit distance approach. Journal of Educational Data Mining, 7(1), 33-50.
[15]	Harding S. M. E., Griffin P. E., Awwal N., Alom B. M., & Scoular C. (2017). Measuring collaborative problem solving using mathematics-based tasks. AERA Open, 3(3), 1-19.
[16]	He Q., Borgonovi F., & Paccagnella M. (2021). Leveraging process data to assess adults’ problem-solving skills: Using sequence mining to identify behavioral patterns across digital tasks. Computers & Education, 166, 104170. doi: 10.1016/j.compedu.2021.104170 URL
[17]	He Q., & von Davier M. (2016). Analyzing process data from problem-solving items with N-grams:Insights from a computer-based large-scale assessment. In R. Yigal, F. Steve, & M. Maryam (Eds.), Handbook of research on technology tools for real-world skill development (pp. 749-776). Hershey, PA: Information Science Reference.
[18]	Hoffman M. D., & Gelman A. (2014). The No-U-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593-1623.
[19]	LaMar M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67-88. doi: 10.1007/s11336-017-9570-0 pmid: 28447309
[20]	Levy R. (2019). Dynamic Bayesian network modeling of game-based diagnostic assessments. Multivariate Behavioral Research, 54(6), 771-794. doi: 10.1080/00273171.2019.1590794 pmid: 30942094
[21]	Li M., Liu Y., Liu H. (2020). Analysis of the Problem-solving strategies in computer-based dynamic assessment: The extension and application of multilevel mixture IRT model. Acta Psychologica Sinica, 52(4), 528-540. doi: 10.3724/SP.J.1041.2020.00528
[22]	Liu H., Liu Y., & Li M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372. doi: 10.3389/fpsyg.2018.01372 pmid: 30123171
[23]	Liu Y., Xu H., Chen Q., & Zhan P. (2022). The measurement of problem-solving competence using process data. Advances in Psychological Science, 30(3), 522-535. doi: 10.3724/SP.J.1042.2022.00522
[24]	Ma W., Iaconangelo C., & de la Torre J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40(3), 200-217. doi: 10.1177/0146621615621717 pmid: 29881048
[25]	Man K., Harring J. R., & Zhan P. (2022). Bridging models of biometric and psychometric assessment: A three-way joint modeling approach of item responses, response times and gaze fixation counts. Applied Psychological Measurement, 46(5), 361-381. doi: 10.1177/01466216221089344 URL
[26]	Newell A., & Simon H. A. (1972). Human problem solving (Vol. 104, No. 9). Englewood Cliffs.
[27]	OECD. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. OECD Publishing. http://dx.doi.org/10.1787/9789264190511-en
[28]	Peng S., Cai Y., Wang D., Luo F., & Tu D. (2022). A generalized diagnostic classification modeling framework integrating differential speediness: Advantages and illustrations in psychological and educational testing. Multivariate Behavioral Research, 57(6), 940-959. doi: 10.1080/00273171.2021.1928474 URL
[29]	Rasch G. (1960). On general laws and the meaning of measurement in psychology. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability: Held at the Statistical Laboratory, University of California, June 20-July 30, 1960 (Vol. 4, pp. 321). University of California Press.
[30]	Rosen Y. (2017). Assessing students in human‐to‐agent settings to inform collaborative problem‐solving learning. Journal of Educational Measurement, 54(1), 36-53. doi: 10.1111/jedm.2017.54.issue-1 URL
[31]	Shu Z., Bergner Y., Zhu M., Hao J., & von Davier A. A. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109-131.
[32]	Tang X., Wang Z., He Q., Liu J., & Ying Z. (2020). Latent feature extraction for process data via multidimensional scaling. Psychometrika, 85(2), 378-397. doi: 10.1007/s11336-020-09708-3 pmid: 32572672
[33]	Tang X., Wang Z., Liu J., & Ying Z. (2021). An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology, 74(1), 1-33.
[34]	van der Linden W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181-204.
[35]	van der Linden W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287-308. doi: 10.1007/s11336-006-1478-z URL
[36]	Vehtari A., Gelman A., & Gabry J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413-1432. doi: 10.1007/s11222-016-9696-4
[37]	Vista A., Care E., & Awwal N. (2017). Visualising and examining sequential actions as behavioural paths that can be interpreted as markers of complex behaviours. Computers in Human Behavior, 76, 656-671. doi: 10.1016/j.chb.2017.01.027 URL
[38]	Watanabe S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11(12), 3571-3594.
[39]	Wilson M., Gochyyev P., & Scalise K. (2017). Modeling data from collaborative assessments: learning in digital interactive social networks. Journal of Educational Measurement, 54(1), 85-102. doi: 10.1111/jedm.2017.54.issue-1 URL
[40]	Xiao Y., He Q., Veldkamp B., & Liu H. (2021). Exploring latent states of problem‐solving competence using hidden Markov model on process data. Journal of Computer Assisted Learning, 37(5), 1232-1247. doi: 10.1111/jcal.v37.5 URL
[41]	Xiao Y., & Liu H. (2023). A state response measurement model for problem-solving process data. Behavior Research Methods, Online First.
[42]	Yuan J., Xiao Y., & Liu H. (2019). Assessment of collaborative problem solving based on process stream data: A new paradigm for extracting indicators and modeling dyad data. Frontiers in Psychology, 10, 369. doi: 10.3389/fpsyg.2019.00369 pmid: 30863344
[43]	Zhan P., Jiao H., & Liao D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262-286. doi: 10.1111/bmsp.2018.71.issue-2 URL
[44]	Zhan P., Man K., Wind S. A., & Malone J. (2022). Cognitive diagnosis modeling incorporating response times and fixation counts: Providing comprehensive feedback and accurate diagnosis. Journal of Educational and Behavioral Statistics, 47(6), 736-776.
[45]	Zhan P., & Qiao X. (2022). Diagnostic classification analysis of problem-solving competence using process data: An item expansion method. Psychometrika, 87(4), 1529-1547. doi: 10.1007/s11336-022-09855-9 pmid: 35389193
[46]	Zhan S., Hao J., & Davier A. V. (2015). Analyzing process data from game/scenariobased tasks: An edit distance approach. Journal of Educational Data Mining, 7(1), 33-50.
[47]	Zhang S., Wang Z., Qi J., Liu J., & Ying Z. (2022). Accurate assessment via process data. Psychometrika, 88(1), 76-97. doi: 10.1007/s11336-022-09880-8 pmid: 35962849
[48]	Zhu M., Shu Z., & von Davier A. A. (2016). Using networks to visualize and analyze process data for educational assessment. Journal of Educational Measurement, 53(2), 190-211. doi: 10.1111/jedm.12107 URL

States in Current Stage (Items)	States in the Next Stage (Options)
S	A (1)	F (0)
A	B (1)	G (0)	S (0)
B	C (1)	H (0)	S (0)
C	D (1)	E (0)	S (0)	J (0)
D	J (1)	E (0)	S (0)
E	D (1)	J (0)	S (0)
F	S (1)	G (0)
G	S (1)	H (0)
H	S (1)	I (0)	J (0)
I	S (1)	J (0)

States in Current Stage (Items)	States in the Next Stage (Options)
S	A (1)	F (0)
A	B (1)	G (0)	S (0)
B	C (1)	H (0)	S (0)
C	D (1)	E (0)	S (0)	J (0)
D	J (1)	E (0)	S (0)
E	D (1)	J (0)	S (0)
F	S (1)	G (0)
G	S (1)	H (0)
H	S (1)	I (0)	J (0)
I	S (1)	J (0)

Model	LOO	WAIC	ppp	Computational time (in seconds)
1P-ASM	11018.208	11007.133	0.511	647.5
2P-ASM	10363.785	10275.475	0.518	958.5
SRM	16804.501	16803.925	0.498	1958.6

Model	LOO	WAIC	ppp	Computational time (in seconds)
1P-ASM	11018.208	11007.133	0.511	647.5
2P-ASM	10363.785	10275.475	0.518	958.5
SRM	16804.501	16803.925	0.498	1958.6

Current State	1P-ASM			2P-ASM
	Action easiness parameter			Action easiness parameter			Action discrimination parameter
	Posterior Mean	Posterior Standard Deviation	95% HPD	Posterior Mean	Posterior Standard Deviation	95% HPD	Posterior Mean	Posterior Standard Deviation	95% HPD
S	0.911	0.046	(0.822, 1.001)	0.969	0.057	(0.860, 1.084)	1.343	0.116	(1.111, 1.570)
A	1.553	0.066	(1.425, 1.682)	1.547	0.077	(1.401, 1.701)	1.457	0.212	(1.043, 1.870)
B	1.432	0.073	(1.290, 1.577)	1.354	0.082	(1.198, 1.521)	1.797	0.398	(0.958, 2.566)
C	1.436	0.080	(1.279, 1.599)	1.207	0.148	(0.940, 1.526)	3.015	0.885	(1.104, 4.759)
E	2.008	0.107	(1.801, 2.215)	1.734	0.159	(1.456, 2.099)	1.615	0.495	(0.463, 2.576)
D	0.361	0.176	(0.015, 0.702)	0.472	0.283	(?0.031, 1.064)	1.472	0.952	(0.225, 3.792)
F	?1.705	0.107	(?1.918, ?1.492)	?1.590	0.123	(?1.829, ?1.348)	1.438	0.250	(0.982, 1.974)
G	?1.888	0.111	(?2.105, ?1.677)	?1.747	0.147	(?2.050, ?1.480)	2.115	0.354	(1.495, 2.875)
H	?0.749	0.075	(?0.898, ?0.599)	?0.292	0.172	(?0.636, 0.037)	2.229	0.426	(1.400, 3.088)
I	?0.368	0.157	(?0.686, 0.062)	0.760	0.470	(?0.101, 1.753)	3.127	0.933	(1.525, 5.179)