关键词: 过程数据, 问题解决, 项目反应理论

Abstract:

In the past decade, computer-based interactive problem-solving tests have become increasingly popular in large-scale assessments. Such tests require examinees to interact with a computer, explore virtual scenarios, and solve practical problems, thus making it possible to record the sequences of actions performed by examinees (i.e., process data). Process data contain rich information about the problem-solving process and can help to gain a deeper understanding of examinees’ problem-solving strategies. Methods for analyzing such process data are still under development. For example, Han et al. (2021) proposed a sequential response model (SRM) that combines comprehensive information from the response process to infer problem-solving ability. Fu et al. (2023) replaced the multinomial logistic modeling in SRM with binary logistic modeling and proposed 1P-ASM with relatively lower model complexity. However, existing studies have ignored the fact that students gradually gather information while completing problem-solving tasks (i.e., learning effect). The probability that an examinee performs the correct behavior is affected by their understanding of the problem situation. If the model does not take this into account, it may result in biased estimate of examinee’s problem-solving ability. To address this issue, this paper puts forward a new model (denoted as 1P-ASM-R*), which extends 1P-ASM to incorporate this learning effect to obtain more accurate ability estimates.

An empirical study was performed to compare 1P-ASM and 1P-ASM-R* in a real-world interactive assessment item (i.e., “Tickets”) in the PISA 2012. The results showed that: (1) the extended model introducing learning effect fitted the empirical data better than the original model; (2) as the examinees delve deeper into the problem, the impact of learning effect on the accuracy of behavioral choices in problem-solving tasks decreased, reflecting a trend of diminishing marginal effect; and (3) introducing learning effect into the model does not affect its ability to capture the characteristics of the problem-solving tasks.

A simulation study was further conducted to explore the psychometric performance of the proposed model in different test scenarios. Three factors were manipulated, they are sample size (200 and 1000), average problem state transition sequence length (short and long), and strength of learning effect (0, 0.1, and 0.3). The problem-solving task structure in the empirical study was used here and 1P-ASM-R* was used to generate the action sequences of the examinees. The results indicated that: (1) when there was no learning effect, 1P-ASM-R* could provide similar fitting performance to the original model and correctly estimate the learning effect parameter as 0. However, when there was a learning effect, 1P-ASM-R* fits the data better, and this advantage became more pronounced as the strength of the learning effect increased; (2) sequence length is one of the factors affecting the parameter recovery of 1P-ASM-R*. The longer the sequence length, the more information the data contains and the higher the parameter estimation accuracy.

In summary, our proposed 1P-ASM-R* model incorporates the learning effect and demonstrates a strong ability to accurately analyze examinees’ problem-solving abilities. The combination of simulation and empirical findings highlights the effectiveness of the model in a variety of contexts. Notably, when the task environment lacks a learning effect, the 1P-ASM-R* model exhibits comparable performance to the original 1P-ASM model. This finding underscores the excellent stability and adaptability of the model, indicating that it can function reliably under different conditions.

Key words: process data, problem-solving, item response theory