Abstract:

In recent years, response time has been attracting rapidly growing attention in psychometric research, likely due to the increasing availability of (item-level) response time data through computer-based testing and online survey data collection. Compared to the conventional item response data that are often dichotomous or polytomous, response time is continuous and can provide much more information. Aberrant response behaviors are frequently encountered during testing, and it could cause various negative effects. Change point analysis (CPA) is a well-established statistical process control method to detect changes in a sequence, and it has provided testing professionals a new lens through to understand test-taking behavior at both the examinee and item levels.

In this paper, we first gave a comprehensive summary and analysis of the application of change-point analysis in the field of psychometrics. Then we took test speededness as an example to illustrate how the CPA method can be used to detect aberrant behavior using item response time data. Two CPA-based statistics were introduced, as well as their rationale. In the simulation study, there were two cases of response time data: one was that the item parameters were known, the other was that the item parameter were unknown. Response time under speededness was simulated using the gradual-change log-normal model for response time. Two CPA-based test statistics, the Likelihood Ratio Test and the Wald Test, were used to detect aberrant response behaviors. The critical values were obtained through Monte Carlo simulations and compared with the approximate critical values in a previous study. Based on the chosen critical values, we examined the performance of the likelihood ratio test and Wald test in detecting speeded responses, specifically in terms of power and empirical Type-I error.

On the one hand, the critical values are almost identical for the Wald and the likelihood ratio test. They vary substantially at different nominal α levels, but do not differ much across different test lengths. On the other hand, results indicate that the proposed method is much more powerful based on the critical values than conventional methods that use item response data. When the item parameters are known, the power was close to 1 for most of the conditions while keeping the type-I error rate well-controlled. When the item parameters are unknown, the power of the statistic decreases slightly, but its lowest value reaches 0.89. In the case of factors other than item parameters, the results under the same conditions are only slightly decreased. See Tables 1 and 2 for more detailed results, which show that unknown item parameters may have a negative impact on performance. Real data analysis also demonstrates the performance of the method.

This study applied CPA based on response time data and offered a very promising approach to detecting aberrant response behavior. Through the simulation study, we demonstrated that it was possible to use fixed critical values in different test lengths, which makes the application of the method straightforward. CPA is very flexible. This study assumed that the log-normal model fitted the response time data, but the method is not bounded by that assumption.

Key words: change point analysis, aberrant response behaviors, response time, test speededness, statistical process control