Abstract:

With the popularity of computer-based testings, the collection of item response times (RTs) and other process data has become a routine in large- and small-scale psychological and educational assessments. RTs not only provide information about the processing speed of participants but also could be utilized to improve the measurement accuracy. This is because the RTs are considered to convey a more synoptic depiction of the participants’ performance beyond responses alone. In multidimensional assessments, various skills are often required to answer questions. The speed at which persons were applying a set of skills reflecting distinct cognitive dimensions could be considered as multidimensional as well. In other words, each latent ability living in its specific dimensions was measured simultaneously with its corresponding working efficiency of applying a facet of skills designed in a multidimensional test. For example, the latent speed corresponding to the latent ability of decoding of an algebra question may differ from encoding. Therefore, a multidimensional RT model is needed to accommodate such scenario, which extends various currently proposed RT models assuming unidimensional processing speed.

To model the multidimensional structure of the latent processing speed, this study proposed a multidimensional log-normal response time model (MLRT) model, which is an extension of the unidimensional log-normal response time model (ULRTM) proposed by van der Linden (2006). Model parameters were estimated via the full Bayesian approach with the Markov chain Monte Carlo (MCMC). A PISA 2012 computer-based mathematics RT dataset was analyzed as a real data example. This dataset contains RTs of 1581 participants for 9 items. A Q-matrix (see Table 1) was prespecified based on the PISA 2012 mathematics assessment framework (see Zhan, Jiao, Liao, 2018). Three dimensions were defined based on the mathematical content knowledge, which are: 1) change and relationships (θ₁), 2) space and shape (θ₂), and 3) uncertainty and data (θ₃). One thing to note is that the defined Q-matrix served as a bridge to link items to the corresponding latent abilities, which shows the multidimensional structure of latent abilities. First, exploratory factor analysis (EFA) was conducted with the real dataset to manifest the multidimensional structure of the processing speed. Second, two RT models, i.e., the ULRTM and the MLRTM, were fitted to the data, and the results were compared. Third, a simulation study was conducted to evaluate the psychometric properties of the proposed model.

The EFA results indicated that the latent processing speed has a three-dimensional structure, which matches with the theoretical multidimensional structure of the latent abilities (i.e., the Q-matrix in Table 1). Furthermore, the ULRTM and the MLRTM yield adequate model data fits according to the posterior predictive model checking values (ppp = 0.597 for the ULRTM and ppp = 0.633 for the MLRTM). Furthermore, by comparing the values of the -2LL, DIC, and WAIC across the ULRTM and the MLRTM, the results indicate that the MLRTM fits the data better. In addition, the results show that (1) the correlations among three dimensions vary from medium to large (from 0.751 to 0.855); (2) the time-intensity parameters estimates of the two models were similar to each other. However, in terms of the time-discrimination parameters, the estimates of the ULRTM were slightly lower than the MLRTM. Moreover, the results from the simulation study show: 1) the model parameters were fully recovered with the Bayesian MCMC estimation algorithm; 2) the item time-discrimination parameter could be underestimated if the multidimensionality of the latent processing speed gets ignored, which meets our expectation, whereas the item time-intensity parameter stayed the same.

Overall, the proposed MLRTM performed well with the empirical data and the simulated data. In addition, the proposed model could facilitate practitioners in the use of the RT data to understand participants’ complex behavioral characteristics.

Key words: item response times, multidimensional latent processing speed, item response theory, computer-based testing, PISA