关键词: 题组反应模型, 多维项目反应模型, 项目内多维题组效应, 多维题组效应模型, Rasch模型

Abstract:

Testlet design has been widely adopted in educational and psychological assessment. A testlet is a cluster of items that share a common stimulus (e.g., a reading comprehension passage or a figure), and the possible local dependence among items within a testlet is called testlet-effect. Various models have been developed to take into account such testlet effect. Examples included the Rasch testlet model, two-parameter logistic Bayesian testlet model, and higher-order testlet model. However, these existing models all assume that an item is affected by only one single testlet effect. Therefore, they are essentially unidimensional testlet-effect models. In practice, multiple testlet effects may simultaneously affect item responses in a testlet. For example, in addition to common stimulus, items can be grouped according to their domains, knowledge units, or item format, such that multiple testlet effects are involved. In essence, an item measures multiple latent traits, in addition to the target latent trait(s) that the test was designed to measure. Existing unidimensional testlet-effect models become inapplicable when multiple testlet effects are involved. To account for multiple testlet effect, in this study we develop the so-called (within-item) multidimensional testlet-effect Rasch model. The parameters can be estimated with marginal maximum likelihood estimation methods or Bayesian methods with Markov chain Monte Carlo (MCMC) algorithms. In this study, a popular computer program for Rasch models, ConQuest, was used. A series of simulations were conducted to evaluate parameter recovery of the new model, consequences of model misspecification, and the effectiveness of model-data fit statistics. Results show that the parameters of the new model can be recovered fairly well; and ignoring the multiple testlet effects resulted in a biased estimation of item parameters, and an overestimation of test reliability. Additionally, it did little harm on parameter estimation to fit a more complicated model (i.e., the multidimensional testlet-effect Rasch model) to data with a simple structure. In conclusion, the new model is feasible and flexible.

Key words: testlet response models, multidimensional item response models, multidimensional testlet-effect models, Rasch models