Abstract:

Cognitive diagnosis, which is also referred as skill assessment or skill profiling, utilizes latent class models to provide fine-grained information about students’ strength and weakness in the learning process. The outcome of cognitive diagnostic models (CDMs) is a profile with binary element for each examinee to indicate the mastery/nonmastery status of every attribute/skill. Therefore, one major advantage of CDMs is the capacity to provide additional information about the instructional needs of students. In the past decades, extensive research has been conducted in the area of cognitive diagnosis and many statistical models based on a probabilistic approach have been proposed. Examples of CDMs include the deterministic inputs, noisy and gate (DINA) model (Junker & Sijtsma, 2001), the deterministic input, noisy or gate (DINO) model (Templin & Henson, 2006), and the linear Logistic model (LLM) (Maris, 1999). In educational measurement, one of the most commonly used formats is the testlet design, which is a cluster of items that share a common stimulus (e.g., a reading comprehension passage or a figure). Under the framework of item response theory (IRT), various testlet response models (TRM) have been proposed, such as the Rasch testlet model (Wang & Wilson, 2005) and the multidimensional testlet-effect Rasch model (MTERM) (Zhan, Wang, Wang, & Li, 2014). However, limited efforts have been contributed to the development of testlet models for CDMs. A question then naturally arises is the searching for a way to account for testlet effect under CDMs. To address this issue, this study proposed two testlet-CDMs. One followed the compensatory approach and the other followed the noncompensatory approach: (1) the compensatory multidimensional testlet-effect CDM (C-MTECDM) was based on the combination of LLM and MTERM, while (2) the noncompensatory multidimensional testlet-effect CDM (N-MTECDM) was based on the combination of (logit)DINA model and MTERM, respectively. Model parameters can be estimated by the Bayesian methods with Markov chain Monte Carlo (MCMC) algorithms, which have been implemented with the freeware WinBUGS. In study 1, a series of simulations were conducted to evaluate parameter recovery of two new models, and results showed that the model parameters could be recovered fairly well under all simulated conditions. In study 2, the two new models were compared with the LLM and the (logit)DINA model, respectively. Results showed that ignoring testlet effect would result in biased item parameter estimations and worse person classification rates. Additionally, fitting a more complicated model (i.e., MTECDM) to data with a simpler structure did litter harm on parameter recovery. In conclusion, the new models is feasible and flexible.

Key words: cognitive diagnosis, testlet, item response theory, multidimensional testlet-effect, Logistic testlet framework, DINA