Abstract:

For a criterion-referenced test, classification consistency and accuracy indices are important indicators to evaluate the reliability and validity of classification results. Some procedures have been proposed to estimate these indices in the framework of unidimensional item response theory (UIRT) based on either the total sum scores or the latent trait estimates. Although multidimensional item response theory (MIRT) has enjoyed tremendous popularity, most research is based on the total sum scores only, and Yao (2016) is a case in point. The present authors believe that under MIRT, the decision rules on the two indices should consider the both depending on the different situations. The two reasons are (1) Classifications from the latent trait estimates are equally or more accurate than from the total sum scores, at least for the logistic model of one-parameter, two-parameters, and the graded response model in UIRT; (2) It may be difficult to estimate the two indices from the total sum scores in some content areas when some items may measure more than two domains (complex structure). In this study, the Guo-based consistency and accuracy indices have been extended to MIRT for complex decision rules. Monte Carlo method was employed to estimate Lee-and Guo-based indices for tackling intractable summations or high-dimensional integrals. A simulation study was conducted under a multidimensional graded response model (MGRM). In the simulation study, one, two and four factors were manipulated. Three levels of correlation (ρ=0.0, ρ=0.50, and ρ=0.8) between pairs of dimensions were considered. The examinee sample size was 1,000 and 3,000 respectively. The ability vectors were generated from the multivariate normal distributions with an appropriately sized mean vector of 0 and covariance matrix Σ, where the diagonal elements of Σ were all 1 and the off-diagonal elements were given by the corresponding correlations. The test length for the one factor model was 10 and 20, for the two factor model was 15 and 30, and for the four factor model was 30 and 60. In order to balance information of each domain or dimension, content balancing techniques were adopted to ensure that the tests fulfill the content or domain requirements. The fully crossed design yielded a total of 28 conditions, where each was replicated 10 times. Simulation results suggested that the Guo-based indices worked well and flexibly because their values matched closely with the simulated consistency and accuracy rates for three decision rules, and the difference between the Lee- and Guo-based accuracy indices was much smaller for decision rule based on total score, which conformed to the theoretical results. The two practical implications of this research are identified. First, the indices can be used in score interpretations and test construction. Since it is convenient to estimate consistency and accuracy indices for domain scores and composite scores when the true cut scores are set on the θ scale, items that measure specific dimension with low indices can be created. Second, they might be useful in developing item selection algorithm in computerized classification testing for making multidimensional classification decisions.

Key words: multidimensional item response theory, decision rule, classification consistency, classification accuracy, reliability, validity