Abstract:

The model parameter standard error (SE; or variance-covariance matrix), which provides an estimate of the uncertainty associated with the model parameter estimate, has both theoretical and practical implications in cognitive diagnostic models (CDMs). The drawbacks of the analytic methods, such as the empirical cross-product information matrix, observed information matrix, and “robust” sandwich-type information matrix, are that they require the positive definiteness of the information matrix and may suffer from boundary problems. Another method for estimating model parameter SEs is to use the computer-intensive bootstrap method, and consequently, no study has systematically explored the performance of the bootstrap in calculating model parameter SEs and confidence intervals (CIs) in CDMs.
The purpose of this research is to present two new highly efficient bootstrap methods to calculate model parameter SEs and CIs in CDMs, namely the parallel parametric bootstrap (pPB) and parallel non-parametric bootstrap (pNPB) methods. A simulation study was conducted to evaluate the performance of the pPB and pNPB methods. Five factors that may influence the performance of the model parameter SEs and CIs were manipulated. The two model specification scenarios considered in this simulation were the correctly specified and over-specified models. The sample size was set to two levels: 1, 000 and 3, 000. Three bootstrap sample sizes were manipulated: 200, 500, and 3, 000. Three levels of item quality (IQ) were considered: high IQ $[P(\mathbf{0})=0.1, P(\mathbf{1})=0.9]$, moderate IQ $[P(\mathbf{0})=0.2, \quad P(\mathbf{1})=0.8]$, and low IQ $[P(\mathbf{0})=0.3, \quad P(\mathbf{1})=0.7]$. The pPB and pNPB methods were used to estimate model parameter SEs and CIs.
The simulation results indicated the following.
(1) The coverage rates of the pNPB-based or pPB-based 95% CIs of the item and the structural parameter SEs are shown in Figures 1 and 2, respectively. In general, for the correctly specified CDMs, under the high- or moderate-item-quality conditions, the coverage rates of the 95% CIs of the model parameter SEs based on the pNPB or pPB method were reasonably close to the expected coverage rate. And the simulation results revealed that the estimated SE was almost identical to the empirical SE. The increase in the bootstrap sample size had only a slight effect on the performance of the pNPB or pPB method. Under the low-item-quality condition, the pNPB method tended to over-estimate SE, whereas a contrary trend was observed for the pPB method.
(2) For the over-specified CDMs, as illustrated in Figures 3 and 4, most of the permissible item parameter SEs and almost all of the permissible structural parameter SEs exhibited good performance in terms of the 95% CI coverage rates. The 95% CI coverage rate results of the permissible and the impermissible of the structural parameter SEs are shown in Figures 5 and 6. Under most of the simulation conditions, the impermissible model parameter SEs did not exhibit good performance in approximating the empirical SEs.
To the best of our knowledge, this is the first study in which the performance of the bootstrap method in estimating model parameter SEs and CIs in CDMs is systematically investigated. The pNPB or pPB appears to be a useful tool for researchers interested in evaluating the uncertainty of the model parameter point estimates. As a time-saving computational strategy, the pNPB or pPB method is substantially faster than the usual bootstrap method. The simulation and real data studies showed that 3, 000 re-samples might be adequate for the bootstrap method in calculating model parameter SEs and CIs in CDMs.

Key words: cognitive diagnostic model, standard error, confidence interval, bootstrap, parallel computing method