Abstract: Bootstrap is a returned re-sampling method used to estimate the variance component and their variability. Adjusted bootstrap method was used by Wiley in p×i design for normal data in 2001. However, Wiley did not compare the difference between adjusted method and unadjusted method when estimating the variability. To expand Wiley’s 2001 study, our study applied Monte Carlo method to simulate four distribution data. The aim of simulation is to explore the effects of four different estimation methods when estimating the variability of estimated variance components for generalizability theory. The four distribution data are normal distribution data, dichotomous distribution data, polytomous distribution data and skewed distribution data. It is common that researchers focus on normal distribution data and neglect non-normal distribution data, yet non-normal distribution data could always be seen in tests such as TOEFL and GRE. There are several methods to estimate the variability of variance components, including traditional, bootstrap, jackknife and Markov Chain Monte Carlo (MCMC). Former research by Li and Zhang (2009) shows that bootstrap method is significantly better than traditional, jackknife, and MCMC methods in estimating the variability for four distribution data. Bootstrap method has superior cross-distribution quality when estimating the variability of estimated variance components. Li and Zhang (2009) also suggest that bootstrap method should be adopted with a “divide-and-conquer” strategy to obtain good estimated standard error and estimated confidence interval and the criteria of such strategy should be set to: boot-p for person, boot-pi for item, and boot-i for person and item. However, it is unclear that which of the bootstrap methods (adjusted and unadjusted) is better for boot-p, boot-pi, and boot-i. Therefore, our study intends to probe into this comparison as well. This aim of the study is to explore whether adjusted bootstrap method is superior to unadjusted method in improving the estimation of variance components and their variability relative for generalizability theory. The simulation is implemented in R statistical programming environment. To simulate skewed data, HyperbolicDist package is used. Some criteria are set to compare the four methods. The bias is considered when variance components and their standard errors are estimated. The smaller the absolute bias is, the more reliable the result is. The criterion of confidence intervals is “80% interval coverage”. If the “80% interval coverage” is closer to 0.80, the confidence interval is more reliable. The results indicate that for four distribution data, adjusted bootstrap method is superior to unadjusted bootstrap method whether in point estimation of variance components or in variability estimation of variance components. For its improvement of the estimation of variance components and their variability for generalizability theory, adjusted bootstrap should be adopted as soon as possible.

Key words: Generalizability Theory, Bootstrap method, Variance component, variability of estimated variance components, Monte Carlo simulation