Abstract:

Generalizability theory (GT), a modern psychometric framework, has been widely applied in practice. This study evaluated and compared the estimation accuracy of three sampling algorithms of MCMC method across two common GT research designs: the one-faceted crossed design (p×i) and the two-faceted crossed design (p×i×h). Using simulation methodologies, we manipulated three independent variables: (1) sampling algorithms (M-H, Gibbs, and HMC), (2) prior distributions (informative, non-informative, and empirical), and (3) missing data ratios (0%, 5%, and 10%). The dependent variables were the estimated variance components, Bias, and Root Mean Square Error (RMSE). The research results indicated that: (1) low ratios of missing data (up to 10%) yielded negligible effects across all three algorithms, demonstrating the robust handling of missingness inherent to MCMC; (2) across all conditions, informative and empirical priors yielded higher estimation accuracy than non-informative priors; (3) HMC algorithm uniquely exhibited “cross-design” stability, maintaining high accuracy across both the p×i and the p×i×h, whereas M-H algorithm and Gibbs sampling demonstrated structural limitations and lower accuracy; and (4) for M-H algorithm and Gibbs sampling to achieve acceptable accuracy in higher-dimensional spaces, a minimum of four levels for the h facet (h≥4) was required. Consequently, we highly recommend that future GT researchers utilizing Bayesian MCMC frameworks adopt the HMC algorithm.

Key words: generalizability theory, estimating variance components, MCMC method, M-H algorithm, Gibbs sampling, HMC algorithm