关键词: 概化理论, 方差分量估计, MCMC方法, M-H算法, Gibbs抽样, HMC算法

Abstract: Generalizability theory (GT), a modern psychometric framework, has been widely applied in practice. Its efficacy relies on the accurate estimation of variance components. While variance components are traditionally estimated using methods such as analysis of variance (ANOVA), Bootstrap, Jackknife, MINQUE, and restricted maximum likelihood (REML method), Bayesian Markov chain Monte Carlo (MCMC) estimation has demonstrated advantages. However, the performance of MCMC depends on the chosen sampling algorithms. Local update algorithms, such as Metropolis-Hastings (M-H) algorithm and Gibbs sampling, are prone to random walk behaviors and slow convergence, particularly when conditional distributions are complex or parameters are highly correlated. Although the Hamiltonian Monte Carlo (HMC) algorithm effectively suppresses random walk behavior and handles high-dimensional, complex models efficiently, its application to the estimation of variance components within GT remains unexamined. To date, no published studies have compared M-H algorithm, Gibbs sampling, and HMC algorithm in this context.
This study evaluated and compared the estimation accuracy of three sampling algorithms of MCMC method across two common GT research designs: the one-faceted crossed design (p×i) and the two-faceted crossed design (p×i×h). Using simulation methodologies, we manipulated three independent variables: (1) sampling algorithms (M-H, Gibbs, and HMC), (2) prior distributions (informative, non-informative, and empirical), and (3) missing data ratios (0%, 5%, and 10%). The dependent variables were the estimated variance components, Bias, and Root Mean Square Error (RMSE).
In simulation study, we generated data using R software, with 1000 batches of data generated for the p×i and the p×i×h. For the generated data, R software implemented the estimation of variance components for M-H algorithm, Gibbs sampling, and HMC algorithm by calling the corresponding software package. In empirical research, a batch of data containing 1429 participants was used to validate the conclusions drawn from the simulation study, and it was found that the empirical research results were almost consistent with the simulation research results.
The research results indicated that: (1) low ratios of missing data (up to 10%) yielded negligible effects across all three algorithms, demonstrating the robust handling of missingness inherent to MCMC; (2) across all conditions, informative and empirical priors yielded higher estimation accuracy than non-informative priors; (3) HMC algorithm uniquely exhibited “cross-design” stability, maintaining high accuracy across both the p×i and the p×i×h, whereas M-H algorithm and Gibbs sampling demonstrated structural limitations and lower accuracy; and (4) for M-H algorithm and Gibbs sampling to achieve acceptable accuracy in higher-dimensional spaces, a minimum of four levels for the h facet (h≥4) was required. Consequently, we highly recommend that future GT researchers utilizing Bayesian MCMC frameworks adopt the HMC algorithm.

Key words: generalizability theory, estimating variance components, MCMC method, M-H algorithm, Gibbs sampling, HMC algorithm