Abstract:

Missing observations are common in operational performance assessment settings or psychological surveys and experiments. Since these assessments are time-consuming to administer and score, examinees seldom respond to all test items and raters seldom evaluate all examinee responses. As a result, a frequent problem encountered by those using generalizability theory with large-scale performance assessments is working with missing data. Data from such examinations compose a missing data matrix. Researchers usually concern about how to make good use of the full data and often ignore missing data. As for these missing data, a common practice is to delete them or make an imputaion for missing records; however, it may cause problems in following aspects. Firstly, deleting or interpolating missing data may result in ineffective statistical analysis. Secondly, it is difficult for researchers to choose an unbiased method among diverse rules of interpolation. As a result of missing data, a series of problems may be caused when estimating variance components of unbalanced data in generalizability theory. A key issue with generalizability theory lies in how to effectively utilize the existing missing data to their maximum statistical analysis capacity. This article provides four methods to estimate variance components of missing data for unbalanced random p×i×r design of generalizability theory: formulas method, restricted maximum likelihood estimation (REML) method, subdividing method, and Markov Chain Monte Carlo (MCMC) method. Based on the estimating formulas of p×i design by Brennan (2001), formulas method is the deduction of estimating variance components formulas for p×i×r design with missing data. The aim of this article is to investigate which method is superior in estimating variance components of missing data rapidly and effectively. MATLAB 7.0 was used to simulate data, and generalizability theory was used to estimate variance components. Three conditions were simulated respectively: (1) persons sample with small size (200 students), medium size (1000 students) and large size (5000 students); (2) item sample with 2 items, 4 items and 6 items; (3) raters sample with 5 raters, 10 raters and 20 raters. The authors also developed some programs for MATLAB, WinBUGS, SAS and urGENOVA software in order to estimate variance components of p×i×r missing data with four methods. Criterions were made for the purpose of comparing the four methods. For example, bias was the criterion when estimating variance components. The reliability of the results increased as the absolute bias decreased. Results indicate that: (1) MCMC method has a strong advantage for estimating variance components of p×i×r missing data over the other three methods. MCMC method is superior to formulas method because of smaller deviation for variance components estimation. It is better than REML method because iteration of MCMC method converge, while REML method does not. Unlike subdividing method, MCMC method does not require variance components to be combined in order to obtain accurate estimations. (2) Item and rater are two important influencing factors for estimating variance components of missing data. If manpower and material resources are limited, priority should be given to increase the number of items in order to increase estimation accuracy. If researchers cannot increase the number of items, the next-best thing is to increase the number of raters. However, the number of raters should be cautiously controlled.

Key words: Generalizability Theory, missing data, estimating variance components, p×i×r design, Markov Chain Monte Carlo (MCMC)