Generalizability Theory (GT) is widely applied in psychological measurement and evaluation. A larger generalizability coefficient often indicates a higher reliability the test may have. Generalizability coefficients can be improved by increasing sample sizes. However, the size of a sample would be subject to budget constraints. Therefore, it is important to examine how to effectively determine the size of a sample considering the budget constraints. The existing literature has been largely limited to traditional methods, such as the differential optimization method, the Lagrange method and the Cauchy Schwartz inequality method.
These traditional methods have limited scope of application and their typical conditions are hard to satisfy. In addition, there is no unified comparison available. Fortunately, with the increased use of high performance computing, the Constrained Optimization Evolutionary Algorithms (COEAs) becomes highly feasible.
This paper expands and compares the four methods—the differential optimization method, Lagrange method, Cauchy Schwartz inequality method, and COEAs—determine the best solution to the optimal sample size problem under the budget constraints in GT. Specifically, this paper compares the applicability of the four methods using three generalizability designs, including p × i × r, (r: p) × i and p × i × r × o designs. The results are presented as follows:
(1) In the optimization performance of two-facet generalizability design of p × i × r and (r: p) × i, the performance of COEAs is slightly better than that of the traditional methods, whereas the performance of three traditional methods is equivalent. Although COEAs and the traditional methods have showed similar accuracy, the former has better compliance concerning budget constraints.
(2) In the optimization performance of three-facet generalizability design of p × i × r × o, the performance of COEAs is obviously better than that of the traditional methods. The least ideal generalizability coefficient is obtained using the differential optimization method, whereas its budget compliance is the best; the generalizability coefficient obtained by Lagrange method is the best, but higher than the budget. The Cauchy inequality method obtains a better generalizability coefficient under special budget constraints. But, the performance of COEAs is slightly better than that of Cauchy Schwartz inequality method, especially closer to the budget constraints.
(3) In terms of the algorithm complexity, COEAs obtains an obviously smaller algorithm complexity than do the traditional methods. The complexity of the three traditional methods is relatively high. However, COEAs does not rely on the derivation of mathematical formulas, and the algorithm is relatively less complex.
(4) In terms of the algorithm applicability, COEAs is significantly better than the traditional methods. The applicability of the three traditional methods is relatively narrow. However, COEAs does not rely on a specific generalizability design or a budget expression, and, therefore, the applicability of COEAs is stronger.
(5) In terms of the algorithm generalizability, COEAs is obviously better than the traditional methods. The limited mathematical principles make it difficult to extend the three traditional methods to more complex generalizability designs, and thus, the feasibility of calculation is poor. Howerve, COEAs has revealed stronger generalizability.
(6) In terms of the possibility of getting the best solution, COEAs is also better than the traditional methods. Because evolutionary algorithm is a probabilistic algorithm, multiple tests can be conducted to obtain better results for optimal sample sizes. Under some conditions, COEAs can determine better solutions, which, however, is impossible for three traditional methods.
(7) These results suggest that COEAs is superior to three traditional methods in estimating the optimal sample size problem under the budget constraints in GT. It is recommended that researchers use COEAs in future research to determine an optimal sample size in their psychological measurement and evaluation.
