Abstract: In the research of psychology and other social sciences, test reliability is often used to reflect measurement stability and consistency. Coefficient α is the most popular indicator of test reliability. Recent years, however, coefficient α was challenged now and again. Is coefficient α still recommended for evaluating test reliability? If not, what should replace it?
With the classical concept of reliability, which is defined as the ratio of true variance to observed variance on a test under consideration, we introduced the relationship between test reliability and coefficient α, and the limitations of coefficient α. The concepts closely related to coefficient α were considered. We clearly defined homogeneity reliability and internal consistency reliability. Homogeneity reflects the presence of a general factor, whereas internal consistency relates the presence of common factors (including a general factor and local factors). For unidimensional tests, homogeneity and internal consistency are the same concept. Investigating the relationship between test reliability, coefficient α, homogeneity reliability, and internal consistency reliability, we showed that homogeneity reliability is not larger than internal consistency reliability, and that the latter is not larger than test reliability; coefficient α usually underestimates internal consistency reliability, and the latter is closer to test reliability.
For ordinary use, the errors of items in a test are reasonably uncorrelated. Under the assumption that the total score of the test is meaningful, we proposed a guideline for evaluating test reliability. If coefficient α is high enough to be accepted, then the test reliability is also acceptable whether the test is unidimensional or not. In this case, using coefficient α to evaluate test reliability is the first choice. If the coefficient α is not large enough, we should calculate internal consistency reliability which is also known as composite reliability in literatures. If the internal consistency reliability is high enough to be accepted, then the test reliability is also acceptable. An operational procedure was summarized for reliability analysis based on the above guideline.
We illustrated how to calculate homogeneity reliability and internal consistency reliability by using factor analysis. A LISREL program was developed to calculate them for a multidimensional test. A simplified version of the program was also made for a unidimensional test. The programs are almost the same as those for ordinary confirmatory factor analysis and can be managed easily by empirical researchers.

Key words: reliability, coefficient a, homogeneity, internal consistency, composite reliability