Abstract: Generalizability Theory (GT) is a new trend in psychological measurement. First the basic ideas and methods of Generalizability Theory was introduced, Then an exploratory application of it was conducted on the problem of scoring in Chinese composition. Twenty high school students were asked to write three compositions of different forms in one month, all of the compositions were rated by six raters. Data was finally analysed by GENOVA. The results indicated that the influence of raters was large, while item effect was small It was also found that different forms of compositions had different scoring error, essay with problem discussion was likely to have the largest scoring error. As to the three essential factors of writing ability, the largest scoring error happened on the content, while the smallest scoring error happened on the structure. Furthermore, the author pointed out some defficiencies of the Kendall coefficient of concordance (W coefficient) and suggest that phi coefficient is a more effective index as rater consistency reliability.

Key words: Generalizability Theory, composition scoring, error controlling, W coefficient