ISSN 1671-3710
CN 11-4766/R
主办:中国科学院心理研究所
出版:科学出版社

Advances in Psychological Science ›› 2022, Vol. 30 ›› Issue (8): 1682-1691.doi: 10.3724/SP.J.1042.2022.01682

• Section of Research Methods • Previous Articles     Next Articles

Research on test reliability in China’s mainland from 2001 to 2020

WEN Zhonglin1(), CHEN Hongxi1, FANG Jie2, YE Baojuan3, CAI Baozhen1   

  1. 1School of Psychology & Center for Studies of Psychological Application, South China Normal University, Guangzhou 510631, China
    2Institute of New Development & Department of Applied Psychology, Guangdong University of Finance & Economics, Guangzhou 510320, China
    3School of Psychology & Center of Mental Health Education and Research, Jiangxi Normal University, Nanchang 330022, China
  • Received:2021-12-29 Online:2022-08-15 Published:2022-06-23
  • Contact: WEN Zhonglin E-mail:wenzl@scnu.edu.cn

Abstract:

With the application of confirmatory factor analysis, research on reliability has entered a new stage. In the first two decades of the 21st century, the studies on test reliability (including point estimation and interval estimation) in China’s mainland show three main lines of development.

The first line is the development from research centered on the coefficient αto the reliability research based on confirmatory factor models, including the homogeneity coefficient, composite reliability, maximum reliability, single-indicator reliability and reliability of the whole item set scores. Studies have shown that the coefficient αis still useful. In most cases, the α coefficient is the lower bound of the reliability of the composite score (total or average score). As long as the coefficient αis high enough, the test reliability will be even higher. But the coefficient αcannot be used to measure the homogeneity and the internal consistency of a test. The homogeneity coefficient based on the bi-factor model can be adopted to measure the homogeneity of a multidimensional scale, and the composite reliability can be adopted to measure the internal consistency (if consistency is understood as the consistency within each dimension). Furthermore, the Delta method can be employed to estimate the confidence intervals of various reliability.

The second line is the expansion of data types collected by scales (or questionnaires), from single-level data to multi-level and longitudinal data. Whether unidimensional or multidimensional, it is recommended to use a multi-level confirmatory factor model to calculate the reliability of multi-level data. As for the longitudinal data, it is recommended to use the test reliability developed on the basis of the linear mixed model, and the longitudinal data can also be used as a special case of the two-level data for reliability analysis.

The third line is the extended use of reliability, involving rater reliability, encoder reliability, attribute-level classification consistency in cognitive diagnostic assessment, and reliability of difference scores. In addition, research of reliability generalization and reliability meta-analysis appeared.

For a common test with item-errors that can be reasonably assumed uncorrelated, the following procedure of reliability analysis is recommended. When the coefficient αis high enough, report the coefficient α; otherwise calculate the composite reliability on the basis of the factor model. If the composite reliability is high enough, report the composite reliability; otherwise the test reliability is considered unacceptable.

If the composite reliability of every variable in a statistical model is very high (over 0.95), modeling with composite scores does not differ much from modeling with latent variables. Otherwise, it is better to use latent variable modeling.

Key words: reliability, coefficient α, homogeneity coefficient, composite reliability, interval estimation

CLC Number: