Abstract:

Intensive longitudinal data are increasingly used in empirical research in psychology and other social sciences. While such data provide a wealth of information for exploring the dynamics of variables, they also pose challenges for estimating the reliability of tests used in intensive longitudinal studies. A large number of previous intensive longitudinal studies have failed to reasonably and adequately assess the reliability of their tests. Earlier reliability estimation methods drawn from cross-sectional studies or based on generalizability theory (GT) have many limitations and are not applicable to intensive longitudinal data. Therefore, the main purpose of this study is to systematically introduce the reliability estimation methods suitable for intensive longitudinal studies and to provide methodological guidance for applied researchers.

Intensive longitudinal data have two main characteristics: multilevel structure and dynamic nature. The multilevel structure refers to the data structure of multiple repeated measures (level 1) nested within individuals (level 2); and the dynamic nature refers to the temporal dependency between consecutive observations. From the perspective of multilevel structure and dynamic nature, we first introduce two reliability estimation methods that consider one of these two characteristics. The reliability estimation method focusing on the multilevel structure is based on multilevel confirmatory factor analysis (MCFA). It can estimate test reliability at the between- and within-person levels separately. However, this method does not take into account individual differences in reliability and temporal dependency between consecutive observations in intensive longitudinal data. The reliability estimation method focusing on the dynamic nature is based on dynamic factor analysis (DFA). It establishes a dynamic factor model for each individual, and includes autoregressive processes to reflect the dynamic nature of intensive longitudinal data. It can estimate person-specific reliabilities, helping researchers to better understand individual differences in test reliability. However, this method confounds the trait and state components of observed scores, which may lead to biased estimates of person-specific reliabilities. In addition, it cannot estimate between-person reliabilities.

Considering the limitations of the above two methods, we next present another reliability estimation method (i.e., a method based on dynamic structural equation modeling (DSEM)) that integrates the multilevel structure and dynamic nature of intensive longitudinal data. This method builds factor models at the between- and within-person levels to estimate reliabilities at both levels, and includes autoregressive processes to reflect the temporal dependency between consecutive observations. It also allows for random effects of model parameters (e.g., factor loadings and autoregressive effects) to enable estimation of person-specific reliabilities. In summary, this method can simultaneously reflect the multilevel structure and dynamic nature of intensive longitudinal data, and examine individual differences in test reliability, which helps researchers to better estimate and understand reliability in intensive longitudinal studies.

Finally, we demonstrate and compare three reliability estimation methods with empirical data on daily procrastination. We summarize and discuss the main features and applicable contexts of these methods, and recommend researchers to examine the reliability of each item and consider individual differences in test reliability in intensive longitudinal studies. We also provide practical suggestions on how to better report reliabilities in applied research. Future research could explore the reliability estimation methods based on other models, and should also pay more attention to the testing and reporting of test reliability in intensive longitudinal studies.

This paper provides a systematic review of reliability estimation methods in intensive longitudinal studies from the perspectives of multilevel structure and dynamic nature. We illustrate the differences between the methods in estimating three types of reliabilities (i.e., between-person reliabilities, within-person reliabilities, and person-specific reliabilities), and interpret the reliability results at both the item and scale levels, which helps researchers to understand and evaluate the test reliability in intensive longitudinal studies more comprehensively. In addition, by analyzing and comparing the main features of different methods, we point out the contexts in which they are applicable and provide practical suggestions for estimating and reporting reliability in intensive longitudinal studies.

Key words: intensive longitudinal study, reliability, multilevel structure, dynamic nature, dynamic structural equation modeling