Abstract:

Multiverse-style analysis (e.g., the vibration of effects, multimodel analysis, multiverse analysis, specification curve analysis) proposes to report combinations of multiple analysis strategies during data analysis and to test the robustness of effects between relevant variables in all analytic strategies. The basic principles and applications of the multiverse-style analysis are described, and the operational steps are presented as an example of the relationship between smartphone use and smartphone stress. The strengths and limitations of the method are discussed, as well as future directions.

Searching the Web of Science on the topic of multiverse-style analysis (e.g., the vibration of effects), we found that the number of papers rose from 9 in 2015 to 40 in 2021. Multiverse-style analysis is gradually being applied in psychology, behavioral sciences, neuroscience, psychiatry, and other fields. Most of these studies used self-reported data. Some neuroscience and biology-related studies used objective data (e.g. physiological indicators such as brain imaging data). Few studies combined self-reported and objective data. Most studies used cross-sectional designs. A few studies used longitudinal or cohort designs. In addition, multiverse-style analysis is gradually being combined with other psychological methods. For example, some researchers have combined it with mediation analysis to determine the robustness of mechanisms among variables. It has also been used with network analysis to reduce the instability of network centrality. People have combined multiverse-style analysis with meta-analysis to form the “combinational meta-analysis”. Finally, different studies have different preferences in the choice of combinations of analytic strategies. For example, some focus on different measurement approaches (self-report or objective measures), while others focus on different estimation methods or concentrate on the diversity of the datasets. In combining multiverse-style analysis with other methods, researchers usually emphasize the strengths of multiverse-style analysis to compensate for the weaknesses of other methods.

Advantages of multiverse-style analysis: (1) It can include multiple data sets, multiple measurement methods and estimation methods, and then perform effect tests. (2) Multiverse-style analysis can be used to resolve controversial issues. Multiverse-style analysis not only has the advantages of meta-analysis but also can be applied to emerging areas where empirical studies are scarce and meta-analysis is not appropriate. Limitations of multiverse-style analysis: (1) As the combination of analysis strategies and sample size increases, it becomes more time consuming to make statistical inferences. (2) The method is still essentially a subjective selection process by the researchers. As such, there may be a potential risk of leading to the problem of "truly arbitrariness". (3) The statistical inference indicators of multiverse-style analysis are not stable. Conflicting results between different statistical indicators may arise. (4) It is difficult for the researcher to report all possible combinations of analytical strategies for an effect based on the available dataset. It is necessary to select the appropriate combination of analytical strategies and build an appropriate dataset based on the available theory and evidence before data analysis.

Future directions: (1) Most existing studies demonstrate the robustness of interesting effects by simply describing all outcomes. Future applied research should consider implementing statistical inference. (2) Deepening the integration of multiverse-style analysis with other research methods, e.g. developing different criteria when integrating multiverse-style analysis with different methods. (3) Select stable statistical inference indicators, give more consideration to parameter estimation (e.g., BIC, AIC) and model estimation methods (e.g., Bayes, Monte-Carlo) when constructing combinations of analytic strategies, and include statistical inference in analytical software or software packages. (4) Combining multiple channels to jointly address the reproducibility crisis (e.g. future research could incorporate multiverse-style analysis during data analysis and pre-registration before data collection). (5) Hold a critical sight towards the different outcomes of different combinations of analytical strategies. There may not be a single standard law in the field of human psychology and behavior, which is influenced by multiple factors (e.g. genes, groups, environment, culture, etc.).

Key words: multiverse-style analysis, replicability crisis, selective analysis, selective report, questionable research practice, smartphone stress