ISSN 1671-3710
CN 11-4766/R

心理科学进展 ›› 2023, Vol. 31 ›› Issue (2): 196-208.doi: 10.3724/SP.J.1042.2023.00196

• 研究方法 • 上一篇    下一篇


黄顺森, 陈豪杰, 来枭雄, 代欣然, 王耘()   

  1. 北京师范大学认知神经科学与学习国家重点实验室, 北京 100875
  • 收稿日期:2022-01-13 出版日期:2023-02-15 发布日期:2022-11-10
  • 通讯作者: 王耘
  • 基金资助:

Multiverse-style analysis: Introduction and application

HUANG Shunsen, CHEN Haojie, LAI Xiaoxiong, DAI Xinran, WANG Yun()   

  1. State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing 100875, China
  • Received:2022-01-13 Online:2023-02-15 Published:2022-11-10
  • Contact: WANG Yun


选择性分析和报告是造成心理科学研究可重复性危机的一个重要因素。近年来研究者提出用多元宇宙样分析的方法, 囊括多种数据分析策略, 减少分析过程中的主观选择性和随意性, 并进行稳健性检验以提高结果的可靠性。以手机使用与手机压力的关系为例, 介绍该方法和操作步骤。该方法已在心理学和认知神经科学等领域得到一定的应用。未来研究应继续发展和完善该方法的统计推断, 使之运用到更多的数据类型和更广的研究领域中。

关键词: 多元宇宙样分析, 可重复性危机, 选择性分析, 选择性报告, 可疑研究操作, 手机压力


Multiverse-style analysis (e.g., the vibration of effects, multimodel analysis, multiverse analysis, specification curve analysis) proposes to report combinations of multiple analysis strategies during data analysis and to test the robustness of effects between relevant variables in all analytic strategies. The basic principles and applications of the multiverse-style analysis are described, and the operational steps are presented as an example of the relationship between smartphone use and smartphone stress. The strengths and limitations of the method are discussed, as well as future directions.

Searching the Web of Science on the topic of multiverse-style analysis (e.g., the vibration of effects), we found that the number of papers rose from 9 in 2015 to 40 in 2021. Multiverse-style analysis is gradually being applied in psychology, behavioral sciences, neuroscience, psychiatry, and other fields. Most of these studies used self-reported data. Some neuroscience and biology-related studies used objective data (e.g. physiological indicators such as brain imaging data). Few studies combined self-reported and objective data. Most studies used cross-sectional designs. A few studies used longitudinal or cohort designs. In addition, multiverse-style analysis is gradually being combined with other psychological methods. For example, some researchers have combined it with mediation analysis to determine the robustness of mechanisms among variables. It has also been used with network analysis to reduce the instability of network centrality. People have combined multiverse-style analysis with meta-analysis to form the “combinational meta-analysis”. Finally, different studies have different preferences in the choice of combinations of analytic strategies. For example, some focus on different measurement approaches (self-report or objective measures), while others focus on different estimation methods or concentrate on the diversity of the datasets. In combining multiverse-style analysis with other methods, researchers usually emphasize the strengths of multiverse-style analysis to compensate for the weaknesses of other methods.

Advantages of multiverse-style analysis: (1) It can include multiple data sets, multiple measurement methods and estimation methods, and then perform effect tests. (2) Multiverse-style analysis can be used to resolve controversial issues. Multiverse-style analysis not only has the advantages of meta-analysis but also can be applied to emerging areas where empirical studies are scarce and meta-analysis is not appropriate. Limitations of multiverse-style analysis: (1) As the combination of analysis strategies and sample size increases, it becomes more time consuming to make statistical inferences. (2) The method is still essentially a subjective selection process by the researchers. As such, there may be a potential risk of leading to the problem of "truly arbitrariness". (3) The statistical inference indicators of multiverse-style analysis are not stable. Conflicting results between different statistical indicators may arise. (4) It is difficult for the researcher to report all possible combinations of analytical strategies for an effect based on the available dataset. It is necessary to select the appropriate combination of analytical strategies and build an appropriate dataset based on the available theory and evidence before data analysis.

Future directions: (1) Most existing studies demonstrate the robustness of interesting effects by simply describing all outcomes. Future applied research should consider implementing statistical inference. (2) Deepening the integration of multiverse-style analysis with other research methods, e.g. developing different criteria when integrating multiverse-style analysis with different methods. (3) Select stable statistical inference indicators, give more consideration to parameter estimation (e.g., BIC, AIC) and model estimation methods (e.g., Bayes, Monte-Carlo) when constructing combinations of analytic strategies, and include statistical inference in analytical software or software packages. (4) Combining multiple channels to jointly address the reproducibility crisis (e.g. future research could incorporate multiverse-style analysis during data analysis and pre-registration before data collection). (5) Hold a critical sight towards the different outcomes of different combinations of analytical strategies. There may not be a single standard law in the field of human psychology and behavior, which is influenced by multiple factors (e.g. genes, groups, environment, culture, etc.).

Key words: multiverse-style analysis, replicability crisis, selective analysis, selective report, questionable research practice, smartphone stress