ISSN 1671-3710
CN 11-4766/R

Advances in Psychological Science ›› 2021, Vol. 29 ›› Issue (3): 381-393.doi: 10.3724/SP.J.1042.2021.00381

• Research Method •     Next Articles

Interpreting nonsignificant results: A quantitative investigation based on 500 Chinese psychological research

WANG Jun1, SONG Qiongya1, XU Yuepei2,3, JIA Binbin4, LU Chunlei5, CHEN Xi6, DAI Zixu7, HUANG Zhiyue8, LI Zhenjiang9, LIN Jingxi10, LUO Wanying11, SHI Sainan12, ZHANG Yingying13, ZANG Yufeng14, ZUO Xi-Nian15, HU Chuanpeng16   

  1. 1Department of Psychology, Sun Yat-Sen University, Guangzhou 510006, China;
    2Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China;
    3Department of Psychology, Chinese Academy of Sciences, Beijing 100049, China;
    4School of Psychology, Shanghai University of Sport, Shanghai 200438, China;
    5College of Teacher Education, Zhejiang Normal University, Jinhua 321000, China;
    6Person, Shanghai 200122, China;
    7School of Psychology, South China Normal University, Guangzhou 510631, China;
    8Tisch School of the Arts, New York University, New York 11201, the United States;
    9School of Education, Soochow University, Suzhou 215123, China;
    10Institute of Education Science, Heilongjiang University, Harbin 150080, China;
    11School of Psychology and Cognitive Sciences, Peking University, Beijing 100871, China;
    12School of Psychology and Cognitive Sciences, East China Normal University, Shanghai 200063, China;
    13Faculty of Psychology, Southwest University, Chongqing 400715, China;
    14Center for Cognition and Brain Disorders, Hangzhou Normal University, Hangzhou 311121, China;
    15National Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing 100875, China;
    16Leibniz Institute for Resilience Research, Mainz 55131, Germany
  • Received:2020-07-14 Online:2021-03-15 Published:2021-01-27

Abstract: Nonsignificant results are common in psychological research and can be easily misinterpreted as evidence for accepting null hypothesis. This misinterpretation may lead to false statistical inferences in empirical research. However, how prevalent this misinterpretation exists in Chinese published psychological studies is unknown. To answer this question, we randomly selected 500 empirical research papers published between 2017 and 2018 in Acta Psychological Sinica, Journal of Psychological Science, Chinese Journal of Clinical Psychology, Psychological Development and Education, Psychological and Behavioral Studies, screened articles in which the abstracts contained any sentences that indicated nonsignificant results (we call these sentences “negative statements” hereafter). We then read those articles and extracted negative- statements-related statistics and their interpretations, and evaluated the correctness of each interpretation. Finally, we calculated Bayes factors based on the available t values in these nonsignificant results. The protocol was pre-registered at OSF (https://osf.io/czx6f). We found that (1) out of 500 empirical research, 36% of their abstracts (n = 180) contained negative statements; (2) in those 180 articles, we extracted 236 nonsignificant results and corresponding interpretations, and found that 41% of these interpretations was incorrect, (3) Bayes factor analysis revealed that only 5.1% (n = 2) of available nonsignificant t-values (n = 39) can provide strong evidence in favor of null hypothesis (BF01 > 10). We compared the results with Aczel et al. (2018) and discussed the potential reasons that caused the misinterpretation. These data suggest that Chinese psychology researchers need to improve their understanding of nonsignificant results and statistical inference.

Key words: nonsignificant results, null-hypothesis significance testing, Bayes factors, meta-research

CLC Number: