Please wait a minute...
Advances in Psychological Science    2020, Vol. 28 Issue (4) : 673-680     DOI: 10.3724/SP.J.1042.2020.00673
Research Method |
What is the minimum number of effect sizes required in meta-regression? An estimation based on statistical power and estimation precision
FANG Junyan,ZHANG Minqiang()
School of psychology, South China Normal University, Guangzhou 510631, China
Download: PDF(1038 KB)   HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks    
Abstract  

Meta-regression is the most frequently used technique for identifying moderators in meta-analysis. In this study, main principles and basic models of meta-analysis and meta-regression were briefly introduced first. Then a Monte Carlo simulation was conducted to investigate the minimum number of the effect size required in meta-regression based on statistical power and estimation precision. The results showed that (1) the Wald-type z test was prone to type I error in meta-regression; (2) at least 20 effect sizes were needed to meet parameter estimation requirements; (3) and inclusion of proper moderators could reduce the number of effect size required. Therefore, it is suggested that (1) meta-analysts should be careful when using the CMA software and the Wald-type z test; (2) at least 20 or more effect sizes are generally needed based on different situations; (3) exploration of moderators is necessary; (4) reviewers can value a meta-analysis research according to the minimum number of effect size required.

Keywords meta-analysis      meta-regression      effect size      minimum number requirement     
ZTFLH:  B841  
Corresponding Authors: Minqiang ZHANG     E-mail: 2640726401@qq.com
Issue Date: 24 February 2020
Service
E-mail this article
E-mail Alert
RSS
Articles by authors
Junyan FANG
Minqiang ZHANG
Cite this article:   
Junyan FANG,Minqiang ZHANG. What is the minimum number of effect sizes required in meta-regression? An estimation based on statistical power and estimation precision[J]. Advances in Psychological Science, 2020, 28(4): 673-680.
URL:  
http://journal.psych.ac.cn/xlkxjz/EN/10.3724/SP.J.1042.2020.00673     OR     http://journal.psych.ac.cn/xlkxjz/EN/Y2020/V28/I4/673
检验方法 τ2 = 0.08
β为0 β(均)较小 β(均)较大 β一个较大, 一个较小
β = 0 β = (0, 0) β = 0.2 β = (0.2, 0.2) β = 0.5 β = (0.5,0.5) β = (0.2,0.5)
Knha-test 20 20 20 20 20 20 20
z-test 23 25 23 25 23 25 25
  
检验方法 τ2=0.32
β为0 β(均)较小 β(均)较大 β一个较大, 一个较小
β = 0 β = (0,0) β = 0.2 β = (0.2,0.2) β = 0.5 β = (0.5,0.5) β = (0.2,0.5)
Knha-test 38 38 38 38 38 38 38
z-test 43 43 43 43 43 43 43
  
检验
方法
τ2 = 0.08 τ2 = 0.32
β(均)较小 β(均)较大 β一个较大
一个较小
β(均)较小 β(均)较大 β一个较大
一个较小
β = 0.2 β = (0.2, 0.2) β = 0.5 β = (0.5, 0.5) β = (0.2, 0.5) β = 0.2 β = (0.2, 0.2) β = 0.5 β = (0.5, 0.5) β = (0.2, 0.5)
Knha-test 30 30 20 70 70 20 20 50
z-test 38 38 30 80 80 20 20 52
  
回归系数取值 剩余异质性较小 剩余异质性较大
包含一个调节变量 包含两个调节变量 包含一个调节变量 包含两个调节变量
β(均)为0 20 20 38 38
β(均)较小 30 30 70 70
β(均)较大 20 20 38 38
β1较小β2较大 —— 20 —— 50
  
回归系数取值 剩余异质性较小 剩余异质性较大
包含一个调节变量 包含两个调节变量 包含一个调节变量 包含两个调节变量
β(均)为0 23 25 43 43
β(均)较小 38 38 80 80
β(均)较大 23 25 43 43
β1较小β2较大 —— 30 —— 52
  
k τ2 = 0.08 τ2 = 0.32
β = 0 β = 0.2 β = 0.5 β = 0 β = 0.2 β = 0.5
20 -0.0004 -0.0001 0.0003 -0.0012 0.0008 0.0056
40 0.0009 0.0004 -0.0001 -0.0031 -0.0011 -0.0009
60 0.0006 0.0000 -0.0009 0.0010 -0.0014 -0.0009
80 0.0004 -0.0003 -0.0003 0.0000 0.0003 -0.0005
100 -0.0007 0.0000 0.0002 -0.0006 0.0004 0.0000
120 0.0000 0.0000 -0.0004 0.0003 -0.0001 0.0002
  
k τ2 = 0.08 τ2 = 0.32
β = 0 β = 0.2 β = 0.5 β = 0 β = 0.2 β = 0.5
20 0.0000 0.0000 0.0003 0.0026 0.0005 -0.0009
40 0.0007 0.0004 -0.0009 0.0000 -0.0004 0.0003
60 -0.0003 0.0007 -0.0001 -0.0005 0.0000 0.0003
80 0.0000 0.0001 0.0008 0.0001 0.0013 0.0017
100 -0.0002 0.0001 -0.0001 0.0005 -0.0009 -0.0014
120 0.0001 0.0003 -0.0006 0.0007 0.0000 0.0002
  
k τ2 = 0.08 τ2 = 0.32
β = (0, 0) β = (0.2, 0.2) β = (0.5, 0.5) β = (0.2, 0.5) β = (0, 0) β = (0.2, 0.2) β = (0.5, 0.5) β = (0.2, 0.5)
20 -0.0008 0.0006 -0.0007 0.0003 0.0001 -0.0007 0.0003 -0.0005
40 0.0003 0.0006 -0.0001 0.0008 -0.0008 0.0001 -0.0017 -0.0005
60 0.0000 -0.0006 -0.0003 0.0000 0.0004 0.0001 0.0003 -0.0002
80 0.0001 0.0002 -0.0004 0.0001 0.0005 0.0005 0.0006 0.0001
100 -0.0002 0.0002 -0.0003 0.0001 0.0004 0.0006 -0.0003 0.0002
120 0.0000 0.0001 0.0001 -0.0001 -0.0003 0.0000 0.0002 0.0004
  
k τ2 = 0.08 τ2 = 0.32
β = (0, 0) β = (0.2, 0.2) β = (0.5, 0.5) β = (0.2, 0.5) β = (0, 0) β = (0.2, 0.2) β = (0.5, 0.5) β = (0.2, 0.5)
20 -0.0003 -0.0002 0.0009 0.0006 0.0005 -0.0002 0.0000 0.0000
40 -0.0008 -0.0001 0.0000 -0.0002 0.0002 -0.0002 0.0002 0.0010
60 0.0001 -0.0002 -0.0004 -0.0001 0.0010 0.0002 0.0005 0.0001
80 0.0000 -0.0003 -0.0001 0.0002 0.0005 -0.0002 -0.0001 0.0001
100 0.0001 0.0000 0.0000 0.0000 0.0006 -0.0003 0.0000 0.0002
120 0.0001 0.0002 0.0005 -0.0001 0.0005 0.0005 0.0000 0.0004
  
  
  
  
  
  
  
1 董圣杰, 曾宪涛, 郭毅 . ( 2012). R软件Metafor程序包在Meta分析中的应用. 中国循证医学杂志, 12( 9), 1141-1147.
2 方杰, 张敏强 . ( 2012). 中介效应的点估计和区间估计:乘积分布法、非参数Bootstrap和MCMC法. 心理学报, 44( 10), 1408-1420.
3 刘俊, 秦传燕 . ( 2018). 企业社会责任与员工绩效的关系:一项元分析. 心理科学进展, 26( 7), 1152-1164.
url: http://journal.psych.ac.cn/xlkxjz/CN/article/article4368.shtml
4 王超, 袁蒙蒙, 姜媛, 方平 . ( 2019). 宜人性对企业家成就的影响:来自元分析的证据. 心理与行为研究, 17( 1), 126-133.
5 张天嵩, 刘江波, 钟文昭 . ( 2009). Stata在探索异质性来源—Meta回归分析中的应用. 循证医学, 9( 1), 48-50.
6 张云权, 马露, 冯仁杰, 朱耀辉, 李存禄 . ( 2015). 模型回归系数的合并分析在R软件metafor包中的实现. 中国循证医学杂志, 15( 3), 367-372.
7 Borenstein M., Hedges L. V., Higgins J. P. T., & Rothstein H. R . ( 2009). Introduction to meta-analysis. UK: John Wiley & Sons, Ltd., Publication.
url: http://dx.doi.org/ohn Wiley
8 Borenstein M., Hedges L. V., Higgins J. P.T., & Rothstein H. R . ( 2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1( 2), 97-111.
url: http://dx.doi.org/10.1002/jrsm.12
9 Cafri G., Kromrey J. D., & Brannick M. T . ( 2010). A meta-analysis: Empirical review of statistical power, type I error rates, effect sizes, and model selection of meta-analyses published in psychology. Multivariate Behavioral Research, 45( 2), 239-270.
url: http://dx.doi.org/10.1080/00273171003680187
10 Card N. A. (2012). Applied meta-analysis for social science research. New York: The Guilford Press.
11 Cheung, M. W. L., & Vijayakumar, R . ( 2016). A guide to conducting a meta-analysis. Neuropsychology Review, 26( 2), 121-128.
url: http://dx.doi.org/10.1007/s11065-016-9319-z
12 ?o?altay N., & Karada? E. . ( 2015). Introduction to meta-analysis. In E. Karada? (Eds.), Leadership and Organizational Outcomes (2nd ed, pp. 19-28). Switzerland: Springer International Publishing.
13 Cohn, L. D., & Becker, B. J . ( 2003). How meta-analysis increases statistical power. Psychological Methods, 8( 3), 243-253.
url: http://dx.doi.org/10.1037/1082-989X.8.3.243
14 Field, A. P . ( 2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods, 6( 2), 161-180.
url: http://dx.doi.org/10.1037/1082-989X.6.2.161
15 Glass, G. V . ( 1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5( 10), 3-8.
16 Huizenga H. M., Visser I., & Dolan C. V . ( 2010). Testing overall and moderator effects in random effects meta-regression. British Journal of Mathematical and Statistical Psychology, 64, 1-19.
17 Knapp, G., & Hartung, G . ( 2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22, 2693-2710.
url: http://dx.doi.org/10.1002/sim.1482
18 López-López J. A., Noortgate W. V. D., Tanner-Smith E. E., Wilson S. J., & Lipsey M. W . ( 2017). Assessing meta- regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation. Research Synthesis Methods, 8( 4), 435-450.
url: http://dx.doi.org/10.1002/jrsm.1245
19 Murphy, K. R . ( 2017). What inferences can and cannot be made on the basis of meta-analysis? Human Resource Management Review, 27( 1), 193-200.
url: http://dx.doi.org/10.1016/j.hrmr.2015.06.001
20 Schmidt, F. L . ( 2017). Statistical and measurement pitfalls in the use of meta-regression in meta-analysis. Career Development International, 22( 5), 469-476.
url: http://dx.doi.org/10.1108/CDI-08-2017-0136
21 Steel, P. D., & Kammeyer-Mueller, J. D . ( 2002). Comparing meta-analytic moderator estimation techniques under realistic conditions. Journal of Applied Psychology, 87( 1), 96-111.
url: http://dx.doi.org/10.1037/0021-9010.87.1.96
22 Suchotzki K., Verschuere B., Bockstaele B. V., Ben-Shakhar G., & Crombez G . ( 2017). Lying takes time: A meta-analysis on reaction time measures of deception. Psychological Bulletin, 143( 4), 428-453.
url: http://dx.doi.org/10.1037/bul0000087
23 Valentine J. C., Pigott T. D., & Rothstein H. R . ( 2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational & Behavioral Statistics, 35( 2), 215-247.
url: http://dx.doi.org/urnal of Educational
24 Viechtbauer, W . ( 2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1-48.
25 Viechtbauer W., López-López J. A., Sánchez-Meca J., & Marín-Martínez F . ( 2015). A comparison of procedures to test for moderators in mixed-effects meta-regression models. Psychological Methods, 20( 3), 360-374.
url: http://dx.doi.org/10.1037/met0000023
[1] ZHANG Jianping,QIN Chuanyan,LIU Shanshi. Does seek feedback improve performance? A meta-analytic review about feedback-seeking behavior and individual performance[J]. Advances in Psychological Science, 2020, 28(4): 549-565.
[2] DING Fengqin,WANG Dongxia. Embodied metaphors of moral concepts and its influential factors: A systematic review and meta-analysis[J]. Advances in Psychological Science, 2019, 27(9): 1540-1555.
[3] XU Yan,LI Chaoping. The relationship between leadership styles and engagement: A meta-analysis[J]. Advances in Psychological Science, 2019, 27(8): 1363-1383.
[4] XIN Sufei,JIANG Wenyuan,XIN Ziqiang. A cross-temporal meta-analysis of changes in medical college students’ mental health: 1993-2016[J]. Advances in Psychological Science, 2019, 27(7): 1183-1193.
[5] ZHANG Yali,LI Sen,YU Guoliang. The relationship between self-esteem and social anxiety: A meta-analysis with Chinese students[J]. Advances in Psychological Science, 2019, 27(6): 1005-1018.
[6] TANG Ming,LI Weiqiang,LIU Fuhui,YUAN Bo. The association between guilt and prosocial behavior: A systematic review and meta-analysis[J]. Advances in Psychological Science, 2019, 27(5): 773-788.
[7] SHAO Rong,TENG Zhaojun,LIU Yanling. How violent video games affect prosocial outcomes: A meta-analysis[J]. Advances in Psychological Science, 2019, 27(3): 453-464.
[8] Jun LIU,Chuanyan QIN. Corporate social responsibility and employee performance: A meta-analysis[J]. Advances in Psychological Science, 2018, 26(7): 1152-1164.
[9] WANG Haiwen, ZHANG Shuhua.  The relationship between emotional labor and job satisfaction: A meta-analysis[J]. Advances in Psychological Science, 2018, 26(4): 599-613.
[10] LV Hongjiang, HAN Chengxuan, WANG Daojin.  The relationship between emotional intelligence and leadership effectiveness: A meta-analysis[J]. Advances in Psychological Science, 2018, 26(2): 204-220.
[11] Lingbo ZHAO,Lizu LAI,Yuzhong LIN,Chunxiao ZHAO,Zhihong REN. The effect and moderators of school-based anti-bullying programs: Meta-analysis and GRADE evidence[J]. Advances in Psychological Science, 2018, 26(12): 2113-2128.
[12] DING Fengqin, ZHAO Huying. Is the individual subjective well-being of gratitude stronger?A meta-analysis[J]. Advances in Psychological Science, 2018, 26(10): 1749-1763.
[13] LAI Lizu, REN Zhihong, TAO Rong.  A meta-analysis on Co-Rumination[J]. Advances in Psychological Science, 2018, 26(1): 42-55.
[14] SHI Guochun, ZHANG Lihua, FAN Huiyong.  The relationship between aggression and self-esteem: A Meta-analysis[J]. Advances in Psychological Science, 2017, 25(8): 1274-1288.
[15] YUAN Bo, DONG Yue, LI Weiqiang.  The trust repair effect of apology: A systematic review and meta-analysis[J]. Advances in Psychological Science, 2017, 25(7): 1103-1113.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
Copyright © Advances in Psychological Science
Support by Beijing Magtech