What is the minimum number of effect sizes required in meta-regression? An estimation based on statistical power and estimation precision

doi:10.3724/SP.J.1042.2020.00673

Abstract

Abstract:

Meta-regression is the most frequently used technique for identifying moderators in meta-analysis. In this study, main principles and basic models of meta-analysis and meta-regression were briefly introduced first. Then a Monte Carlo simulation was conducted to investigate the minimum number of the effect size required in meta-regression based on statistical power and estimation precision. The results showed that (1) the Wald-type z test was prone to type I error in meta-regression; (2) at least 20 effect sizes were needed to meet parameter estimation requirements; (3) and inclusion of proper moderators could reduce the number of effect size required. Therefore, it is suggested that (1) meta-analysts should be careful when using the CMA software and the Wald-type z test; (2) at least 20 or more effect sizes are generally needed based on different situations; (3) exploration of moderators is necessary; (4) reviewers can value a meta-analysis research according to the minimum number of effect size required.

Key words: meta-analysis, meta-regression, effect size, minimum number requirement

CLC Number:

B841

FANG Junyan, ZHANG Minqiang. What is the minimum number of effect sizes required in meta-regression? An estimation based on statistical power and estimation precision[J]. Advances in Psychological Science, 2020, 28(4): 673-680.

Figures/Tables 15

References 25

1	董圣杰, 曾宪涛, 郭毅 . ( 2012). R软件Metafor程序包在Meta分析中的应用. 中国循证医学杂志, 12( 9), 1141-1147.
2	方杰, 张敏强 . ( 2012). 中介效应的点估计和区间估计:乘积分布法、非参数Bootstrap和MCMC法. 心理学报, 44( 10), 1408-1420.
3	刘俊, 秦传燕 . ( 2018). 企业社会责任与员工绩效的关系:一项元分析. 心理科学进展, 26( 7), 1152-1164.
4	王超, 袁蒙蒙, 姜媛, 方平 . ( 2019). 宜人性对企业家成就的影响:来自元分析的证据. 心理与行为研究, 17( 1), 126-133.
5	张天嵩, 刘江波, 钟文昭 . ( 2009). Stata在探索异质性来源—Meta回归分析中的应用. 循证医学, 9( 1), 48-50.
6	张云权, 马露, 冯仁杰, 朱耀辉, 李存禄 . ( 2015). 模型回归系数的合并分析在R软件metafor包中的实现. 中国循证医学杂志, 15( 3), 367-372.
7	Borenstein M., Hedges L. V., Higgins J. P. T., & Rothstein H. R . ( 2009). Introduction to meta-analysis. UK: John Wiley & Sons, Ltd., Publication.
8	Borenstein M., Hedges L. V., Higgins J. P.T., & Rothstein H. R . ( 2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1( 2), 97-111.
9	Cafri G., Kromrey J. D., & Brannick M. T . ( 2010). A meta-analysis: Empirical review of statistical power, type I error rates, effect sizes, and model selection of meta-analyses published in psychology. Multivariate Behavioral Research, 45( 2), 239-270.
10	Card N. A. (2012). Applied meta-analysis for social science research. New York: The Guilford Press.
11	Cheung, M. W. L., & Vijayakumar, R . ( 2016). A guide to conducting a meta-analysis. Neuropsychology Review, 26( 2), 121-128.
12	Çoğaltay N., & Karadağ E. . ( 2015). Introduction to meta-analysis. In E. Karadağ (Eds.), Leadership and Organizational Outcomes (2nd ed, pp. 19-28). Switzerland: Springer International Publishing.
13	Cohn, L. D., & Becker, B. J . ( 2003). How meta-analysis increases statistical power. Psychological Methods, 8( 3), 243-253.
14	Field, A. P . ( 2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods, 6( 2), 161-180.
15	Glass, G. V . ( 1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5( 10), 3-8.
16	Huizenga H. M., Visser I., & Dolan C. V . ( 2010). Testing overall and moderator effects in random effects meta-regression. British Journal of Mathematical and Statistical Psychology, 64, 1-19.
17	Knapp, G., & Hartung, G . ( 2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22, 2693-2710.
18	López-López J. A., Noortgate W. V. D., Tanner-Smith E. E., Wilson S. J., & Lipsey M. W . ( 2017). Assessing meta- regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation. Research Synthesis Methods, 8( 4), 435-450.
19	Murphy, K. R . ( 2017). What inferences can and cannot be made on the basis of meta-analysis? Human Resource Management Review, 27( 1), 193-200.
20	Schmidt, F. L . ( 2017). Statistical and measurement pitfalls in the use of meta-regression in meta-analysis. Career Development International, 22( 5), 469-476.
21	Steel, P. D., & Kammeyer-Mueller, J. D . ( 2002). Comparing meta-analytic moderator estimation techniques under realistic conditions. Journal of Applied Psychology, 87( 1), 96-111.
22	Suchotzki K., Verschuere B., Bockstaele B. V., Ben-Shakhar G., & Crombez G . ( 2017). Lying takes time: A meta-analysis on reaction time measures of deception. Psychological Bulletin, 143( 4), 428-453.
23	Valentine J. C., Pigott T. D., & Rothstein H. R . ( 2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational & Behavioral Statistics, 35( 2), 215-247.
24	Viechtbauer, W . ( 2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1-48.
25	Viechtbauer W., López-López J. A., Sánchez-Meca J., & Marín-Martínez F . ( 2015). A comparison of procedures to test for moderators in mixed-effects meta-regression models. Psychological Methods, 20( 3), 360-374.

检验方法	τ² = 0.08
	β为0		β(均)较小		β(均)较大		β一个较大, 一个较小
	β = 0	β = (0, 0)	β = 0.2	β = (0.2, 0.2)	β = 0.5	β = (0.5,0.5)	β = (0.2,0.5)
Knha-test	20	20	20	20	20	20	20
z-test	23	25	23	25	23	25	25

检验方法	τ² = 0.08
	β为0		β(均)较小		β(均)较大		β一个较大, 一个较小
	β = 0	β = (0, 0)	β = 0.2	β = (0.2, 0.2)	β = 0.5	β = (0.5,0.5)	β = (0.2,0.5)
Knha-test	20	20	20	20	20	20	20
z-test	23	25	23	25	23	25	25

检验方法	τ²=0.32
	β为0		β(均)较小		β(均)较大		β一个较大, 一个较小
	β = 0	β = (0,0)	β = 0.2	β = (0.2,0.2)	β = 0.5	β = (0.5,0.5)	β = (0.2,0.5)
Knha-test	38	38	38	38	38	38	38
z-test	43	43	43	43	43	43	43

检验方法	τ²=0.32
	β为0		β(均)较小		β(均)较大		β一个较大, 一个较小
	β = 0	β = (0,0)	β = 0.2	β = (0.2,0.2)	β = 0.5	β = (0.5,0.5)	β = (0.2,0.5)
Knha-test	38	38	38	38	38	38	38
z-test	43	43	43	43	43	43	43

检验方法	τ² = 0.08					τ² = 0.32
	β(均)较小		β(均)较大		β一个较大一个较小	β(均)较小		β(均)较大		β一个较大一个较小
	β = 0.2	β = (0.2, 0.2)	β = 0.5	β = (0.5, 0.5)	β = (0.2, 0.5)	β = 0.2	β = (0.2, 0.2)	β = 0.5	β = (0.5, 0.5)	β = (0.2, 0.5)
Knha-test	30	30	√	√	20	70	70	20	20	50
z-test	38	38	√	√	30	80	80	20	20	52