Evaluation of predictors’ relative importance: Methods and applications

doi:10.3724/SP.J.1042.2023.00145

Abstract

Abstract:

Psychological researches are more concerned with high-dimensional data than ever. The evaluation of predictors’ relative importance can explore or test the ordering of predictors’ importance, which promotes the effective use of variables with limited resources. This article reviews many aspects of relative importance, including its measures, inference, models, and empirical studies, in order to help researchers select appropriate measures and inference methods and provide directional suggestions for the relative importance studies.

Previous studies proposed various measures of relative importance with different interpretations and calculations. The traditional measure, standardized regression coefficient, considers only the unique effect of a predictor after controlling other predictors but ignores the independent effect of the predictor on the outcome variable. By contrast, the dominance analysis and relative weight measures take into account both the independent and unique contributions of a predictor on the outcome variable. These two measures are recommended because they decompose the R squared such that the contribution to the variation of the outcome variable is attributed to each predictor. Specifically, dominance analysis can be used when the research concerns different importance patterns, whereas relative weight is recommended when evaluating a large number of predictors. After estimating the importance measures, researchers can use the bootstrap sampling or Bayesian testing approach for the inference of the importance of predictors and their orderings. Bootstrap sampling is commonly used to infer the importance of a single predictor or the difference between the importance of two predictors. When comparing the importance of three or more predictors, the Bayesian approach can be used to test the importance orderings.

Besides linear regression models, relative importance analysis has been extended to logistic regression models, multivariate multiple regression models, and multilevel models but not to structural equation models and generalized linear mixed models. Furthermore, the robustness of relative importance has not been analyzed when categorical data is involved in these models. Relative importance analysis can be implemented in many statistical software, such as SPSS, R and Python. However, an integrated software incorporating different measures and inference methods is still absent. Although relative importance analysis has been widely used in psychological studies, researchers may select inappropriate measures and inference methods in different models. Therefore, a real data example is used to illustrate how the relative importance can be evaluated. Finally, we propose that further researches could focus on the applications of relative importance analysis in different models and various types of data together with the software development.

Key words: relative importance, dominance analysis, relative weight, Bootstrap, Bayes factor

CLC Number:

B841

ZHU Xun, GU Xin. Evaluation of predictors’ relative importance: Methods and applications[J]. Advances in Psychological Science, 2023, 31(1): 145-158.

Figures/Tables 13

变量	考试成绩	学科能力	复习时长	焦虑程度
考试成绩	1.00
学科能力	0.80	1.00
复习时长	0.40	0.50	1.00
焦虑程度	0.00	0.20	0.10	1.00

子集	$R 2 ?$	$R 2$ 增量
子集	$R 2 ?$	$X 1$	$X 2$	$X 3$
$k = 0$ 平均	0	0.64	0.16	0.00
$X 1$	0.64		0.00	0.03
$X 2$	0.16	0.48		0.00
$X 3$	0.00	0.67	0.16
$k = 1$ 平均		0.57	0.08	0.01
$X 1 X 2$	0.64			0.03
$X 1 X 3$	0.67		0.00
$X 2 X 3$	0.16	0.51
$k = 2$ 平均		0.51	0.00	0.03
总平均		0.57	0.08	0.01

子集	$R 2 ?$	$R 2$ 增量
子集	$R 2 ?$	$X 1$	$X 2$	$X 3$
$k = 0$ 平均	0	0.64	0.16	0.00
$X 1$	0.64		0.00	0.03
$X 2$	0.16	0.48		0.00
$X 3$	0.00	0.67	0.16
$k = 1$ 平均		0.57	0.08	0.01
$X 1 X 2$	0.64			0.03
$X 1 X 3$	0.67		0.00
$X 2 X 3$	0.16	0.51
$k = 2$ 平均		0.51	0.00	0.03
总平均		0.57	0.08	0.01

效应	自变量	CC	百分比(% R²)
独特效应	学科能力	0.51	76
	复习时长	0.00	0
	焦虑程度	0.03	4
共同效应	学科能力-复习时长	0.16	24
	学科能力-焦虑程度	-0.03	-4
	复习时长-焦虑程度	0.00	0
	学科能力-复习时长-焦虑程度	0.00	0
总		0.67	100

效应	自变量	CC	百分比(% R²)
独特效应	学科能力	0.51	76
	复习时长	0.00	0
	焦虑程度	0.03	4
共同效应	学科能力-复习时长	0.16	24
	学科能力-焦虑程度	-0.03	-4
	复习时长-焦虑程度	0.00	0
	学科能力-复习时长-焦虑程度	0.00	0
总		0.67	100

指标	非负	方差解释	$R 2$ 的分解	比较变量组
$r j 2$	√	×	×	×
$β ? j 2$	√	×	×	×
$r j β ? j$	×	×	√	×
$Δ R 2 (X j)$	√	√	×	√
$C C$	×	√	√	√
$Δ R 2 ˉ (X j)$	√	√	√	√
$S (X j)$	√	√	√	√
$d (X j)$	√	√	√	√
$w j$	√	√	√	√

指标	非负	方差解释	$R 2$ 的分解	比较变量组
$r j 2$	√	×	×	×
$β ? j 2$	√	×	×	×
$r j β ? j$	×	×	√	×
$Δ R 2 (X j)$	√	√	×	√
$C C$	×	√	√	√
$Δ R 2 ˉ (X j)$	√	√	√	√
$S (X j)$	√	√	√	√
$d (X j)$	√	√	√	√
$w j$	√	√	√	√

$BF$	支持假设的证据
1~3	不明显
3~20	积极的
20~150	强
>150	非常强

$BF$	支持假设的证据
1~3	不明显
3~20	积极的
20~150	强
>150	非常强

变量	$P o s t n u m b$	$P r e n u m b$	$A g e$	$P e a b o d y$	$β^j$
$P o s t n u m b$	1.00
$P r e n u m b$	0.68	1.00			0.57
$A g e$	0.30	0.36	1.00		0.06
$P e a b o d y$	0.50	0.59	0.24	1.00	0.15

变量	$P o s t n u m b$	$P r e n u m b$	$A g e$	$P e a b o d y$	$β^j$
$P o s t n u m b$	1.00
$P r e n u m b$	0.68	1.00			0.57
$A g e$	0.30	0.36	1.00		0.06
$P e a b o d y$	0.50	0.59	0.24	1.00	0.15

变量	估计值	标准误	百分位置信区间	BCa置信区间	Wald统计量	$P$ 值
$P r e n u m b$	0.32	0.05	[0.23,0.40]	[0.23,0.40]	7.10	0.000
$A g e$	0.04	0.02	[0.01,0.08]	[0.01,0.08]	2.13	0.017
$P e a b o d y$	0.12	0.03	[0.07,0.18]	[0.07,0.18]	4.27	0.000

变量	估计值	标准误	百分位置信区间	BCa置信区间	Wald统计量	$P$ 值
$P r e n u m b$	0.32	0.05	[0.23,0.40]	[0.23,0.40]	7.10	0.000
$A g e$	0.04	0.02	[0.01,0.08]	[0.01,0.08]	2.13	0.017
$P e a b o d y$	0.12	0.03	[0.07,0.18]	[0.07,0.18]	4.27	0.000

	$H 1$	$H 2$	$H 3$	$H 4$	$H 5$	$H 6$
$B F i u (β ? j 2)$	0.44	5.77	0.00	0.00	0.00	0.00
$B F i u (d j)$	0.01	5.54	0.00	0.00	0.00	0.00

	$H 1$	$H 2$	$H 3$	$H 4$	$H 5$	$H 6$
$B F i u (β ? j 2)$	0.44	5.77	0.00	0.00	0.00	0.00
$B F i u (d j)$	0.01	5.54	0.00	0.00	0.00	0.00

References 51

[1]	方杰, 温忠麟, 张敏强, 孙配贞. (2014). 基于结构方程模型的多重中介效应分析. 心理科学, 37(3), 735-741.
[2]	胡传鹏, 孔祥祯, Wagenmakers, E. J., Ly, A., 彭凯平. (2018). 贝叶斯因子及其在JASP中的实现. 心理科学进展, 26(6), 951-965.
[3]	胡咏梅, 元静. (2021). 学校投入与家庭投入哪个更重要?——回应由《科尔曼报告》引起的关于学校与家庭作用之争. 华东师范大学学报 (教育科学版), 39(1), 1-25.
[4]	宋勃东, 李永娟, 董好叶, 方平, 王岩. (2015). 无惧失败预测幸福: 成就动机对心理幸福感的预测作用. 心理科学, 38(1), 203-208.
[5]	胥彦, 李超平. (2019). 领导风格与敬业度关系的元分析. 心理科学进展, 27(8), 1363-1383.
[6]	张纯霞, 冯廷勇. (2018). 冲动性对学习拖延的影响——自我控制的中介作用. 心理学进展, 8(2), 272-282.
[7]	周浩, 龙立荣. (2014). 分配制度公平对员工分配公平感的影响: 中国组织情境下的实证研究. 心理与行为研究, 12(5), 675-681.
[8]	Ainsworth, S., & Oldfield, J. (2019). Quantifying teacher resilience: Context matters. Teaching and Teacher Education, 82, 117-128. doi: 10.1016/j.tate.2019.03.012
[9]	Azen, R., & Budescu, D. V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8(2), 129-148. pmid: 12924811
[10]	Azen, R., & Budescu, D. V. (2006). Comparing predictors in multivariate regression models: An extension of dominance analysis. Journal of Educational and Behavioral Statistics, 31(2), 157-180.
[11]	Azen, R., & Traxel, N. (2009). Using dominance analysis to determine predictor importance in logistic regression. Journal of Educational and Behavioral Statistics, 34(3), 319-347.
[12]	Budescu, D. V. (1993). Dominance analysis: A new approach to the problem of relative importance of predictors in multiple regression. Psychological Bulletin, 114(3), 542-551. doi: 10.1037/0033-2909.114.3.542 URL
[13]	Braeken, J., Mulder, J., & Wood, S. (2015). Relative effects at work: Bayes factors for order hypotheses. Journal of Management, 41(2), 544-573. doi: 10.1177/0149206314525206 URL
[14]	Braun, M. T., Converse, P. D., & Oswald, F. L. (2019). The accuracy of dominance analysis as a metric to assess relative importance: The joint impact of sampling error variance and measurement unreliability. Journal of Applied Psychology, 104(4), 593-602. doi: 10.1037/apl0000361 pmid: 30321030
[15]	Casillas, A., Robbins, S., Allen, J., Kuo, Y. L., Hanson, M. A., & Schmeiser, C. (2012). Predicting early academic failure in high school from prior academic achievement, psychosocial characteristics, and behavior. Journal of Educational Psychology, 104(2), 407-420. doi: 10.1037/a0027180 URL
[16]	Dalal, R. S., Baysinger, M., Brummel, B. J., & LeBreton, J. M. (2012). The relative importance of employee engagement, other job attitudes, and trait affect as predictors of job performance. Journal of Applied Social Psychology, 42(1), E295-E325.
[17]	Darlington, R. B. (1968). Multiple regression in psychological research and practice. Psychological Bulletin, 69(3), 161-182. pmid: 4868134
[18]	Groemping, U., & Matthias, L. (2021). Package ‘relaimpo’. Relative Importance of Regressors in Linear Models. R Fundation for Statistical Computing: Vienna, Austria.
[19]	Gu, X., Mulder, J., Deković, M., & Hoijtink, H. (2014). Bayesian evaluation of inequality constrained hypotheses. Psychological Methods, 19(4), 511-527. doi: 10.1037/met0000017 pmid: 25046447
[20]	Gu, X., Mulder, J., & Hoijtink, H. (2018). Approximated adjusted fractional Bayes factors: A general method for testing informative hypotheses. British Journal of Mathematical and Statistical Psychology, 71(2), 229-261. doi: 10.1111/bmsp.12110 URL
[21]	Gu, X. (2021). Evaluating predictors’ relative importance using Bayes factors in regression models. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000431
[22]	Gu, X. (2022). Assessing the relative importance of predictors in latent regression models. Structural Equation Modeling: A Multidisciplinary Journal. Advance online publication. https://doi.org/10.1080/10705511.2021.2025377
[23]	Hakanen, J. J., Bakker, A. B., & Turunen, J. (2021). The relative importance of various job resources for work engagement: A concurrent and follow-up dominance analysis. Business Research Quarterly. https://doi.org/10.1177/23409444211012419
[24]	Hoijtink, H., Mulder, J., van Lissa, C., & Gu, X. (2019). A tutorial on testing hypotheses using the Bayes factor. Psychological Methods, 24(5), 539-556. doi: 10.1037/met0000201 pmid: 30742472
[25]	Johnson, J. W. (2000). A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35(1), 1-19. doi: 10.1207/S15327906MBR3501_1 pmid: 26777229
[26]	Johnson, J. W., & LeBreton, J. M. (2004). History and use of relative importance indices in organizational research. Organizational Research Methods, 7(3), 238-257. doi: 10.1177/1094428104266510 URL
[27]	Kraha, A., Turner, H., Nimon, K., Zientek, L., & Henson, R. (2012). Tools to support interpreting multiple regression in the face of multicollinearity. Frontiers in Psychology, 3(44), 1-16.
[28]	Kruskal, W. (1987). Relative importance by averaging over orderings. The American Statistician, 41(1), 6-10.
[29]	Lachowicz, M. J., Preacher, K. J., & Kelley, K. (2018). A novel measure of effect size for mediation analysis. Psychological Methods, 23(2), 244-261. doi: 10.1037/met0000165 pmid: 29172614
[30]	LeBreton, J. M., & Tonidandel, S. (2008). Multivariate relative importance: Extending relative weight analysis to multivariate criterion spaces. Journal of Applied Psychology, 93(2), 329-345. doi: 10.1037/0021-9010.93.2.329 pmid: 18361636
[31]	Luchman, J. N. (2021). Determining relative importance in Stata using dominance analysis: Domin and domme. The Stata Journal, 21(2), 510-538. doi: 10.1177/1536867X211025837 URL
[32]	Luo, W., & Azen, R. (2013). Determining predictor importance in hierarchical linear models using dominance analysis. Journal of Educational and Behavioral Statistics, 38(1), 3-31.
[33]	MacCann, C., Jiang, Y., Brown, L. E., Double, K. S., Bucich, M., & Minbashian, A. (2020). Emotional intelligence predicts academic performance: A meta-analysis. Psychological Bulletin, 146(2), 150-186. doi: 10.1037/bul0000219 pmid: 31829667
[34]	Modersitzki, N., Phan, L. V., Kuper, N., & Rauthmann, J. F. (2021). Who is impacted? Personality predicts individual differences in psychological consequences of the COVID- 19 pandemic in Germany. Social Psychological and Personality Science, 12(6), 1110-1130. doi: 10.1177/1948550620952576 URL
[35]	Mulder, J., Wagenmakers, E. J., & Marsman, M. (2022). A generalization of the Savage-Dickey density ratio for testing equality and order constrained hypotheses. The American Statistician, 76(2), 102-109. doi: 10.1080/00031305.2020.1799861 URL
[36]	Mulder, J., Williams, D. R., Gu, X., Tomarken, A., Böing-Messing, F., Olsson-Collentine, A.,... van Lissa, C. (2021). BFpack: Flexible Bayes factor testing of scientific theories in R. Journal of Statistical Software, 100(18), 1-63.
[37]	Nandlall, S. D., & Millard, K. (2019). Quantifying the relative importance of variables and groups of variables in remote sensing classifiers using Shapley values and game theory. Institute of Electrical and Electronics Engineers Geoscience and Remote Sensing Letters, 17(1),42-46.
[38]	Nimon, K. F., & Oswald, F. L. (2013). Understanding the results of multiple linear regression: Beyond standardized regression coefficients. Organizational Research Methods, 16(4), 650-674. doi: 10.1177/1094428113493929 URL
[39]	Ozdemir, B. (2015). Partitioning variance into constituents in multiple regression models:Commonality analysis. In R. E. Millsap, D. M. Bolt, L. A. Ark, & W.-C. Wang (Eds.), Quantitative psychology research: Vol. 89: Springer proceedings in mathematics & statistics (pp. 395-405). Switzerland: Springer International.
[40]	Preacher, K. J., & Hayes, A. F. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40(3), 879-891. pmid: 18697684
[41]	Preacher, K. J., & Kelley, K. (2011). Effect size measures for mediation models: Quantitative strategies for communicating indirect effects. Psychological Methods, 16(2), 93-115. doi: 10.1037/a0022658 pmid: 21500915
[42]	Richardson, M., Passmore, H. A., Lumber, R., Thomas, R., & Hunt, A. (2021). Moments, not minutes: The nature- wellbeing relationship. International Journal of Wellbeing, 11(1), 8-33. doi: 10.5502/ijw.v11i1.1267 URL
[43]	Rights, J. D., & Sterba, S. K. (2019). Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures. Psychological Methods, 24(3), 309-338. doi: 10.1037/met0000184 pmid: 29999378
[44]	Stevens, J. P. (2002). Applied multivariate statistics for the social sciences. New York: Routledge.
[45]	Thomas, D. R., Zumbo, B. D., Kwan, E., & Schweitzer, L. (2014). On Johnson's (2000) relative weights method for assessing variable importance: A reanalysis. Multivariate Behavioral Research, 49(4), 329-338. doi: 10.1080/00273171.2014.905766 pmid: 26765801
[46]	Tonidandel, S., LeBreton, J. M., & Johnson, J. W. (2009). Determining the statistical significance of relative weights. Psychological Methods, 14(4), 387-399. doi: 10.1037/a0017735 pmid: 19968399
[47]	Tonidandel, S., & LeBreton, J. M. (2009). Determining the relative importance of predictors in logistic regression: An extension of relative weight analysis. Organizational Research Methods, 13(4), 767-781. doi: 10.1177/1094428109341993 URL
[48]	Tonidandel, S., & LeBreton, J. M. (2011). Relative importance analysis: A useful supplement to regression analysis. Journal of Business and Psychology, 26(1), 1-9. doi: 10.1007/s10869-010-9204-3 URL
[49]	Vaessen, A., & Blomert, L. (2010). Long-term cognitive dynamics of fluent reading development. Journal of Experimental Child Psychology, 105(3), 213-231. doi: 10.1016/j.jecp.2009.11.005 pmid: 20042196
[50]	Young, H. R., Glerum, D. R., Wang, W., & Joseph, D. L. (2018). Who are the most engaged at work? A meta‐analysis of personality and employee engagement. Journal of Organizational Behavior, 39(10), 1330-1346. doi: 10.1002/job.2303 URL
[51]	Zhang, Y., Liu, G., Zhang, L., Xu, S., & Cheung, M. W. L. (2021). Psychological ownership: A meta-analysis and comparison of multiple forms of attachment in the workplace. Journal of Management, 47(3), 745-770. doi: 10.1177/0149206320917195 URL