Abstract:

The measurement of shared unit property constructs is ubiquitous in multilevel organizational research, of which the most frequently used approach is to aggregate the ratings of several unit members to the unit level. The data aggregation adequacy testing (DAAT) is a statistical hurdle to ensure the validity and representativeness of aggregated scores. Well-established indicators of DAAT include within-group agreement index, r_WG, and within-group reliability indices, ICC(1) and ICC(2); nonetheless, some key issues are still open to debate, for instance, the superiority of the two families of indicators, the null distribution and data screening decision of r_WG, and appropriate cut-off values. To address the above questions, the current research firstly conducted a content analysis of 166 studies adopting DAAT procedure published on 9 Chinese journals in the field of management and psychology since 2014, coupled with 85 studies from Journal of Applied Psychology as a comparison. Common problems in routine practice of DAAT were identified and related suggestions were proposed as follows: (1) Disentangling and differentiating the role of DAAT indicators; specifically, r_WG should be used as the exclusive indicator of aggregation adequacy, whereas ICC(1) and ICC(2) should be deemed as indices of validity and reliability, respectively. (2) Making prudent and justifiable decisions in choosing null distributions when calculating r_WG index, and excluding groups with low within-group agreement. (3) Applying more reasonable and moderately flexible cut-off values instead of arbitrary and rough practical standards. Last but not the least, researchers should always prioritize theoretical considerations in the process of framework building and DAAT, and unload disproportionate dependence on statistical results.

Key words: multilevel research, shared unit property, aggregation, within-group agreement, within-group reliability