Abstract:

Manipulation checks are important method to ensure the construct validity of managerial psychology experiment, which are also regarded as the basic technology of modern psychology experiments. Although manipulation checks have many advantages and have received widespread attention, current experimental method in managerial psychology literatures mainly focus on the overall experiment design and the strategy of data analysis, rarely investigating the details of manipulation checks design. Although some psychological research published in top international journals has discussed the problem of construct validity and the statistical analysis of manipulation checks, these limited studies have not carried out an integrated review from the key design process of manipulation checks and provided targeted coping strategies. This may lead researchers to undertake manipulation checks designs without solid theoretical support, and even conduct subjective design based on incomplete understanding and draw biased experimental conclusions.
In order to effectively improve the normativity of manipulation checks in the managerial psychology field in China, this study summarized the functions of direct manipulation checks, discriminant manipulation checks and instructional manipulation checks, and then coded 144 experiments drawn from 49 studies in the field of managerial psychology which were published in 4 top-tier journals in 2020 (i.e., Acta Psychologica Sinica, Journal of Psychological Science, Management World, and Nankai Business Review). The present study analyzed the current status, problems and coping strategies of manipulation checks in the managerial psychology field in China from the perspective of the implementation form, the error control, the location choice, the implementation timing and the analytic strategy.
Results showed that the application of manipulation checks in the managerial psychology field were not normative enough in China. Specifically, the way to implement manipulation checks appeared monistic with self-report dominating among these experiments. In terms of error control, many studies did not use maturity measurement scale, and seldomly employed new manipulation checks strategies (e.g., discriminant manipulation checks and instructional manipulation checks) to exclude competitive explanations and invalid experimental samples. For the choice of location, only a few managerial psychology studies paid attention to the influence of manipulation checks location on experiments, and most managerial psychology experiments placed manipulation checks in front of dependent variable. Furthermore, in the aspect of the implementation timing, most managerial psychology experiments completed manipulation checks during the experiment process, and manipulation checks were less implemented in the form of pre manipulation checks (e.g., pilot experiment) and post manipulation checks (e.g., qualitative coding). Finally, as for the strategy of experimental data analysis, the utilization rate of manipulation checks data was low. For example, most managerial psychology studies only focused on analysis of variance of the independent variable between groups and ignored the strength and mediation analysis strategy of the experimental manipulation.
Manipulation checks without enough normativity cannot strongly guarantee the construct validity of experiment and may even interference the experimental process and get incorrect conclusions. To avoid it, future managerial psychology research should pay attention to the application of non-interventional forms to reduce the interference of manipulation checks on the experimental process. Besides, implementing error control strategies should be advocated to eliminate invalid data and competitive explanations. At the same time, the present study suggest that researchers should consider the characteristics of manipulation checks when they decide the appropriate position of manipulation checks included in the experiment material. Researchers are also encouraged to use pre and post manipulation checks to ensure flexibility. In the data analysis phase, we recommend researchers use manipulation strength and mediation analysis to guarantee the internal and external validity of the managerial psychology experiment. By doing so, this study expects the normativity of manipulation checks will be improved to a great extent.

Key words: managerial psychology experiment, direct manipulation checks, discriminant manipulation checks, instructional manipulation checks