Question 0: How Appropriate Is Meta-Analysis for the Generalization We Want to Make?

It is not our intention to suggest that meta-analyses should be abandoned by I-O psychologists or that the technique has no place in our field. Meta-analysis is an undeniably useful tool for detecting a signal among background noise attributable to artifacts and error that could stall practical and theoretical advancement. However, we believe that the applications and interpretations of meta-analysis can deviate too far from the tool's original intention to examine validity generalization (i.e., establishing degree of context invariance for practically important predictor–criterion relationships; Hunter & Schmidt, Reference Hunter and Schmidt2004) and into making claims of scientific inference (i.e., understanding and explaining psychological and social phenomena). Overextending meta-analyses into domains where the core phenomena are inherently within-person and/or socially dynamic risks overgeneralization or “freezing” of a field of inquiry when the detected signal is interpreted as “proof” of a theoretical explanation.

So under what conditions are meta-analyses likely to be most appropriate? We posit that when the relations of interest are unidirectional, linear, and generally stable over time (or the time window for generalization is narrowly specified, e.g., first 6 months on the job), then the inferences drawn from meta-analyses are likely to be highly informative. For example, Schmidt and Hunter's (Reference Schmidt and Hunter1998) meta-analysis on selection methods for personnel psychology examined the validity of different selection procedures in predicting subsequent job and training performance. In that case, the evaluated relationships are likely to be unidirectional (e.g., job/training performance cannot precede selection), adequately linear (though perhaps not perfectly for all predictors examined), and generally stable over a defined period of time. Note that Tett et al.’s (Reference Tett, Hundley and Christiansen2017) recommendations for evaluating the degree of situational specificity and the potential for identifying moderators of these relationships remain relevant even under these conditions. The broader point, however, is that the phenomena/relations of interest and the interpretative affordances granted by the meta-analytic engine are appropriately aligned. In sum, meta-analysis serves a valuable and practically important purpose for summarizing generalizable cumulative knowledge under the conditions outlined above and can provide evidence-based guidance for evaluating procedures/relations that help the field avoid reinventing the wheel.

We believe this point is also relevant to Tett et al.’s (Reference Tett, Hundley and Christiansen2017) sentiment that meta-analysis should serve a less terminal role in in the scientific process; to this end, we posit that meta-analysis is most appropriate to use as an intermediary rather than concluding step in our quest for scientific inference. It is commonly recognized that variability in rho suggests the need to explore moderators of a relationship in future research. For example, after early meta-analyses in I-O psychology identified variability in the relationships between employment interviews and job performance, subsequent research found that interview structure moderated the effectiveness of interviews (McDaniel, Whetzel, Schmidt, & Maurer, Reference McDaniel, Whetzel, Schmidt and Maurer1994; Wiesner & Cronshaw, Reference Wiesner and Cronshaw1988). However, the inclusion of an ever-growing number of moderators should not take precedence over the need to re-evaluate and—most importantly—update one's initial theory or explanation for how, why, and when a relationship should manifest. For example, meta-analytic evidence that reveals high situational specificity should cause the researcher to question the original expectation of a between-individual, linear relationship. In any case, it strikes us as both more instructive and constructive to the goals of scientific inference to view situational specificity not as a “failure” of generalization, but as an indication that the precision of the theory motivating the empirical results needs to be improved. Consequently, meta-analysis is more appropriate as a means for reconnecting the scientific process feedback loop and a diagnostic indicator of the extent to which one's theoretical rationale and interpretations of empirical observations are aligned.

Expanding to Where Life Does Happen

Although we recognize the functional roles and potential benefits of meta-analysis, we also believe it can inadvertently steer our field away from “where life does happen.” That is, because the nature of meta-analytic inference emphasizes between-individual, linear, and cross-sectional generalizations by design, it affords few opportunities to generate, evaluate, and update psychological theory at the dynamic and within-person level where most of our primary phenomena of interest actually function. For example, subtle features of the relationship between self-efficacy and performance that are illuminated when analyzed within-individual are lost or misinterpreted when treated as only a between-individual effect (Vancouver, Thompson, Tischner, & Putka, Reference Vancouver, Thompson, Tischner and Putka2002; Vancouver, Thompson, & Williams, Reference Vancouver, Thompson and Williams2001). Furthermore, concepts such as job satisfaction, racial stratification, and team cohesion are rarely effects of singular causes and undoubtedly themselves induce subsequent effects—yet it is not hard to come by meta-analytic treatments of these variables as static, between-person antecedents and outcomes in linear, cross-sectional relationships in our literature. Consequently, we believe that significant improvements to the quality of our scientific inferences can be achieved by efforts to step below the aggregate generalizations afforded by meta-analytic techniques and to consider the dynamic and within-unit intricacies of the phenomena of interest to psychological researchers.

Somewhat paradoxically, such dynamism is commonly discussed or assumed in our descriptive theories. Unfortunately, these processes are difficult to capture and thus are rarely examined using archetypal construct-to-construct empirical studies (Cronin, Weingart, & Todorova, Reference Cronin, Weingart and Todorova2011; Kozlowski, Chao, Grand, Braun, & Kuljanin, Reference Kozlowski, Chao, Grand, Braun and Kuljanin2013). Meta-analysis is a useful method for evaluating construct-to-construct relationships, but is limited in its ability to evaluate the validity and generalizability of dynamical theories. Yet, when the underlying dynamism and complexity in psychological concepts are recognized, we return to the process of scientific inquiry and the question, “Why?” Specifying the process underlying the connection between two constructs often drives further specification when new questions are raised. Theories become more precise and more transparent. The strengthening of construct conceptualization and the explication of linking mechanisms improves construct and internal validity and increases the accessibility of the “black box” between the theorized inputs and outputs (Davis, Eisenhardt, & Bingham, Reference Davis, Eisenhardt and Bingham2007). Furthermore, the development of one's empirical tests of theory become better grounded, potential errors in the theory are more apparent and appropriately addressed by the field, and unanswered research questions are potentially unearthed. Thinking and theorizing in terms of dynamics naturally facilitates discourse and revision of theory in a manner that can be difficult to achieve through meta-analytic means.

Tackling the underlying dynamism of psychological constructs is itself a complex task, and predicting outcomes of these processes is even more so. As simple assumptions and logic that form the structure of a theory compound on one another and are carried out over time, the consequences become nearly impossible to derive (Axelrod, Reference Axelrod, Conte, Hegselmann and Terna1997). Meta-analysis, whether used to explore variability in a relationship or to summarize between-person, linear, and cross-sectional tests of a theory, does not support the exploration of dynamics. The processes within the black box get lost in aggregation. Alternative methods are needed to pursue this new paradigm of theoretical development.

One potentially valuable tool to facilitate the accumulation, evaluation, and updating of dynamic theory is computational modeling and simulation. Building a computational model involves translating theory into algorithmic representations, which can subsequently be used to simulate data and conduct virtual experiments by feeding the algorithm theoretically driven values to represent different levels of the constructs (Davis et al., Reference Davis, Eisenhardt and Bingham2007; Harrison, Lin, Carroll, & Carley, Reference Harrison, Lin, Carroll and Carley2007). This translation necessitates specificity and simulation allows processes that are difficult to study in the lab or in the field to develop in a much shorter window of time than would be required in reality. Computational modeling and simulation are useful both in the early stages of theory development, prior to empirical validation (see Kozlowski, Chao, Grand, Braun, and Kuljanin [Reference Kozlowski, Chao, Grand, Braun and Kuljanin2016] and Grand, Braun, Kuljanin, Kozlowski, and Chao [Reference Grand, Braun, Kuljanin, Kozlowski and Chao2016] for an example in the context of multilevel emergent effects) or to pursue the processes that underlie empirically collected data (see Vancouver, Weinhardt, and Schmidt [Reference Vancouver, Weinhardt and Schmidt2010] for an explanation of the Schmidt and DeShon [Reference Schmidt and DeShon2007] reversal effect in dynamic goal prioritization). Thus, computational modeling offers a useful method to support scientific inference and explore the generalizability of theory in contexts that may be ill-suited for meta-analysis (e.g., when phenomena are within-person, nonlinear, and dynamic).