Watts et al. (Reference Watts, Gray and Medeiros2021) consider many important factors that portend industrial-organizational (I-O) psychologists’ overlooking organizational intervention side effects; their discourse reasonably encourages further exploration of an underconsidered issue in I-O psychology. However, we argue that our understanding of organizational interventions lacks the scope necessary to employ medical metaphor, and we question whether I-O-centric efforts are equally tested and tenable to medical treatments. Rather than assume an existing intervention effectively produced an observed outcome (and consequentially label unintended effects as side effects), we propose that a multilevel intervention research (MLIR) approach may best reveal the nuances of intervention effects. Although a multilevel perspective is not new to I-O psychology, its absence from the focal article warrants note. If we appropriate the “side effects” concept from health care research, so too should we appropriate MLIR, which garners increasing attention in health care (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018), to comprehensively consider intervention effectiveness. Directing attention to MLIR encourages I-O psychologists to incorporate multilevel intervention strategies comprehensively addresings the myriad effects of organizational interventions—intended and otherwise.
Reevaluating how I-O investigators label organizational intervention effects
Watts et al. (Reference Watts, Gray and Medeiros2021) use a medical metaphor as rationale for I-O psychologists underexploring side effects. Though reasonably illustrative, the comparion obscures clinical nuances and assumptions regarding a clinical investigaton’s validity and implications. Notably, medications undergo stepwise testing known as trial phases, whereby investigators test a drug or intervention among a small, homogenous group to establish its safety and efficacy under “ideal circumstances,” culminating in a government-approved intervention study tracking the intervention’s safety in the general population (Phase I through Phase IV trials; National Institutes of Health, 2017). This strategy presumes the existence of a homogenous, morbidity-free cohort that investigators may randomize to an intervention exposure. Clinical trial methodology drives health care’s capacity to attribute observed outcomes to the drug/intervention; “side effects” are defined subsequent to achieving this degree of certainty (American Cancer Society, 2021).
This approach is both is fallable and variably applied in health care; it nevertheless informs tens of thousands of federally regulated clinical trials (Zarin et al., Reference Zarin, Fain, Dobbins, Tse and Williams2019), an influence that Watts et al. (Reference Watts, Gray and Medeiros2021) acknowledge I-O has not achieved (p. 22). I-O psychology’s comparatively smaller scope of influence may indicate that our perspective and methods do not yet produce organizational interventions informed by a context that tenably define and distinguish unexpected effects from side effects. Previous assessments via meta-analyses (e.g., Neuman et al., Reference Neuman, Edwards and Raju1989; Vanhove et al., Reference Vanhove, Herian, Perez, Harms and Lester2016) suggest there is a dearth of organizational interventions tested in contexts one can tenably characterize as homogenous or morbidity free (the clinical trial standard). Indeed, evidence suggests I-O psychologists are challenged to not only implement interventions in organizations but even detect their effects (Karanika-Murray & Biron, Reference Karanika-Murray, Biron, Karanika-Murray and Biron2015).
Much of the discussion of unintended effects of organizational interventions assumes an effectively designed and implemented intervention producing intended effects and unanticipated, untoward side effects. However, this assumption may lack foundation. By labeling certain effects or consequences as “unintended” or “side effects,” I-O psychologists may relinquish their responsibility to explore the possibility that the intervention was fundamentally flawed in its conception or design. Although Watts et al. (Reference Watts, Gray and Medeiros2021) assert that side effects encompass undesired changes in secondary criteria (i.e., not effects that occur when an intervention fails to achieve an intended result), a few of the examples seemingly describe cases where the side effect was an intervention’s singular effect. For instance, some claim that scientific management represents a set of pervasive beliefs about presumed, but unverified, organizational phenomena; consequentially, scientific-management-informed interventions “may serve to demoralize employees, thus resulting in a drop in worker productivity” (Buckley et al., Reference Buckley, Baur, Hardy, Johnson, Johnson, MacDougall, Banford, Bagdasarov, Peterson and Peacock2015, p. 78). If decreased motivation and productivity are the only observable intervention effects, such effects, though unintended, are not “side” but primary and direct intervention products. Similarly, bias training, presumed to encourage more bias awarness, may exacerbate untoward consequences of implicit bias without creating desired changes (Anand & Winters, Reference Anand and Winters2008). Consequently, investigators may be hasty in considering presence or influence of side effects because the intervention “trial phase testing” did not yield a desired effect, encouraging further consideration/testing. Below, we argue that a multilevel perspective may more comprehensively address these challenges.
A multilevel approach to assessing intervention effectiveness
I-O psychologists may consider experiences beyond the I-O psychology domain to advance organizational interventions. Just as the focal article leveraged medical metaphor, so too may I-O psychology leverage insights gleaned from medical intervention research and practice. Below, we introduce the MLIR approach that health service researchers often consider. We also briefly describe implementation mapping (Fernandez et al., Reference Fernandez, Ten Hoor, van Lieshout, Rodriguez, Beidas, Parcel, Ruiter, Markham and Kok2019) to exemplify this approach. Then, we use the multilevel intervention approach to inform and advance the focal article’s proposed future directions for better addressing unintended organizational intervention effects.
The multilevel intervention research in health care
Despite its importance to I-O psychology, multilevel theory (Kozlowski & Klein, Reference Kozlowski, Klein, Klein and Koslowski2000) is rare in I-O intervention research. Existing work often addresses a single level and relegates multilevel interventions to future directions (e.g., Bodner & Bliese, Reference Bodner and Bliese2018). This may be why Watts et al. (Reference Watts, Gray and Medeiros2021) omit multilevel considerations in their discussion of organizational interventions. Still, as multilevel intervention work evolves, I-O psychology will benefit from considering intervention development, implementation, and evaluation approaches espoused by other disciplines for addressing similar multilevel challenges. Consistent with the focal article, we draw attention to intervention approaches in health care (e.g., Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018), which consider multiple content and process effects to develop and evaluate interventions. Specifically, MLIR is a health service research domain informed by multilevel theory that seeks to positively influence health and prevent disease (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018). Researchers increasingly adopt this perspective to address clinical challenges in intervention work (e.g., oncology; Oh et al., Reference Oh, Vinson and Chambers2021). MLIR emerged from a need to improve intervention work that often produced ineffective outcomes owing to salient yet unaddressed factors across levels of analysis (Clauser et al., Reference Clauser, Taplin, Foster, Fagan and Kaluzny2012).
MLIR considers micro-, meso-, and macro-level influences within a nested, interdependent structure of individuals, communities, and national policies that contribute to intervention implementation and effectiveness (Bronfenbrenner, Reference Bronfenbrenner1979). Hall et al. (Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018) posit that MLIR “requires particular attention to conditions [that] may vary across research domains and implementation contexts” and “typically targets factors at two or more different levels of analysis, with specific constructs that can be measured and evaluated at multiple levels” (p. 972). MLIR addresses these complexities, helping scholars understand and detect instances in which barriers existing at one level may reduce the efficacy of an intervention implemented at another level as well as account for factors across levels that may contribute to an increased likelihood of intervention success and sustainability (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018). In MLIR, rather than label unintended effects as side effects, effort is placed into understanding how and why all effects occur across levels.
A multilevel intervention approach example: Implementation mapping
The implementation mapping is a five-step approach to planning multilevel interventions in the public health field advocated by the National Cancer Institute (NCI, 2020). We summarize implementation mapping steps below to provide an example for how I-O psychologists may adopt an MLIR approach (see Fernandez et al., Reference Fernandez, Ten Hoor, van Lieshout, Rodriguez, Beidas, Parcel, Ruiter, Markham and Kok2019 for more details).
-
1. Conduct a needs assessment or problem analysis that determines who will adopt the intervention and who will implement it, identifying relevant levels of analysis for each.
-
2. Create matrices of intervention-driven change objectives, including adoption and implementation goals, that map predictors at each level to outcomes of interest at each level.
-
3. Select theory-based methods best suited to study multilevel predictors and outcomes as well as the mechanisms driving potential cross-level effects, including implementation strategies.
-
4. Build an implementation protocol that maps intervention components and implementation strategies to the theoretical model and acknowledges within- and cross-level influences.
-
5. Plan for the evaluation of overall program effectiveness by developing a mixed-methods approach to measuring and assessing intervention processes, intervention effects, and implementation outcomes (e.g., adoption, sustainability) across levels of analysis.
A multilevel way forward
Although MLIR provides many advantages, its adoption requires coordinating limited resources and investing substantial effort across a variety of stakeholders (Nastasi & Hitchcock, Reference Nastasi and Hitchcock2009). MLIR may not resolve all of the complexities of conducting intervention work. Still, MLIR can advance the “way forward” described in the focal article (Watts et al., Reference Watts, Gray and Medeiros2021, pp. 24–30). Below, we offer initial insights designed to further the conversation.
Multilevel intervention research questions
Approaching interventions from an MLIR framework involves examining factors across levels that may function as “third variables” influencing intervention components and outcomes of interest. MLIR positions the consideration of potential side effects as fundamental to developing intervention research questions. Instead of asking, how will X intervention effect Y outcome of interest or what Z mediates or moderates X relative to Y (Watts et al., Reference Watts, Gray and Medeiros2021, p. 25), an MLIR question examines all possible effects of an intervention, including measured and potentially unmeasured confounders (Brookhart et al., Reference Brookhart, Stürmer, Glynn, Rassen and Schneeweiss2010). We appreciate the MLIR questions posed by Hall et al. (Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018): “Are the levels interacting, or do they compete? How do they interact? How can they be disentangled?” (p. 973).
Training future multilevel researchers
Instead of supplementing coursework on organizational interventions with special topics courses about side effects (Watts et al., Reference Watts, Gray and Medeiros2021, p. 30), we believe that teaching a multilevel intervention perspective will help to better educate future researchers. A multilevel approach that considers emergence and cross-level effects will help researchers understand all potential intervention-related statistical effects that should be studied, including mediation, moderation, and confounding effects (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018).
Incentivizing multilevel research
Federal sponsors increasingly solicit research proposals requiring multilevel considerations (Jacobsen et al., Reference Jacobsen, de Moor, Doria-Rose, Geiger, Kobrin, Sampson and Smith2021). Solicitations encourage investigators to consider all relevant multilevel factors potentially related to intervention implementation and effectiveness (NCI, 2017). I-O journals may adopt this perspective when reviewing manuscript submissions. Although we agree with Watts et al. (Reference Watts, Gray and Medeiros2021) that a special issue on “side effects” may increase visibility of the issue, sustained attention to MLIR factors is more likely to solidify comprehensive treatment of intervention effects into I-O research culture.
Intervention guidelines and reporting
The focal article acknowledges the complexity of organizations as social systems (Watts et al., Reference Watts, Gray and Medeiros2021, p. 18) but seemingly addresses system complexity as one of many factors complicating assessment of side effects. MLIR, in contrast, presumes system complexity predicates the entire foundation of intervention design and evaluation (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018). A “warning label” that lists side effects of an intervention (Watts et al., Reference Watts, Gray and Medeiros2021, p. 29) might be relabled to disclose all intervention effects, allowing employers to determine an intevention’s impact across levels.
Conclusion
Through this commentary, we remind I-O psychologists of the importance of multilevel considerations for the study of organizational interventions and provide MLIR as an example of how interventions are approached in health care research that may be more applicable to I-O interventions than medical drugs. Design, implementation, and evaluation processes challenge investigators who are invested in studying organizational intervention effects. The health service MLIR approach may help I-O investigators more comprehensively address these complexities in a way that also mitigates concerns about “side effects.” As multilevel theory, which provides the basis of MLIR, is a critical component of I-O education and training, I-O scholars will likely recognize many MLIR aspects we explore here. These multilevel intervention tools and exercises may help I-O psychologists better expose and test their latent assumptions to more comprehensively consider intervention effectiveness. Our suggestions may also encourage transdisciplinary collaboration benefitting the constituents I-O psychologists serve. Our field may then clarify the mechanisms that produce anticipated outcomes and minimize unanticipated, untoward outcomes across intervention phases from conceptualization to iterative evaluation.
You have
Access
Watts et al. (Reference Watts, Gray and Medeiros2021) consider many important factors that portend industrial-organizational (I-O) psychologists’ overlooking organizational intervention side effects; their discourse reasonably encourages further exploration of an underconsidered issue in I-O psychology. However, we argue that our understanding of organizational interventions lacks the scope necessary to employ medical metaphor, and we question whether I-O-centric efforts are equally tested and tenable to medical treatments. Rather than assume an existing intervention effectively produced an observed outcome (and consequentially label unintended effects as side effects), we propose that a multilevel intervention research (MLIR) approach may best reveal the nuances of intervention effects. Although a multilevel perspective is not new to I-O psychology, its absence from the focal article warrants note. If we appropriate the “side effects” concept from health care research, so too should we appropriate MLIR, which garners increasing attention in health care (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018), to comprehensively consider intervention effectiveness. Directing attention to MLIR encourages I-O psychologists to incorporate multilevel intervention strategies comprehensively addresings the myriad effects of organizational interventions—intended and otherwise.
Reevaluating how I-O investigators label organizational intervention effects
Watts et al. (Reference Watts, Gray and Medeiros2021) use a medical metaphor as rationale for I-O psychologists underexploring side effects. Though reasonably illustrative, the comparion obscures clinical nuances and assumptions regarding a clinical investigaton’s validity and implications. Notably, medications undergo stepwise testing known as trial phases, whereby investigators test a drug or intervention among a small, homogenous group to establish its safety and efficacy under “ideal circumstances,” culminating in a government-approved intervention study tracking the intervention’s safety in the general population (Phase I through Phase IV trials; National Institutes of Health, 2017). This strategy presumes the existence of a homogenous, morbidity-free cohort that investigators may randomize to an intervention exposure. Clinical trial methodology drives health care’s capacity to attribute observed outcomes to the drug/intervention; “side effects” are defined subsequent to achieving this degree of certainty (American Cancer Society, 2021).
This approach is both is fallable and variably applied in health care; it nevertheless informs tens of thousands of federally regulated clinical trials (Zarin et al., Reference Zarin, Fain, Dobbins, Tse and Williams2019), an influence that Watts et al. (Reference Watts, Gray and Medeiros2021) acknowledge I-O has not achieved (p. 22). I-O psychology’s comparatively smaller scope of influence may indicate that our perspective and methods do not yet produce organizational interventions informed by a context that tenably define and distinguish unexpected effects from side effects. Previous assessments via meta-analyses (e.g., Neuman et al., Reference Neuman, Edwards and Raju1989; Vanhove et al., Reference Vanhove, Herian, Perez, Harms and Lester2016) suggest there is a dearth of organizational interventions tested in contexts one can tenably characterize as homogenous or morbidity free (the clinical trial standard). Indeed, evidence suggests I-O psychologists are challenged to not only implement interventions in organizations but even detect their effects (Karanika-Murray & Biron, Reference Karanika-Murray, Biron, Karanika-Murray and Biron2015).
Much of the discussion of unintended effects of organizational interventions assumes an effectively designed and implemented intervention producing intended effects and unanticipated, untoward side effects. However, this assumption may lack foundation. By labeling certain effects or consequences as “unintended” or “side effects,” I-O psychologists may relinquish their responsibility to explore the possibility that the intervention was fundamentally flawed in its conception or design. Although Watts et al. (Reference Watts, Gray and Medeiros2021) assert that side effects encompass undesired changes in secondary criteria (i.e., not effects that occur when an intervention fails to achieve an intended result), a few of the examples seemingly describe cases where the side effect was an intervention’s singular effect. For instance, some claim that scientific management represents a set of pervasive beliefs about presumed, but unverified, organizational phenomena; consequentially, scientific-management-informed interventions “may serve to demoralize employees, thus resulting in a drop in worker productivity” (Buckley et al., Reference Buckley, Baur, Hardy, Johnson, Johnson, MacDougall, Banford, Bagdasarov, Peterson and Peacock2015, p. 78). If decreased motivation and productivity are the only observable intervention effects, such effects, though unintended, are not “side” but primary and direct intervention products. Similarly, bias training, presumed to encourage more bias awarness, may exacerbate untoward consequences of implicit bias without creating desired changes (Anand & Winters, Reference Anand and Winters2008). Consequently, investigators may be hasty in considering presence or influence of side effects because the intervention “trial phase testing” did not yield a desired effect, encouraging further consideration/testing. Below, we argue that a multilevel perspective may more comprehensively address these challenges.
A multilevel approach to assessing intervention effectiveness
I-O psychologists may consider experiences beyond the I-O psychology domain to advance organizational interventions. Just as the focal article leveraged medical metaphor, so too may I-O psychology leverage insights gleaned from medical intervention research and practice. Below, we introduce the MLIR approach that health service researchers often consider. We also briefly describe implementation mapping (Fernandez et al., Reference Fernandez, Ten Hoor, van Lieshout, Rodriguez, Beidas, Parcel, Ruiter, Markham and Kok2019) to exemplify this approach. Then, we use the multilevel intervention approach to inform and advance the focal article’s proposed future directions for better addressing unintended organizational intervention effects.
The multilevel intervention research in health care
Despite its importance to I-O psychology, multilevel theory (Kozlowski & Klein, Reference Kozlowski, Klein, Klein and Koslowski2000) is rare in I-O intervention research. Existing work often addresses a single level and relegates multilevel interventions to future directions (e.g., Bodner & Bliese, Reference Bodner and Bliese2018). This may be why Watts et al. (Reference Watts, Gray and Medeiros2021) omit multilevel considerations in their discussion of organizational interventions. Still, as multilevel intervention work evolves, I-O psychology will benefit from considering intervention development, implementation, and evaluation approaches espoused by other disciplines for addressing similar multilevel challenges. Consistent with the focal article, we draw attention to intervention approaches in health care (e.g., Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018), which consider multiple content and process effects to develop and evaluate interventions. Specifically, MLIR is a health service research domain informed by multilevel theory that seeks to positively influence health and prevent disease (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018). Researchers increasingly adopt this perspective to address clinical challenges in intervention work (e.g., oncology; Oh et al., Reference Oh, Vinson and Chambers2021). MLIR emerged from a need to improve intervention work that often produced ineffective outcomes owing to salient yet unaddressed factors across levels of analysis (Clauser et al., Reference Clauser, Taplin, Foster, Fagan and Kaluzny2012).
MLIR considers micro-, meso-, and macro-level influences within a nested, interdependent structure of individuals, communities, and national policies that contribute to intervention implementation and effectiveness (Bronfenbrenner, Reference Bronfenbrenner1979). Hall et al. (Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018) posit that MLIR “requires particular attention to conditions [that] may vary across research domains and implementation contexts” and “typically targets factors at two or more different levels of analysis, with specific constructs that can be measured and evaluated at multiple levels” (p. 972). MLIR addresses these complexities, helping scholars understand and detect instances in which barriers existing at one level may reduce the efficacy of an intervention implemented at another level as well as account for factors across levels that may contribute to an increased likelihood of intervention success and sustainability (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018). In MLIR, rather than label unintended effects as side effects, effort is placed into understanding how and why all effects occur across levels.
A multilevel intervention approach example: Implementation mapping
The implementation mapping is a five-step approach to planning multilevel interventions in the public health field advocated by the National Cancer Institute (NCI, 2020). We summarize implementation mapping steps below to provide an example for how I-O psychologists may adopt an MLIR approach (see Fernandez et al., Reference Fernandez, Ten Hoor, van Lieshout, Rodriguez, Beidas, Parcel, Ruiter, Markham and Kok2019 for more details).
1. Conduct a needs assessment or problem analysis that determines who will adopt the intervention and who will implement it, identifying relevant levels of analysis for each.
2. Create matrices of intervention-driven change objectives, including adoption and implementation goals, that map predictors at each level to outcomes of interest at each level.
3. Select theory-based methods best suited to study multilevel predictors and outcomes as well as the mechanisms driving potential cross-level effects, including implementation strategies.
4. Build an implementation protocol that maps intervention components and implementation strategies to the theoretical model and acknowledges within- and cross-level influences.
5. Plan for the evaluation of overall program effectiveness by developing a mixed-methods approach to measuring and assessing intervention processes, intervention effects, and implementation outcomes (e.g., adoption, sustainability) across levels of analysis.
A multilevel way forward
Although MLIR provides many advantages, its adoption requires coordinating limited resources and investing substantial effort across a variety of stakeholders (Nastasi & Hitchcock, Reference Nastasi and Hitchcock2009). MLIR may not resolve all of the complexities of conducting intervention work. Still, MLIR can advance the “way forward” described in the focal article (Watts et al., Reference Watts, Gray and Medeiros2021, pp. 24–30). Below, we offer initial insights designed to further the conversation.
Multilevel intervention research questions
Approaching interventions from an MLIR framework involves examining factors across levels that may function as “third variables” influencing intervention components and outcomes of interest. MLIR positions the consideration of potential side effects as fundamental to developing intervention research questions. Instead of asking, how will X intervention effect Y outcome of interest or what Z mediates or moderates X relative to Y (Watts et al., Reference Watts, Gray and Medeiros2021, p. 25), an MLIR question examines all possible effects of an intervention, including measured and potentially unmeasured confounders (Brookhart et al., Reference Brookhart, Stürmer, Glynn, Rassen and Schneeweiss2010). We appreciate the MLIR questions posed by Hall et al. (Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018): “Are the levels interacting, or do they compete? How do they interact? How can they be disentangled?” (p. 973).
Training future multilevel researchers
Instead of supplementing coursework on organizational interventions with special topics courses about side effects (Watts et al., Reference Watts, Gray and Medeiros2021, p. 30), we believe that teaching a multilevel intervention perspective will help to better educate future researchers. A multilevel approach that considers emergence and cross-level effects will help researchers understand all potential intervention-related statistical effects that should be studied, including mediation, moderation, and confounding effects (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018).
Incentivizing multilevel research
Federal sponsors increasingly solicit research proposals requiring multilevel considerations (Jacobsen et al., Reference Jacobsen, de Moor, Doria-Rose, Geiger, Kobrin, Sampson and Smith2021). Solicitations encourage investigators to consider all relevant multilevel factors potentially related to intervention implementation and effectiveness (NCI, 2017). I-O journals may adopt this perspective when reviewing manuscript submissions. Although we agree with Watts et al. (Reference Watts, Gray and Medeiros2021) that a special issue on “side effects” may increase visibility of the issue, sustained attention to MLIR factors is more likely to solidify comprehensive treatment of intervention effects into I-O research culture.
Intervention guidelines and reporting
The focal article acknowledges the complexity of organizations as social systems (Watts et al., Reference Watts, Gray and Medeiros2021, p. 18) but seemingly addresses system complexity as one of many factors complicating assessment of side effects. MLIR, in contrast, presumes system complexity predicates the entire foundation of intervention design and evaluation (Hall et al., Reference Hall, Oh, Perez, Rice, Patel, Czajkowski and Klesges2018). A “warning label” that lists side effects of an intervention (Watts et al., Reference Watts, Gray and Medeiros2021, p. 29) might be relabled to disclose all intervention effects, allowing employers to determine an intevention’s impact across levels.
Conclusion
Through this commentary, we remind I-O psychologists of the importance of multilevel considerations for the study of organizational interventions and provide MLIR as an example of how interventions are approached in health care research that may be more applicable to I-O interventions than medical drugs. Design, implementation, and evaluation processes challenge investigators who are invested in studying organizational intervention effects. The health service MLIR approach may help I-O investigators more comprehensively address these complexities in a way that also mitigates concerns about “side effects.” As multilevel theory, which provides the basis of MLIR, is a critical component of I-O education and training, I-O scholars will likely recognize many MLIR aspects we explore here. These multilevel intervention tools and exercises may help I-O psychologists better expose and test their latent assumptions to more comprehensively consider intervention effectiveness. Our suggestions may also encourage transdisciplinary collaboration benefitting the constituents I-O psychologists serve. Our field may then clarify the mechanisms that produce anticipated outcomes and minimize unanticipated, untoward outcomes across intervention phases from conceptualization to iterative evaluation.