The framework proposed by Banks et al. (Reference Banks, Knapp, Lin, Sanders and Grand2022) in the focal article provides a useful structure for guiding assessment scientists in taking actions to support ethical decision making in organizations. In this commentary we discuss how assessment scientists can broadly apply the framework throughout the assessment life cycle. Specifically, we build upon the idea that lack of ethical knowledge is an important cause of ethical failure (Popp & Popp, Reference Popp and Popp2021) by providing specific ethical considerations that can be employed by practitioners in the context of assessment science within each stage of the proposed framework.
Recently the business case for inclusive hiring is becoming more prominent, introducing the need for increased focus on diversity, equity, and inclusion as an ethical issue for practitioners. Organizations are aspiring to move beyond compliance with professional best practices to create a fully inclusive assessment experience that benefits both organizations in support of their diversity goals and hiring outcomes for a diverse talent pool. In the recent Top 10 Workplace Trends survey (SIOP, 2021), thousands of industrial-organizational psychologists voted Inclusive Practices to Get, Keep, and Grow Talent as the #4 trend organizations will face in 2021, highlighting the opportunity and ethical responsibility for practitioners to leverage inclusive assessments to help organizations achieve these goals.
First stage: Recognize the ethical issue
Practitioners should recognize the ethical responsibility to serve a diverse talent pool. From a cultural inclusion perspective, practitioners should ensure a robust localization process that follows the International Test Commission Guidelines for Translating and Adapting Tests (ITC; 2017). This process involves content decentering to remove any culture-specific material and a rigorous process of translation and back translation to ensure that the content will be equivalent for candidates across cultures. In addition to cultural decentering, one must consider the different demographic groups comprising the diverse talent pool. For instance, individuals with disabilities are an underemployed population, with only 17.9% of individuals with a disability employed in the United States in 2020 (U.S. Bureau of Labor Force Statistics, 2021). With the goal of disability inclusion, practitioners should consider the accessibility of both the assessment platform and the content itself when developing assessment programs and openly share best practices for doing so with other practitioners. In light of the movement for diversity and inclusion, it is more important than ever for practitioners to recognize the ethical responsibility to implement inclusive and equitable assessments to support data-driven decisions in organizations and promote equal opportunity to help organizations achieve their diversity goals.
Second stage: Gather information
As detailed in the proposed framework, once these ethical issues are identified, practitioners should then gather more information. Practitioners should continually seek out information on the latest developments in best practice guidance, especially where this information is not yet established, unclear, or rapidly changing. For example, as new assessment methodologies emerge (e.g., artificial intelligence), practitioners should seek out guidance as it evolves from industry leaders for how they can be ethically used. Where limited best practice guidance is available (e.g., understanding the appropriateness of different assessment types for a neurodiverse talent pool), practitioners should commit to gathering and sharing information that will inform evidence-based best practices for the field.
Third stage: Identify proximal and distal stakeholders
The proximal stakeholders for assessments are the organizations that are making talent decisions. A practitioner’s ethical responsibility to these stakeholders is ensuring that assessments are fit for purpose (determined through job analysis) and accurately predict performance to deliver value and drive business outcomes (determined through validation). The distal stakeholders of inclusive assessments are candidates. In light of the Great Resignation (World Economic Forum, 2021) and current labor shortage, ensuring a positive candidate experience is increasingly important. Providing candidates with a realistic job preview, combined with the opportunity to demonstrate their job-related skills and behaviors can help to ensure transparency and a good match between the candidate’s skills and requirements of the job, as this mismatch may be a driving force behind the current labor shortage. As noted by the authors in the focal article, we must avoid becoming so focused on our ethical responsibilities to the proximal stakeholders that we neglect consideration of “how a professional decision might influence the well-being of distal stakeholders” (Banks et al., Reference Banks, Knapp, Lin, Sanders and Grand2022, p. X). Practitioners must balance the competing demands from both proximal stakeholders (organizations) wanting highly predictive assessments while maintaining a focus on the experience of distal stakeholders (candidates). A positive candidate experience can be achieved by implementing assessments that are short enough so that they are not perceived as cumbersome for the candidate but lengthy enough that they allow the candidate to demonstrate a range of job-related knowledge, skills, abilities, and other characteristics and collect enough information to accurately evaluate potential for success on the job. Additionally, ensuring a positive experience for the distal stakeholders may help to attract talent that is more committed and engaged, ultimately benefiting the proximal stakeholders as well.
Fourth stage: Identify alternative actions
With the goal of measuring as many competencies as necessary to obtain a comprehensive understanding of a candidate’s potential to perform on the job, practitioners should carefully design the assessment program by mapping the content to job-related requirements. This should involve a thorough consideration of available options, including different assessment types and emerging assessment innovations and technologies, with the goal of using multiple assessment types in combination to maximize incremental validity and obtain a full picture of a candidate’s fit for the role. Furthermore, consideration should be given to pairing together assessment components that will balance the needs for predictive validity with achieving diversity goals. For example, when cognitive ability tests are used, they can be paired with noncognitive assessments, which are less likely to exhibit subgroup score differences to mitigate the risk of adverse impact in the overall assessment score.
Fifth stage: Compare alternative actions
As discussed in the previous section, combining multiple assessment types can be a powerful strategy for enabling inclusive outcomes. When evaluating options to optimize assessment design, assessment scientists must first and foremost ensure the integrity of the psychometric properties in accordance with professional best practices. The ethical responsibility of assessment scientists lies in preserving the psychometric properties and legal defensibility of the assessment while making informed decisions that best balance the goals of the organization and the needs of the candidates. Therefore, careful consideration should be given to the weighting of the assessment components. Implementing differential weightings can further reduce potential for adverse impact while preserving predictive validity. For example, assessments that generally exhibit smaller score differences across subgroups (e.g., personality or situational judgment) can be weighted more heavily in an overall score than those that tend to exhibit larger score differences (e.g., cognitive ability). This approach can reduce adverse impact in the overall score, on which decisions are made, potentially without substantially decreasing predictive validity. Alternative weighting schemes should be explored in order to balance predictive validity and fairness with the goal of achieving an inclusive assessment experience for a diverse talent pool. This stage is more of an art than a science, and there is likely no one “right” approach. Decisions made at this stage should be documented along with the rationale to support the defensibility of the assessment and inform ongoing monitoring and evaluation.
Sixth stage: Implement action(s) and monitor outcomes
Once the assessment program is designed, practitioners have an ethical responsibility to continually monitor and evaluate associated outcomes. This includes collecting ongoing validation evidence to document how an assessment predicts performance across the contexts in which it is used and monitoring how the validity may deteriorate over time. For assessments that have been translated, this involves statistical confirmation that the localized assessment is still functioning in a similar way to inform decisions across cultures, as detailed by the ITC (2017) guidelines. When enough data on localized versions of assessment content becomes available, measurement equivalence should be established through confirmatory factor analysis procedures. An analysis of differential item functioning can determine whether there are problematic variations in the measurement of items across cultures. Additionally, local validation studies can be conducted across different language versions of the assessment to bolster the evidence of the relationship between the predictor and criterion across cultures. Practitioners should also continually monitor adverse impact across multiple levels (i.e., candidates who are hired vs. not hired as well as the distribution of demographics within various score bands) to detect systematic differences in how the assessment functions across subgroups. As noted by Banks et al. (Reference Banks, Knapp, Lin, Sanders and Grand2022), ethical decision making requires one to “revisit and reconsider the conclusions generated in particular phases of the framework as a situation unfolds, additional information is uncovered, and/or consequences are realized” (p. X). Accordingly, assessment scientists have a responsibility to continuous monitoring, validation, and an openness to new iterations of assessment programs based on these findings.
In conclusion, the framework for ethical decision making proposed in the focal article is a useful guide for industrial-organizational psychologists. This commentary is intended to extend the practical application for assessment scientists by providing some illustrative examples of how this model can be used to elicit behaviors in the assessment development and implementation process that support inclusive and ethical decision making in organizations.
You have
Access
The framework proposed by Banks et al. (Reference Banks, Knapp, Lin, Sanders and Grand2022) in the focal article provides a useful structure for guiding assessment scientists in taking actions to support ethical decision making in organizations. In this commentary we discuss how assessment scientists can broadly apply the framework throughout the assessment life cycle. Specifically, we build upon the idea that lack of ethical knowledge is an important cause of ethical failure (Popp & Popp, Reference Popp and Popp2021) by providing specific ethical considerations that can be employed by practitioners in the context of assessment science within each stage of the proposed framework.
Recently the business case for inclusive hiring is becoming more prominent, introducing the need for increased focus on diversity, equity, and inclusion as an ethical issue for practitioners. Organizations are aspiring to move beyond compliance with professional best practices to create a fully inclusive assessment experience that benefits both organizations in support of their diversity goals and hiring outcomes for a diverse talent pool. In the recent Top 10 Workplace Trends survey (SIOP, 2021), thousands of industrial-organizational psychologists voted Inclusive Practices to Get, Keep, and Grow Talent as the #4 trend organizations will face in 2021, highlighting the opportunity and ethical responsibility for practitioners to leverage inclusive assessments to help organizations achieve these goals.
First stage: Recognize the ethical issue
Practitioners should recognize the ethical responsibility to serve a diverse talent pool. From a cultural inclusion perspective, practitioners should ensure a robust localization process that follows the International Test Commission Guidelines for Translating and Adapting Tests (ITC; 2017). This process involves content decentering to remove any culture-specific material and a rigorous process of translation and back translation to ensure that the content will be equivalent for candidates across cultures. In addition to cultural decentering, one must consider the different demographic groups comprising the diverse talent pool. For instance, individuals with disabilities are an underemployed population, with only 17.9% of individuals with a disability employed in the United States in 2020 (U.S. Bureau of Labor Force Statistics, 2021). With the goal of disability inclusion, practitioners should consider the accessibility of both the assessment platform and the content itself when developing assessment programs and openly share best practices for doing so with other practitioners. In light of the movement for diversity and inclusion, it is more important than ever for practitioners to recognize the ethical responsibility to implement inclusive and equitable assessments to support data-driven decisions in organizations and promote equal opportunity to help organizations achieve their diversity goals.
Second stage: Gather information
As detailed in the proposed framework, once these ethical issues are identified, practitioners should then gather more information. Practitioners should continually seek out information on the latest developments in best practice guidance, especially where this information is not yet established, unclear, or rapidly changing. For example, as new assessment methodologies emerge (e.g., artificial intelligence), practitioners should seek out guidance as it evolves from industry leaders for how they can be ethically used. Where limited best practice guidance is available (e.g., understanding the appropriateness of different assessment types for a neurodiverse talent pool), practitioners should commit to gathering and sharing information that will inform evidence-based best practices for the field.
Third stage: Identify proximal and distal stakeholders
The proximal stakeholders for assessments are the organizations that are making talent decisions. A practitioner’s ethical responsibility to these stakeholders is ensuring that assessments are fit for purpose (determined through job analysis) and accurately predict performance to deliver value and drive business outcomes (determined through validation). The distal stakeholders of inclusive assessments are candidates. In light of the Great Resignation (World Economic Forum, 2021) and current labor shortage, ensuring a positive candidate experience is increasingly important. Providing candidates with a realistic job preview, combined with the opportunity to demonstrate their job-related skills and behaviors can help to ensure transparency and a good match between the candidate’s skills and requirements of the job, as this mismatch may be a driving force behind the current labor shortage. As noted by the authors in the focal article, we must avoid becoming so focused on our ethical responsibilities to the proximal stakeholders that we neglect consideration of “how a professional decision might influence the well-being of distal stakeholders” (Banks et al., Reference Banks, Knapp, Lin, Sanders and Grand2022, p. X). Practitioners must balance the competing demands from both proximal stakeholders (organizations) wanting highly predictive assessments while maintaining a focus on the experience of distal stakeholders (candidates). A positive candidate experience can be achieved by implementing assessments that are short enough so that they are not perceived as cumbersome for the candidate but lengthy enough that they allow the candidate to demonstrate a range of job-related knowledge, skills, abilities, and other characteristics and collect enough information to accurately evaluate potential for success on the job. Additionally, ensuring a positive experience for the distal stakeholders may help to attract talent that is more committed and engaged, ultimately benefiting the proximal stakeholders as well.
Fourth stage: Identify alternative actions
With the goal of measuring as many competencies as necessary to obtain a comprehensive understanding of a candidate’s potential to perform on the job, practitioners should carefully design the assessment program by mapping the content to job-related requirements. This should involve a thorough consideration of available options, including different assessment types and emerging assessment innovations and technologies, with the goal of using multiple assessment types in combination to maximize incremental validity and obtain a full picture of a candidate’s fit for the role. Furthermore, consideration should be given to pairing together assessment components that will balance the needs for predictive validity with achieving diversity goals. For example, when cognitive ability tests are used, they can be paired with noncognitive assessments, which are less likely to exhibit subgroup score differences to mitigate the risk of adverse impact in the overall assessment score.
Fifth stage: Compare alternative actions
As discussed in the previous section, combining multiple assessment types can be a powerful strategy for enabling inclusive outcomes. When evaluating options to optimize assessment design, assessment scientists must first and foremost ensure the integrity of the psychometric properties in accordance with professional best practices. The ethical responsibility of assessment scientists lies in preserving the psychometric properties and legal defensibility of the assessment while making informed decisions that best balance the goals of the organization and the needs of the candidates. Therefore, careful consideration should be given to the weighting of the assessment components. Implementing differential weightings can further reduce potential for adverse impact while preserving predictive validity. For example, assessments that generally exhibit smaller score differences across subgroups (e.g., personality or situational judgment) can be weighted more heavily in an overall score than those that tend to exhibit larger score differences (e.g., cognitive ability). This approach can reduce adverse impact in the overall score, on which decisions are made, potentially without substantially decreasing predictive validity. Alternative weighting schemes should be explored in order to balance predictive validity and fairness with the goal of achieving an inclusive assessment experience for a diverse talent pool. This stage is more of an art than a science, and there is likely no one “right” approach. Decisions made at this stage should be documented along with the rationale to support the defensibility of the assessment and inform ongoing monitoring and evaluation.
Sixth stage: Implement action(s) and monitor outcomes
Once the assessment program is designed, practitioners have an ethical responsibility to continually monitor and evaluate associated outcomes. This includes collecting ongoing validation evidence to document how an assessment predicts performance across the contexts in which it is used and monitoring how the validity may deteriorate over time. For assessments that have been translated, this involves statistical confirmation that the localized assessment is still functioning in a similar way to inform decisions across cultures, as detailed by the ITC (2017) guidelines. When enough data on localized versions of assessment content becomes available, measurement equivalence should be established through confirmatory factor analysis procedures. An analysis of differential item functioning can determine whether there are problematic variations in the measurement of items across cultures. Additionally, local validation studies can be conducted across different language versions of the assessment to bolster the evidence of the relationship between the predictor and criterion across cultures. Practitioners should also continually monitor adverse impact across multiple levels (i.e., candidates who are hired vs. not hired as well as the distribution of demographics within various score bands) to detect systematic differences in how the assessment functions across subgroups. As noted by Banks et al. (Reference Banks, Knapp, Lin, Sanders and Grand2022), ethical decision making requires one to “revisit and reconsider the conclusions generated in particular phases of the framework as a situation unfolds, additional information is uncovered, and/or consequences are realized” (p. X). Accordingly, assessment scientists have a responsibility to continuous monitoring, validation, and an openness to new iterations of assessment programs based on these findings.
In conclusion, the framework for ethical decision making proposed in the focal article is a useful guide for industrial-organizational psychologists. This commentary is intended to extend the practical application for assessment scientists by providing some illustrative examples of how this model can be used to elicit behaviors in the assessment development and implementation process that support inclusive and ethical decision making in organizations.