Foster et al. (Reference Foster, Steel, Harms, O’Neill and Wood2024) assert that current selection systems have been able to account for most of the variance in job performance ratings and emphasize that variance attributed to the person being rated is only part of the total variance in job performance. However, the authors offer the assessmentFootnote 1 that selection tests are often devalued due to the limited variance selection tests account for in overall job performance ratings. The authors discuss a number of recommendations, including how selection tests should focus on being highly predictive but also underscore additional benefits, specifically considering such sources of performance-related variance as ratee main effects and variance due to the rater. However, solely focusing on ratee main effects versus other sources of variance (i.e., design bias) lacks consideration of how perceptions of individual differences and issues of accessibility may adversely impact applicants in need of accommodations.Footnote 2 Specifically, performance-relevant variance of disabled job applicants who often face the same selection tests and batteries that able-bodied applicants face (Colella et al., Reference Colella, DeNisi and Varma1998) may be muddled by lack of accessibility or rater biases (e.g., horn vs. halo effects). Moreover, these selection tests, although highly predictive (e.g., general mental ability; Schmidt & Hunter, Reference Schmidt and Hunter2004), may not fully capture the performance domain of disabled job applicants. In this commentary, we will (a) discuss legal issues and considerations relevant to employment selection and applicants with disabilities, (b) consider how selection tests can be designed to not only provide predictive validity but also accommodate disabled applicants, and (c) consider potential remedies for rater effects that may negatively influence performance ratings of disabled applicants.
The Americans with Disabilities Act and selection tests
The Americans with Disabilities Act of 1990 (ADA) states that employers are not allowed to discriminate against a job applicant or employee in a way that adversely impacts their employment opportunities based on the status of their disability (Equal Employment Opportunity Commission, 2008). Moreover, the ADA also prevents organizations from using “standards, criteria, or methods of administration, which are not job-related and consistent with business necessity, and have the effect of discriminating on the basis on disability” (Equal Employment Opportunity Commission, 2008). Therefore, within the US context, employers must consider the adverse impact of their selection instruments and systems on disabled applicants and provide reasonable accommodations in the selection process.
Job applicants must clear two hurdles to qualify for protections under the ADA: disability and qualification (Gutman, Reference Gutman, Hanvey and Sady2015). Being disabled can encompass one of three prongs: “a current physical or mental impairment, a record of physical or mental impairment, or being regarded as having a physical or mental impairment that substantially limits a major life activity” (Gutman, Reference Gutman, Hanvey and Sady2015, p. 165). Additionally, applicants must meet two requirements, including “possession of knowledge, skills, and abilities (KSA), needed to perform the essential job functions/duties, and performance of all essential job functions with or without reasonable accommodation” (Gutman, Reference Gutman, Hanvey and Sady2015, p. 165). Unfortunately, selection devices and performance evaluation tools that are limited to suiting able-bodied candidates and employees may undermine both outcomes for those with disabilities.
These protections suggest organizations must consider potential limitations that selection instruments and systems impose on disabled applicants, which may adversely impact their job prospects. We note that legally, ethically, and practically, selection tests can and should assess essential job-related KSAs with or without reasonable accommodations. However, as Foster et al. (Reference Foster, Steel, Harms, O’Neill and Wood2024) also suggest, in the quest to create selection instruments that predict the highest amount of variance, we often overlook other sources of variance, such as method effects and rater biases (Arthur et al., Reference Arthur, Edwards and Barrett2002; Hazer & Bedell, Reference Hazer and Bedell2000; Sackett & Lievens, Reference Sackett and Lievens2008). We argue that personnel selection systems can be designed and implemented to minimize adverse impacts while continuing to enhance the effectiveness of selection tests in predicting job performance. Specifically, we consider both method effects in selection testing and rater effects and encourage organizations to create multiple forms of each and/or include opportunities for accommodations in the design process to enhance and ensure accessibility.
Method effects and test accommodations for disabled applicants
How a test is conducted contributes to method effects in personnel selection and can have a sizeable impact on disabled applicants (Lievens & Sackett, Reference Lievens and Sackett2017). Although not examining the impact on disabled applicants, Lievens and Sackett (Reference Lievens and Sackett2017) use a modular approach to discuss how selection methods can produce similar results but offer the opportunity to enlist different approaches to measuring the underlying KSAs. Similarly, we posit that test accommodations can assist disabled applicants by altering the testing method while maintaining the overall equivalency of the test. For example, test accommodations can include providing larger text or higher brightness settings for low-vision applicants taking a multiple-choice test, providing subtitles and transcripts for video assessments, and providing alternative formats for interviews for applicants who are wheelchair bound (e.g., video vs. in person). Such accommodations may help to reduce error variance that is attributable to the disability but not specific to the job. Take, for example, the multiple-choice test and larger text size; in most circumstances, an employee is able to alter text size without the constraint of static text size. However, without such an accommodation, a low-vision applicant is being tested on vision, not on the psychological constructs purportedly being measured. Although test accommodations for disabled applicants have implications both legally and practically, the literature that examines this issue is limited. We believe this is an area for research that is both necessary and timely, given the need for employment practices that are more inclusive.
Job applicants are often assessed using a selection battery or system, such as a general mental ability test, personality test, and structured interview. As Foster et al. (Reference Foster, Steel, Harms, O’Neill and Wood2024) suggest, this may improve the predictive ability of the selection process. Although we also advocate for the use of multiple selection predictors (e.g., general mental ability, conscientiousness, emotional intelligence), it is important for both scientists and practitioners to consider multiple predictor methods. Following a modular approach (Lievens & Sackett, Reference Lievens and Sackett2017), job analysts should consider multiple method choices within each method factor that may be utilized not only for individuals with disabilities but provide additional assessment strategies in the recruitment of disadvantaged employees. For example, although a personality assessment might be tested using a questionnaire (i.e., Minnesota Multiphasic Personality Inventory), it may be beneficial to provide this assessment in both written and auditory formats, validating their equivalency.
Tester accommodations can also assist disabled applicants by supplementing deficiencies that limit an individual’s ability to carry out tasks normally but do not impede job requirements. Technology and more specifically artificial intelligence technology has advanced dramatically in the last few decades (Campion & Campion, Reference Campion and Campion2023), which has provided disabled individuals with opportunities that were previously not available. For example, text-to-speech technology now allows disabled individuals to provide voice responses to selection instruments such as interviews and recorded response tests.
Mitigating rater effects in disabled applicant selection testing
Selection assessments are often evaluated by raters who hold biases and may contribute to errors in assessing disabled applicants fairly (Colella et al., Reference Colella, DeNisi and Varma1998). Although a number of studies have shown that rater assessments of disabled applicants can negatively influence selection decisions (Leasher et al., Reference Leasher, Miller and Gooden2009; Moss, Reference Moss2021; Premeaux, Reference Premeaux2001), how to mitigate these biases has received less attention in the literature. A commonly suggested method to mitigate rater bias is through structured processes and training in selection procedures (Levashina et al., Reference Levashina, Hartwell, Morgeson and Campion2014; Van Iddekinge et al., Reference Van Iddekinge, Raymark and Roth2005). Structuring the process by specifying the job-related KSAs under assessment and including multiple raters to (a) understand how rater biases produce negative ratee × rater interactions and (b) reduce the likelihood that a rater holding negative biases exerts excessive influence on the outcomes of the process may assist in mitigating variance attributable to instrument- and process-related biases. Moreover, bringing awareness of and preemptively addressing commonly held biases toward disabled applicants may also mitigate biased selection decisions.
Another potential method to improve the selection process for disabled employees while also reducing error variance is providing applicants with information and resources on the interview process. For example, in a study by Hebl and Skorinko (Reference Hebl and Skorinko2005), the authors showed that interviewees who were able to manage their disabled identity were evaluated more positively by raters than when they did not disclose. Lyons et al. (Reference Lyons, Martinez, Ruggs, Hebl, Ryan, O’Brien and Roebuck2018) found similar results and found that applicants who disclosed their physical disability and accentuated their disability in a positive light were evaluated more positively than those who engaged in other strategies, such as nondisclosure. Thus, providing information on how to mitigate bias to disabled applicants may help reduce error variance attributable to rater effects. Although such resources may help disabled applicants manage their identity during the selection process, this may not be necessarily a route an organization may take as the ADA prohibits asking applicants to disclose their disability (Gutman, Reference Gutman, Hanvey and Sady2015). In light of this, we believe that the findings of I-O psychologists can still have a positive impact on disabled job seekers and an obligation to partner with and share their findings with policy, community, and advocate groups that are tasked with helping disabled individuals find employment.
Conclusion
In this commentary, we discussed why scientists and practitioners should consider disabled applicants in the design and implementation of selection instruments and systems, complementing Foster et al. (Reference Foster, Steel, Harms, O’Neill and Wood2024) broader call for assessing and reflecting on current selection practices and evaluating sources of variance. As discussed, assessment instruments and processes may be susceptible to discriminating against disabled applicants; however, there are potential ways to design and implement assessment processes that may mitigate the adverse impact and provide better prediction in performance by mitigating variance associated with method effects that are not job-related. We agree with Foster et al.’s (Reference Foster, Steel, Harms, O’Neill and Wood2024) notion that rater effects minimize the impact of selection instruments, and we further suggest that I-O psychologists should consider both employer and applicant remedies that may help to reduce rater effects that negatively impact the performance ratings of disabled applicants. Taken together, we believe that I-O psychologists have the opportunity to make selection tests work better for disabled job applicants because they haven’t.
