Examining personality testing in selection for neurodiverse individuals

In their focal article, LeFevre-Levy, Melson-Silimon, Harmata, Hulett, and Carter (Reference LeFevre-Levy, Melson-Silimon, Harmata, Hulett and Carter2023) bring attention to neurodiversity as a topic that has been seldomly addressed within the organizational sciences. The authors describe how neurodiversity can have positive impacts on organizations (e.g., those with autism spectrum disorder can excel in highly technical fields; Baron-Cohen et al., Reference Baron-Cohen, Bolton, Wheelwright, Scahill, Short, Mead and Smith1998), as well as the obvious benefits of employment for neurodiverse applicants themselves (e.g., unemployment rates for neurodivergent individuals are high). However, neurodivergent people must first be hired for either party to reap these benefits, and this may be challenging given current selection systems used by organizations (Austin & Pisano, Reference Austin and Pisano2017; Bruyère & Colella, Reference Bruyère and Colella2022; Taylor & Seltzer, Reference Taylor and Seltzer2011). LeFevre-Levy and colleagues state that “common selection practices for new hires may not work adequately for neurodiverse applicants” (pp. 23), and that there is a “need to examine the evaluation and psychometric properties of selection systems in specific groups to ensure that selection measures are equally valid across neurotypical and neuroatypical populations” (pp. 24). We agree and when evaluating these statements are reminded of recent public discourse surrounding the practice of personality testing and possible negative impacts on neurodiverse populations.

Despite robust evidence that personality tests are capable of identifying employees that are well suited for a job (Wilmot & Ones, Reference Wilmot and Ones2021),Footnote ¹ there are those who hold negative perceptions of personality testing and particularly as they apply to hiring neurodivergent individuals (Brown, Reference Brown2021; Claypool, Reference Claypool2021). A quick online search of “neurodivergence in hiring” reveals a surfeit of books, webinars, blog posts, Reddit discussions, and even documentaries that all share anecdotal stories of how personality tests unfairly screen out neurodivergent individuals. The general takeaway is that personality tests are prejudiced, algorithmic tools that coldly determine who to hire and that result in the exclusion of certain groups from the workforce (e.g., those with mental illness or disability). Although we do not support this view, we do believe that research needs to be conducted to better understand how personality testing impacts groups such as neurodivergent adults. Thus, we view the focal article as a timely and important call-to-action. In this commentary, we support and extend LeFevre-Levy and colleagues’ argument for the need to investigate the psychometric properties and implications of using selection systems when hiring neurotypical populations, particularly in respect to personality testing.

How do neurodiverse applicants react to personality tests?

Although preemployment personality assessments are generally designed to assess normal range personality and not clinical traits, neurodiverse individuals may still respond differently than neurotypical individuals. Meta-analytic evidence (Lodi-Smith et al., Reference Lodi-Smith, Rodgers, Cunningham, Lopata and Thomeer2019) suggests that individuals with autism spectrum disorder (ASD) receive lower personality scores. For example, scores to extraversion items (e.g., “I am quick to warm up to others”) are often lower than what is seen with neurotypical populations (Schriber et al., Reference Schriber, Robins and Solomon2014; Wakabayashi et al., Reference Wakabayashi, Baron-Cohen, Wheelwright, Goldenfeld, Delaney, Fine, Smith and Weil2006). Despite this, there is little empirical research that addresses neurodiverse applicant reactions to personality tests. More generally, there is also little research as to whether unseen characteristics (i.e., deep-level diversity) impact applicant reactions to personality tests.

More interesting is to consider what factors might affect applicant reactions to personality tests specifically for neurodiverse individuals. One factor is the contextualization of the assessment. Research suggests work-contextualized assessments result in more positive perceptions of fairness and face validity across different groups (Fisher et al., Reference Fisher, Cunningham, Kerr and Allscheid2017; Heggestad & Gordon, Reference Heggestad and Gordon2008). Contextualizing personality items (e.g., “At work, I am quick to warm up to others”) helps individuals establish a similar frame of reference. This may be particularly important for influencing fairness perceptions for neurodivergent individuals if self-reported personality is expressed differently across situations (i.e., work, home, school; Bellini & Peter, Reference Bellini and Peters2008; Grob et al., Reference Grob, Lerman, Langlinais and Villante2019; Holtz et al., Reference Holtz, Ployhart and Dominguez2005; Webb et al., Reference Webb, Miller, Pierce, Strawser and Jones2004; Weiss & Harris, Reference Weiss and Harris2001).

Applicant reactions are also likely to vary across personality traits. For example, applicant reactions to traits such as conscientiousness may not be as negatively perceived because they are not as emotionally loaded and are more ostensibly job related. Specifically for individuals with ASD, emotionally or socially oriented personality items (e.g., extraversion) may prime social stigmas pertaining to ASD, such as stereotype threat (Ployhart et al., Reference Ployhart, Ziegert and McFarland2003) and impact perceptions of the assessment.

Do personality tests exhibit adverse impact and bias in neurodivergent populations?

The aforementioned concerns likely have a connection to a more immediate research need as it pertains to personality testing of neurodivergent individuals: investigating adverse impact (AI) and test bias within preemployment settings. LeFevre-Levy and colleagues suggest that “research should investigate how construct validity varies between neurodiverse groups to ensure the selection tool is measuring job-related KSAOs without construct contamination” (pp. 24). In other words, LeFevre-Levy are calling for research on statistical test bias.

A good review of test bias can be found in Meade and Tonidandel (Reference Meade and Tonidandel2010). Statistical test bias can be broken into two approaches of evaluation. The first is measurement bias, which investigates internal properties of the test and occurs if the relationship between the latent trait and item responses differs for members of different demographic groups. We know of little research that has investigated measurement bias for neurodivergent individuals. However, Lodi-Smith et al. (Reference Lodi-Smith, Rodgers, Cunningham, Lopata and Thomeer2019) suggest that differences in personality scores across neurotypical and neurodivergent individuals may be evidence of differential scale functioning instead of true mean-level differences and suggest that test bias is a crucial area of future research.

The second test bias strategy is to examine predictive bias, which occurs if test scores are differentially related to an important criterion. This is most frequently tested using the Cleary method (Reference Cleary1968), which in essence tests whether the group-specific regression lines are the same. For example, if personality scores operate similarly for neurotypical and neurodivergent respondents, we’d expect the relationship between trait scores and important outcomes (e.g., job performance) to be similar. However, we simply do not have knowledge whether personality tests produce predictive bias against neurodivergent respondents. Given the vitriol against personality tests (Brown, Reference Brown2021; Claypool Reference Claypool2021), such research is necessary before boldly claiming personality tests are biased against this population.

Last, as a result of test bias, or even if test bias does not exist, personality tests may still produce AI against neurodivergent respondents. AI occurs when the pass rates differ between different demographic groups and is driven (in part) by whether there are mean differences in test scores between those groups (Murphy & Jacobs, Reference Murphy and Jacobs2012). AI will be a concern to neurodivergent individuals, given it reduces their chances of being hired. That said, if there is not test bias, the observed differences in scores may still be valid for neurotypical and neurodivergent respondents, meaning that AI may not be problematic from a psychometric perspective. Once again, we do not know of research that has investigated AI across these groups within testing contexts, though existing meta-analytical evidence suggests that it is likely to exist (Lodi-Smith et al., Reference Lodi-Smith, Rodgers, Cunningham, Lopata and Thomeer2019). This is important to understand, given that both AI or test bias poses threats to the legality of using personality assessment in this context and prevents organizations from reaching a truly inclusive workplace.

The question thus becomes, what might we expect regarding AI and test bias against neurodivergent individuals? Neurological differences, including those that result from ASD, create diversity in cognitive abilities and behavioral patterns among working adults (American Psychiatric Association, 2013; Chen et al., Reference Chen, Leader, Sung and Leahy2015a,b). Given these differences in behavioral functioning, many hiring methods might be challenging for neurodivergent individuals (e.g., interview, cognitive ability test, personality test). Specific to personality tests, neurodivergent individuals may not tailor their responses to personality items to portray themselves as ideal candidates the same way that neurotypical respondents might. Atypical individuals may respond too true to self, specifically for social components of personality tests (e.g., extraversion, agreeableness, emotional stability; Baez, Reference Baez2013; Bruyère, & Colella, Reference Bruyère and Colella2022; Patton, Reference Patton2019). If neurodiverse individuals receive lower scores on personality tests, they may be disproportionately screened out even though neurodiverse respondents may possess other skills that might allow them to excel at the job. Despite these concerns, there has been little research investigating psychometric properties across neurotypical and neurodivergent groups. In agreement with the perspective espoused by Lefevre-Levy and colleagues, we believe research on this topic is vital. Based on this, we propose the following research agendas: