The World Health Organization (WHO, 2001) has determined that over one billion people live with disabilities (PWDs)Footnote ¹. The United Nations Convention on the Rights of Persons with Disabilities states that PWDs are entitled to all of the rights and freedoms afforded all people. This includes the right to employment and all of the benefits associated with this entitlement (United Nations, 2006). Employment is understood to be a fundamental and necessary component of the well-being of all people, including PWDs (Dutta et al., Reference Dutta, Gervey, Chan, Chou and Ditchman2012). Policy makers and disability activists consider that employment is central to the social inclusion of PWDs (Shah & Priestley, 2011).

Many writers suggest that there are three principal factors that influence the employment rate of PWDs. First described in detail by Stone and Colella (Reference Stone and Colella1996), these broad categories include environmental factors such as specific legislative climates, the policy and behavioural norms found within individual work situations, and the personal characteristics of those with disabilities and those with whom they interact in the workplace. Within these categories, the negative workplace attitudes of employers and co-workers toward PWDs contribute most significantly to these low rates of employment (Antonak & Livneh, Reference Antonak and Livneh2000; Burke et al., Reference Burke, Bezyak, Fraser, Pete, Ditchman and Chan2013; Vornholt, Uitdewilligen, & Nijhuis, Reference Vornholt, Uitdewilligen and Nijhuis2013).

A review of the effectiveness of instruments available to measure these attitudes, particularly whether measurable changes occur subsequent to disability-related education, training, and other interventions, will allow organizations to more reliably determine the value of specific interventions, and, consequently, to most strategically allocate future resources. Instrument effectiveness is determined, in large part, to the degree to which a particular measure demonstrates both theoretical and empirical validity evidence (Hubley & Zumbo, Reference Hubley, Zumbo and Geisinger2013; Messick, Reference Messick1990). As noted by Hubley and Zumbo (Reference Hubley and Zumbo1996), ‘without validity, a test, measure, or observations and any inferences made from it are meaningless’ (p. 207).

As detailed below, each of the nine instruments discovered within this critical systematic review provides evidence of empirical validity; however, none demonstrates evidence of theoretical validity. Evidence of empirical validity includes psychometric data based on inferences generated from test scores (AERA et al., 2014). Evidence of theoretical validity includes ‘theoretical rationales [which] support the adequacy and appropriateness’ of assessments or interpretations of test scores (Messick, Reference Messick1990, p. 5).

In many developed countries, including the United States and Canada, unemployment rates for PWDs remain nearly double those within the general population (Chan et al., Reference Chan, Strausser, Maher, Lee, Jones and Johnson2010; Copeland, Chan, Bezyak, & Fraser, Reference Copeland, Chan, Bezyak and Fraser2010; Jasper & Waldhart, Reference Jasper and Waldhart2013; Schur, Kruse, & Blanck, Reference Schur, Kruse and Blanck2005). Developing countries, such as South Africa, report that employment rates for PWDs are less than 1 per cent (Wiggett-Barnard & Swartz, Reference Wiggett-Barnard and Swartz2012).

As an example, the integration of PWDs into the Canadian workforce continues to be of significant concern, with PWD representation rates having remained virtually unchanged since the enactment of the Canadian Employment Equity Act in 1986 (Rioux & Patton, Reference Rioux, Patton and Agócs2014). According to Statistics Canada, the unemployment rate for PWDs in 2011 was 11 per cent, compared with 6 per cent for the general population (Turcotte, Reference Turcotte2014).

Workplace attitudes toward PWDs are a subset of the more general construct of attitudes, which are described as latent psychological processes that are acquired through experience and which are displayed or evoked by specific circumstances (Antonak & Livneh, Reference Antonak and Livneh2000). Attitudes are commonly understood to contain cognitive, affective, and behavioural components (Findler, Vilchinsky, & Werner, Reference Findler, Vichinsky and Werner2007). Eagley and Chaiken (Reference Eagly and Chaiken1993) offered a succinct definition of an attitude as ‘a psychological tendency that is expressed by evaluating a particular entity with some degree of favour or disfavour’ (p. 1).

A comprehensive understanding of the underlying influences that drive these psychological tendencies contributes to the theoretical validity evidence supporting the interpretations of the several measures considered within this review. Newly named as ontological validity, this form of validity evidence is based on a recognition and consideration of those psychological tendencies which formulate attitudes. However, unlike the several forms of validity evidence which are based on test content or the inferences generated from respondents’ scores on various instruments (Fuhr & Bacharach, Reference Fuhr and Bacharach2014; Kane, Reference Kane2013), ontological validity is based on a consideration of these influences as they are delineated by the instrument developers. Without this delineation, it is impossible to determine whether an instrument provides a satisfactory level of theoretical validity evidence necessary to support an understanding of score meanings. Ontological validity is not alone in being chronologically distant from a specific test application. In that regard, it is similar to validity evidence based on the unintended consequences of tests (AERA et al., 2014). However, evidence of ontological validity is gathered prior to administration, while evidence based on consequences of tests is gathered following a specific test application (Hubley & Zumbo, Reference Hubley and Zumbo2011).

Young (Reference Young1990) noted that those engaged in theoretical considerations of social structures have an obligation to ‘acknowledge the particularity of the voice of their writing’ (p. 14). Young suggested that without a consideration and clear articulation of the worldview which informs such theoretical constructions, there is no basis against which they may be compared and evaluated. Hacking (Reference Hacking1999), in a consideration of various understandings of disability, argued that care must be taken to articulate the basis of these understandings in order to support the intended interpretations. Messick (Reference Messick1995) also acknowledged that an explicit articulation of ‘the theoretical rationale underlying score interpretation’ (p. 743) is a valuable evidentiary contribution to the overall understanding of score meanings.

Systematic reviews are intended to add value to the category of knowledge under consideration. Typically, systematic reviews are used to compare interventions or other processes, either by degree or kind, most often following the commonly used PICO model. That model considers four categorizations of data, including those of Population, Intervention, Comparison, and Outcome within the investigation (Gough, Oliver, & Thomas, Reference Gough, Oliver and Thomas2012).

This study aimed to address the question – ‘What current measures of workplace attitudes toward PWDs provide theoretical and empirical validity evidence sufficient to support their interpretations of test scores?’ If, as indicated above, employer and co-worker attitudes contribute significantly to the workplace success of PWDs, an analysis of the validity evidence associated with available instruments is critical.

Method

Databases searched were Medline, PsycInfo, Google Scholar, Cinahl, and Cochrane Collaboration. Keywords were generated from within the four PICO categories of Population, Intervention, Comparator and Outcome (Boland, Cherry, & Dickson, Reference Boland, Cherry and Dickson2014) and included variants of disability, attitude, procedure/design/employment, and outcome.

This systematic review included 13 scans of five academic databases. Inclusion criteria required that participants were adults, that both disability and competitive employment were specifically mentioned, and that these studies provided quantitative analysis. These scans were limited to measures that were either published between January, 2005 and March, 2015 or were published prior to this period but remain in active use. Initially, 1,192 articles were generated. All but 49 were removed following reviews of titles and abstracts. Full-text reviews eliminated another 40, resulting in nine self-report measures of attitudes toward people with disabilities. Of these, four were developed specifically to measure employer attitudes toward PWDs. The remaining five, although designed to measure attitudes toward PWDs within a variety of settings, either indicated a possible goal of measuring employment-related attitudes toward people with disabilities or have subsequently been applied for this purpose (see Figure 1).

FIGURE 1

Records flow chart. *Additional records identified through the review of titles of references in articles retrieved from databases.

Titles and abstracts of peer reviewed articles were used as the first two screening tools. Remaining articles were then reviewed in their entirety. All screens used a set of inclusion criteria that included – age (adult), date of publication (2005–2015), specific references to disability and competitive employment, and some demonstration of quantitative analysis. Measures developed prior to 2005 were also included if still considered to be in active use. This determination was based on journal references to the application of each of these since 2005, as discovered during the course of the systematic review.

The phrases work disability, Return To Work (RTW), and self-reports of disabled persons were excluded as being beyond the scope of this review. Preliminary searches, in the first two instances, had resulted in material focused on disabilities acquired during the course of employment, with an emphasis on disability management procedures. In the latter case, searches had generated a range of qualitative analyses. Although qualitative research allows for an in-depth analysis of meanings associated with various findings (Creswell, Reference Creswell2013; Mason, Reference Mason2002), the construct of workplace attitude as a principal component of the employment challenges faced by PWDs is already an accepted premise of this review (Stone & Colella, Reference Stone and Colella1996). Therefore, these qualitative analyses were excluded.

Results and Discussion

Table 1 presents the findings by individual study. These include three principal categories of evidence:

• • Evidence of empirical validation – each study was reviewed for content validity, Cronbach's alpha, test/retest reliability, inter-item correlations, factor analyses, and considerations of response bias.

• • Evidence of theoretical validity – interpreted by analysis of individual scale items for etiological basis (medical model, social model, and embodiment model).

• • Weight of evidence framework – an overall quality assessment of each study based on format, content, focus, and psychometric information.

TABLE 1

Models, Theoretical Assumptions, and Psychometrics

The weight of evidence model (Gough, Reference Gough2007) was employed to determine the relative strength of empirical evidence provided by each of the studies (Boland et al., Reference Boland, Cherry and Dickson2014) as related to co-worker attitudes toward PWDs. A relative quality assessment for each of the included studies was developed within a three-dimensional framework. These dimensions include the quality of execution of the study according to generic standards, the appropriateness of the study itself for answering the research question of the systematic review, and the applicability of the study (including its population, sample size, and context) to the review question (Gough et al., Reference Gough, Oliver and Thomas2012). Each of these dimensions, summarized as format, content, and focus, is detailed in Table 2.

TABLE 2

Weight of Evidence Chart

Adapted from the weight of evidence framework (Gough et al., Reference Gough, Oliver and Thomas2012, p. 162).

Three threads of development are apparent. First, the recognition and analyses of attitude as a significant contributing factor moves from a unidimensional to a multidimensional interpretation (for example, Antonak, Reference Antonak1982; Stone & Colella, Reference Stone and Colella1996; Yuker, Block, & Younng, Reference Yuker, Block and Younng1970). Second, data gathering methodologies have developed indirect measurement techniques to address considerations of bias within self-reports of such a socially sensitive topic (Antonak, Reference Antonak1982; Greenwald, McGhee, & Schwartz, Reference Greenwald, McGhee and Schwartz1998). Third, there has been a recognition that the cognitions, affects, and behaviours of those whose attitudes impact the employment of PWDs are not necessarily congruent within individuals (Domzal, Houtenville, & Sharma, Reference Domzal, Houtenville and Sharma2008; Findler et al., Reference Findler, Vichinsky and Werner2007; Ozawa & Yaeda, Reference Ozawa and Yaeda2007; Pruett & Chan, Reference Pruett and Chan2006).

None of these studies delineated a specific theoretical model of disability upon which these instruments were based or from which they drew support. Of the four scales which provided complete lists of items specific to PWDs, each contains a majority of items that represent a medical model of understanding disability and a minority of items informed by either the social or embodiment models, or which were categorized as ‘unknown’. These categorizations followed a review of each item to determine whether responsibility for disability was stated or implied as having derived from flaws or limitations found within the PWD (medical model), as residing outside the individual and within the social environment (social model), or as a combination of the two (embodiment model). Several items were assigned to an ‘unknown’ category if this determination of responsibility was not apparent.

A review of the included measures reveals that a general definition of disability was not provided to participants. In some cases (Findler et al., Reference Findler, Vichinsky and Werner2007; Yuker et al., Reference Yuker, Block and Younng1970), the concept of disability was represented by a single image or a narrow subset of disability, with attitudinal results being extrapolated to the general construct of disability.

Of the nine identified instruments, four selected employers as respondents and sought information specifically about attitudes in employment situations. The remaining five measures had selected students or rehabilitation professionals to respond to more generalized statements about attitude toward PWDs, with the resulting measures deemed to be applicable to a variety of circumstances, including employment situations. However, as noted by Waltz, Strickland and Lenz (Reference Waltz, Strickland and Lenz2005), the referent group selected for generating data suitable for scale validation should accurately reflect the population for whom the scale was designed. None of these nine instruments selected a broad representation of co-workers as a referent group.