%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Los Angeles County's civil service rule specifies a 70% cutoff score regardless of the situation (type of job, type of assessment, or outcome of interest). This civil service rule would be difficult to defend if it were challenged in court, and the rule places the county at increased risk in the event of employment litigation, particularly with public safety jobs (police, fire, sheriff, etc.). Additionally, it is unlikely that this cutoff would optimally balance the county's interests in fair employment practices and expected job performance (SIOP, 2003). Given the propensity for public safety candidates and employees to file lawsuits related to hiring and promotions, and given the fact that public safety agencies in this county have been subjected to a number of employment-related lawsuits in the past, the current rule is problematic. Prior to this rule's development, as well as after its implementation in 1988, there has been substantial litigation related to cutoff scores, and courts have identified what they consider to be more and less appropriate methods for setting cutoff scores.Footnote 1
Using an earlier and shorter version of this article, the author formally contacted the county's centralized human resources (HR) department in an attempt to have the cutoff score rule revised or to otherwise explore how other county agencies reporting to the centralized HR oversight group might negotiate some flexibility in setting cutoff scores. When requesting rationale regarding why cutoff scores must be 70%, little was provided other than the 70% cutoff is a civil service rule.
Such a cutoff score policy places industrial and organizational (I-O) practitioners in an ethical dilemma. As discussed in this article, it is not possible to satisfy legal requirements and/or professional guidance in establishing reasonable support for cutoff scores while simultaneously complying with this mandated civil service cutoff score rule. The I-O practitioner has to make a decision in situations of this nature. At one extreme, the practitioner might simply fall in line, stop asking questions, and implement a 70% cutoff score for all assessments. This option minimizes friction with centralized HR but also increases legal liability for this agency in the event that exams are subsequently challenged. At the other extreme, the practitioner might attempt to skirt the civil service rule by not providing cutoff score information on exams that are going to be advertised (i.e., exam bulletins that will be posted and must be reviewed and approved by the centralized HR function). If the practitioner avoided providing cutoff score information at this point in the exam process, he or she could then collect data during test administration to guide setting of a more optimal cutoff score. This option would maximize friction with centralized HR and delay or prevent advertisements while minimizing legal risk to the local agency (because any cutoff scores used would have independent support). There are many other examples one can imagine along this hypothetical continuum.
This case study reviews county and state cutoff score rules, discusses Uniform Guidelines (EEOC, 1978) and professional guidance on setting cutoff scores (Standards for Educational and Psychological Testing, AERA, APA, & NCME 1999; Principles for the Validation and Use of Personnel Selection Procedure, SIOP, 2003), and briefly touches on four legal cases related to cutoff scores. It closes with some examples of how this topic has been handled for two major categories (sworn and civilian) of employees in this agency, focusing primarily on risk management and risk tolerance as factors to consider when deciding whether to seek exceptions to civil service rules.
Passing Grades
County Civil Service Rule
This county's civil service rule provides limited flexibility in setting cutoff scores. One subsection of the rule requires a cutoff score of 70% unless otherwise provided for in the bulletin advertising a job, whereas another subsection discusses cutoff scores for examinations comprising two or more parts. Although the latter subsection does not specify a minimum cutoff score, it requires that any cutoff score(s) be announced at the time the examination is posted. In practice, the centralized HR agency interprets this rule to mean that, if there are multiple components in a selection process, all components must have a cutoff of 70%. Additionally, the civil service rule as implemented and interpreted in practice requires that any cutoff score be developed without collecting data from examinees (i.e., when the exam is announced and prior to implementation). In contrast, the most recent version of cutoff score rules published in 2014 by the state of California (State of California Government Code Section 18937; discussed below) provides greater flexibility than does the county's current rule.
State of California
The state of California (http://law.justia.com/codes/california/2016/code-gov/title-2/division-5/part-2/chapter-4/article-2/section-18937) has taken steps to provide greater flexibility in establishing cutoff scores and making adjustments to cutoff scores where necessary. State of California Government Code Section 18937 is quoted in part below.
18937. The passing mark for an examination may be other than the true percentage or average published as a part of the announcement of the examination, if deemed by the department or a designated appointing authority to be justified in order to provide an adequate list of eligibles or to adjust for the apparent difficulty of an examination. . . . (emphasis added; amended by Stats. 2013, Ch. 427, Sec. 37. [AP 1062] effective January 1, 2014)
As Section 18937 demonstrates, the State of California's policy allows state agency HR decision makers to adjust cutoff scores after they have been announced if justification is provided. Neither does it mandate a specific, set percentage correct cutoff score. If such a policy were to be adopted by this county, it would allow HR practitioners to revise a published cutoff score if sufficient justification is provided. Although the state code still appears to require that cutoff scores be published at the time the exam is announced, and hence before examinee data are collected, it allows state agencies to adjust published cutoff scores based on an agency's need to obtain sufficient candidates or in the case of an exam being overly difficult. Although not perfect, this rule provides greater flexibility for I-O practitioners than the current county rule.
Why a Mandated 70% Cutoff May Be Problematic
There are a number of reasons why the blanket use of a 70% (or any other fixed percentage) cutoff score is difficult to defend. The most important reason is that cutoff scores generally cannot be established without considering a number of factors, including the actual distribution of examinee test scores, the requirements for successful job performance, and the professional judgment required to balance these and many other competing interests and outcomes (Cascio, Alexander, & Barrett, Reference Cascio, Alexander and Barrett1988; EEOC, 1978; Kehoe & Olson, Reference Kehoe, Olson and Landy2005; SIOP, 2003).
Because this county's rule mandates that cutoff scores must be specified when the examination is announced, and before examinee data are collected, it is not possible to rely on data when deciding where to set a cutoff score. Likewise, if subject matter experts (SMEs) are involved in setting a cutoff score, as in the judgment-based Angoff method (Angoff, Reference Angoff and Thorndike1971), the bulletin announcing an examination is likely to be published well before SMEs have reviewed any related test materials. Examination announcements frequently occur before finalizing the test or other assessment. If an agency is using a test or assessment for the first time or if that assessment has undergone substantial revision since its last use, test developers may have no idea how a test will function (i.e., its relative difficulty and score distribution) and may have no idea where a reasonable and defensible cutoff score should be set.
From a psychometric perspective, I-O practitioners should expect cutoff scores to vary depending on the type of test being used (Cascio et al., Reference Cascio, Alexander and Barrett1988). For example, 70% may be a reasonable cutoff score for a test of job-related knowledge because such tests are often interpreted on a criterion-referenced basis. Conversely, on a cognitive ability test or personality inventory, such as is often used in entry level hiring, it is inappropriate to use a percent-correct cutoff score. Such measures are interpreted on a normative rather than on a criterion-referenced basis (Cascio et al., Reference Cascio, Alexander and Barrett1988; Kline, Reference Kline2006; Nunnally & Bernstein, Reference Nunnally and Bernstein1994), and cutoff scores for such assessments are typically set based on percentile equivalents rather than on a percent-correct basis.
Guidelines and Regulations Regarding Cutoff Scores
Uniform Guidelines
The Uniform Guidelines on Employee Selection Procedures (EEOC, 1978), Part 1607, Section 5.H, state “Where cutoff scores are used, they should be normally set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the workforce.” As Kehoe and Olson (Reference Kehoe, Olson and Landy2005) observed: “While this guideline is open to considerable interpretation, it clearly establishes the expectation that the defense of challenged cut scores will be grounded in the relationship of the cut score to work proficiency” (p. 411). Using a 70% cutoff for all tests, regardless of the type of test or the type of job involved, does not meet the Uniform Guidelines requirement of “consistent with normal expectations of acceptable proficiency within the workforce.” Standards of acceptable work proficiency are likely to differ across jobs, types of tests or assessments, and outcomes of interest.
Professional Guidance on Setting Cutoff Scores
In contrast, whereas the Uniform Guidelines (EEOC, 1978) require cutoff scores to bear some relationship to acceptable work proficiency, professional organizations recognize and discuss a number of additional factors or potential outcomes to consider when setting cutoff scores. The AERA Standards (1999) provide technical guidance regarding test development and use in employment as well as educational settings, whereas the SIOP Principles (2003) are more narrowly focused on selection procedures used for employment decision-making. Kehoe and Olson (Reference Kehoe, Olson and Landy2005) and Cascio et al. (Reference Cascio, Alexander and Barrett1988) argue that the Standards and the Principles should be the primary sources of professional guidance for those setting and using cutoff scores.
The Principles (SIOP, 2003) share the Standards’ (AERA et al., 1999) philosophy that cutoff scores may be influenced by multiple factors. According to the Principles (2003):
Professional judgment is necessary in setting any cutoff score, and typically is based on a rationale that may include such factors as estimated cost-benefit ratio, number of vacancies and selection ratio, expectancy of success versus failure, performance and diversity goals of the organization, or judgment as to the knowledge, skill, ability, or other characteristics required by the work. (p. 47)
In addition to expected proficiency, cost–benefit ratios, number of vacancies, expectancy of success versus failure, and so on, those setting cutoff scores might consider factors such as expected level of adverse impact against protected group members, cost of competing selection processes, and the utility of the selection process (Hoffman & Thornton, Reference Hoffman and Thornton1997).
Professional standards and the Uniform Guidelines (EEOC, 1978) support the notion that the process of setting cutoff scores requires professional judgment and require that practitioners provide some rationale that supports the choice of cutoff. It is doubtful that sole reliance on a civil service rule specifying a 70% (or any other fixed percentage) cutoff score is sufficient justification (Guardians Association, 1980; U.S. and Vulcan Society, 2009) to support the choice of a cutoff score. Because cutoff scores cannot be set without data and without the application of professional judgment (Cascio et al., Reference Cascio, Alexander and Barrett1988; SIOP, 2003), an employer runs the risk of losing in court if a cutoff set in such a vacuum faces legal challenge.
Federal Court Cases
A number of court cases have dealt with the issue of setting cutoff scores. In consideration of space limitations, this review touches on just four cases: Griggs v. Duke Power (1971), National Education Association v. South Carolina (1978), Guardians Association of New York City Police Department Inc. v. Civil Service (1980), and U.S. and Vulcan Society v. New York (2009). The Guardians and Vulcan cases are particularly relevant to this discussion because both deal with hiring under civil service rules, both deal with tests developed under a content-oriented validation strategy, both deal with public safety jobs (police and fire, respectively), and both discuss use of a mandated civil service cut score of 70%. Public safety hiring is likely to be a significant area of interest and potential litigation in public sector agencies.
Griggs (1971)
Griggs was the first Supreme Court case dealing with Title VII of the Civil Rights Act of 1964. Duke Power Company had implemented minimum test and education requirements for a number of different jobs, and plaintiffs alleged these requirements discriminated against African Americans. The court reviewed the test and education requirements and judged them to be discriminatory and unrelated to job performance. Furthermore, the court stated that the Civil Rights Act of 1964 requires “the removal of artificial, arbitrary, and unnecessary barriers to employment.” The court also referred to business necessity in setting employment standards, stating that “if an employment practice . . . cannot be shown to be related to job performance, the practice is prohibited.”
It is common for employment tests to result in adverse impact against protected group members, making it necessary for organizations to demonstrate that their employment processes are valid and job related. The choice of cutoff score often plays a significant role in determining the level of adverse impact resulting from a given assessment process (Hoffman & Thornton, Reference Hoffman and Thornton1997; Ployhart & Holtz, Reference Ployhart and Holtz2008). Use of an unvarying cutoff score of 70% regardless of the type of job, test, or the test's placement within a larger selection process could be viewed as “artificial” or “arbitrary” in the event of a legal challenge and as if the practitioner had no basis for specifying that cutoff score other than that its being mandated by civil service rules.
National Education Association v. South Carolina (1978)
As discussed earlier, professional practice standards describe a variety of tradeoffs that practitioners may consider during the process of setting cutoff scores. In the National Education Association teacher certification case, the Supreme Court upheld a cutoff score setting process that considered the expected test performance of minimally competent teachers, rating agreement among experts who were involved in setting the cutoff score, test reliability, labor market issues, and workforce diversity. The use of a preset or standard cutoff, such as the 70% required by this county's civil service rule, would probably not have been defensible in the National Education Association case.
Guardians Association of New York City Police Department Inc. v. Civil Service (1980)
The Guardians case dealt with the hiring of entry level police officer candidates for the City of New York, with plaintiffs alleging that the test used in making selection decisions was discriminatory against Black and Hispanic candidates. The court criticized the city for its job analysis process, test development process, unwillingness to rely on experts with a specialization in testing methodology, and methods for setting cutoff scores. In one administration of the test, the City set a cutoff score that would provide a sufficient number of candidates to fulfill its expected business needs. In a second administration of the test, the city set the cutoff score at the city of New York civil service mandated cutoff score of 70%. The court rejected both cutoff scores because the City did not provide sufficient supporting rationale.
In its ruling against the city of New York and its Civil Service Commission, the court stated:
No matter how valid the exam, it is the cutoff score that ultimately determines whether a person passes or fails. A cutoff score unrelated to job performance may well lead to the rejection of applicants who were fully capable of performing the job. When a cutoff score unrelated to job performance produces disparate racial results, Title VII is violated. . . . Consequently, there should generally be some independent basis for choosing the cutoff. [emphasis added]
Vulcan Society v. New York (2009)
This case dealt with the hiring process used by the city of New York in selecting entry-level firefighters. The Vulcan court was highly critical of the procedures used by the city of New York in conducting the job analysis, developing the test, and setting cutoff scores. The ruling stated:
The evidence presented by the City indicates that it relied on inappropriate grounds in selecting the cutoff scores for Exams 7029 and 2043. For Exam 7029, the cutoff score was based merely on the number of entry-level firefighter job openings expected by the FDNY. . . . In making this selection, the City ignored Guardians’ warning that the City should not “simply choose as many candidates as it needed, and then set the cutoff score so that the remaining candidates would fail.” 630 F.2d at 105
For Exam 2043, the city used the standard civil service cutoff score of 70%. The court was even more pointed in its critique of the cutoff procedure used by the City for Exam 2043, stating: This choice of cutoff score ignored Guardians warning against relying solely on a civil service default score” (emphasis added). The Guardians court clarified the role of Title VII in such instances, stating: “Title VII explicitly relieves employers from any duty to observe a state hiring provision which purports to require or permit any discriminatory employment practice. . . . More importantly, the City has presented no evidence that its chosen cutoff scores bear any relationship to the necessary qualifications for the job of entry-level firefighter” (emphasis added).
Civil Service Adaptation to 70% Cutoff Score: Transformed Cutoff Scores
Some agencies that mandate a 70% cutoff score develop and implement “work-around” methods for using cutoff scores other than 70% while attempting to make those cutoffs appear to comply with civil service rules. Public sector agencies may use a mathematical transformation to achieve this goal. A cutoff score of, say, 50% is redefined to a numerical value of 70 using a transformation. Although such a transformed score is then called “70” and is often (and incorrectly) referred to as 70%, it does not reflect 70% because there is only one way to calculate a percentage score. In our view, such a transformed score of 70 is misleading and does not actually comply with the 70% civil service requirement.
Table 1 provides an example showing how scores on a hypothetical interview could be rescaled (this table is based on training material). The hypothetical interview consisted of four dimensions with each dimension rated on a 7-point scale, so the maximum possible score was 28 points and the minimum possible score was 4 points. A raw score of 12 (43%) was then defined as a passing score of 70, and all raw scores were converted into scaled scores. The scaling requires a mathematical adjustment whereby the range of scaled score points (30 points; 100 – 70) is divided by the number of actual scale points (16 points; 28 raw score points – 12 raw score points). This fraction (30/16) is added to each scaled score above the new 70, while this fraction is subtracted from each scaled score below the new 70. As the table shows, the raw interview score associated with a mathematically based 70% percent score is between 20 and 21 points, whereas the scaled scores for these interview scores are 85.0 and 86.9, respectively. A perfect interview score of 28 points (mathematically based 100%) is equivalent to a scaled score of 100, an interview score of 12 points (mathematically based 43%) is equivalent to a scaled score of 70, and the minimum possible interview score of 4 points (mathematically based 14%) is equivalent to a scaled score of 41. Although it is true that the rank order of the mathematically based percent scores and scaled scores does not differ (r = 1.00), the scaled scores bear little resemblance to the percent scores. If any point on a score distribution can be redefined to a value of 70, then the entire process of setting the cutoff score has little connection to reality.
Example of Scaling Raw Score (Other Than 70%) to Value of 70
Note: Assumes interview with four dimensions being rated, 7-point scale, raw score of 12 scaled to 70.
Potential Problems With Transformed Scores
Valle and Hoffman (Reference Valle and Hoffman2013) described several problems associated with a different process for creating transformed scores in an item banking and test scoring program. Similar to the transformation highlighted in Table 1, the scoring function was created to allow agencies to treat any observed test score as a 70 and thus ostensibly comply with a civil service rule requiring a cutoff of 70%. Should candidates attempt to calculate their percentage correct score after a cutoff score has been transformed, they will find that the percentage correct score does not match the output of the mathematical transformation. For candidates scoring above the new, transformed cutoff score, their transformed score will be higher than their actual percent correct score. For candidates scoring below the new, transformed cutoff score, their transformed score will be lower than their actual percent correct score. For candidates scoring very low on an exam, negative transformed scores have been observed. It is difficult to explain negative test scores to candidates, and such a score transformation process is unlikely to be viewed as transparent by candidates or by applicants. It is likely that such a score transformation process would raise similar questions if it were the subject of legal challenge.
As discussed above, such transformed scores do not reflect reality. Calling a number “70” and implying it is 70% is misleading. Using a 70% cutoff without further justification cannot be supported (Guardians, 1980; EEOC, 1978; Vulcan Society, 2009), and multiple court cases have ruled that default cutoff scores cannot be defended if they lead to adverse impact against protected groups and there is no other evidence supporting the choice of cutoff score (Griggs, 1971; Guardians, 1980; Vulcan Society, 2009). A major concern with such score transformations is their lack of transparency.
Practice Recommendations for Addressing Civil Service Rules
Kehoe, Brown, and Hoffman (Reference Kehoe, Brown, Hoffman and Schmitt2012) observed that selection professionals working in public sector settings may need to find work-arounds to civil service rules if those rules place an organization at increased risk of litigation. In addition to cutoff scores, public sector rules might require top-down ranking, use of specific types of exams, specific weighting of predictors, specific ranges of scores for banding purposes, and so on. When selection procedures result in adverse impact against members of protected groups and rank ordering is used, courts may hold those procedures to higher standards of reliability and validity (Guardians, 1980; Vulcan Society, 2009). Some civil service rules may require the researcher to ignore job analysis findings, situational constraints, published literature, and professional/legal concerns about the quality and relevance of various predictors or methods of combining predictors. For example, some agencies require the use of training and experience evaluations (T&E) despite research demonstrating that many types of T&Es perform rather poorly as measurement tools (Gatewood, Feild, & Barrick, Reference Gatewood, Feild and Barrick2011). Kehoe et al. (Reference Kehoe, Brown, Hoffman and Schmitt2012) made the following point regarding complying with civil service rules:
Our advice to practitioners in public sector settings is to follow legal and professional guidelines first and then pay attention to county or city codes. For example, do not use a 70% rule as the basis for setting passing scores without collecting data supporting the choice of cutoff score. Practitioners should attempt to pay attention to legal and professional requirements rather than blindly following civil service procedures that were written without regard to legal requirements or professionally accepted practices. (p. 906)
The ramifications of ignoring or skirting civil service rules depend on the particular setting, and such leeway may not be possible. In situations where civil service provisions run counter to legal requirements or sound professional practice, further discussions are needed to determine when, how, and if it may be possible to deviate from civil service rules. Exactly who the selection professional should consult depends on the particular organization, its structure, and whether there is a past history of legal challenge. Parties with whom the selection professional might consult are quite varied and could include a centralized human resources group if there is one, internal or external legal counsel, and other practicing selection professionals from inside or external to the organization.
The prospect for relief from civil service provisions may improve where the selection professional can argue that specific practices required by civil service code (e.g., blanket use of 70% cut score, use of specific and narrow types of tests, rules for the weighting of specific exam components, overly specific rules on what type of job analysis is acceptable) may not be legally defensible. If the agency has been subject to legal challenge in the past and questionable civil service rules contributed to that challenge, relief from such rules may be easier to achieve. Our discussion now turns to examples from this agency that demonstrate where case-specific exceptions to civil service rules have been granted.
Sworn Versus Civilian HR Processes
In this public sector/public safety agency, I-O practitioners provide technical support to generalists who handle day-to-day HR operations such as hiring and promotions. There are a number of important differences between sworn (police) and civilian (or nonsworn) HR processes in this agency. There are far more civilian job classifications (over 350), with most classifications having relatively few incumbents. There are fewer sworn classifications, 60 or so, and most sworn promotions involve just three classifications: ranks of sergeant, lieutenant, and captain. Table 2 contrasts civilian and sworn jobs on a number of key comparisons. As the table shows, sworn HR programs are at greater risk of being challenged, including union complaints, formal grievances, civil service appeals, and employment litigation. Consequently, in this agency, relatively more internal resources are focused on sworn jobs, whereas relatively fewer resources are focused on civilian jobs.
Sworn Versus Civilian HR Processes in This Agency
Exams for civilian (nonsworn) classifications, whether entry level or promotional, typically have small applicant pools. The stakes are lower because civilian candidates, whether internal (promotional) or external (typically entry-level), are far less likely to grieve or appeal HR decisions or to file lawsuits. To date, there is no history of employment-related lawsuits in the civilian arena. Because of this lower risk of litigation, this agency has not sought many exceptions from civil service rules for civilian exams. Another reason to seek fewer exceptions is the large number of civilian exams involved. This agency handles around 60 civilian test development and validation projects on an annual basis, so the logistics of seeking exceptions with this many exams would be complex, particularly given the lower risk for legal challenge with these jobs.
Sworn exam processes, both entry-level and promotional, represent much higher stakes for this agency than do civilian exams. Sworn classifications as a whole have an extensive and ongoing history of union grievances, internal appeals, and civil service appeals. Additionally, this agency has a past history of litigation, including a 20-year consent decree that initially focused on promotion of women to the rank of sergeant. That consent decree later expanded to include hiring of women as entry-level deputies, placement of women into highly prized “bonus deputy” assignments, promotion of women to ranks higher than sergeant, and the setting of challenging goals regarding female representation in all sworn ranks. In general, the sworn exams have a much greater legal liability than the civilian exams; therefore, internal I-O staff focus most of their efforts on the agency's ability to defend sworn exams against legal challenge.
Any time an exception is made regarding civil service rules, that exception must be sought from centralized HR on a case-by-case basis. Rather than seek exceptions to civil service rules on all or most exams, I-O practitioners in this agency have adopted a strategic approach of focusing on a few exams with the greatest legal exposure. Consequently, we are less likely to raise issues related to civilian exams, whether entry level or promotional, and are more likely to seek exceptions from civil service rules for sworn exams.
Case Examples
Entry-level sworn exams
Because of the volume of candidates required to staff entry-level deputy sheriff openings (15,000–20,000 applicants yearly), this agency has sought and been granted exceptions from the centralized HR function related to civil service rules for entry-level sworn (deputy) hiring. The exceptions granted include variations from normal pass/fail procedures on the qualifying written test, the use of alternative banding procedures rather than the banding procedure specified by civil service rules, and moving a limited number of qualifying background questions to an earlier step in the hiring process. These exceptions to civil service rules have made it easier to staff this high priority classification and helped this agency reduce its staffing cycle time by more than 150%.
Promotional sworn exams
Because this agency typically conducts only two “high-stakes,” sworn promotional exams each year, and because these exams are more likely to be challenged, it is more feasible for internal I-O staff to seek exceptions to civil service rules for these exams. Civil service rules allow either multiple hurdle or compensatory strategies when dealing with multiple predictors but require that a weighted composite score be used as the basis for final banding of candidates. The use of a composite score to drive decision making could be considered one of the better practices required in this county's civil service rules.
The initial hurdle on these sworn promotional exam processes has typically included a written job knowledge test (JKT), and the agency can rely on the judgment of a large panel of SMEs, making it is easier to support such a cutoff score. There is ample support for using SME-based cutoff methods, such as the Angoff (Reference Angoff and Thorndike1971) method, and legal cases have examined the relevance of this method for setting a cutoff score (Contreras v. City of Los Angeles, 1981; National Education Association v. South Carolina, 1978). The Contreras court ruled that seven SMEs was a sufficient number for setting Angoff cutoff scores, whereas the National Education Association v. South Carolina court ruled that 10 SMEs was a sufficient number. This agency typically relies on 15–20 SMEs when setting such cutoff scores.
Candidates who pass the initial JKT hurdle participate in all subsequent assessments. Historically, these assessments have included structured panel interviews and direct supervisor ratings of promotability. More recently, the last step in these promotional programs has included a situational judgment test, a multiple-choice in-basket exercise, and a performance-based writing sample. Rather than setting a cutoff score on each component and then attempting to support each cutoff, this agency combines all three scores using a compensatory approach. Candidates are then banded following civil service rules.
Cascio et al. (Reference Cascio, Alexander and Barrett1988) discussed the Principles recommendation that cutoff scores should be supported by a sufficient rationale or justification, and noted that: “One interpretation of this principle is that, if possible, try to avoid using any cutoff score. Simply use top-down selection” (p. 20). By placing predictors within a weighted composite and then banding based on that composite score, this agency avoids having to set and then support multiple cutoff scores.
How I-O Practitioners Can Influence Policy
There are a few things I-O practitioners may wish to consider when attempting to influence policy. Although these suggestions are focused more on a public-sector setting, there should be substantial overlap with private-sector settings.
-
1. Because there are more exceptions or changes needed than is practical to address, choose wisely. One can't change the world overnight, so ask yourself whether this topic is one worth addressing.
-
2. Don't accept that “because it is a civil service rule” is a sufficient rationale for accepting a questionable HR practice.
-
3. Don't be hesitant to (respectfully) ask for the rationale behind a policy. Far too often, the answer is some variation of “this is how it has always been done” and not due to an actual policy requirement.
-
4. Barriers to change may come from within one's own organization rather than from a centralized or “parent” organization; if the rationale you receive is some variation of “the centralized HR agency requires this,” ask to speak to someone at the agency about the issue. Tactfully question such assertions and raise questions through proper channels to people in the parent organization. Although it may seem faster and more efficient to try to address your questions directly with the decision maker, this can be counterproductive if there is a large difference in organizational level between you and the decision maker. Get your boss (or your boss's boss) involved and let the system work for you.
-
5. Don't expect immediate change; although change is slow in many organizations, it is probably slower in public-sector agencies.
-
6. If you don't succeed in obtaining a desired/needed change, raise the question again after some time has elapsed. Sometimes one has to wait until there is a change in leadership, either in your own organization or the parent organization, before change efforts can be successful. As an example, this agency's implementation of SJTs in sworn promotional exams was proposed and rejected on at least two prior occasions before it was approved. These rejections came from management within this agency and were not the result of concerns raised by the centralized HR function [see #4 above].
Summary
A common theme in Title VII court cases dealing with cutoff scores is that a cutoff must bear some manifest relationship to expected job performance, and the choice of cutoff score may also be influenced by other important considerations. The Guidelines (EEOC, 1978) require that any selection process that results in adverse impact against a protected group must be demonstrated to be valid and job related. Depending on the type of test, type of job, and placement of the test within a larger selection process, cutoff scores may need to be more selective (pass fewer applicants) or less selective (pass more applicants) than would be possible under an arbitrary 70% rule. With many types of assessments, such as tests of aptitude or personality inventories, the notion of a 70% score has no meaning because these assessments are interpreted on a normative basis rather than on an absolute basis. With personality assessments, it may be necessary to reject candidates who score too high or too low on specific dimensions, making a 70% cutoff score even more problematic.
Civil service rules related to cutoff scores often do not provide sufficient flexibility to allow I-O practitioners to use their professional judgment, the input of subject matter experts, labor market conditions, expected adverse impact, and expected future job performance in setting cutoff scores. Attempting to apply the same fixed cutoff score regardless of the situation is risky from a legal perspective and poor practice from a professional perspective. The Vulcan Society (2009) court stated, “Title VII explicitly relieves employers from any duty to observe a state hiring provision which purports to require or permit any discriminatory employment practice” (emphasis added). This statement by the Vulcan court and similar statements in other cases provides I-O practitioners with room to discuss ways to ameliorate problems imposed by well-intentioned but inflexible civil service rules.
