THE PROBLEM

We think about diversity from the perspective of the “leaky pipeline,” in which a critical leak is known to occur between undergraduate and graduate education. In the political science profession as a whole, minorities earned 40% of undergraduate degrees but comprised only 29% of incoming PhD students. By comparison, women earned 48% of undergraduate degrees and comprised 45% of incoming PhD students.Footnote ¹ The situation in our department at the University of California, Santa Barbara (UCSB) aligns with this pattern. About 41% of our undergraduate majors are underrepresented minorities, defined by the University of California (UC) as those who identify as Black, Latino/Hispanic, and Native American/Alaskan Native.Footnote ² Yet, in recent years, only about 26% of those admitted to our PhD program have been underrepresented minorities, with great variance among years. We lack data at the graduate level on first-generation status, but we expect that the proportion is much lower than the 44% of our undergraduate majors who are first generation. These numbers frame our challenge for improving diversity. Doing so is especially important because UCSB is both a Hispanic-Serving Institution and an Asian American Native American Pacific Islander Serving Institution.

INTRODUCTION TO OUR REFORMS

We undertook the overhaul of our admissions process with the short-term goal of reducing the undergraduate-to-graduate pipeline leak for the class entering in Fall 2021 and the long-term goal of a graduate student population that resembles our undergraduate student body. Our approach was informed by the principle of holistic review, which is “a process by which programs consider a broad range of characteristics, including noncognitive and personal attributes, when reviewing applications for admissions” (Kent and McCarthy Reference Kent and McCarthy2016, 1). Holistic review is “widely viewed as a useful strategy for improving diversity of higher education” and is practiced in many settings, including undergraduate and medical school admissions (Kent and McCarthy Reference Kent and McCarthy2016, 1). Although it has its critics, we were persuaded that holistic review has received enough support for us to adopt it in the spirit of experimentation. We intend to build a multiyear body of evidence about its effects.

Our reforms were guided by Kent and McCarthy’s (Reference Kent and McCarthy2016) review of holistic graduate admissions as well as by the general review of evaluation and decision making in higher education by Posselt et al. (Reference Posselt, Hernandez, Villareal, Rodgers and Irwin2020). We additionally drew from descriptions of holistic review developed outside of political science, especially the highly influential Fisk–Vanderbilt Bridge Program (Stassun et al. Reference Stassun, Sturm, Holley-Bockelmann, Burger, Ernst and Webb2011).Footnote ³ Some departments at UCSB use holistic review, and our institution provides National Science Foundation (NSF)–supported guidance about making the change.Footnote ⁴ This meant that we had institutional support, examples of how others achieved it, and good reasons to believe that our traditional process should be updated.

TRADITIONAL GRADUATE ADMISSIONS IN POLITICAL SCIENCE

Our previous practice was to rate applicants on a single evaluative dimension that frankly was not well defined. Although admissions committee members in recent years universally voiced support for diversity, equity, and inclusion, our process did not formally incorporate these considerations. Admissions committee members assigned a score to each applicant using their individual preferences for how to weigh the various indicators in the written file—for example, test scores, Grade Point Average (GPA) and relevant coursework, references, status of undergraduate or graduate institutions, personal statement, writing sample, and any reported contributions to diversity. In addition to being dissatisfied with diversity outcomes from this approach, we were skeptical of the reliability and validity of the scoring.

Based on our informal review of the information displayed on their websites, our previous approach appears consistent with those of many other political science departments.Footnote ⁵ Among the top 25 departments, six (23%) did not disclose any information about how admissions decisions are made or which evaluative criteria are used. Of the remaining 20 departments (77%) that provided information, all described use of a comprehensive process—for example, no specific weighting of Graduate Record Exam (GRE) scores or minimum GPA, careful review of all components of the application, and no formula—that resembles holistic review. However, only nine departments (35%) stated that they consider diversity, personal characteristics, or other noncognitive attributes. Moreover, only one department described a process that, in our view, qualifies as holistic through the use of specific, stated evaluative criteria that surpass traditional cognitive indicators.

HOLISTIC REVIEW: KEY PARAMETERS

Many different practices are associated with the term “holistic review.” This section outlines the key parameters that we identified from the resources discussed previously, the decisions that we made and our reasoning for them, and alternatives.

Second-Round Holistic Review

This subsection describes the key decisions we made with respect to the holistic review of the screened applicants.

Evaluation Criteria and Rubric

Our next two decisions concerned which evaluation criteria to use and the associated rubric. Even departments that are interested only in evaluating cognitive criteria along traditional lines should benefit from this step due to its potential for minimizing bias and making the evaluation process more transparent, consistent, and efficient (Kent and McCarthy Reference Kent and McCarthy2016, vi, 13). To evaluate the semi-finalist applicants, we adopted five dimensions of evaluation, including elements related to diversity and “noncognitive” qualities relating to adjustment, motivation, and perceptions (Kent and McCarthy Reference Kent and McCarthy2016). These five dimensions were academic preparation, relevant research experience, perseverance, self-knowledge, and leadership-outreach activities/DEI. We drew heavily on the criteria used by existing, well-regarded programs with an established diversity-promoting record, notably the Fisk–Vanderbilt Bridge Program (Stassun et al. Reference Stassun, Sturm, Holley-Bockelmann, Burger, Ernst and Webb2011).Footnote ⁹ Other criteria are also in use around the country. For example, the UC–San Diego political science department uses “diversity” as the sole noncognitive criterion along with otherwise more conventional cognitive criteria.Footnote ¹⁰ Solely cognitive criteria also could be used. Which criteria are the most helpful in delivering “inclusive excellence,” to use Posselt’s (Reference Posselt2013) phrase, is a question that unfortunately is well ahead of evidence at this point.Footnote ¹¹ Yet, the same can be claimed about traditional approaches to admissions that examine multiple indicators of academic performance. We believe that the discipline needs experience gained from programmatic experimentation and data collection.

Our next two decisions concerned which evaluation criteria to use and the associated rubric. Even departments that are interested only in evaluating cognitive criteria along traditional lines should benefit from this step due to its potential for minimizing bias and making the evaluation process more transparent, consistent, and efficient.

We scored each applicant as high, medium, or low on each of the five criteria using a rubric that we adapted from the sources cited previously (figure 1). We wanted to minimize implicit bias and values, especially those associating quality with prestige (Posselt Reference Posselt2013), and also to avoid overweighing what can be quantified (e.g., GPA). We experienced a tendency toward rating at the mean, but we were mostly satisfied with the process. In the future, we will revise the rubric to produce more spread and we will engage in an initial round of norming. We also welcome and anticipate further discussion of appropriate criteria and their operationalization.

Figure 1

Rubric for Second-Round Holistic Review

Adapted from Stassun et al. (Reference Stassun, Sturm, Holley-Bockelmann, Burger, Ernst and Webb2011).

Interviews

Next, we decided whether to conduct interviews. We chose to interview all semi-finalists using a written interview protocol derived from the Fisk–Vanderbilt Bridge Program. Some of our questions are listed in figure 2. These interviews provided useful information in addition to what was in the written files, especially for the noncognitive criteria. Because of how much light our conversations shed on applicants’ qualities beyond their academic record, we will continue to interview in the future. After the interviews, one committee member expressed amazement that the department had ever managed to admit people without having spoken with them first.

Figure 2

Sample Interview Questions

Adapted from Stassun et al. (Reference Stassun, Sturm, Holley-Bockelmann, Burger, Ernst and Webb2011).

For other departments considering the adoption of interviews, we recommend discussing the risk of introducing a new avenue of bias and how this might be mitigated, such as by using a standardized set of questions. We were aware of this risk, but we believed that the importance of learning more about the applicants’ noncognitive qualities would outweigh the risk of implicit or explicit bias in the interview. An alternative for those who are concerned about interview bias involves application instructions. We encourage departments to direct applicants and their letter writers to address whatever evaluation criteria are being used, especially noncognitive criteria. This is especially useful for letter writers to understand that the admissions committee is requesting more than a traditional assessment of academic performance. Due to the timeline for developing our new process, we could not communicate our holistic-review criteria in advance to candidates and letter writers, but we will do so in future admissions cycles. If adequate information can be elicited in the written materials, interviews might have less importance.

Score Aggregation

After the applicants were rated, informed by our interviews, our final step was to explore approaches to aggregating each semi-finalist’s five scores. These approaches ranged from simple and weighted averages to a count of the number of criteria on which the applicant scored “high.” Of course, many alternatives can be imagined. In our case, these approaches converged fairly well, mostly all pointing to the same set of semi-finalists, who we then divided into finalists: admitted and waitlisted candidates. In a case in which various approaches to aggregating scores do not converge, admissions committees must apply another set of principles in this final stage.

OUTCOMES

We were extremely pleased with the final list of admitted and waitlisted candidates, and we were sufficiently satisfied with the process to continue gaining experience with it. Our top candidates evidenced the type of noncognitive qualities that we anticipated were associated with successful completion of a doctoral program, as well as academic excellence. The diversity of the finalists exceeded that of our existing graduate population, which meant that we met our short-term goal. Specifically, a large majority (73%) of the finalists were women; a solid majority (64%) were underrepresented minorities; slightly more than a quarter (27%) were racial or ethnic minorities not classified as underrepresented; and slightly more than a third (36%) were first-generation students. Also, a substantial number were in the intersectional category of underrepresented minority women.