Reassessing SCIPE Validity

To validate their five trait estimates, Hall et al. (Reference Hall2021) subject those estimates to two examinations related to the concept of ideology. First, they analyze the correlation between their trait scores and the ideological direction of each justice’s vote in each case (Spaeth et al. Reference Spaeth2010).Footnote ¹ Second, they analyze the correlation between their trait scores and a justice’s Clerk-Based Ideology (CBI) score, which come from the political campaign contributions made by their former law clerks (Bonica et al. Reference Bonica2017).

We suspect few scholars of the Court would identify these two particular measures as offering the most direct or intuitive assessment of justice ideology, so we examine four additional approaches to measuring judicial preferences. First, we look at the simple proportion of conservative votes cast by each justice over the course of his or her career. We count the number of conservative votes and divide that into the total number of votes cast by the justice that had a determinable ideological direction. This is the approach used in basically every existing study that introduces a new approach to measuring preferences, ranging from Segal-Cover scores to Martin-Quinn scores (Martin and Quinn Reference Martin2002) to even the validation of CBI scores themselves.

Second, we examine each justice’s Segal-Cover score, which is derived from a content analysis of the preconfirmation hearing newspaper editorials written about each nominee (see also Segal et al. Reference Segal1995). These scores have the added bonus of being the only measure of ideology not derived from any aspect of how a justice behaves once on the Court (cf. voting on cases or hiring law clerks). That strength is arguably also their greatest weakness, however, since a good number of justices seem to show evidence of ideological change over the course of their careers (e.g., Epstein, Martin, Quinn, and Segal Reference Epstein2007).

Our final two additions are more sophisticated approaches that summarize judicial votes through the use of item response theory models. In particular, we estimate a two-parameter item response model to produce a single, career-level estimate of each justice’s ideal point. That is, as our third measure, we estimate a justice’s static Martin-Quinn score (Martin and Quinn Reference Martin2002).Footnote ² Our final measure is a career-averaged ideology score as estimated by the approach suggested by Bailey (Reference Bailey2007, Reference Bailey2013). The key difference between these latter two approaches is that the Martin and Quinn–inspired version uses only the actual votes cast by justices on cases, whereas Bailey’s formulation creatively leverages instances where a justice takes a position on a case in which she was not a voting member of the Court (e.g., Justice Thomas writing that he believes the Court ruled wrongly in a previously decided case, decades before he joined the Court).

All four of our additional measures as well as the CBI score are near continuous in nature, so we use ordinary least squares regression to examine the relationship between alternative ideology measures and SCIPE scores. And, because the individual votes justices cast in each case are dichotomous (1 if liberal, 0 if conservative), we use logistic regression for those data.Footnote ³

In addition to the individual trait estimates, Hall et al. (Reference Hall2021) also include controls for a justice’s sex and birth year. We estimate our models both with and without these controls, both to assess sensitivity to their exclusion and provide practical guidance to would-be users of SCIPEs. Analyzing the data using both approaches is important since, although Hall et al. include these controls in their validation models (and three of the four are statistically significant), neither they nor Hall (Reference Hall2018) includes them in any of the substantive applications/chapters.

Figures 1 and 2 show our results. In figure 1, each of the individual panels corresponds to one of the Big Five traits. The Y-axis within each panel identifies which measure of ideology we examine. Each X-axis reports the coefficients from our OLS regressions. In order to facilitate apples-to-apples comparisons among the different dependent variables, we standardized all of them prior to estimating each model. SCIPEs are already standardized, so a coefficient of 0.75 means that a one standard deviation increase in a given trait translates into a 0.75 standard deviation increase in the ideology measure. Finally, within the plotting space, we identify estimates for models that do (circles) and do not (squares) include controls for justice sex or birth year. The horizontal whiskers denote the confidence intervals around the coefficient estimates. The thicker lines show the 80% values, the vertical ticks show the 90% values, and the thinner line extends to the 95% values (all two-tailed tests).

Figure 1.

Summary of ideology models with SCIPE trait estimates. The horizontal whiskers report 95% (thin lines), 90% (vertical ticks), and 80% (thick lines) confidence intervals (all two-tailed). See text for additional description.

Consider the conscientiousness trait. Scholarship shows that it consistently correlates with ideological conservatism both among the mass public and among political elites like state legislators (Dietrich et al. Reference Dietrich2012) and members of Congress (Ramey, Klinger, and Hollibaugh Reference Ramey2017).Footnote ⁴ We therefore should expect to see a positive and statistically significant correlation between SCIPE-assessed conscientiousness and conservative ideology.

The SCIPE-assessed conscientiousness measure provides no such correlation, however. In figure 1, we examine three measures that directly tap into ideology, as evidenced in the votes cast by a justice. Only one—the Bailey score—provides even weak evidence of an association. What is more, that weak association only appears when simultaneously controlling for a justice’s sex and age. In other words, the measures by themselves are insufficient. Consistent with Hall et al. (Reference Hall2021), we find decently strong evidence of an association with the CBI measure, but this too requires the inclusion of the demographic control variables.

Next, consider agreeableness. Scholarship on the correlation between ideology and agreeableness is mixed, but Dietrich et al. (Reference Dietrich2012, 203), who conducted a comprehensive review of roughly two dozen studies, note that “several studies have found [a] modest link … between agreeableness and liberal identification.” These authors found that more agreeable state legislators were more liberal. Ramey et al.’s (Reference Ramey2017) study of Congress found similar results. As figure 1 indicates, however, no such relationship exists between SCIPEs and any of the five measures of judicial ideology. None of the coefficients is signed in the correct direction, and, in any event, all have very wide confidence intervals.

Dietrich et al. (Reference Dietrich2012) classify neuroticism as having a “modest” negative relationship with ideological conservatism (or, if you prefer, a positive correlation with liberalism). Across five dependent variables and two different model specifications, however, we find only a single result that is statistically significant and in the expected direction. As reported by Hall et al. (Reference Hall2021), the CBI score paired with demographic controls yields a statistically significant coefficient. As our plot shows, however, this effect is weakened both in magnitude and significance level if one omits the demographic control variables. And, of course, none of the four other measures of ideology exhibits any connection with neuroticism.

Next, the trait of openness has, by Dietrich et al.’s (Reference Dietrich2012, 203) accounting, “consistently … been found to be a strong predictor of ideological liberalism.” As such, we should expect to find a negative relationship between openness and conservatism, since all five of our measures are scaled with positive values being conservative. This is not what we see in figure 1, however. Indeed, not a single one of the 10 models produced a result that even approaches statistical significance.

Next, we consider extraversion. Predictions here are probably the most challenging of any of the traits. On one hand, Dietrich et al.’s (Reference Dietrich2012, 203) comprehensive canvassing of the literature lead them to conclude that this trait “routinely produces null results when included as a predictor of ideology.” On the other hand, it is unclear where Hall et al. (Reference Hall2021) put their chips, as they first note that one mass public study found a negative relationship between liberalism and extraversion. They then go on to observe that the analysis of members of Congress by Ramey et al. (Reference Ramey2017) found a positive association.

Starting with the five aggregate measures, we find some moderately consistent evidence that extraversion is positively associated with liberalism, with results in the range of statistical significance for a justice’s Martin-Quinn score, Bailey score, and CBI score. As with the earlier significant results, however, all of these results are sensitive to the inclusion/exclusion of the demographic control variables. And, given that the Hall et al. measures are derived from the same software as the Ramey et al. study, it is possible this is an artifact of the particular method as opposed to something substantively informative.

We next turn our attention to figure 2, which follows the same approach to predict the ideological direction of justices’ votes. It shows coefficient estimates for the SCIPE variables from two logistic regression models. Circles represent models that include controls, and squares denote models that exclude them. We use point color to mark whether a variable is (black) or is not (gray) statistically significant.Footnote ⁵ All of the black points are significant at the .05 level and the two gray points have two-tailed p-values greater than .20.

Figure 2.

Summary of justice vote models with SCIPE trait estimates. The dependent variable is coded 1 if a justice voted liberally and 0 if he or she voted conservatively in a case. For both models, $N=\mathrm{72,657}$ . Gray points are statistically insignificant ( $p>.20$ ), and black points are significant at the .05 level. Significance calculated using naive standard errors (see n. 5 for additional discussion).

For conscientiousness, the vote-level results in figure 2 underscore the SCIPE’s sensitivity to demographic controls. We observe a strong and statistically significant effect in the expected direction only when the models include demographic controls. That is, increased conscientiousness decreases the probability of a liberal vote only when the models include demographic controls. If the models exclude demographic controls—and contain only the SCIPEs—the coefficient size shrinks nearly to zero and the standard errors correspondingly increase, rendering them statistically indistinguishable from zero.

For agreeableness, the vote-level models in figure 2 also show sensitivity to the inclusion of demographic controls, though here the effect is actually reversed as compared to conscientiousness. A model that includes controls fails to show any significant correlation, but one that excludes them does. In fact, the coefficient size increases by a factor of 10, changes signs, and becomes statistically significant in the expected direction when excluding the demographic controls.

For neuroticism, in the context of the vote-level models, we again confront a confusing pair of results. The inclusion of demographic controls determines whether the effect of the SCIPE neuroticism estimate is positive and statistically significant (controls included) or negative and statistically significant (controls excluded).

For openness, the results are more encouraging if one considers the individual vote models, where the effect is positive and largely unchanged regardless of how one handles the demographic controls or the calculation of standard errors. Yet, as Hall et al. (Reference Hall2021) themselves note, this is due in no small part to the fact that these models contain more than 70,000 observations.

As to the vote-level analyses on extraversion, we again observe that the sign of the effect depends critically on whether one includes the demographic controls. When they are included, extraversion is positively correlated with casting a liberal vote; when the controls are excluded, however, the effect switches signs, decreases in absolute magnitude, and yet still retains its statistical significance.

Taken together, these findings show that the SCIPEs do not appear to provide valid estimates of Supreme Court justices’ personality traits. This is consistent with our previous work, which also shows that the SCIPE trait estimates fail to perform in the way one would expect across a half dozen nonideological outcome variables.Footnote ⁶ We turn next to offering some data-driven insights into why this might be the case.

The Danger of Using Concurring Opinions to Estimate Justices’ Personalities

As a preliminary matter, it is worth emphasizing that we fully agree with the general approach utilized by Hall et al. (Reference Hall2021). That is, a text-based approach is likely to be the optimal (though by no means the only) way of assessing judicial personalities from afar.Footnote ⁷ As Pennebaker et al. (Reference Pennebaker2015, 1) tell us: “The ways people use words in their daily lives can provide rich information about their beliefs, fears, thinking patterns, social relationships, and personalities” (emphasis added). Herein lies the challenge. Scholars are generally not privy to the conversations, text messages, and emails political elites like Supreme Court justices generate in their daily lives, and so we need to look elsewhere for that information. Where we part ways from the approach of Hall (Reference Hall2018) and Hall et al. (Reference Hall2021) is in where one ought to go looking to find such information.

Concurrences Primarily Reflect the Court’s Majority Opinion

As the sole source of input for their trait estimates, Hall et al. examine the concurring opinions justices have written while serving as a Supreme Court justice. We are skeptical (Black et al. Reference Black2020). Perhaps the biggest problem is that Hall et al. (Reference Hall2021) simply ask too much of a single type of document to be able to provide valid insights into personality. An approach that puts all of its eggs in a single basket is likely to be insufficient.

To appreciate the magnitude of the gap between the ideal and Hall et al.’s approach, it is useful to consider the primary sources that the researchers who develop text-to-trait models employ to identify the textual correlates of personality traits. That is, when scholars are trying to discern what fundamental patterns exist, from where do their “daily words” come?

Typically, scholars will employ texts of a more “natural” cast to estimate personality. Personality Recognizer—the approach utilized by Hall et al. (Reference Hall2021)—repurposed a corpus of essays written by undergraduate psychology students who responded to a prompt that solicited their “thoughts, feelings and sensations are at this moment. … Your goal in this assignment is to reveal in your writing the way your mind works naturally” (Pennebaker and King Reference Pennebaker1999, 1301). Concurring opinions, by contrast, seem to us anything but an open-ended opportunity to express one’s self. As Corley (Reference Corley2010, 96) notes, concurring opinions often communicate a justice’s “understanding of the majority opinion” (emphasis added). Similarly, Ray (Reference Ray1990, 783) observes that “by writing separately, a concurring author always offers an internal commentary on the court’s judgment.” In other words, concurring opinions do not provide an open-ended opportunity for expression. Majority opinions often substantially frame and constrain the content of concurring opinions in a given case.Footnote ⁸

In defending concurrences, Hall (Reference Hall2018, 38) quotes approvingly from a longtime circuit court judge: “[SCIPE’s] focus on concurrences reflects Judge Frank Coffin’s description of the ‘feeling of unjudicial glee as one shucks off the normal restraint of writing for a panel and proceeds to thrust and parry with gay abandon.’” The same colorful quote also appears in Hall et al. (Reference Hall2021). However, when read with the added context of the paragraph from which the quote originates, one cannot help but doubt whether Coffin was actually talking about concurrences, as Hall and Hall et al. suggest. Here is that quotation along with its source paragraph, which is absent from both Hall and Hall et al.’s presentation of it:

A concurrence is like a fencing foil; it elegantly makes its usually bloodless points. A dissent, on the other hand, is more like a broadsword. It takes more resolution and commitment to wield it and there is the expectation of drawing at least a little blood. In any event, there is a feeling of unjudicial glee as one shucks off the normal restraint of writing for a panel and proceeds to thrust and parry with gay abandon. For this very reason, we judges are well advised to resist the temptation unless we find a compelling interest and no more effective alternative. Sometimes, however, a dissent is the precise instrument that should be used. [Note: Coffin then goes on to enumerate the five conditions in which he believes a dissent is appropriate.]

(Coffin Reference Coffin1994, 227)

Perhaps most dispositive, however, is the fact that the entire paragraph itself—“thrust and parry” quote included—appears in a section that is labeled with the heading “Dissenting Opinions” and not in the distinctive “Concurring Opinions” section on the previous page. Interestingly, what thoughts Coffin does offer about concurring opinions are all consistent with our majority-response view of concurring opinions; that is, they center on how the concurring opinion will expand or limit the majority opinion (Coffin Reference Coffin1994, 226–27).

Even if we accept the Coffin quote as deployed by Hall and Hall et al., it still identifies what we believe to be a critical and inherent limitation of looking only at concurrences (or dissents for that matter): a justice or judge is not writing in a manner that elicits that jurist’s instant “thoughts, feelings, and sensations,” but rather is fundamentally constrained by the content, topics, and, to a large degree, the specific words utilized by the majority opinion (i.e., the would-be opponent in Coffin’s fencing match).

At any rate, this question is testable. If we are correct, the personality content of concurring opinions should appear, on average, like the majority opinions. Moreover, it should also appear more like the majority opinion than other concurring opinions written by that same justice. In our earlier work, we tested this argument by analyzing opinion-level personality content in the Court’s majority opinion and any of the concurring opinions that accompanied it. We then also examined the personality content in pairs of concurring opinions written by the same author. What we found when we compared the two was that the correlation in personality content of majority-concur pairings was at least three times larger than the analogous correlation between pairings of concurring opinions written by the same author (Black et al. Reference Black2020). Or, stated a bit differently, opinions written by two different justices in the same case had more alike in terms of their apparent personality than opinions written by the same justice across two different cases.

Here we build upon this previous analysis in two important ways. First, when selecting a different concurring opinion, we originally only constrained the pool of possibilities to be other concurrences written by the same justice. This meant that we could be pairing an economic activity concurrence with one, say, about the First Amendment. Given that the majority and concurring opinion comparison come from the exact same case, there is a guaranteed agreement in terms of issue area. This, in turn, could be inflating the correlations we found between a case’s majority and concurring opinions. To address this issue, we now limit the pool of possible concurrences to be those in the same issue area (while still requiring that it was the same justice authoring it).

Second, the results we report in our book come from a single set of random pairings. That is, we took each of the more than 2,700 concurring opinions in our data and randomly paired them with one other concurrence by the same author and computed the resulting correlations for each of the Big Five traits. Justices in our data, however, often authored quite a few concurring opinions (the median is 65). One plausible concern is that by hanging our hat on a single set of results, perhaps we just pulled a particularly “unlucky” draw of concurrences with which to make those calculations. To address this concern, we then repeated this entire process a total of 10,000 times. That is, we paired each of the 2,700 (or so) concurrences with a random one written by the same justice and within the same issue area and calculated the trait correlations. We then did it a second time, and then a third, and so on.

Figure 3 displays the results from this analysis. Each of the five panels represents one personality trait. Within each panel, we show the level of correlation between the majority and concurring opinions in a case with the dashed line. The violin plots in each panel portray the distribution of correlations we recovered from the 10,000 sampling iterations that we performed. The gray area shows the distribution like a kernel density plot. The vertical whisker spans the full range of the 10,000 iterations. The black rectangle identifies the 25th and 75th percentiles. The white circle shows the median. The text annotations above the dashed line show the factor by which the majority-concur correlations exceeds the concur-concur correlations, using the median and, in brackets, the minimum and maximum values for the 10,000 samples.

Figure 3.

Comparison of opinion-level personality content. The dashed lines report the correlation between the content of a majority opinion and a concurring opinion in the same case. The violin plots show the correlation between the concurring opinion and another, randomly chosen concurring opinion written by the same justice in the same issue area. We repeated this random assignment process a total of 10,000 times. The circles, black rectangles, and vertical whiskers correspond to the median, interquartile range, and minimum/maximum, respectively. Text annotations above the dashed line report how much bigger the majority-concur correlations are compared to the concur-concur correlations.

Median values for the concur-concur correlations range from a low of .14 for agreeableness to a high of .19 for neuroticism, which, to be fair, are elevated relative to the values we found in our initial analysis (they ranged from .08 to .13). That being said, our ultimate conclusion remains the same: the same case pairings continue to overwhelmingly dominate the same justice pairings across the board. Four out of five traits observe correlations that are at least twice as strong as compared to the median result from our 10,000 samples. Importantly, in not one of the 10,000 samples we analyzed did the concur-concur correlations match, let alone surpass the majority-concur correlation in strength. Thus, by even this rather conservative test, we find the fundamental theory that underlies SCIPEs to be wanting.Footnote ⁹

The Impact of a Concurrence’s Legal Purpose

The above suggests a general lack of consistency within the corpus of a justice’s concurring opinions—even those that were written on the same general topic. Again, this is not especially surprising given what published research on the topic more than a decade ago has already demonstrated: when justices write concurring opinions, they do so for a wide range of substantive legal purposes (Corley Reference Corley2010). Although this point isn’t addressed in Hall (Reference Hall2018), Hall et al. (Reference Hall2021) do acknowledge this—as well as other features—as sources of “potential” bias but then, without providing any assessment of the impact of such a bias, assert that their approach is still valid.

In our earlier work, we already examined the impact of these biases by drawing out a number of comparisons among different types of concurring opinions. Our results suggested that they are not just hypothetical but are both substantial and systematic. Justices vary, for example, in the rate at which they write a regular versus a special concurrence. Not only that, but the trait estimates one would arrive at vary significantly if you examine one type of concurrence versus another (Black et al. Reference Black2020). Relatedly, one of the main assertions made by Hall et al. (Reference Hall2021) in defense of concurrences is that their authors lack an incentive to accommodate the requests of other colleagues. Here, too, our previously published work demonstrated that justices vary quite a bit in terms of whether their concurrences are joined by other colleagues; Justice Douglas, for example, wrote for only himself in more than 90% of his nearly 130 concurrences. At the other end of spectrum, Justice Brennan concurred nearly 170 times in his career and the majority of those were joined by one or more of his brethren. And, once again, we find considerable discrepancies in the resulting trait estimates one would produce by using one set of concurrences versus another (Black et al. Reference Black2020).

So, it turns out we already know quite a bit about systematic bias in the singular documents that create SCIPEs. Here we build on these existing efforts by probing, in more detail, the importance of the legal purpose of concurring opinions. To do so, we turn to Corley (Reference Corley2010), who identified a total of six different categories of concurrences: doctrinal, emphatic, expansive, limiting, reluctant, and unnecessary.Footnote ¹⁰ Doctrinal concurrences are what judicial scholars often refer to as “special concurrences.” Here, the justice joins the majority’s result but not the rationale it uses to reach that result. And so the justice writes a concurring opinion to explain how her rationale differs from the majority’s. Emphatic concurrences seek to clarify a particular aspect of the majority opinion. Expansive concurrences and limiting concurrences seek to enlarge or restrict the scope of the majority opinion, respectively. Reluctant concurrences indicate that a justice may join the majority opinion, but with reservations. Finally, unnecessary concurrences exist when a justice concurs specially but does not write an opinion clarifying why she disagrees with the majority’s rationale.

Corley performs a painstaking content analysis of nearly 300 concurrences written during the Court’s 1986–89 terms, coding each concurring opinion as belonging to one of the six previously described categories.Footnote ¹¹ Her work further documents considerable variation in each justice’s tendency to write or join different types of concurring opinions (Corley Reference Corley2010, 32). In terms of doctrinal concurrences, for example, just over 45% of Justice Brennan’s concurring activity took place in such opinions as compared to only 26% for Justice Kennedy; when it came to expansive concurrences, however, Kennedy showed a stronger preference for those (22% of his activity) as compared to Brennan (only 10% of his activity).

We next seek to ascertain whether this varying mixture of concurrences translates into the same sort of differences we previously uncovered with respect to overall personality trait estimation. Again, the intuition is fairly straightforward: if these different opinions are producing equivalent information about the writer’s personality, then there should be a reasonable degree of consistency among the estimates we obtain. To that end, we utilize Corley’s coding of the concurring opinions to assess how sensitive estimates of justice personalities are depending on the type of concurring opinion used to generate those inputs.

To do so, we started by gathering the 269 written concurrences coded by Corley. Unnecessary concurrences, by definition, do not include a written opinion, and so we necessarily had to exclude those. We then aggregated all of a justice’s opinions by each concurring opinion type, which yielded a total of 43 merged opinions.Footnote ¹² There is considerable variation in the resulting length of each of these files. Justice Scalia’s corpus of doctrinal concurrences weighs in at more than 45,000 words in length. Scalia’s body of restrictive concurrences, by contrast, contains just over 400 words. We follow Hall et al. (Reference Hall2021, n. 3) and exclude four justice-concurring combinations that contain fewer than 500 words.Footnote ¹³ In addition to these justice-type pairings, we also generated a single file for each justice that contains all of her concurrences. This is equivalent to the approach utilized by Hall et al., and so we include it for comparison purposes.

Opinion files in hand, we then simply process this corpus of files using Personality Recognizer as utilized by Hall et al. Because we have six different types of inputs (one for each concurrence type plus one containing all concurrences), we have a total of six different sets of estimates for each of the Big Five traits for the 10 justices in the Corley data. Figure 4 plots all of these estimates. Each of the five panels corresponds to an individual trait. The X-axis shows the standardized trait score. Within the plot, we use different symbols to identify what specific inputs were used to generate that estimate.Footnote ¹⁴ The Y-axis identifies each justice’s initials. The parenthetical values located after those initials report the absolute value of the difference between the smallest and largest trait estimate for that justice. Consider Justice Blackmun (HAB) and the conscientiousness trait. Using the six different types of inputs, we estimated six different conscientious scores for him, which are, in ascending order: −1.8 (expansive), −1.0 (all combined), −0.1 (doctrinal), 0.0 (emphatic), 0.7 (limiting), and 1.3 (reluctant). It is worth reiterating that these scores are all standardized, and so a score of −1.8 means Blackmun is estimated to be 1.8 standard deviations below the mean and a score of 1.3, by contrast puts him 1.3 standard deviations above the mean for the trait. The absolute difference between these two extremes is 3.1 units, which is quite large.

Figure 4.

Personality traits as estimated by different concurring opinion types

Even a casual eyeballing of these results shows that most justices seem to bounce around in relation to their colleagues. Moreover, this movement based on input does not follow a consistent or predictable pattern. For example, in terms of his conscientiousness, Blackmun’s emphatic concurrences (the triangle) evince considerably more conscientiousness than do his reluctant concurrences (the diamond). The opposite is true, however, for Justice Brennan (WJB), whose reluctant concurrences show higher conscientiousness than his emphatic ones. And, as the annotations by their initials reveal, Brennan, much like Blackmun, also observes a lot of variation in his concurrence-assessed conscientiousness (2.4 units between the minimum and maximum).

To provide more of a macro-level assessment, however, we turn to figure 5, which visualizes absolute-value differences, like those of Blackmun’s and Brennan’s we just described. In that sense, it provides us with an opportunity to examine whether the “potential for such bias” that Hall et al. (Reference Hall2021) acknowledge is simply potential as opposed to realized. Each of the Big Five traits appears along the Y-axis. The values portrayed in each of the box plots come from the parentheticals in figure 4. That is, for the conscientiousness box plot, we include the 3.1 value for Blackmun and the 2.4 value for Brennan plus the other eight justices in the data.

Figure 5.

Box plots of the absolute value differences in each justice’s concurrence-type trait estimate. Each plot portrays 10 data points, which are identified in the parentheticals of figure 4.

Within each box plot, the thicker black line denotes the median value and the shaded gray area identifies the interquartile range of these values. Of particular note, the median amount of difference between a justice’s smallest and largest estimates for a trait is never less than 1.4 standard deviations. Another consistent feature of these results is that there are large gaps for all 10 of the justices in the data. To wit, all justices have at least one trait where their minimum/maximum gap is at least one standard deviation in magnitude. Eight of the 10 justices have one or more gaps that are at least two standard deviations, and fully four of those eight also have one or more gaps that are three or more standard deviations. This indicates a high degree of trait sensitivity driven solely by the documents included in the estimation process, which itself suggests that these different documents are, consistent with the literature on concurrences, in fact very distinctive.

What Kind of Personality Measures Would Be Appropriate or Ideal?

We have raised serious questions about the personality measures from Hall (Reference Hall2018) and Hall et al. (Reference Hall2021). But the next question is, what kind of text-based measures would be appropriate to employ? As a baseline, it would be ideal to generate personality measures for all of the justices under consideration. We recognize, however, there may be some limitations that make it challenging to produce measures on all justices. Hall et al. (Reference Hall2021) are up-front about not being able to generate measures for Chief Justice Vinson and Justice Minton because they did not produce enough concurrences. While being forced to omit a few of the justices is not the end of the world—after all, this happens to most everyone studying judicial behavior at some point in time (ourselves included)—it is, however, peculiar when Hall et al. later argue against using alternative sources of justices’ language because “these alternative sources pose serious availability problems” (353). In reality, however, this is not much of a challenge. In our earlier efforts, we gathered enough alternative texts to estimate personality traits for all the justices during their timespan, including Vinson and Minton. And, in preparing this article we were easily able to update our estimates to include newly appointed justices. Thus, this serious problem appears to be not so serious after all.

We also wish to highlight several important features of the ideal measurement strategy. First, the personality estimates should be exogenous to the justices’ opinions written while on the Supreme Court. Scholars should want to avoid a circular measurement procedure. We would not want to use justice behavior to explain justice behavior, which is circular and inappropriate.Footnote ¹⁵ As the authors of the leading study on the estimation of Supreme Court justices’ ideal points remind us, “The circularity concern [of using votes-based ideal points to explain votes] is quite important as a purely technical matter. Strictly speaking, the scores should not be used in this context” (Martin and Quinn Reference Martin2005, 2). Similarly, Ho and Quinn (Reference Ho2010, 847–48) state:

Much like Segal and Cover’s (Reference Segal1989) decision three decades ago to craft an exogenous indicator of judicial ideology, researchers ought to employ texts that are exogenous to the behavior scholars are most likely to want to use the resulting personality measures to explain. “One cannot demonstrate that attitudes affect votes when attitudes are operationalized from those same votes” (558). Thus, scholars seeking to estimate the personalities of political actors should take pains to ensure the texts they employ do not come from the same behaviors they intend scholars to examine later.

Second, the text inputs should predominantly reflect the sole work of the justices. Of course, due to the justices’ involvement in government and politics, the things they say and write over their lifetimes are often for particular audiences. This creates the problem that some document types are likely to reveal more about personality traits than other types (Hall Reference Hall2018, 36). This tension can be successfully addressed (a) by collecting texts that come from justices’ prenomination speeches and writings (or even lower court opinions when other document types are in short supply) and (b) by collecting as much textual data as possible under conditions where the justices are not strictly beholden to the same organizational, group, or societal interest.Footnote ¹⁶

To satisfy those two prongs suggests there is considerable virtue in collecting texts from various sources as a way to account for the fact that personality traits will be more or less visible under different situations. The key, however, to making this strategy successful is that there needs to be an appropriate standardization process in place where documents with more constraints are weighed accordingly. For example, the standardization process should be able to account for the fact that less personality can be extracted from a lower court opinion compared to a speech, presumably because there are more constraints on a lower court opinion. Our existing approach does so by comparing the personality content of actual texts with ones generated completely at random (Black et al. Reference Black2020, 38–41).

Third, the estimation process should be able to add quickly any newly seated justices. By relying on concurring opinions as the input method, Hall et al. (Reference Hall2021) must wait a significantly long time before adding new personality estimates. This point becomes sharper when considering how quickly changes happen to the Court’s membership; as of this writing, we have observed three new justices (Gorsuch, Kavanaugh, and Barrett) taking the bench in a span of four years. Having to wait a little while for scores seems reasonable, since a justice’s first term or two will not give scholars much data with which to work. But Hall et al. provide scant practical guidance about when a justice will have written enough concurrences to fully reveal his or her personality to scholars. An ideal approach is one that is able to produce an estimate shortly after a nominee is announced—an approach that is consistent with the standard Segal and Cover (Reference Segal1989) adopted.

Fourth, scholars ought to employ the most recent developments in text-to-trait technology and continue to refine their measures as far as technological advances will allow. For example, the recent approach designed by IBM—Watson Personality Insight (WPI)—provides advantages for the task of text-driven personality recognition. WPI infers personality traits from text based on an open-vocabulary approach (rather than a closed-vocabulary approach) using a word-embedding technique called GloVe (Global Vectors for Word Representation) to obtain a vector representation of the input texts, which then feeds into a model that uses an algorithm that was trained on thousands of individuals who provided both text and answers to personality surveys (IBM 2017). This approach is an improvement for multiple reasons. Perhaps most importantly, text-based personality measurement with an open-vocabulary approach outperforms a closed approach (Schwartz et al. Reference Schwartz2013). This method lets the underlying data decide which individual words, multiword phrases, and overall topics best predict an individual’s personality traits. It creates trait estimates with greater accuracy and efficiency than older approaches (Arnoux et al. Reference Arnoux2017).

Unfortunately, as is always the case with a reliance on any cutting-edge technology, today’s hot fad eventually becomes yesterday’s old news. For example, Black et al. (Reference Black2020) employed WPI, but that program will soon be sunsetted. So while the processes used to generate personality estimates by Black et al. are valid (as are the conscientiousness scores themselves), they will eventually need to be replaced when scholars seek to examine more justices.Footnote ¹⁷ Fortunately, research at the intersection of natural language processing, machine learning, and personality psychology is taking place at a highly vigorous rate. In one recent example, Obschonka et al. (Reference Obschonka2020) discuss how to estimate personality traits from an available dataset established by Schwartz et al. (Reference Schwartz2013) that contains over 70,000 Facebook users who also completed a personality survey (and made their posts publicly available).Footnote ¹⁸ This provides a viable roadmap for entrepreneurial scholars to build the next generation of personality estimates (see also Kern et al. Reference Kern2014; Park et al. Reference Park2015).

Fifth, an ideal estimate of personality traits also would contain some measure of uncertainty or standard error of the estimate. The judicial equivalent would be moving from Segal-Cover scores, which only contain ideological point estimates, to Martin-Quinn scores, which have standard errors. Having standard errors would be valuable not only from a measurement standpoint but also from a substantive point of view. It could contribute to a better understanding of how much variation there is in personality traits across situations and time.

Finally, but most importantly, any set of measures should be subjected to a thorough validation before proceeding. One ought not expect that future research will be uniformly successful at tapping into all of the Big Five traits. It could be, for example, that agreeableness is particularly difficult to pin down. Because we do not have self-report measures and we cannot use reports by others, that means the primary method of validation is to compare how well measures predict or correlate with various behaviors with which the traits are known to correlate. Importantly, reliance on just one or two behavioral indicators drawn from a single concept like ideology is not optimal, especially when existing work on the topic is far from clear about what relationships one should expect. We have previously illustrated this point when validating the measure of conscientiousness by examining its relationship to other indicators beyond ideology (Black et al. Reference Black2020).Footnote ¹⁹ Relying on only a couple of indicators is a risky proposition, as we demonstrate above with our reassessment of ideology.

An Application and a Test

So far, we have identified a number of issues with SCIPEs, both in terms of their construction as well as their resulting empirical validity. We have also sketched out some thoughts about how an alternative approach could address some of these concerns. In this section, we illustrate the pitfalls of applying the SCIPEs to studies of judicial decision making, and to compare them to measures that are better, we apply them to an analysis of the Supreme Court’s agenda-setting process. We examine whether conscientiousness makes justices more likely to vote to grant review when there is conflict among the circuit courts. Conscientiousness is particularly important to the practice of judging.Footnote ²⁰ As part of this inquiry, we compare the results of empirical models that include the Hall et al. (Reference Hall2021) SCIPE scores versus our own (Black et al. Reference Black2020).

For substantive background, there are two kinds of legal conflict that are relevant at the Court’s agenda-setting stage and also to this analysis: strong conflict and weak conflict. Strong conflict exists when there is a square conflict between or among lower courts and the conflict does not appear to be clearing itself up. In other words, if the same legal question has come up in multiple circuits and those circuits have reached opposing answers, there is strong legal conflict involving that issue. In such a case, we would expect all justices to be sensitive to the conflict and vote to resolve it (e.g., Black and Owens Reference Black2009).

Weak conflict is another matter. Here, there is tension among lower court decisions, but the nature of the conflict is qualitatively different from strong conflict. The need to address it is perhaps not as pressing—at least for most justices. The two circuit court decisions purportedly at loggerheads might not address an identical legal question. This means a conflict might be characterized as “shallow,” or indirect. Or, it could be that the circuit courts appear to be working out the conflict on their own, by virtue of one answer to a legal question gaining favor over another and moving the circuits toward uniformity.

Conscientious justices will treat petitions with weak legal conflict more seriously than do less conscientious justices (Black et al. Reference Black2020). Conscientious people take their professional obligations seriously. Conscientious workers “are predisposed to be organized, exacting, disciplined, diligent, dependable, methodical, and purposeful … [they] thoroughly and correctly perform work tasks [and] take initiative in solving problems” (Witt et al. Reference Witt2002, 164). One of the justices’ central duties is to ensure uniformity in the law. It is a task that the Court—and only the Court—can accomplish. In cases with strong conflict, all justices will be inclined toward granting review. But it is in cases with weak legal conflict that the conscientious justice’s heightened level of duty and problem-solving initiative should stand out. In sum, we expect that high-conscientious justices are more likely than low-conscientious justices to vote to grant review to cases that present such legal conflict.

Following Black and Owens (Reference Black2009) and Black et al. (Reference Black2020), we test this hypothesis with a random sample of 360 paid, non-death-penalty petitions appealed from the federal court of appeals that made the Court’s discuss list during the 1986–93 terms.Footnote ²¹ From these 360 petitions we recovered a total of 3,024 individual justice votes. The data on the justices’ votes originate from the digital images of Justice Blackmun’s docket sheets, which we retrieved from Epstein, Segal, and Spaeth (Reference Epstein2007).

Dependent Variable. The dependent variable, Grant, measures whether each justice cast a vote to grant (1) or deny (0) review to each certiorari petition in the sample.

Conscientiousness. We focus on each justice’s conscientiousness; larger values correspond to greater conscientiousness. We explore the empirical results using the SCIPE scores from Hall et al. (Reference Hall2021) and our own indicators (Black et al. Reference Black2020). Our personality measures are derived using the text of a justice’s published articles, public speeches, and lower court opinions penned prior to his or her confirmation. We translated these texts into personality trait scores using WPI while introducing a novel standardization process to account for differences in personality content across document type.Footnote ²² For this article, we have updated these measures to incorporate Justices Gorsuch, Kavanaugh, and Barrett. We also control for justices’ scores on the four other personality trait dimensions: openness, extraversion, agreeableness, and neuroticism.

Legal Conflict. We create four dummy variables to measure the presence and extent of lower court conflict within the case. We derive these measures from reading the cert pool memo in each case.Footnote ²³ Strong Conflict represents instances when the pool memo writer acknowledges a clear and deep split among the lower courts. Weak Conflict is present when the law clerk, while assessing the presence of alleged conflict, suggests that immediate review may not be necessary. Alleged Conflict occurs if the petitioner in the case alleges a conflict among lower courts but the pool memo writer denies the existence of this conflict. No Conflict is the baseline category in our models, and it represents instances where the petitioner did not allege any legal conflict among the lower courts.

With these four binary variables in hand, we then interact them with conscientiousness. We also control for multiple factors that previous research has shown to influence the policy and legal motivations behind Supreme Court agenda setting (see, e.g., Black and Owens Reference Black2009). (See the appendix for full coding details.)