Ghostwriters

Most obviously, much of the language in a judicial opinion was not crafted by the justice who ostensibly authored the opinion. This ghostwriter problem manifests in Supreme Court opinions in at least three ways.

First, opinion authors often quote or closely paraphrase language from lower court opinions or prior judicial rulings in order to explain case facts and relevant precedents. This problem is most prevalent in majority opinions (opinions of the Court and judgments of the Court), which “tend to be relatively stylized and formulaic,” and “routinely and boringly recite case facts and procedural history as boilerplate” (Wahlbeck, Spriggs, and Sigelman Reference Wahlbeck2002, 179). Thus, we exclude majority opinion language from our estimation of justice personality traits in order to avoid this borrowed language.

Second, some opinions are the result of negotiation among the justices, as they bargain over opinion content (Wahlbeck, Spriggs, and Maltzman Reference Wahlbeck1998). Because the support of at least five justices is necessary for a majority opinion to set precedent, the authors of majority opinions are the most likely to accommodate bargaining requests from other justices (Maltzman, Spriggs, and Wahlbeck Reference Maltzman2000). In contrast, the authors of separate opinions have less incentive to accommodate such requests. For example, one appellate court judge has described writing separate opinions as a “feeling of unjudicial glee as one shucks off the normal restraints of writing for a panel and proceeds to thrust and parry with gay abandon” (Coffin Reference Coffin1994, 227). However, on the Supreme Court it is possible for more than one justice to dissent, and dissenting justices often join together to support a single opinion that serves as a sort of rival majority opinion (often called a “lead dissent”; see, e.g., Driver Reference Driver2011, 1268; Karlan Reference Karlan2018, 149). Indeed, the senior justice in the minority sometimes even “assigns” the lead dissent to a specific justice (see, e.g., Toobin Reference Toobin2012, 259). Consequently, the authors of dissenting opinions (opinions dissenting, dissenting in part, or concurring in part and dissenting in part)—especially long dissents, which contain the most opinion language—often face incentives to accommodate other justices.

In contrast, we suspect that the authors of concurring opinions (opinions concurring, concurring in part, or concurring in the judgment) face weaker incentives to accommodate bargaining requests than do the authors of majority or dissenting opinions. And the extant literature supports the notion that concurring opinions are the least susceptible to bargaining. For example, prior research has found that the size of the majority coalition is associated with the clarity of majority and dissenting opinions, but not concurring opinions (Owens and Wedeking Reference Owens2011, 1050). And the rhetorical content of majority and dissenting opinions is more influenced by strategic interactions with other justices than is the content of concurring opinions (Hume Reference Hume2006, 837). Additionally, unlike majority and dissenting opinions, neither the clarity nor the rhetorical content of concurring opinions is influenced by the opinion author’s ideology (Hume Reference Hume2006, 837; Owens and Wedeking Reference Owens2011, 1050). This is not to suggest that strategic considerations have no influence on the content of concurring opinions. Attracting support for a concurrence almost never has any precedential value, but justices might still alter the language in their concurring opinions based on strategic considerations. For example, Corley, Howard, and Nixon (Reference Corley2005, 336–37) found that concurring opinion authors respond to the content of majority opinions (though this pattern of strategically responding to other opinions is just as strong among dissenting opinion authors and even stronger among majority opinion authors). Nonetheless, taken together, the literature suggests that the influence of bargaining and negotiation is likely weaker—and is certainly no stronger—in concurring opinions than it is in majority or dissenting opinions. Thus, we derive our personality estimates from the justices’ language in only their concurring opinions.

Third, because justices often rely on their law clerks to assist in drafting opinions, clerks may sometimes influence opinion language (Wahlbeck et al. Reference Wahlbeck2002). For example, Ward and Weiden (Reference Ward2006) found that 38% of surveyed law clerks thought they had been able to change the minds of their justices regarding opinion content (see also Peppers Reference Peppers2006; Peppers and Ward Reference Peppers2012). Therefore, opinion language may also reflect the clerks’ personalities. However, two aspects of prior research suggest that clerk influence poses only minor concerns for our estimates.

Most importantly, Wahlbeck et al. (Reference Wahlbeck2002, 179) find that “opinions drafted by different clerks for the same justice [are] significantly more homogeneous than different-justice pairs but not significantly less homogeneous than same-clerk pairs.” In other words, opinions authored by the same justice are significantly more similar to each other than to those authored by different justices; however, opinions drafted by the same clerk for a justice are not significantly more similar to each other than opinions drafted by different clerks for the same justice. This finding—which the authors describe as “especially pertinent”—suggests that, regardless of the clerks’ influence, justices retain primary control over the language in their published opinions.

Moreover, clerks’ influence should be limited to the time they serve as clerks—usually a single term. Thus, estimating career-level personality scores for each justice mitigates the concern that individual clerks influence our estimates. Indeed, two studies that purport to establish clerks’ influence over opinion language do so by demonstrating that the content of the justices’ opinions became more variable after the mid-20th century (the period of our analysis) than it was before that time (when justices relied less on clerks; Rosenthal and Yoon Reference Rosenthal2011; Carlson, Livermore, and Rockmore Reference Carlson2016). Thus, any clerk influence should simply increase the noise in our estimates.

The only way clerk influence might systematically bias our estimates is if a justice consistently selected clerks with certain personality traits. However, such a practice may itself reflect the justice’s personality. And, if justices consistently selected clerks with traits they did not themselves possess, our estimates may still illuminate the justices’ behavior even without accurately capturing their traits. For example, imagine a justice who consistently selected clerks who were low in neuroticism because the justice wanted clerks to inject emotional stability into his or her chamber. As a result, we might expect these clerks to systematically influence the language in the justice’s opinions, making the justice appear less neurotic than he or she actually is. However, we should also expect these clerks to systematically influence the justice’s other behavior, possibly making the justice act less neurotic when making critical decisions. Consequently, one might interpret SCIPEs as reflecting the personality of a justice’s chamber rather than the individual justice. And, since we are primarily interested in the official decisions that emerge from the justice’s chamber (rather than, say, a justice’s behavior at a private party), such estimates may be especially valuable.

Finally, we would note that almost any publicly available source of elite language, such as speeches, journal articles, or legal briefs, is crafted with the assistance of staff or peers. For example, law clerks even influence the questions that justices ask during oral arguments (Johnson, Stras, and Black Reference Johnson2014). Thus, save for access to the justices’ private diaries or personal correspondence, the ghostwriter problem is largely unavoidable.

Multiple-Source Bias

Next, if justices use different language in different types of opinions, then personality estimates based on multiple opinion types may be endogenous to their behavior. For example, if justices tend to use less agreeable language in dissenting opinions, then those justices who wrote more dissenting opinions may appear to be less agreeable due to the measurement process rather than their actual personalities. Our estimation process seeks to minimize this multiple-source bias by examining only concurring opinions. However, multiple-source bias may yet affect our estimates because our corpus of language includes different types of concurring opinions: our analysis combines opinions that were joined by other justices with those that were not, concurrences geared toward the general public with those aimed at niche audiences, and brief concurrences making minor comments with lengthy diatribes urging alternative theories of legal interpretation. And the extant literature has established that concurring opinions vary widely in content based on factors such as workload, the number of justices concurring, and agreement with the majority opinion author (see, e.g., Way and Turner Reference Way2006; Corley, Steigerwalt, and Ward Reference Corley2010).

Therefore, combining language from different types of concurrences to estimate justice personality traits may also introduce multiple-source bias in our estimates. On the one hand, we acknowledge that this criticism is valid and the potential for such bias certainly exists. On the other hand, we doubt that any corpus of language could entirely avoid such a problem. Any attempt to completely avoid every possible type of multiple-source bias would face serious measurement problems and, inevitably, reduce the corpus to a small and unrepresentative amount of text. Ultimately, we contend that an appropriate measurement strategy must weigh the costs and benefits of combining multiple types of text. We strike a balance between these competing interests by eliminating the multiple-source bias that is most obviously problematic and would likely bias analyses of interest (i.e., bias from combining majority and minority opinions). For example, we anticipate scholars using our estimates to test the influence of personality on the decision to dissent or the decision to file a separate opinion, and our estimates are suitable for such tasks. In contrast, it would be inappropriate to use our estimates to test the influence of personality on the more nuanced decision to write a regular concurrence versus an opinion concurring in the judgment. But we also doubt there would be substantial interest in such a research question. Thus, while multiple-source bias is not entirely avoidable, our approach minimizes the problem for most analyses of interest.

Strategic Manipulation

Finally, justices may strategically manipulate the language in their concurring opinions in order to present themselves in a more favorable light. In his groundbreaking book, Judges and Their Audiences, Lawrence Baum illustrates the various ways in which justices alter their behavior to enhance their reputation, prestige, and prominence among the media, the public, elected officials, academics, and other judges (Reference Baum2006; see also Posner Reference Posner1993; Miceli and Cosgel Reference Miceli1994; Gulati and McCauliff Reference Gulati1998; Epstein and Knight Reference Epstein2013). Altering opinion language may be one of the most low-cost strategies a justice can employ to achieve such goals. Additionally, justices may alter their opinion language to evade congressional review (Owens, Wedeking, and Wohlfarth Reference Owens2013) or mitigate public backlash for unpopular rulings (Black et al. Reference Black2016). Therefore, strategic manipulation of language may bias our estimates.

However, almost all publicly available language is potentially subject to strategic manipulation. Speeches, journal articles, and legal documents are almost always crafted in an effort to attract praise and persuade others. In the absence of access to truly private writings, such as personal diaries, strategic manipulation is always a concern for text-based personality estimates. Yet such strategic manipulation would only bias our estimates if justices can accurately anticipate the relationship between the psycholinguistic characteristics of their writing and their latent personality traits. It seems highly unlikely that justices consider, for example, the number of six-letter words or commas used in their writing in an effort to shape their reputation. Moreover, text-based estimates should always be compared to their most common alternative—self-reported survey responses. And survey-based research has found that political elites tend to report more “desirable” personality traits, such as low neuroticism and high openness, conscientiousness, extraversion, and agreeableness (Dietrich et al. Reference Dietrich2012). Ultimately, we believe that strategically manipulating one’s revealed traits is arguably more difficult to maintain through nuanced adjustments to language over decades of opinion writing than while completing a personality survey over a half hour.

We do not mean to suggest that our focus on concurring opinions is without limitations. Instead, we wish to highlight these limitations so that scholars will recognize the strengths and weaknesses of our estimation approach and use our measures appropriately (e.g., scholars can use our estimates to predict whether justices file concurring opinions with minimal concerns about endogeneity, but our estimates should not be used to predict the type of concurring opinions that justices file). Moreover, as we emphasize below, because all of the text and models we utilize to create our estimates are fully transparent and publicly available, we invite scholars to develop alternative estimates that might be more appropriate for use in specific contexts. Our only goal in this article is to establish the feasibility of measuring judicial personality from text. We leave to future work the question of which source of text (or combination of sources) is best suited to this task.

Alternative Sources of Justice Language

Finally, despite the admitted limitations of using concurring opinions, the use of alternative sources of language is fraught with more substantial challenges. Here, we consider four alternative sources of justices’ language: public speeches, testimony during confirmation hearings, oral arguments, and preconfirmation documents. We argue that these alternative sources pose serious availability problems as well as the ghostwriter problem, multiple-source bias, and the potential for strategic manipulation.

First, many justices deliver public speeches on legal, political, or social issues. However, the availability of these speeches varies widely depending on the type of venue in which the speech was delivered, the willingness of the justice or the justice’s hosts to share the speech in an accessible format, and the time period during which the speech was delivered. Indeed, the text of public speeches is completely unavailable for many justices. Public speeches may also suffer from the ghostwriter problem if justices receive help in crafting their remarks from speechwriters, clerks, or other individuals. Public speeches are also susceptible to multiple-source bias because they vary widely in audience and context (e.g., a speech to a graduating class vs. a speech to the Federalist Society). And justices almost certainly craft their public speeches in an effort to strategically manipulate their perception among elites, the media, and the public (see Baum Reference Baum2006).

Justices also sometimes speak during their confirmation hearings; however, congressional testimony also varies widely in availability. Many justices spoke very briefly during their hearings, and many had no hearing at all. The testimony that does exist from justices’ confirmation hearings is also susceptible to the ghostwriter problem because presidential aides and allied senators heavily coach nominees on their testimony. And confirmation testimony is susceptible to multiple-source bias because confirmation hearings vary widely in nature. Consider, for example, the confirmation hearings for Justices Thomas and Kavanaugh—both of whom angrily defended themselves against charges of sexual harassment—compared to hearings for justices who never faced such charges. The differences in the justices’ language during these hearings was undoubtedly influenced by the types of questions they faced and, consequently, combining language from different types of hearings would bias the estimates. And confirmation hearings are obviously rife with strategic manipulation, as justices try to persuade senators and the public to support their confirmation.

Most modern-day justices regularly speak during oral arguments; however, justices rarely did so prior to the 1970s and were not individually identified in oral argument transcripts prior to 2004.Footnote ² As noted above, even if fully available, this language would also face the ghostwriter problem given that clerks influence the questions that justices ask during oral arguments (Johnson et al. Reference Johnson2014). Additionally, oral arguments would suffer from the same multiple-source bias that affects opinion language. That is, the questions come from justices who plan to or will eventually write the majority opinion, join the majority, dissent, or write or join a concurrence, and combining questions from different types of justices would create problems similar to those that arise when combining language from different types of opinions. Finally, because oral arguments offer justices a valuable opportunity to influence one another, justices alter their questioning in an effort to persuade their colleagues (Johnson Reference Johnson2004; Jacobi and Schweers Reference Jacobi2017); therefore, the language in oral arguments may also be strategically manipulated.

Finally, justices produce a considerable number of writings before their nomination to the Supreme Court, such as lower-court opinions, academic articles, and public speeches. But these writings suffer from similar problems. Prenomination writings vary widely in availability. Ghostwriters in the form of law clerks, coauthors, and speechwriters influence the language in these documents. Multiple-source bias affects each type of prenomination writing just as it does postnomination writings, and combining different types of writings would only compound this bias. And, undoubtedly, justices strategically manipulated their prenomination writings—not just to achieve proximate goals, such as evading review by a higher court, satisfying journal editors, or impressing an audience, but also to help the justice win a seat on the high court.

To be clear, we do not contend that concurring opinions are the only legitimate source of language from which justices’ personality traits might be validly estimated; our argument is far more modest. We contend that concurring opinions are at least no worse than other possible sources of the justices’ language and offer some clear advantages over alternative sources, including wide availability, a relatively consistent context, and some easily avoidable biases (e.g., identifying majority vs. dissenting opinions). Most importantly, we believe the goal of text-based estimation approaches is not to establish any one source of language as undeniably superior to its alternatives, but rather to directly and transparently identify the relative advantages and disadvantages of the approach so that scholars can be sure to use the estimates with care and caution.

Overview

We used the Personality Recognizer (Mairesse et al. Reference Mairesse2007) to generate SCIPEs for each Big Five personality trait (openness, conscientiousness, extraversion, agreeableness, and neuroticism) based on all of the justices’ concurring opinions written throughout their careers.Footnote ³ The use of language to capture individual differences (among various psychological constructs) has been validated in a series of experiments using Linguistic Inquiry Word Count (LIWC) and other computerized textual analysis methods (Tausczik and Pennebaker Reference Tausczik2010). Similarly, the Personality Recognizer was developed using Pennebaker and King’s (Reference Pennebaker1999) corpus of over 1.9 million words from laboratory experiments by comparing written and spoken language to self-assessed and observed personality ratings and then training machine learning models to estimate personality traits.Footnote ⁴

We first processed the language through the 2001 versions of both LIWC and the MRC Psycholinguistic Database (MRCPD) in order to collect word counts and proportions across all LIWC and MRCPD categories.Footnote ⁵ We then standardized these results and inserted them into Mairesse et al.’s (Reference Mairesse2007) pretrained models.Footnote ⁶ Finally, we generated 95% confidence intervals using 1,000 sentence-level bootstrap replications (Lowe and Benoit Reference Lowe2011). Specifically, we assume each justice j uses N_j sentences during their tenure. For each justice, we resampled N_j sentences with replacement from their corpus of language during their career.

Skeptical readers might worry that applying this approach to legal writing is inappropriate because the underlying models were trained using unstructured essays. However, two factors mitigate this concern. First, this approach has been employed to measure personality traits in a variety of contexts. For example, Ramey et al. (Reference Ramey2016) estimate personality scores for members of the US Congress based on their floor speeches and find that the scores are associated with a variety of congressional behavior. Second, if anything, the artificial uniformity of legal writing should obscure individual writing styles and make it more difficult to uncover true personality traits. Therefore, applications to legal writing should only bias results against finding any effects.

Importantly, the distributions of linguistic features in our opinion corpus are not radically different from those in the training data. The correlation between the mean LIWC category usage from the Pennebaker and King (Reference Pennebaker1999) data and the mean usages in our corpus is approximately .991, and Spearman’s ρ is about .789. Thus, while the subject matter is different, justices’ usage of the various linguistic features does not differ substantively from the laboratory writers. Moreover, the estimates are precise. Figure 1 presents the 95% bootstrapped confidence intervals for each justice-trait on the trait-specific scale. As can be seen, the confidence interval widths are typically about one standard deviation (SD), meaning the 95% intervals are approximately the point estimate ± one-half SD. Thus, we can have confidence in the precision of our estimates.

Figure 1. SCIPEs and 95% confidence intervals.

Horizontal lines indicate 95% bootstrapped confidence intervals.

Face Validity

To assess the face validity of our estimates, we test whether the relationships between SCIPEs and judicial ideology are consistent with those found in the political psychology literature, much of which indicates that liberalism is positively associated with openness and neuroticism, and negatively associated with conscientiousness (Gerber et al. Reference Gerber2011). Evidence for the other traits is more mixed. For example, Mondak et al. (Reference Mondak2010) found that liberalism was not associated with extraversion and only weakly associated with agreeableness. But Gerber et al. (Reference Gerber2011) found that liberalism was negatively associated with both traits among the mass public. And, among elites, Ramey et al. (Reference Ramey2016) found positive relationships between liberalism and both extraversion and agreeableness.

We use two dependent variables to test the relationship between SCIPEs and justice ideology—the proportion of cases during this time period in which a justice voted in the “liberal” direction (Spaeth et al. Reference Spaeth2014) and the justice’s Clerk-Based Ideology (CBI) scores (Bonica et al. Reference Bonica2017), which are estimates of judicial ideology based on clerk hiring behavior.Footnote ⁷ Our independent variables are our SCIPEs, along with justices’ gender and birth year, as the latter have well-known correlates with personality traits. We estimate three models: a binomial logit model with the proportion of cases in which each justice voted in the liberal direction as the dependent variable, and two linear regression models with the CBI scores as the dependent variable.Footnote ⁸ Results are presented in table 1.

Table 1.

Justice Personality and Ideology

Note.—The dependent variable in the Liberal Direction Proportion model is the proportion of cases in which each justice voted in the liberal direction during the time under analysis. The dependent variable in the CBI models is each justice’s CBI score, inverted so that positive values indicate higher degrees of liberalism. Fit statistics for the “Overimputed” model are the mean values for all individual imputed models. Observations are at the justice level. Standard errors in parentheses. Two-tailed significance tests.

* p < .05.

While the voting model attains higher levels of statistical significance across all traits, in part due to the higher effective number of observations, all models produce results consistent with each other, and consistent with our expectations (where significance is attained). Consistent with prior research, openness, agreeableness, and neuroticism (the last of which attains significance in all models) are associated with liberalism, and conscientiousness is associated with conservatism.Footnote ⁹ The positive relationship we find between extraversion and liberalism contrasts with Mondak et al.’s (Reference Mondak2010) null results and Gerber et al.’s (Reference Gerber2011) opposite findings, but it comports with Ramey et al. (Reference Ramey2016). Overall, our results are consistent with the literature and support SCIPEs’ validity. Next, we turn to two empirical applications to test SCIPEs’ predictive validity.