Two Approaches to Measuring Personality Traits of US Supreme Court Justices

While there are obvious methodological disagreements between Hall et al. and Black et al. they share a number of substantive and methodological commitments. For instance, both view personality psychology as important for the study of judicial behavior. They also share a focus on the so-called Big Five personality traits (or one of these five traits, in the case of Black et al.). Finally, both sets of authors are committed to using the legal writing of the justices as the primary source of information about the personality traits of the justices.Footnote ¹

Hall et al.’s SCIPE measure relies on a pretrained support vector machine regression from Mairesse et al. (Reference Mairesse2007) to predict each of the Big Five personality traits from features extracted from the text of each justice’s concurring opinions. The Mairesse et al. study relies on data from Pennebaker and King that consists of 2,479 essays from students in introductory psychology courses at the University of Texas and Southern Methodist University in the 1990s (Pennebaker and King Reference Pennebaker1999, 1306; Mairesse et al. Reference Mairesse2007, 463). These students were asked to write their thoughts in a stream of consciousness mode for 20 minutes. The students also completed a standard five-factor personality inventory that was used to measure their Big Five personality traits (Pennebaker and King Reference Pennebaker1999, 1306). Mairesse et al. extract syntactic and semantic features from these essays and then train machine-learning methods to predict the personality traits from the linguistic features extracted from the essays.

Hall et al. take each justice’s concurring opinions, extract the syntactic and semantic features that are the inputs to one of the machine-learning methods employed by Mairesse et al.,Footnote ² insert those features into the mappings that were estimated from the undergraduate psychology student data by Mairesse et al., and take the resulting outputs as the estimates of the personality traits of the justice (i.e., the justice’s five SCIPE scores).

Hall et al. go on to assess the face validity of their SCIPE measures by looking at associations between the SCIPE measures and two measures of judicial ideology. They argue that the associations previously found between Big Five personality traits and political liberalism among members of the mass public should also be present among Supreme Court justices. Hall et al. close with two substantive applications that make use of their SCIPE measures.

As noted above, Black et al. are sympathetic to many aspects of the Hall et al. research agenda. The primary points of disagreement between Black et al. and Hall et al. relate to (a) the validity of Hall et al.’s SCIPE measures, (b) what is the best source of text data from which to infer personality traits, and (c) what sorts of machine-learning methods should be used to infer personality traits from text.

Like Hall et al., Black et al. begin their assessment of the validity of Hall et al.’s SCIPE measures by examining the associations between SCIPE scores and judicial ideology. They depart from Hall et al.’s approach by looking at four additional measures of judicial ideology and by checking to see whether associations between SCIPE scores and these measures of judicial ideology change when adjusting for the justice’s sex and birth year. What they find is that the SCIPE measures tend not to be associated with these other measures of judicial ideology in the expected ways, and when they do exhibit a significant association in the expected direction, that significance depends on whether age and birth year are entered as controls. Black et al. conclude that “these findings show that the SCIPEs do not appear to provide valid estimates of Supreme Court justices’ personality traits” (Reference Black2021, 377).

Black et al. go on to question whether concurring opinions are the best source of data for inferring personality traits of Supreme Court justices. They note that a concurring opinion is written in reaction to a majority opinion and is thus “substantially frame[d] and constrain[ed]” by the majority opinion (Reference Black2021, 379). Indeed, they show that opinion-level personality content is more similar between the majority opinion and a concurring opinion on the same case than between concurring opinions written by the same author (380).

Drawing on work by Corley (Reference Corley2010), Black et al. also note that concurrences are written for different purposes. Black et al. break up each justice’s concurring opinions into the categories identified by Corley and then use Hall et al.’s methods to calculate six different versions of SCIPEs for each justice—each based on a different subset of concurrences. They find that the estimated personality traits exhibit a great deal of within-justice variability that is driven by the subset of the concurrences used as input data, and this variability doesn’t have a consistent pattern.

While they are critical of Hall et al.’s SCIPE measures, Black et al. see value in text-based measures of personality traits, and they go on to discuss desiderata for such measurement approaches. They conclude with a substantive example that examines the conditional relationship between lower court legal conflict, justice-specific conscientiousness, and votes to grant certiorari. Here they use the SCIPE measure of conscientiousness as well as their own measure of conscientiousness derived from an alternative machine-learning approach. They demonstrate that the estimated relationship is different depending on which measure of conscientiousness is used.

General Thoughts on Measuring Personality Traits of Supreme Court Justices

As both Hall et al. and Black et al. note, there is ample evidence to suggest that Supreme Court justices have distinctive personalities and that, at least in some cases, these personality traits affect the functioning of the Court. However, the fact that we can point to such anecdotal evidence doesn’t necessarily mean that attempts to measure the personality traits of justices will have a major impact on the study of judicial behavior.

As I see it, there are four basic challenges that researchers interested in the personality traits of Supreme Court justices should consider:

1. 1. Personality traits are widely thought to be stable once an individual reaches adulthood. They are thus essentially immutable characteristics if the unit of analysis is the individual justice. This raises subtle issues about how personality can, and should, enter into causal explanations of judicial behavior. Given this, how should we conceptualize the role of personality traits on judicial behavior, and how should personality enter into our theoretical explanations of judicial behavior?

2. 2. Are the Big Five traits the right personality traits to focus on?

3. 3. Accurately measuring the personality traits of a small number of elites who have had to pass through many filters, some of which plausibly filter on personality, is different from measuring the personality traits of members of the mass public. How comfortable should we be in extrapolating from studies of the mass public to a highly idiosyncratic set of elites?

4. 4. It seems very difficult to truly validate any text-based measure of a justice’s personality traits. How can we be sure that a proposed measure of personality is actually measuring what it is purported to measure?

The fact that personality traits are widely thought to be stable once an individual reaches adulthood implies that personality traits cannot be plausibly manipulated at the individual justice level. Thus, one needs to be very careful when talking about the “causal effect” of neuroticism, conscientiousness, and so forth on any outcome (see Holland Reference Holland1986; Greiner and Rubin Reference Greiner2011). The counterfactual of making Justice McReynolds more open-minded and agreeable is no more plausible than the counterfactual of changing Justice McReynolds’s sex at birth.

I see two ways that personality traits (and other immutable characteristics) might enter into an explanatory or causal account of judicial behavior. The first is to think of personality traits as characteristics that are associated with the direction and size of heterogeneous treatment effects of some other (at least hypothetically) manipulable treatment variable on some outcome. Seen this way, personality traits may provide a concise summary of how the effect of some other variable varies across different types of justices.

While they don’t frame things quite this way, Black et al.’s application looking at lower court conflict, conscientiousness, and cert votes could be reframed this way. Here the primary question is how lower court conflict affects cert votes. Conscientiousness enters in as a way to describe how the strength of this relationship between lower court conflict and a vote in favor of granting cert varies across justices with different levels of conscientiousness.

The other approach is to reconceptualize what it means to intervene on and manipulate a particular personality trait. While contemplating an intervention at the justice level that only changes a justice’s personality and leaves every other personal characteristic intact is not, in my view, plausible, one could consider how personality traits are one of many things that the president and Senate select on when putting a justice on the Court. The fact that all nine justices hear each case and that there is limited turnover on the Court makes it difficult to go very far with this approach at the Supreme Court level. However, if high-quality measures of personality traits become available for federal appeals court judges, then one could easily imagine how one might exploit the random assignment of cases and judges to three-judge panels to investigate panel effects related to personality traits.

Finally, it is worth noting that if personality traits are not of direct substantive interest, the fact that they are specific to a justice and not varying over time suggests that they can be adjusted for with justice-specific fixed effects or random effects. Given the measurement issues discussed below, this simple approach has some benefits.

Both Hall et al. and Black et al. focus their attention on the so-called Big Five personality traits. The advantage of this is that the basic ideas behind the Big Five or the five-factor model have been around since the 1960s and there has been a large amount of work in psychology that makes use of the framework, typically at the mass level. That said, there has been criticisms of the Big Five/five-factor framework within psychology. These criticisms include questions about the underlying theoretical basis (or lack thereof) for the five-factor approach, issues relating to the methods used for scale development and measurement, and evidence that there are aspects of personality that do not fit easily within the five-factor framework. See Block (Reference Block2010) for an overview of these criticisms.

If scholars of judicial behavior do begin embracing concepts from personality psychology, it seems important that they make informed choices about how to use those concepts. To be clear, I am not claiming that either Hall et al. or Black et al. do not understand the limitations of the five-factor approach. What I am stating is that it is important for other, future researchers to fully understand the limitations of whatever concepts and measures they import.

Most of the work on the Big Five/five-factor framework in psychology takes place at the mass level. This is quite different from the focus of Hall et al. and Black et al., as these articles focus on a small number of elites (Supreme Court justices). This difference is important for at least two reasons.

First, the measurement approaches that one would use for, say, undergraduate psychology students will typically not be available for the justices. The measurement strategies that Hall et al. and Black et al. employ both use machine-learning methods that have been trained to detect the relationships between word usage and traditional measures of the Big Five traits among members of the mass public. They then use these mappings to infer justice-specific personality traits based on the word usage of the justices.

Note that such an approach will typically have more measurement error than the mass-level trait measures in the training set data. The reason for this that the mass-level measures are themselves estimates and will thus contain some amount of measurement error. The mapping from word usage to personality traits that is generated by the machine-learning method is actually a mapping from word usage to the mass-level estimates of the traits and not the traits themselves. This mapping is also an estimate, and has two sources of uncertainty: the measurement error that shows up in the mass-level estimates used in the training data and the estimation uncertainty from estimating the mapping. Accordingly, one would expect it to have more measurement error than the mass-level estimates that were used in the training data.

If one were certain that this measurement error was uncorrelated with other right-hand-side variables and the outcome of interest, then one might argue that this excess measurement error is not a serious problem since it will attenuate the regression coefficient estimates toward zero and is thus conservative. That said, it is not obvious to me that it would be reasonable to assume that the measurement error is uncorrelated with other variables in this way. In particular, if one includes all five of the personality traits on the right-hand side of a regression, it seems plausible that the measurement error will be correlated among the five traits.

Second, Supreme Court justices, at least in the modern era, go through an extensive vetting and selection process, and this can create problems for how researchers form hypotheses about judicial personality traits. Many of the hypotheses about the relationship between personality traits and either attitudes or behaviors are drawn from work at the mass level, where the selection effects, if any, are very different.

To see how this can create problems, consider the following hypothetical example. Suppose that Republican presidents want to put highly conscientious conservatives on the Court and that Democratic presidents want to put highly conscientious liberals on the Court. Further, both types of presidents are successful in doing so. If that is the case, then there would be essentially no relationship between true conscientiousness and ideology among the justices on the Court, even though there may well be a significant relationship between conscientiousness and ideology in the mass public. One could imagine other selection stories that give rise to any number of associations on the Court that are not present in the mass public.

This is related to the final point I want to make, which is that it is very difficult to do a true validation of any text-based measure of a justice’s personality traits. The approach of both Hall et al. and Black et al. is to assess face validity by checking to see whether associations found at the mass level also appear among the justices. However, as noted above, there is reason to be skeptical that mass-level associations between personality traits and attitudes such as political liberalism or ideology should be the same among the justices.

As Quinn et al. (Reference Quinn2010, 216) note, there are other types of measurement validity, and some of these could, in principle, be applied to personality trait measurement. However, some of these forms of validity, such as convergent and discriminant construct validity, involve comparisons to other measures of the construct of interest. Given the relative paucity of good measures of judicial personality traits, it seems difficult to do much with either form of construct validity. If one has a strong theoretical understanding of the expected relationship between the personality trait of interest and some other variable, then predictive validity and hypothesis validity become possible. However, it is not clear to me that our theoretical understanding of how personality operates within the confines of the Supreme Court is sufficiently well-developed for such approaches to be useful.