Updating the PAJID scores

To update PAJID scores, we replicate Brace, Langer, and Hall’s (Reference Brace, Langer and Hall2000) original methodology using more recent data.Footnote ³ First, we identified 1,666 unique individuals who worked on each state supreme court between 1970 and 2019 in the 50 American states. We then coded a dichotomous variable equal to “1” if a justice was a Democrat at the time of their selection, “0” otherwise. We also identified whether a state supreme court selects its members via popular election or elite appointment at the time a justice was selected.

The final variable in our PAJID update relies upon Berry et al.’s (Reference Berry, Fording, Ringquist, Hanson and Klarner2010) measure of state citizen and elite ideology in a given state and year (“Berry scores”).Footnote ⁴ Berry scores are measured on a scale from 0 to 100, where smaller values represent conservatism and larger values represent liberalism. For states using elective judicial selection methods, we incorporate Berry et al.’s (Reference Berry, Fording, Ringquist, Hanson and Klarner2010) citizen ideology value, and for states using appointive selection methods, we incorporate Berry et al.’s (Reference Berry, Fording, Ringquist, Hanson and Klarner2010) elite ideology value. We label either of these measures as a “preferences” indicator.

The first step in calculating the PAJID scores is to estimate a logistic regression that models the likelihood a given judge, $ j $ , is a Democrat given the preferences of their selectors:

(1) $$ \hat{\mathit{\Pr}}\left( Democra{t}_j=1\right)={\Lambda}^{-1}\left({\hat{\beta}}_0+{\hat{\beta}}_1 Preference{s}_j\right). $$

Using Equation (1), we then calculate the predicted probability a given judge is a Democrat, $ {\hat{p}}_j=\hat{\mathit{\Pr}}\left( Democra{t}_j=1\right) $ . Next, using $ \hat{p} $ , we calculate a pseudo-residual that is the difference between a justice’s partisanship and the predicted probability they are a Democrat:

(2) $$ {\tilde{u}}_j= Democra{t}_j-{\hat{p}}_j. $$

Equation (2) simply measures the degree to which the preferences of a given judge’s selectors fail to account for their partisanship.

Finally, a justice’s PAJID score is calculated accordingly:

(3) $$ PAJI{D}_j=\left( Preference{s}_j\hskip0.35em \ast \hskip0.35em {\tilde{u}}_j\right)+ Preference{s}_j. $$

The logic of Equation (3) is as follows. If the preferences of selectors in Equation (1) perfectly predicted partisanship, we would arrive at a value of $ {\tilde{u}}_j=0 $ , and no adjustment would be necessary to an individual’s PAJID score given the preferences of their selectors. Now, provided we find some $ {\tilde{u}}_j>0 $ , this would mean that we have a Democratic judge, but our model under-predicted the likelihood of them being a Democrat. Equation (3) would then add to a judge’s preferences an amount proportional to the size of the error in $ {\tilde{u}}_j $ . A similar logic holds for calculating the preferences of Republican judges with $ {\tilde{u}}_j<0 $ .

Assessing the validity of the updated PAJID scores

In Figure 1, we present median PAJID scores for each state supreme court between 1970 and 2019. Geographic and temporal trends speak to the face validity of the updated scores. For example, southern justices are approximately 21.6 percent less liberal compared to their counterparts in northern states ( $ t=7.53\Big) $ . And across all states and years, Democratic justices have a mean PAJID score of 68.0 compared to just 19.2 among Republicans – a 254.7 percent difference ( $ t=68.2 $ ). Given that the updated PAJID scores comport with one’s general expectations and knowledge of state politics and partisan power, we conclude they are facially valid measures of state supreme court ideology.

Figure 1.

Median state supreme court PAJID scores by state (1970–2019).

Beyond facial validity, we also examine convergence validity in our updated PAJID estimates. Convergence validity assesses whether a given measure of a concept is associated with other common measures of that concept. To do so, we examine how PAJID scores compare with (1) Partisanship, (2) Ideology as measured by Bonica and Woodruff (Reference Bonica and Woodruff2015), and (3) Ideology as measured by Windett, Harden, and Hall (Reference Windett, Harden and Hall2015). We present the results of this analysis in Table 1, which contains correlation coefficients among the variables of interest.

Table 1.

Convergence validity analysis of state supreme court justice preferences

Note: Table entries represent Pearson’s correlation coefficients.

First, the results demonstrate a strong, positive correlation between a justice’s PAJID score and partisanship. This is intuitive given that PAJID scores are calculated using a judge’s partisanship. Next, we find that that PAJID scores are negatively associated with Bonica and Woodruff (BW) and Windett, Harden, and Hall’s (WHH) estimates. This is also expected, given that PAJID is measured on a conservative-to-liberal scale, while the other two are measured on a liberal-to-conservative scale. It is worth noting that the strength of association between Bonica and Woodruff (BH) and Windett, Harden, and Hall’s (WHH) scores are relatively higher compared to either’s association with PAJID. This likely reflects the differences in how these measures are estimated compared to PAJID. Even still, the degree of strength that PAJID correlates with these more recent ideological scores indicates moderate convergence.

Our final validity check examines the construct validity of the updated PAJID scores. Construct validity assesses whether a given measure is associated with outcomes in a theoretically related concept. Drawing upon literature related to the attitudinal and strategic models which hold that judges’ votes in cases are a function of their policy preferences, we examine whether updated PAJID scores are reliable predictors of judicial behavior. We also consider how favorably they compare to other, more recent ideological indicators.

For this analysis, we used a novel dataset of state supreme court cases related to abortion and capital punishment. To populate our sample, we searched Westlaw using keycite terms related to abortion and death penalty cases heard in the state supreme courts between 1970 and 2018. Our data include 213 cases (139 abortion and 74 death penalty), with a total of 1,511 judge-votes. Each justice’s vote is coded as either liberal (pro-abortion or anti-death penalty) or conservative (anti-abortion and pro-death penalty). Among all votes, 52.3 percent were in a conservative direction, while 47.8 percent were in a liberal direction.

Next, we employed logistic regression to model the likelihood a state supreme court justice cast a liberal vote in a case. Our primary focus is the effect of a justice’s policy preference upon their vote. We estimate four separate logistic regressions using a different preference measure in each to predict the directionality of a justice’s vote.Footnote ⁵ The results from these models appear in Table 2.

Table 2.

Construct validity analysis of justice voting across four preference measures

Note: Table entries represent logistic regression coefficients (standard errors in parentheses). The dependent variable is whether a justice cast a liberal vote in a given decision (“1” if yes, “0” otherwise).

* denote statistical significance ( $ p<0.05 $ ).

Our results demonstrate that the updated PAJID scores, in addition to each of the other three measures, are significantly associated with judicial voting behavior – a good indication of construct validity. There are comparable proportional reductions in error (PRE) across all four models, which would seem to indicate that each preference measure captures a similar phenomenon. Given heterogeneous sample sizes, we also examine the change in the predicted probability of a liberal vote across each model. For the sake of comparison, we examine changes in the predicted probability given a shift in a preference measure from its minimum to its maximum.

The simplest measure under consideration is partisanship. A shift from minimum to maximum partisanship (Republican to Democrat) is associated with a 15.0 percent predicted change in the probability that a state supreme court justice casts a liberal vote, ceteris paribus. Next, a change from PAJID’s minimum to maximum is associated with a 23.1 percent predicted change in the probability of a liberal vote. A similar shift in Bonica and Woodruff’s (BW) measure is associated with a 99.6 percent predicted change in vote choice. Finally, a similar change in Windett, Harden,651 and Hall’s (WHH) measure is associated with a 409.0 percent change in vote choice.

From the above results, we can reaffirm Windett, Harden, and Hall’s (Reference Windett, Harden and Hall2015) conclusion regarding the efficiency of those scores compared to other alternatives. Nevertheless, WHH scores are only available for 25.6 percent of the observations in Table 2. While Bonica and Woodruff’s (Reference Bonica and Woodruff2015) measure is the next most discriminating, it also has limits given sparse availability prior to 1990. Consequently, we conclude that if either of these former measures is of limited availability, PAJID scores are more efficient compared to rote partisanship and are of sufficient predictive power to offer additional robustness checks on more sophisticated measures.