Although the literature on agenda-setting at the US Supreme Court is voluminous, justice-vote-level analyses of certiorari are limited in several respects. First, multivariate justice-vote-level analyses have largely been based on samples of cases from 1986 to 1993 (Benesh et al., Reference Benesh, Armstrong II and Wallander2020 [625 cases]; Black and Boyd, Reference Black and Boyd2012a [305 cases]; Black and Boyd, Reference Black and Boyd2012b [447 cases]; Black and Owens, Reference Black and Owens2009 [358 cases]; Johnson, Reference Johnson2018 [c. 300 cases]). As Lane and Black (Reference Lane and Black2017: 4) points out, this is because data from these terms are readily available as part of the Digital Archive of the Papers of Harry A. Blackmun (Epstein et al., Reference Epstein, Segal and Spaeth2007).
The limited temporal scope of these analyses is worrisome for two reasons. More obviously, there are questions about generalizability—whether the patterns demonstrated plausibly apply to other periods in the Court's history. This is not only of historical interest. Our lack of knowledge about the temporal stability of previous findings limits what we can say about how likely it is that the results demonstrated continue to hold today.Footnote 1
There is a less obvious implication of these studies’ narrow temporal scope: difficulty disentangling the attributes of justices from the identities of justices. The fewer justices there are included in the sample, the greater is the risk that observed justice-level characteristics are confounded with unobserved, potentially idiosyncratic influences associated with individual justices. The risk of such confounding is particularly high for covariates that vary minimally or not at all within justices, during a given time period—for example, ideology and similar variables. With only nine or so justices in a sample, effects attributed to these variables may be driven by a single justice; when this occurs, it is effectively impossible to determine whether one should credit the justice-specific covariate or a given justice's idiosyncrasies. Increasing the temporal scope of an analysis, and thus the number of justices included, reduces—although, as we will see, does not eliminate—the threat of this ambiguity arising.
Perhaps even more important than the limited temporal breadth of the modern literature on agenda-setting is that scholars have done very little to explore justice-level heterogeneity on certiorari. Sorely lacking are systematic examinations of how the effects of known influences on the Court's collective cert decision vary across justices—whether as individuals or as a function of observed characteristics. Although some research has assessed the impact of a few individual-level variables on the cert vote, little or no work has sought to explain how individual-level variables, including justice fixed effects, condition the effects of case-level variables on the cert vote. Accordingly, our primary objective here is to determine the extent of justice-level variability; secondarily, we seek to explain this variability as a function of justices’ attributes.
Finally, there is another issue related to sampling. Typically, in the studies mentioned above, cases are drawn from a set of dockets that have made the Discuss List—a relatively small, and over time shrinking, fraction of all cases. This, at the very least, raises potential inferential difficulties associated with selection bias. By considering all paid cases in our analyses—discussed or not—we avoid these statistical difficulties, and in addition present a more complete picture of the Court's cert decision.
1. Background
The major historical shift in the Supreme Court's agenda-setting was the Judges’ Bill of 1925, which gave the Court substantial control over its plenary docket for the first time. Thereafter, the Court promulgated formal rules about the criteria for granting certiorari, that is, discretionary review of lower courts’ decisions.
These criteria have remained largely unchanged from 1925 to the present, and thus over the terms we cover, 1939, 1968, and 1982. They are specified in Rule 10 today (Rule 38 in 1939), and are familiar. The Court prefers cases involving “decisions issued by either state supreme courts or US courts of appeals that conflict with either state courts of last resort, US circuit courts, or the Supreme Court,” as well as “decisions in which either state supreme courts or federal circuit courts decide an issue that has never been settled by the Supreme Court,” and, finally, decisions wherein “one of these courts departs from the accepted and usual course of judicial proceedings (Lane and Black, Reference Lane and Black2017: 5).”Footnote 2
Of course, scholars have long recognized that the formal rules do not constitute a complete explanation of certiorari. The Court frequently denies petitions that appear to meet the rule's criteria and grants others, absent those criteria. Clearly, other, informal forces are at work. Nonetheless, these formal criteria are a starting point and guide petitioners, as suggested by the leading volume on Supreme Court practice (Stern and Gressman, Reference Stern and Gressman1950; Shapiro et al., Reference Shapiro, Geller, Bishop, Hartnett and Himmelfarb2019; see also Robinson and Kirkham, Reference Robertson and Kirkham1936).
Though a unanimous denial is the most common outcome, we know there is disagreement among justices about whether a given case is “cert-worthy” (Caldeira et al., Reference Caldeira, Wright and Zorn1999). These votes, however, are not made public, except in the rare cases where one or more justices publicly dissents from a denial of certiorari, and, even then, one cannot infer that all those who did not publicly dissent favored a grant. Thus, as scholars, we have access to individual votes only if preserved in justices’ publicly archived papers. This has led directly to the rather limited temporal scope of the research on Court agenda-setting.
2. Previous research
The foundational piece in the modern literature on Supreme Court agenda-setting is Caldeira and Wright (Reference Caldeira and Wright1988). In a multivariate, Court-level analysis, it demonstrates that the Court is more likely to grant certiorari if (1) the United States is a petitioner; (2) the court below reverses the decision below it; (3) the legal question is associated with a live inter-circuit or other conflict enumerated in Rule 10; (4) the decision below is ideologically discordant with the CourtFootnote 3; and (5) amicus briefs are present on certiorari—whether in favor of review or in opposition. Caldeira and Wright (Reference Caldeira and Wright1990) additionally demonstrates that a dissent in the court below increases the likelihood of a cert grant, though not if the analysis is limited to discussed cases. The effects of these “standard factors” have been confirmed repeatedly in subsequent research, some based on justice-vote level data (see in particular Schoenherr and Black (Reference Schoenherr and Black2019), but also, e.g., Black and Boyd (Reference Black and Boyd2012b), Black and Owens (Reference Black and Owens2009, Reference Black and Owens2012), and McGuire and Caldeira (Reference McGuire and Caldeira1993)).
Yet, as we noted above, the extent to which the effects of these standard factors may vary, that is, be heterogeneous, has not been systematically explored. The occasional study has assessed whether the effect of a single factor is conditional on a case-level variable (e.g., Black and Boyd (Reference Black and Boyd2012b: 300) argues, though does not formally test, that the effect of amicus briefs is conditioned by litigant status). But, explorations of whether the factors influencing a justice's vote are heterogeneous as a function of a justice's identity or characteristics have been limited to studies of votes on the merits (e.g., Segal and Spaeth, Reference Segal and Spaeth1996; Collins, Reference Collins2008b; Bartels, Reference Bartels2011).
This inattention to justice-level heterogeneity is despite a growing (e.g., Rainey, Reference Rainey2016)—though depending on one's perspective, still limited (Feller and Holmes, Reference Feller and Holmes2009; Lam, Reference Lam2013)—literature in political science concerned with effect heterogeneity. Thus, in an initial effort to explore potential heterogeneity, we examine whether justices place different weights on the standard factors, using data on all paid cert petitions granted or denied by the Court during October Terms (OT) 1939, 1968, and 1982.
3. Data
We analyze the 1939, 1968, and 1982 terms of the Supreme Court. Our chief consideration in selecting these terms was the broad timespan covered. To our knowledge, 1939 is the earliest term for which Court records (i.e., justice papers) exist that allow for collection of the data we need to construct the requisite variables. Although these terms were not randomly selected in any formal sense, there is no particular reason to believe that they were much different from others in their temporal neighborhood. Of course, it would be yet more desirable to have even more terms—and we are in the process of gathering data for the 1940–1945 terms—but data collection for just these three terms represents years of effort. And, as discussed above, the temporal breadth of our analysis greatly exceeds any existing systematic analysis of Court agenda-setting.
For the 1939 term, for information on the relatively objective, descriptive features of cases, we relied on petitions and briefs in opposition found in Supreme Court Briefs and Records (Hein), lower court opinions, and United States Law Week. For data on subjective indicators, we read and coded the “cert memoranda” in the papers of William O. Douglas (Library of Congress), primarily, and, in the absence of them, those in the papers of Stanley F. Reed (University of Kentucky).
For the 1968 term, we gathered our data from Records and Briefs of the Supreme Court on microfiche (issue area, presence of amici curiae, the United States as a petitioner, reversals between or dissents in the lower courts), United States Law Week (dates of actions, resolutions of cases), and, for conflict and allegations of conflict, John M. Harlan II and Thurgood Marshall's certiorari memoranda (located in the Mudd Collection at Princeton University and in the Library of Congress, respectively). The data for the OT 1982 term come from, and are as described in Caldeira and Wright (Reference Caldeira and Wright1990). Individual cert votes are as recorded in the docket books of Douglas (for OT 1939), Warren (1968), and Brennan (1982).
The operationalization of our variables accords with usual practice (see e.g., Caldeira and Wright, Reference Caldeira and Wright1990), and we describe the definitions and theoretical definitions concisely below.
Grant, whether a justice votes to grant (= 1) or deny (= 0) a cert petition, is our outcome variable,
US Petitioner is coded 1 if the “United States,” a federal agency, or its representative (in an official capacity) is one of the petitioners. Without fail, research has shown that the Solicitor General's office, which represents the United States before the Court, is particularly adept at making legal arguments (Johnson et al., Reference Johnson, Wahlbeck and Spriggs2006: 107) and is highly successful as a litigant both on certiorari and on the merits.
Intermediate Reversal is coded 1 if the court immediately below (nearly always either a federal court of appeal or state supreme court) reverses the court below it (usually a trial court, less often an agency or state intermediate court). When lower courts disagree among each other about the outcome of a given case, it is a signal that the case is either legally or ideologically nontrivial; as such, justices are expected to categorize the petition as “one perhaps worthy of a closer look (Caldeira and Wright, Reference Caldeira and Wright1988: 1115).”
Dissent Below is coded 1 if, in the court immediately below the Supreme Court, one or more judges dissent. Much as with a reversal in the court below, a lower court dissent signals that the controversy in nontrivial; depending on the author and content of the dissent, it may signal more (Caldeira and Wright, Reference Caldeira and Wright1990: 821–2). But, at the least, it is a “sign of distinction” that may prompt justices to more carefully scrutinize the cert petition (Caldeira and Wright, Reference Caldeira and Wright1990: 814).
Actual Conflict is coded 1 if the case involves a square conflict between two or more cases, occurring in different circuits, state supreme courts, or between the lower court and the Supreme Court—that is, the conflicts enumerated in Supreme Court Rule 10 (formerly Rule 38). The Supreme Court's formal rules emphasize the institution's role in resolving legal inconsistencies. And accounts have long indicated that the Court is in fact attentive to such conflicts (e.g., Ulmer, Reference Ulmer1984; Caldeira and Wright, Reference Caldeira and Wright1988). For OT 1939 and 1968, we relied on assessments in certiorari memoranda (primarily those of Douglas and Marshall); for OT 1982, we relied on the New York University Law Review's Supreme Court Project (NYU Law Review Supreme Court Project, 1984a, 1984b) to code this variable.
Alleged Conflict is coded 1 if the law clerk who is writing the certiorari memorandum (for OT 1939 and OT 1968) notes that the petitioner has alleged a conflict enumerated in Rule 10; and, for OT 1982, if the NYU Law Review's Supreme Court Project has made a similar judgment. For a justice who is particularly sensitive to inter-court conflicts, an allegation of conflict, even when not squarely presented, may nonetheless catch her attention. Certainly, the emphasis on conflicts placed by the Court, its formal rules, and practitioners’ manuals, encourages savvy lawyers to make such allegations.
Civil Liberties is coded 1 if the primary substantive issue area of the case involves civil liberties (including criminal procedure and civil rights). Accounts strongly suggest that the Supreme Court, over time, gives differential emphasis to cases involving civil liberties (Armstrong and Johnson, Reference Armstrong and Johnson1982).
Amicus Present is coded 1 if amicus curiae in support of or opposed to certiorari appear in a case. We code this variable as 1 whether the amicus favors or opposes the writ because the mere filing of such a brief indicates a case's salience. These briefs provide the Court with evidence of a case's social, political, economic, and legal importance, as well as (motivated) information on political and legal stakes (see Caldeira and Wright, Reference Caldeira and Wright1988; Collins, Reference Collins2008a).
Incompatible Decision Below is coded 1 if a conservative justice is reviewing a liberal decision below or a liberal is reviewing a conservative outcome below. Otherwise, it is coded 0. It is well understood that a justice voting on certiorari takes into account the ideological or policy implications of his votes; one part of this calculus is whether he believes the lower court's decision is correct (Caldeira et al., Reference Caldeira, Wright and Zorn1999). Thus, the Court's rules to the contrary, “error correction” is a consideration for justices at the agenda-setting stage.
We generally follow the coding rules from Spaeth (Reference Spaeth2001) to categorize the ideological direction of a lower court decision.Footnote 4 We classify a justice as liberal if, conditional on covariates, she was more likely to grant a conservative case below than a liberal one. Correspondingly, we classify a justice as conservative if, conditional on covariates, he was more likely to favor review of a liberal decision below than a conservative one. In practice, this operationalization means that the effect of Incompatible Decision Below in our analyses is constrained to be non-negative; of course, the effect's magnitude, which we are primarily interested in, is unaffected by the operationalization (i.e., can take on any possible value).
4. How much justice-level heterogeneity?
First, we examine the extent to which individual justices place different weights on the standard factors that have been shown to influence the Court's collective vote on certiorari. There is some reason to expect heterogeneity, as anecdotal accounts suggest that justices are not homogenous in how they approach decision-making, both on certiorari and on the merits. Indeed, observers and the justices themselves have commented on the Court's individualism, referring to its “nine little law offices,” operating largely on their own, with little interaction except for written communications. As Justice Jackson remarked, this mode of work cultivates individualistic thinking (Jackson, Reference Jackson1955: 16).
Accounts in Cordray and Meriwether Cordray (Reference Cordray and Meriwether Cordray2004) and Perry (Reference Perry1991) also suggest that justices assess cert-worthiness in non-uniform ways. Perhaps the most-cited example is Justice White's strong inclination to grant cases involving inter-circuit conflicts, which is well-known due to White's frequent public dissents from denial of certiorari (Broyde, Reference Broyde1987; Hutchinson, Reference Hutchinson1998; Sullivan, Reference Sullivan2002). And, to the extent that the merits vote is not independent of the cert vote (Caldeira et al., Reference Caldeira, Wright and Zorn1999), we might expect that certain justice idiosyncrasies demonstrated at the merits stage—for example, late-career Justice Douglas’ propensity to vote against the US Government (see Wolfman et al., Reference Wolfman, Silver and Silver1973)—could influence their agenda-setting vote as well.
Our modeling strategy for this exploratory analysis is straightforward. For each of the 27 justices,Footnote 5 we estimate a regression in which each covariate (US Petitioner, Actual Conflict, Alleged Conflict, Intermediate Reversal, Dissent Below, Amicus Present, Incompatible Decision Below) is interacted with a binary justice variable—that is, a variable equaling 1 if the observation is a vote cast by the given justice, and 0 if it is a vote cast by any of the others. (By way of illustration, one of the regressions is presented in Table 4 of the Supplementary Materials.) This approach allows us to derive, for each justice-covariate combination, estimates of effect size on the probability of an individual justice voting to grant. Less important, but still interesting, it allows us to test formally whether a given justice differs, at the 0.05 level, in the weight he places on a given covariate in deciding whether to grant cert, from the average weight for all other justices combined.Footnote 6 To ease comparison of effect sizes across justices, we calculate the change in predicted probabilities from a baseline of approximately 0.15 (0.15 ± 0.05) as the covariate in question goes from 0 to 1.Footnote 7
Figure 1 shows the effect sizes by justice for two covariates, Actual Conflict and Incompatible Decision Below. For each variable, we see a broad range of effect sizes, although the range is greater for Actual Conflict. There are justices who clearly stand out from their colleagues: Murphy (OT 1939) for Actual Conflict and Douglas (1968) and Rehnquist (1982) for Incompatible Decision Below; but, even disregarding these three, we see great variance in the weight that justices place on these covariates. Interestingly, though the weight that late-period (1982) Justice White places on Actual Conflict is significantly greater than that of the average justice in the sample, he is not among the most extreme justices in this respect.Footnote 8
Individual justice effect sizes for effects summarized in Table 1. Solid circle indicates that the effect for a given justice is statistically significantly different from the average effect for all other justices; hollow circle indicates otherwise.
Mean, minimum, and maximum effect sizes, over justices, on cert votes in OT 1939, 1968, and 1982
The amount of justice-level heterogeneity for these two variables is not atypical. Table 1 summarizes the range of effect sizes for each covariate. Clearly, there is significant heterogeneity for each of the variables—the spread is particularly wide for US Petitioner, Actual Conflict, Incompatible Decision Below, and Amicus Present. In the Supplementary Materials, we graphically present the effect estimates, for each covariate, for each justice in our sample. These graphs illustrate that the wide range of effect sizes in Table 1 is not due to outliers; instead, justices tend to vary substantially in how much weight they put on each covariate—especially, the covariates that have large effects on the Court's collective decision.
In sum, then, we have demonstrated significant justice-level heterogeneity, and now ask whether this heterogeneity can be systematically explained by justice-level characteristics. In particular, after a review of relevant literature, we focus on justice experience and ideological extremism as potential explanators of effect heterogeneity.
5. Explaining justice-level heterogeneity
As discussed above, the literature on justice-level heterogeneity at the agenda-setting stage is very sparse, so we turn to studies that assess judicial behavior at the merits stage to derive our hypotheses. One body of literature that addresses questions analogous to our own includes works that seek to measure whether justice attributes affect the extent to which a justice's vote (or other behavior on the merits) is influenced by legal factors (as opposed—explicitly or implicitly—to ideological factors). Since several of our covariates can be fit into the “legal” or the “ideological” categories, we might ask whether results from the merits stage also hold at the cert stage.
Before delving into that literature, it is worth spending some time to categorize our covariates as legal or ideological. Probably the most straightforwardly legal covariate is Actual Conflict, which is explicitly referred to in the Court's Rules as a facially non-ideological reason for granting cert (see further Perry, Reference Perry1991; Lindquist and Klein, Reference Lindquist and Klein2006; Black and Owens, Reference Black and Owens2009). A strong argument can also be made for US Petitioner as a legal covariate. The most direct evidence is from Johnson et al. (Reference Johnson, Wahlbeck and Spriggs2006: 107), which shows that legal arguments from the Solicitor General and from attorneys representing the federal government are higher quality than those from other lawyers; for evidence that this holds not only at the merits stage, but also on cert, see Budziak and Lempert (Reference Budziak and Lempert2018: 47). The categorization of amicus briefs is more contested. Still, there is a line of theoretical (see Black and Owens, Reference Black and Owens2009: 1070) and empirical (see Collins, Reference Collins2008a) work that argues for interpreting amicus briefs as indicators that a case is legally salient.
The status of Intermediate Reversal and Dissent Below is ambiguous. Though these types of disagreements in the lower courts may signal that the case is not legally trivial, it may also signal ideological divisions in a given case. Thus, we do not categorize these variables as either legal or ideological. Nor do we think that issue area (Civil Liberties) is a legal factor per se. And finally, Incompatible Decision Below is defined in terms of ideology, so this covariate is the most obviously ideological (and non-legal). In sum, then, we categorize Actual Conflict and US Petitioner as clearly legal, Amicus Present as arguably so, and Incompatible Decision Below as clearly ideological.
One area of focus in the relevant merits-stage literature has been justice experience. Systematic analyses dating back at least to Snyder (Reference Snyder1958) have suggested that shorter-tenured justices decide cases in a less ideologically extreme manner. Considering the relative weight of ideological and legal factors as a function of tenure, Hurwitz and Stefko (Reference Hurwitz and Stefko2004) hypothesizes that justices who are newer to the Court will weight precedent relatively more heavily than ideological considerations, compared to their more experienced colleagues. The authors justify this expectation on two grounds:
First, as Snyder suggested, newcomer justices are generally less comfortable pursuing ideologically extreme agendas from the bench than veteran justices, such that newcomers would be more likely to make decisions that conform to existing precedent. Second, appointees to the Supreme Court are typically drawn from lower level courts, [so] their experience on inferior courts, where institutional constraints on personal agendas are more prevalent, will have conditioned and socialized new justices to make decisions in ways their new institution does not necessarily command.
Their empirical results are consistent with these expectations: the longer a justice serves, the less likely is her merits vote to follow Supreme Court precedent and more likely to be consistent with her ideology. Subsequently, in a wider-ranging analysis, Wedeking (Reference Wedeking2012) confirms Hurwitz and Stefko (Reference Hurwitz and Stefko2004). Drawing on this line of literature, we hypothesize that the “legal” variables that affect the cert vote—Actual Conflict, US Petitioner, Amicus Present—will affect less experienced justices to a greater degree than they affect more experienced justices.
Ideological extremism has received less attention. Still, there is some evidence that ideological moderates weigh legal considerations more heavily, relative to ideological considerations, than their more extreme colleagues. Enns and Wohlfarth (Reference Enns and Wohlfarth2013: 1090) theorizes that the “swing justice” in a given case, who is typically an ideological moderate, “will be the least responsive to attitudinal considerations and most responsive to legal and strategic factors” in casting a merits vote. Enns and Wohlfarth (Reference Enns and Wohlfarth2013) find evidence for this proposition, although testing it involves serious measurement challenges. Segal and Spaeth (Reference Segal and Spaeth1996), which shows that, of 15 justices considered, only Powell and Stewart—two moderates—are appreciably affected by stare decisis when voting on the merits, is also consistent with the idea that legal factors affect moderates more strongly. We therefore hypothesize that justices closer to the Court's median will be more affected by the legal variables than will those further from the median.
To test these hypotheses, we estimate a logit regression predicting whether a justice voted to grant cert (= 1) or not (= 0) with the standard covariates—Actual Conflict, US Petitioner, Amicus Present, Dissent Below, Intermediate Reversal, Alleged Conflict, Incompatible Decision Below, and Civil Liberties—each interacted, in turn, with two variables: (1) the justice's Martin and Quinn (Reference Martin and Quinn2002) Distance to the Median justice and (2) his Tenure on the Court, in years, at the time of the vote. We relegate the regression table to the Supplementary Materials (Table 4); here, we focus on our quantity of interest, the difference in the effect of each covariate between the 10th and the 90th percentiles of Tenure and Distance to the Median, respectively.Footnote 9 In other words, we calculate and test the statistical significance of a set of second differences. As an example, the second difference for US Petitioner as a function of Tenure is formally defined as, $\Delta \Delta [ \Pr ( Y) ] = [ \Pr ( Y\vert {\rm Tenure} = 1,\; \, {\rm US\ Petitioner} = 1) -\Pr ( Y\vert {\rm Tenure} = 1,\; \, {\rm US\ Petitioner} = 0) ]\, -$
$[ \Pr ( Y\vert {\rm Tenure} = 26,\; \, {\rm US\ Petitioner} = 1) -\Pr ( Y\vert {\rm Tenure} = 26,\; \, {\rm US\ Petitioner} = 0) ]$
(see Berry et al., Reference Berry, DeMeritt and Esarey2010). For comparability of effect sizes, we again calculate them from a baseline value of approximately 0.15.
We find initial support for our hypotheses related to both experience and extremism. In particular, we find that more experienced and more ideologically extreme justices weigh Actual Conflict and US Petitioner—the two largest, and the most clearly legal, factors affecting the cert vote—significantly less heavily than do more junior and more moderate justices. For example, the presence of the US as petitioner increases the probability of a grant vote by a justice with Tenure at the 10th percentile by 0.50. This increase for a senior justice, with Tenure at the 90th percentile, is just 0.30. The second difference of 0.20 is significant at the 0.05 level.
Additionally, there is a bit of evidence that whether there is Amicus Present, a variable we classified as “arguably legal,” is weighed less heavily by longer-tenured and more extreme justices—although the second differences are large in magnitude and in the expected direction, they are not significant at the 0.05 level.
Two other results are worth noting. First, we find that experience does not affect the extent to which justices take into account the ideological direction of the decision below—the second difference for Incompatible Decision Below as a function of Tenure is ≈ 0. This is in contrast to some (but not all) of the literature on extremism in merits voting as a function of experience. Second, although—as it stands to reason—more extreme justices weigh the ideological direction of the decision below more heavily than do relative moderates, the difference is not huge and only marginally statistically significant. Thus, it appears that even relative moderates put a fair amount of weight on the ideological implications of the cert vote.Footnote 10
Next, we evaluate the robustness of these results. Specifically, we address the concern we raised in the introduction: in short, whether the significant second differences are driven by a single justice. Recall that this is of consequence because, if some results are driven by a single justice, we can have little confidence in attributing the significant second differences to the observed justice attributes (i.e., experience and extremism) rather than unmeasurable justice-level idiosynracies. To use the formal terms, we seek to address whether any justice is unusually influential, as an outlier with high leverage. The broader temporal scope of our analysis reduces, but does not eliminate, the risk that our results are not robust in this sense.
We proceed by re-estimating the previous model (see Table 5 in the Supplementary Materials) 27 times, jackknifing (omitting) from the sample a different justice each time. Then, we use the estimates from each model to estimate 27 sets of second differences, and their associated significance levels, as we did in Table 2. We consider significant second differences robust if, for every one of the 27 jackknife replications, the p value remains below 0.05. Our approach is analogous to the DFBETAS method of detecting influential observations (e.g., Kennedy, Reference Kennedy2003: 379), adapted to our quantity of interest (second differences as opposed to OLS coefficients) and considering influential justices (clusters) rather than influential observations.
Effect sizes and second differences for eight covariates on the probability of a cert grant vote as a function of, respectively, Justice Tenure and Distance to Median
A junior and moderate justice is one who is at the 10th percentile of tenure and extremism, respectively. A senior and extreme justice is at the 90th percentile of the respective measure. see text for details, and Table 5 in the Supplementary Materials for the regression on which these estimates are based.
$^\ast$
p < 0.05; $^{\dagger}$
p < 0.1.
As Table 3 shows, the results for US Petitioner are robust, but not the results for Actual Conflict. No matter which justice is left out, the second differences remain large in magnitude and statistically significant at the 0.05 level for US Petitioner—we can confidently conclude that more junior and more moderate justices give more weight to a petition when the US seeks review. However, for actual conflict, when Douglas (1968) is omitted, the magnitude of the second differences decrease precipitously, and become statistically non-significant. To put it somewhat informally, Douglas is unusually influential here, due to the relatively low weight he places on US Petitioner, and his relatively extreme values for Distance to Median and Tenure. To be clear, the results in Table 2 should not be considered invalid: by all rights, Douglas should be included in the sample—it is not that there was some error in recording his data, or that he should be thought of as drawn from a different population than the other justices. However, we cannot say with confidence that those results for Actual Conflict are in fact driven by experience and extremism, as opposed to some justice-specific idiosyncrasy.Footnote 11 By contrast, we can be confident that our results for US Petitioner are not driven by such idiosyncrasies.
Minimum second differences in probability of a cert grant vote, and maximum associated p-values, calculated across 27 jackknife replications omitting a different justice each time, for two covariates as functions of Justice Tenure and Distance to Median
See text for details.
6. Conclusion
We have explored heterogeneity in the certiorari vote on the Supreme Court, in the widest-spanning analysis of individual justices’ votes in the literature. Two central findings emerge.
First, there is substantial justice-level heterogeneity in the weight that justices place on the standard factors shaping the cert vote (US Petitioner, Actual Conflict, Alleged Conflict, Intermediate Reversal, Dissent Below, Amicus Present, Incompatible Decision Below). Thus, Court-level analyses obscure substantial and interesting variation in justices’ individual agenda-setting calculations.
Second, some of this heterogeneity is associated with justices’ experience and ideological extremism, partially in theoretically predicted ways. The two unambiguously legal variables shaping the cert vote, Actual Conflict and US Petitioner, are weighed more heavily by newer justices and less extreme justices. This is consistent with expectations that more junior justices will emphasize formal, legal criteria when voting to grant cert, and that relative moderates will give more weight to legal factors over ideological ones. However, we cannot rule out the possibility that these results for Actual Conflict are driven by justice-specific idiosyncrasies, as opposed to experience and extremism per se.
This leads to a note of caution, applicable broadly to individual-level studies where there are relatively few justices, and covariates that are constant within justices, or nearly so, are of interest. To reduce the likelihood that results are driven by unmeasured justice-specific confounders (what we have called idiosyncrasies), it is worth re-estimating results with each individual justice, in turn, omitted from the sample. Since we have seen that this danger is present even in our relatively large sample with 27 justices included, it is a threat worth considering in studies that are narrower in temporal scope.
From our initial exploration of justice-level heterogeneity in agenda setting, several promising avenues for future research are apparent. For example, researchers could add more terms and justices to analyses such as those presented above, although data limitations will make this difficult. Researchers might also test whether heterogeneity on the cert vote can be explained by systematic justice-level factors other than experience and extremism (or perhaps by case-level variables).
More broadly, there is no reason to assume that the extent of agent-level heterogeneity we show at the Supreme Court is unique to that institution. To be sure, actors in other institutions face different costs and incentives than do members of the Court. But, the extent of agent-level heterogeneity is an empirical question, and one that we believe has been under-explored. Our basic approach might be fruitfully applied to bodies of similar sizes, such as other courts, administrative commissions, and congressional committees.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/psrm.2021.69. To obtain replication material for this article, please visit https://doi.org/10.7910/DVN/QPPL9H
Acknowledgments
Revision of version presented at the 2020 Western Political Science Association Annual Conference, where we received thoughtful comments from Joanna Wuest and other attendees. We thank Larry Baum and Jeff Budziak for helpful feedback and Audra J. Smith for excellent research assistance. For use of data for OT 1982, we thank John Wright.
