%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Political scientists frequently examine how choices made by actors shape the outputs of the political process. Empirically minded scholars tend to focus on actors’ choices rather than the content of the resulting outputs: discrete choices lend themselves to systematic classification and are thus easily amenable to examination using quantitative techniques (e.g., Poole and Rosenthal Reference Poole and Howard1997). This does not suggest that empirically minded scholars are uninterested in the substantive content of outputs. However, systematically classifying content raises many difficult methodological questions. Classifying the choice made by a legislator on a roll-call vote is relatively simple; systematically classifying the content of the bill itself is more difficult. Yet comprehensive explanation of the political process requires an understanding of both the choices made by actors and the content of those choices.
Nowhere has this distinction emerged more prominently than in scholarly examinations of the judiciary. The most important developments in the field of judicial politics have emerged through investigations of the choices made by judges (Pritchett Reference Pritchett1948; Rohde and Spaeth Reference Rohde and Harold1976; Baum Reference Baum1997; Epstein and Knight Reference Epstein and Jack1998; Segal and Spaeth Reference Segal and Harold2002). While these examinations have advanced the collective understanding of judicial politics in a number of ways, they are not without limitations. Of greatest practical concern, the federal judicial hierarchy is not governed by dispositional votes. The substantive content of judges’ choices—taking the form of legal policies articulated in written opinions—not only dictates the appropriate disposition of the case but shapes both the dispositional choices and substantive opinion content generated by judges when confronting similar cases. This has led to calls for a shift in focus from judicial votes to the content of opinions (Maltzman, Spriggs, and Wahlbeck Reference Maltzman, Spriggs II and Paul2000, 154).
Greater scholarly attention to the study of opinion content is unquestionably a laudable goal. Unfortunately, the challenges of systematically classifying that content have severely constrained empirical analyses of opinions. Recent technological developments in automated textual (text) analysis have permitted scholars to overcome some of these constraints. These tools have been employed most readily to examine the text characteristics of the language used in opinions, which we refer to collectively as “writing style.” Scholars can now test many hypotheses involving opinion content that would have previously been impossible using large samples (Coleman and Phong Reference Coleman and Quy2010; Owens and Wedeking Reference Owens and Justin2011; Owens, Wedeking, and Wohlfarth Reference Owens, Justin and Patrick2013; Goelzhauser and Cann Reference Goelzhauser and Damon2014; Hansford and Coe Reference Hansford and Chelsea2014; Johnson Reference Johnson2014; Black et al. Reference Black, Hall, Owens and Eve2016, Reference Black, Ryan J. Owens and Patrick2016b; Bryan and Ringsmuth Reference Bryan and Eve2016).
This line of inquiry helps break down the barriers to the systematic analysis of opinion content. The valuable insights to be gained from these advances, and the potential for systematic classifications of opinion content using larger samples than previously possible, have led to a rapid proliferation of studies using automated text analysis. The primary focus of these applications has been to examine how opinion writing style varies based on the preferences of other actors. For example, these studies examine whether Supreme Court justices write in different styles depending on the preferences of Congress (Owens et al. Reference Owens, Justin and Patrick2013), lower federal judges (Corley and Wedeking Reference Corley and Justin2014; Black et al. Reference Black, Ryan J. Owens and Patrick2016b), and the public (Black et al. Reference Black, Ryan J. Owens and Patrick2016a, Reference Black, Ryan J. Owens and Patrick2016b). While the conditions under which justices employ specific writing styles are beginning to crystallize in this still-nascent literature, many questions concerning writing style as a concept have been overlooked.
Here we focus on one such question: What are the determinants of judicial writing style? As discussed below, scholarship examining the effects of writing style generally assumes that it is a function of purposeful choices made by judges. From this perspective, judges strategically use writing style to further specific goals. We suggest that while strategic behavior is one potential determinant of writing style, there may be other, less studied factors that have a more substantial role in shaping judicial writing style. Specifically, we propose that characteristics of the judge and of the case have the potential to affect the writing style used in opinions, at least as much as strategic choice.
To investigate the determinants of writing style, we conduct several analyses using a stratified random sample of 11,771 courts of appeals opinions from 1950 to 2002. Each analysis follows the same general research design. First, we identify sets of covariates associated respectively with strategic behavior, judicial characteristics, and case-specific considerations to create groups of strategic, judge, and case variables. Second, we scale opinions on a large number of text characteristics, including those elements of writing style most commonly cited in the judicial politics literature (namely, reading difficulty, cognitive complexity, and affective language). Third, we regress each set of variables on each text characteristic score to determine the proportion of the variance explained by each set of variables.
Our first analysis provides an example of this research design by investigating the variance of one text characteristic: reading difficulty, that is, (a lack of) “clarity,” the most frequently examined Supreme Court opinion text characteristic (e.g., Coleman and Phong Reference Coleman and Quy2010; Owens et al. Reference Owens, Justin and Patrick2013; Goelzhauser and Cann Reference Goelzhauser and Damon2014; Hansford and Coe Reference Hansford and Chelsea2014; Johnson Reference Johnson2014; Black et al. Reference Black, Ryan J. Owens and Patrick2016b). We present a set of regression models to assess what proportion of the variance in the reading difficulty of courts of appeals opinions is explained by strategic, judge, and case variables. The results suggest that variables associated with judges and cases, respectively, explain a much larger proportion of the variance than those associated with strategic considerations.
Our second analysis extends this application beyond reading difficulty to 97 additional possible text characteristics. The results suggest that while no set of variables explains a substantial percentage of the variance in writing style, those related to the case explain the greatest portion of variance for nearly all characteristics. By contrast, strategic variables explain very little variance.
Our final analysis investigates the role of case variables in greater detail. Here we restrict our research design only to opinions accompanied by separate opinions (i.e., concurrences and dissents). If case variables are the primary determinants of writing style, we should expect the writing styles of majority and separate opinions to be similar. The results confirm this expectation. Ceteris paribus, separate opinions’ text characteristics statistically predict the text characteristics of corresponding majority opinions for nearly every text characteristic. Moreover, for approximately half of all text characteristics, separate opinions’ text characteristics explain a higher percentage of variance in the corresponding majority opinions’ text characteristics than all other case, judge, and strategic variables combined. We conclude by discussing the implications of these results for scholarship examining opinion content using automated text analysis and identify several other important questions regarding writing style beyond the scope of our analyses.
Determinants of Writing Style
Many variables likely determine the style used by judges to craft opinions. We organize these plausible determinants into three distinct categories: strategic, judge, and case. We give thorough attention to strategic variables, as scholarship using automated text analysis has primarily approached questions about writing style from this perspective. But, as we argue below, there are good reasons to expect that judge or case variables have the potential to influence writing style at least as much as strategic variables.
Strategic Determinants
Most scholarship using empirical measures of writing style hypothesizes that judges use writing styles to shape the behavior of other actors in an effort to achieve their goals. This expectation is consistent with the strategic approach to judicial behavior, which argues that judges recognize that their ability to achieve their goals depends on several considerations, including the preferences of other actors, the choices those actors are expected to make, and the institutional context in which they operate (Epstein and Knight Reference Epstein and Jack1998, 9). In fact, many analyses explicitly discuss variation in writing style in terms of “strategic opinion writing” (e.g., Black et al. Reference Black, Ryan J. Owens and Patrick2016a, 705–10; Reference Black, Ryan J. Owens and Patrick2016b, 12–14) or “strategic behavior” (e.g., Owens et al. Reference Owens, Justin and Patrick2013, 37–41). For scholars operating in this strategic framework, the primary question of interest centers on the effectiveness of writing style in shaping the behavior of other actors.
Several analyses have produced results that are consistent with a strategic account of opinion writing. Owens et al. (Reference Owens, Justin and Patrick2013) find that Supreme Court opinions are less “clear” (as measured by reading difficulty) when the Court’s majority is ideologically distant from relevant pivot points in Congress. They conclude that this pattern is evidence of strategic manipulation of writing style designed to avoid congressional review. Black et al. (Reference Black, Ryan J. Owens and Patrick2016b) find that the Court produces opinions with lower levels of reading difficulty in several different circumstances: when the lower courts are ideologically scattered and distant from the Court, when the Court rules against administrative agencies with poor performance records, when the Court rules against states with less professionalized legislatures, and when the Court rules against prevailing public opinion. This last finding is replicated by Black et al. (Reference Black, Ryan J. Owens and Patrick2016a). Corley and Wedeking (Reference Corley and Justin2014) and Black et al. (Reference Black, Ryan J. Owens and Patrick2016b) demonstrate, respectively, that higher levels of certainty and lower levels of reading difficulty in Supreme Court opinions are associated with more positive treatment by lower federal courts. The authors interpret their results as evidence of the relationship between writing style and compliance.
The evidence generated by this line of scholarship suggests that any explanation of writing style would be incomplete without a consideration of strategic behavior. However, it would be premature to conclude that writing style for all judges is exclusively, or even primarily, a function of strategy. It is clear that justices can manipulate their styles in an attempt to achieve certain policy goals, but the circumstances under which judges might engage in these forms of strategic behavior are frequently limited. For example, justices may seek to avoid congressional review of their decisions, but, as Owens et al. (Reference Owens, Justin and Patrick2013, 44) note, “justices need not always worry about the preferences of Congress,” since the Supreme Court is often unconstrained, given the preferences of other actors, especially in constitutional cases.
Similar considerations apply to the US Circuit Courts of Appeals. Congressional review—let alone override—of lower-court decisions is particularly rare (Lindquist and Yalof Reference Lindquist and David2001). Also, lower-court compliance is unlikely to be a concern for appeals judges when authoring nonprecedential opinions (a majority of courts of appeals decisions). Still, there is potential for Supreme Court and en banc review, which provides opportunity for strategic behavior.
Judge-Related Determinants
Characteristics of the author offer one plausible alternative explanation for variance in writing styles. Scholars in the fields of information science and technology have recently relied on author-based patterns to attribute authorship to unsigned documents (e.g., Zheng et al. Reference Zheng, Li, Chen and Zan2006, 379). Work in this field demonstrates that writing styles can accurately predict author characteristics like gender (Koppel, Argamon, and Shimoni Reference Koppel, Argamon and Anat2002), educational experiences (Corney et al. Reference Corney, Olivier, Anderson and George2002), and cultural backgrounds, among others, offering support for the expectation that writing styles are shaped by these traits.
Systematic efforts to assess the effect of judge characteristics on writing style by judicial politics scholarship are lacking. Previous analyses tend to compare only justices themselves and not judicial characteristics. Owens and Wedeking (Reference Owens and Justin2011, 1043–44) provide a descriptive comparison of opinion cognitive complexity by 17 Supreme Court justices. The results offer evidence that cognitive complexity does vary by justice, but no explanation as to why is developed (other than the observation that cognitive complexity is not correlated with ideology). Li et al. (Reference Li, Azar, Larochelle, Hill, Cox, Berwick and Andrew2013) use algorithmic attribution techniques to determine the authorship of unsigned Supreme Court opinions. The accuracy of the method is noteworthy, but questions about why judges possess certain writing styles are not considered.
The lack of systematic consideration of judge characteristics, particularly in light of scholarship suggesting that author traits affect writing style, leaves our understanding of opinion writing style incomplete. The absence of such scholarship is likely due to the near-singular focus of scholarship examining writing style on US Supreme Court opinions. Given the small number of authors available for analysis (and the lack of diversity among them), there is simply not enough variation (at the Supreme Court) to systematically assess how judge characteristics shape writing styles. We attempt to overcome that limitation here.
Case-Related Determinants
Case variables offer a second alternative explanation for variance in writing styles. All cases vary in terms of characteristics largely outside of the litigants’ or judges’ control. The substantive content of judicial opinions is inherently constrained by case characteristics; judges presumably must address the facts of the case, the legal question, potentially relevant precedents, and so on. While writing style is conceptually distinct from substantive content, it is not likely to be totally independent. Judges writing opinions that interpret statutes or apply relevant precedents crafted in particular writing styles may have no choice but to match (at least to some degree) the style of the source to achieve consistency and precision in their opinions. A related pattern may exist for the facts of a case itself; some factual patterns may not lend themselves to explanation or discussion using a particular writing style, thus leading judges to adopt different styles than they might select if completely unconstrained by case facts.
Empirical scholarship on the effect of case variables on writing style is minimal. Owens and Wedeking (Reference Owens and Justin2011) provide a descriptive comparison of opinion cognitive complexity using one example of a case variable: issue areas. They find that while justices are reasonably consistent in cognitive complexity, there is appreciable within-author variation in cognitive complexity across issue areas. While this provides a preliminary investigation of how writing style is shaped by case variables, greater systematic attention is clearly warranted.
Determinants of writing style can generally be organized by strategic, judge, and case variables. Although the overwhelming majority of scholarship examining writing style has focused on variables associated with strategic behavior, the circumstances under which such behavior is expected or warranted may be limited. However, it is probably more intuitive to see how opinions could be affected by characteristics of the judge and the case, suggesting that these variables likely explain a relatively significant amount of variance in writing styles. But exactly how much variance is explained by these three sets of variables? To determine the answer, we first must identify characteristics associated with each.
Identifying Variables
We adopt an intentionally inclusive approach to defining the variables associated with each potential influence on writing style. We are not primarily interested in testing hypotheses about associations between individual covariates and elements of writing style. Rather, we seek to determine how much variance in writing style is associated with a given set of variables. To do so, it is important to include as many plausible covariates as possible; failing to include a potentially relevant variable could lead our analyses to underestimate the amount of variance explained by a plausible influence. Our research design should therefore be understood as seeking to identify the maximum amount of variance explained by each influence. While this approach means that the theoretical justification for including certain variables may be reasonably contested, we believe it is the most appropriate design to address our research question.
Consistent with this inclusive approach, we examine courts of appeals opinions to assess the influences on writing style. Prior empirical examinations of writing style have focused exclusively on opinions of the US Supreme Court. The Court frequently warrants disproportionate attention because of its unique position atop the federal judiciary. However, focusing exclusively on the Court would dramatically limit our ability to assess the influences on writing style—particularly judge-related variables. There is simply not enough variance in the characteristics of Supreme Court opinion authors to make a sensible determination of how judge variables affect writing style. The diversity of jurists serving on the courts of appeals helps overcome this concern.
Strategic Variables
The characteristics we identify as being consistent with strategic opinion writing emerge from three different audiences toward which an author could behave strategically: other members of the courts of appeals, members of the Supreme Court, and members of Congress. We address each of these audiences individually.
The most immediate audience judges must consider when engaging in strategic behavior are other members of the courts of appeals. Of greatest importance for an author are the other panel members. To achieve a majority, an author must convince at least one other member of the three-judge panel to agree to the opinion. If an author were to craft an opinion inconsistent with the preferences of the other panel members, they could find themselves in the minority. To achieve a favored policy outcome, authors must consider the preferences of other panel members.
They must also consider the preferences of a circuit. When panels make errors or fail to comply with the preferences of the circuit as a whole, the circuit can review their decisions en banc (George Reference George1999). If an author or panel were to create a legal policy sufficiently inconsistent with the preferences of the circuit, the other members could intervene through en banc review and move the policy away from the preferences of the author or panel and to the preferences of the circuit. Scholarship confirms this intuition: voting behavior on panels is conditioned on the preferences of the full circuit (Kim Reference Kim2008; Kastellec Reference Kastellec2011).
To account for the opportunities for strategic behavior within the courts of appeals, we include three variables. First, Author-Panel Distance is the ideological distance between the opinion author and the median member of the decision-making panel. Second, Author-Circuit Distance is the ideological distance between the author and the median member of the circuit. Finally, Panel-Circuit Distance is the ideological distance between the median member of the decision-making panel and the median member of the circuit. The inclusion of both Author-Circuit Distance and Panel-Circuit Distance reflects the potential that both the opinion author and panel median may affect the content of the legal policies articulated in the opinion of the court. To create these measures (and all measures involving ideology), we rely on Judicial Common Space scores (Epstein et al. Reference Epstein, Martin, Segal and Chad2007). These scores are particularly well suited for analyses such as ours, as they place lower-court judges, Supreme Court justices, and members of Congress into a common policy space.
Strategic courts of appeals judges must also consider the preferences of the Supreme Court. Just as the decisions of courts of appeals panels can be reviewed by the entire circuit en banc, the Supreme Court can review panel decisions through its writ of certiorari. If an author or panel were to create a legal policy sufficiently inconsistent with the preferences of the Supreme Court, members of the Supreme Court could intervene and move the policy away from the preferences of the author or panel and to the preferences of Court (see Clark [Reference Clark2009] for a discussion of the relationship between en banc and Supreme Court review). To account for this opportunity to engage in strategic behavior, we include two additional variables that mirror those previously defined: Author-Court Distance is the ideological distance between the author and the median member of the Supreme Court.Footnote 1 Panel-Court Distance is the ideological distance between the median member of the decision-making panel and the median member of the Court. Again, both measures are included to reflect the potential influence of both the opinion author and the panel median.
Finally, strategic courts of appeals judges may also consider the preferences of Congress. Just as the circuit as a whole or the Supreme Court can review the decision of a panel, so too (theoretically) can Congress. Owens et al. (Reference Owens, Justin and Patrick2013) offer some support for this possibility: they demonstrate that Supreme Court opinions are written less clearly when the ideological composition of Congress heightens the potential for review. While congressional review of courts of appeals decisions is rare (Lindquist and Yalof Reference Lindquist and David2001), in an effort to be inclusive, we include four variables: Author-House Distance, which is the ideological distance between the author and the median member of the House;Footnote 2 Author-Senate Distance, which is the ideological distance between the author and the median member of the Senate; Panel-House Distance, which is the ideological distance between the median member of the panel and the median member of the House; and Panel-Senate Distance, which is the ideological distance between the median member of the panel and the median member of the Senate.
Judge Variables
The lack of scholarship on the relationship between judge characteristics and writing style makes our selection of those characteristics particularly challenging. In our efforts to be inclusive, we have identified as many plausible characteristics for which we could find consistent and reliable data. This is not to suggest that we expect each of these characteristics to individually affect how judges craft opinions; rather, we aim to develop as complete a picture as possible of the cumulative effects of judge characteristics on writing style. Most of the measures created to capture judge characteristics are derived from the Attributes of US Appeals Court Judges Database (Gryski and Zuk Reference Gryski and Gary2008). We outline these measures in table 1.
Judge Factor Variables
Case Variables
The lack of scholarship examining the effect of case characteristics on writing style likewise limits the strength of theoretical justification available for the inclusion of certain measures. Owens and Wedeking (Reference Owens and Justin2011) provide some guidance by demonstrating that opinion cognitive complexity varies consistently across some legal issue areas. To account for this potential variance, we include dummy variables for eight general issue areas identified in the US Appeals Courts Database (Songer Reference Songer2008). These include Criminal Case, Civil Rights Case, First Amendment Case, Due Process Case, Privacy Case, Economic Activity and Regulation Case, and Miscellaneous Case. Labor relations serves as the excluded category.
In addition to issue areas, we theorize that the circuit in which the case originated may affect writing styles. The types of legal disputes conceivably raised in the courts of appeals are not randomly distributed across the country; certain types of cases are undoubtedly more common in some judicial circuits than others. To account for this systematic variation, we include fixed effects for each circuit (save one).
Finally, we adopt a similar strategy when it comes to the time period in which the case originated. Just as the types of legal disputes raised in the courts of appeals are not randomly distributed across geographic location, they are also not randomly distributed over time. Different issues emerge on legal agendas, and changes in judicial interpretation or statutory law have the potential to shape the types of disputes courts of appeals judges are resolving. To account for this variation, we include fixed effects for each year in our data set (again, save one). While the strategy of including fixed effects for both circuit and year is an admittedly broad one for capturing case characteristics, it is consistent with our decision to be as inclusive as possible when considering variables that make up our influences on writing style. These fixed effects for time and circuit naturally absorb a great deal of unmodeled heterogeneity. While one could argue that these fixed effects do account for properties of cases not covered by judge characteristics or strategic considerations, we describe below an alternative approach that excludes these fixed effects, to ameliorate concerns about our strategy.
Identifying Text Characteristics
Just as we adopt an intentionally inclusive approach to defining the potential influences on writing style, we do the same when identifying potentially relevant text characteristics. The literature to date incorporates several features. Prominently, many scholars (Coleman and Phong Reference Coleman and Quy2010; Owens et al. Reference Owens, Justin and Patrick2013; Goelzhauser and Cann Reference Goelzhauser and Damon2014; Hansford and Coe Reference Hansford and Chelsea2014; Johnson Reference Johnson2014; Black et al. Reference Black, Ryan J. Owens and Patrick2016b) focus on the reading difficulty of judicial texts, often termed “clarity.” While specific formulas for calculating readability vary (the Flesch-Kincaid Grade Level and Coleman-Liau Index being among the most common), all these metrics focus on some combination of character, syllable, word, and sentence counts. Therefore, a text that on average contains longer sentences, more multisyllabic words, and a high ratio of characters to words will be considered more difficult to read. We use the Flesch-Kincaid Grade Level (FKGL) as a metric of reading difficulty in the analysis to follow.Footnote 3
In addition to reading difficulty, scholars have analyzed other stylistic features of judicial texts. Technological advances, notably Linguistic Inquiry and Word Count (LIWC) software (e.g., Tausczik and Pennebaker Reference Tausczik and James2010), allow for the consideration of dozens if not hundreds of theoretically distinct linguistic features. Most of these stylistic features involve some form of dictionary-based classification. A predetermined list of words assumed to adequately encompass some characteristic is compared to a text to determine the relative prevalence of that characteristic in the text. LIWC 2015 includes dictionaries for dozens of characteristics, including concepts such as affective processes, certainty, insight, and causation (Pennebaker et al. Reference Pennebaker, Ryan L. Boyd and Kate2015).
Many scholars apply LIWC categories to judicial texts (Owens and Wedeking Reference Owens and Justin2011; Bryan and Ringsmuth Reference Bryan and Eve2016). These scholars have variously considered categories such as affective (emotional) language (Black et al. Reference Black, Hall, Owens and Eve2016; Bryan and Ringsmuth Reference Bryan and Eve2016), negative language (Wedeking and Zilis Reference Wedeking and Michael2015), certainty (Corley and Wedeking Reference Corley and Justin2014), and, combining several categories, cognitive complexity (Owens and Wedeking Reference Owens and Justin2011). For inclusivity, we analyze all of LIWC’s categories. Generally speaking, these variables are the percentage of words in the preprocessed documents that fall in one of the predefined LIWC categories.
Additionally, for comparability with prior work (Owens and Wedeking Reference Owens and Justin2011), we construct an index of cognitive complexity by factor analyzing 10 LIWC categories: causation, insight, discrepancy, inhibition, tentativeness, certainty, inclusiveness, exclusiveness, negations, and percentage of words containing six or more letters. The first eigenvalue of this factor analysis was well above 1, while no other factors presented an eigenvalue above 1, indicating that a single factor captures this underlying construct well (Kaiser Reference Kaiser1960). We use the 2007 LIWC dictionaries for this index for direct comparability with prior published work (i.e., Owens and Wedeking Reference Owens and Justin2011).
We suggest that another textual characteristic, informality, deserves consideration as a textual feature as well.Footnote 4 (In)formality is a stylistic element that practitioners (e.g., Scalia and Garner Reference Scalia and Bryan2008) emphasize. Writing about linguistic variation in general, Heylighen and Dewaele (Reference Heylighen and Jean-Marc1999, 2) state that the “most frequently mentioned” aspect of linguistic style is formality. Formality manifests itself in a variety of observable ways in written language, and scholars have taken several approaches toward quantifying a text’s formality. Two approaches from the linguistics literature for measuring formality deserve brief mention. First, Brooke and Hirst (Reference Brooke and Graeme2014) take a vocabulary-based approach, presenting a method of ranking words by formality. A key practical advantage of the method is the low degree of supervision required: the analyst need not specify the absolute location of training words on a formality scale but only give the relative formality ranking of words in training-word pairs.Footnote 5
Sheika and Inkpen (Reference Sheika and Diana2012) alternatively propose several elements of formal and informal language that we also incorporate into our measure of informality. Informal style, in contrast to formal style, uses sentences written in active voice, intensifiers, phrasal verbs, contractions, emotional language, and idiomatic language. Elements of formal style include impersonal style (third person and passive voice), complex words and sentences, technical vocabularies, polite words and formulas, and objective style. Formal style avoids abbreviations, contractions, colloquialisms, vague expressions, and slang words (Sheika and Inkpen Reference Sheika and Diana2012, 6–8).
For this litany of features, we constructed regular expressions in the R programming language (R Core Team Reference Team.2016) to count the instances of the passive voice, intensifying adverbs (such as “very”), phrasal verbs (based on the list of the 100 most common such verbs in Gardner and Davies [Reference Gardner and Mark2007]), contractions, and idiomatic language (based on Spears Reference Spears2008).Footnote 6 We then scaled these counts by the total number of words in an opinion and standardized these values across our sample. We measured emotional language as the absolute value of an opinion’s sentimentality score, as calculated by the polarity function (which uses a voluminous dictionary of positive and negative words) in the qdap package (RinkerReference Rinker2013) in the R programming language. We then factor analyzed all of these metrics to construct a single composite index, Informality.
Data and Results
To identify a set of courts of appeals opinions, we use the Appeals Court Database (Songer Reference Songer2008) and the Kuersten and Haire (Reference Kuersten and Susan2007) update.Footnote 7 Our sample covers the years 1950–2002 and includes 11,771 signed, published courts of appeals decisions.Footnote 8 The majority opinion serves as our unit of analysis. Our approach is straightforward. For every text characteristic we consider (reading difficulty, informality, cognitive complexity, and every characteristic in LIWC), we identify the amount of variance in that characteristic explained by strategic, judge, and case variables.
Our research design does not test the statistical significance of individual coefficients, a departure from prevailing norms in much of quantitative social science. Instead, we analyze how different factors account for some percentage of the total variability in our dependent variable. This strategy recalls Pritchett’s (Reference Pritchett1941) efforts to explain the total variability in Supreme Court dissensus along left-wing/right-wing lines, as one example. Klingman and Lammers’s (Reference Klingman and William1984) analysis of how to best explain variance in the “general policy liberalism” of American state politics also fits squarely in the methodological tradition we contribute to here. Our emphasis on substantive explanation also draws inspiration from contemporary advice against overreliance on p-values as a sole basis for scientific conclusions (Wasserstein and Lazar Reference Wasserstein and Nicole2016, 131).
Example Regression: Reading Difficulty
We provide an example of our approach in table 2, using reading difficulty as an example. Table 2 shows the results of several OLS regression models. Column 1 of table 2 regresses all of the strategic variables on clarity. Column 2 regresses all of our judge characteristics on clarity. Column 3 regresses our case issue area dummy variables on clarity. Column 4 includes issue area dummies as well as circuit fixed effects and year fixed effects (these latter effects are not presented for brevity). Column 5 includes all of these variables. We focus on adjusted R 2 as our quantity of interest in these five models. That is, taken together, how much total variance in reading difficulty does each category of cases account for?
OLS Regression of Reading Difficulty on Strategic, Judge, and Case Characteristics, All Signed Majority Opinions
Note.— Dependent variable is Flesch-Kincaid Grade Level reading difficulty score. Higher values of the dependent variable indicate a less readable/clear writing style. Robust standard errors in parentheses. Circuit and year fixed effects included but not reported in models 4 and 5 for brevity. N =11,771.
* p < .05.
Because the numbers of variables associated with strategic, judge, and case characteristics differ substantially, we assess the amount of variance explained by each using adjusted R 2. Letting n be the number of observations and k the number of right-hand-side variables (including the constant), adjusted R 2 is defined as 1 − (1 − R 2)(n − 1)/(n − k); since this quantity decreases in k, there is a penalty for the inclusion of extra covariates in a model. This penalty ensures that one set of variables will not receive credit for explaining a higher proportion of the variance simply because of the sheer number of covariates included in the model.Footnote 9
Table 2 shows that we cannot explain even a fifth of the total variation in reading difficulty with all of these covariates combined. Further, when we pay specific attention to the strategic variables, these covariates, taken together, explain an order of magnitude less variation than do case characteristics. Moreover, the individual coefficients on most of these variables fail to attain statistical significance or are signed in the wrong direction.
Of course, some courts of appeals cases are not “ripe” for strategy. These courts regularly dispose of cases that are routine, noncontroversial, and not especially prone to strategic ideological machinations (Bowie and Songer Reference Bowie and Donald2009). However, there are circumstances under which courts of appeals judges should be more likely to behave strategically. Following a similar logic to Owens et al. (Reference Owens, Justin and Patrick2013, 35), which concludes that Supreme Court justices strategically obfuscate the language of majority opinions to avoid unfavorable review from Congress, we theorize that opinion authors will write less clearly when the probability that their decision will be reviewed (en banc, by the Supreme Court, or by Congress) is greatest. As such, table 3 replicates the reading difficulty analysis of table 2 but only on a subset of cases in our data that are more likely to be reviewed. We define cases likely to be reviewed somewhat broadly: any case that features an amicus brief or a dissent or that reverses the trial court is included in this subset. Despite narrowing our focus to cases that more likely contain genuine controversy and disagreement, table 3 shows a strikingly similar pattern as table 2. Our strategic variables, taken together, explain very little variation in reading difficulty, and the individual coefficients (which should all be signed positively under a strategic model of opinion writing) are mostly not statistically significant, and several are signed in the wrong direction. For reading difficulty, at least, strategic calculations do not influence writing style in the circuits in the same way observed by other scholars in the Supreme Court (Black et al. Reference Black, Ryan J. Owens and Patrick2016b).
OLS Regression of Reading Difficulty on Strategic, Judge, and Case Characteristics, Strategy-Ripe Majority Opinions Only
Note.— Dependent variable is Flesch-Kincaid Grade Level reading difficulty score. Higher values of the dependent variable indicate a less readable/clear writing style. Robust standard errors in parentheses. Circuit and year fixed effects included but not reported in models 4 and 5 for brevity. N = 5,748.
* p < .05.
All Signed Majority Opinions
We continue with an analysis of all majority opinions in the data set, applying our approach exemplified above to nearly 100 different stylistic features. Table 4 shows the mean and median adjusted R 2 from the 98 sets of regressions—one set for each of our 98 text features, with each set including a regression of a given text characteristic on strategic, judge, and case variables, respectively. The results demonstrate that most of the variance in writing style remains unexplained despite our intentionally inclusive variable selection process. No set of variables explains a substantial amount of the variance in writing style. In fact, the collective impact of all covariates is limited; variables related to the case, judge, and strategic opportunities together explain approximately one-tenth of the total variance in writing style. These preliminary results suggest that despite the many advances in measuring writing style, most of the variance in the actual content of legal opinions remains unexplained.
Average Adjusted R 2 (across 98 Regressions) for All Text Characteristics, Signed Majority Opinions (N = 11,771)
Despite the limited variance that our covariates collectively explain, there are meaningful differences in the relative amount of variance that our sets of covariates explain, as shown in table 4. The largest substantive gap is between the variance explained by case variables and all other potential explanations. On average, case variables explain just over 8% of the variance in writing style. While not large in absolute magnitude, this represents a substantial increase compared to judge and strategic variables. Judge variables explain a little over 1% of the variation in writing style, while strategic variables explain less than one-half of 1% of the variation in writing style. Even measured crudely by issue area, year, and circuit, case variables explain about 25 times as much variance in writing style, on average, compared to variables associated with strategic behavior.
Table 4 indicates only a general pattern across dozens of textual features. Naturally, not all of these features are of equal import. Thus, figure 1 shows the adjusted R 2 associated with regressing the most theoretically relevant features in the literature—reading difficulty, cognitive complexity, informality, and affective language—on judge, case, and strategic variables, respectively. This more focused exercise explores whether the patterns in table 4 hold for the text features that are particularly meaningful theoretically.
Adjusted R 2 for four common textual features, all signed majority opinions
The general pattern observed in table 4 is largely replicated in figure 1. Case variables explain the most variance for three of the four text characteristics. Judge characteristics explain a higher proportion of the variance in cognitive complexity. This pattern may not be entirely surprising—cognitive complexity is theoretically distinct from other elements of writing style, as it seeks to measure the complexity of ideas employed (Owens and Wedeking Reference Owens and Justin2011, 1038). For the three elements associated exclusively with writing style—affective language, informality, and reading difficulty—variables related to the case explain substantially more variance. For each of these characteristics, case variables explain more than twice as much variance as variables associated with the judge or strategic considerations. For all four text characteristics (including cognitive complexity), case variables explain more variance than strategic considerations by an order of magnitude or more.
The results in table 4 and figure 1 suggest that the most important determinants of writing style are case variables. These variables consist of fixed effects for year, circuit, and issue area. A skeptical reader may question whether year and circuit truly represent characteristics of the case itself; is it possible that the quantity of variance explained by case variables is primarily a function of these two components? To address this concern, refer to table 5. The mean and median adjusted R 2 for the regressions of judge and strategic variables are unchanged. However, the regression for the case variables model now only includes fixed effects for issue area; both year and circuit are excluded from the analysis. While the total quantity of variance explained by issue area alone is lower, on average, it still explains substantially more variance than judge and strategic variables combined. A similar pattern holds for reading difficulty, cognitive complexity, informality, and affective language (not displayed). As with the more expansive definition of case variables, issue areas explain less variance than judge characteristics in cognitive complexity but more variance in informality and reading difficulty. However, judge variables explain more variance in affective language than issue areas alone. All of the results, taken together, suggest that case variables explain the greatest portion of the variance in judicial writing style, although the finding is less definitive if we define case variables narrowly.
Average Adjusted R 2 (across 98 Regressions) for All Text Characteristics, Signed Majority Opinions: Issue Area Only (N = 11,771)
Majority Opinions with Separate Opinions
Given the apparent influence of case variables on writing style, we examine their role further through an analysis of separate opinion writing. While fixed effects for year, circuit, and issue area capture many characteristics of cases, there are surely factors idiosyncratic to each individual case or narrow issue area or topic that are out of reach of such measures. To capture these more idiosyncratic factors, we compare writing style within cases by comparing majority and separate opinions authored in a common decision. Adopting this approach is useful for isolating the impact of case variables from other potential motivations and capturing case factors that cannot be captured by the blunter measures that are available.
We note that the motivation for strategic behavior differs substantially for separate opinion writers. Authors of concurrences and dissents do not typically need to worry about whether their legal and policy position is ideologically acceptable to other actors as majority authors might. Separate opinion writing also varies in terms of judge characteristics; separate opinion authors frequently have different individual characteristics than the majority opinion author. However, we know that separate opinions and a majority opinion share a common set of (unobserved) case variables. Comparing these types of opinions thus allows the most direct test of the effect of case variables on writing style.
To assess the magnitude of this effect, we follow a similar approach as outlined above. For all text characteristics, we regress judge variables, case variables, and strategic variables and report the adjusted R 2 values. We differ from our previous approach in two respects. First, we necessarily restrict our analysis to only cases that include at least one separate opinion. While this approach limits our sample size considerably—the bulk of majority opinions in the courts of appeals fail to inspire any accompanying separate opinions—we believe the inferences that can be drawn from this unique subsample of cases warrant narrowing the set of observations analyzed. Second, we include the adjusted R 2 from a fourth regression: one that regresses the separate opinion’s value of a text characteristic on that characteristic for the majority opinion. If the text characteristic of separate opinion(s) explains variance in the text characteristics of the majority opinion, then this explained variance can reasonably be attributed to shared idiosyncratic case factors.
Table 6 presents the results for the 1,283 signed opinions accompanied by at least one separate opinion. As was the case for the larger sample that included unanimous opinions, for majority opinions that attracted at least one separate opinion, case variables (fixed effects for year, circuit, and issue area) explain substantially more variance, on average, than do judge or strategic variables.Footnote 10 Most importantly, the text characteristics of separate opinions explain the largest quantity of variance in writing style. The observed magnitude is noteworthy. For example, the text characteristics of the separate opinion(s) explain nearly 17 times the amount of variance in the text characteristic of majority opinions as do judge characteristics and 40 times the amount of variance as do strategic considerations. Additionally, our blunter measure of case variables accounts for only about half as much variance in textual features of majority opinions, compared with the same text characteristic in separate opinions.
Average Adjusted R 2 (across 98 Regressions) for All Text Characteristics, Signed Majority Opinions with Separate Opinions (N = 1,283)
To probe the existence of these patterns in the most theoretically relevant text characteristics, figure 2 shows the adjusted R 2 values of regressions of reading difficulty, cognitive complexity, informality, and affective language on judge, case, and strategic variables, as well as the corresponding text characteristic in the case’s separate opinion(s), respectively. The results demonstrate that for these four commonly used text characteristics, variables related to the case explain far more variance than those relating to the judge or strategic behavior. Levels of informality and affective language in separate opinion(s) explain the greatest portion of variance in the informality and affective language in corresponding majority opinions. This is particularly noteworthy for affective language; the level of affective language used in a separate opinion explains approximately 35% of the variance in the levels of affective language in the majority opinion. For both text characteristics, our blunter measure of case variables (i.e., year, circuit, and issue area) explains the second-largest proportion of variance. For cognitive complexity and reading difficulty, the opposite pattern occurs: case variables explain the largest portion of the variance, with the levels of cognitive complexity and reading difficulty, respectively, in the separate opinion accounting for the second-largest proportion of variance. As figure 2 demonstrates, while some differences exist across different text characteristics, a common theme emerges: case variables explain the largest portion of variance in judicial writing style. Judge variables and variables capturing strategic behavior explain comparatively less.
Adjusted R 2 for four common textual features, signed majority opinions with separate opinions.
Conclusion
Automated text analysis now commands great interest from scholars of judicial politics. This level of interest is unquestionably warranted; the ability to systematically classify the content of judicial opinions creates the opportunity to test hypotheses previously out of reach in large samples. Many scholars have taken advantage of these opportunities, and our collective understanding of judicial behavior has grown as a result (Owens and Wedeking Reference Owens and Justin2011; Corley and Wedeking Reference Corley and Justin2014; Goelzhauser and Cann Reference Goelzhauser and Damon2014; Nelson Reference Nelson2014; Black et al. Reference Black, Hall, Owens and Eve2016, Reference Black, Ryan J. Owens and Patrick2016b; Bryan and Ringsmuth Reference Bryan and Eve2016).
In the pursuit of testing these novel hypotheses, it is easy to overlook foundational questions central to this burgeoning literature. In this analysis, we address what is among the most fundamental questions to consider: What determines the writing styles used to craft judicial opinions? Most judicial scholarship using automated text analysis examines strategic choices in writing style. Our results suggest that much of the variance observed in writing styles remains unexplained, even when accounting for an inclusive array of potential explanatory variables. The explained variance in stylistic differences does not appear to be driven by strategic variables, nor is it a function of judge characteristics. Rather, it appears that variables associated with the case are the most substantial influence. This result does not call into question the results of previous scholarly endeavors; we instead hope to have provided a greater understanding of both the potential promise and limitations of text analysis of judicial opinions.
In that spirit, we close by identifying several questions important to the application and interpretation of text analysis in the context of judicial opinions. We also offer some preliminary reactions to these questions in the hope of continuing an important dialogue in what is clearly a fruitful avenue for future research by judicial politics scholars.
First, what is the substantive impact of judicial writing style? Previous scholarship examining text characteristics has found statistically significant associations between these characteristics and a variety of behaviors associated with strategic calculations. However, what is the substantive impact of these relationships? Our results here suggest that the substantive differences in style explained by strategic variables are likely small. Scholars working in this area might consider paying special attention to the substantive, rather than statistical, significance of these relationships. Specifically, one might present changes in predicted outcomes or probabilities over plausible ranges of a text variable, or one might present measures of model fit from specifications that include only the text variable predictor(s). If strategic calculations explain less than one-half of 1% of a document’s reading difficulty or affective tone, can such a difference be perceived by the judge’s audience? If not, what does this say about the likelihood that these differences arise because of a judge’s strategic decision? To advance the field of judicial politics with respect to text analysis of judicial opinions, these questions require substantial attention.
Second, could excessive noise in measures of writing style account for the relatively low variance we see explained? If so, there are several standard recommendations to address such concerns. Scholars should pause before adopting measures of writing style developed in contexts other than legal opinions and consider whether modification is appropriate before application to judicial texts. Relatedly, one ought to consider whether meaningful systematic variation in the measure is likely to exist across legal opinions, which, we suspect, occupy a relatively narrow stylistic range across some attributes, given authors’ shared training. When possible, use multiple indicators for a single construct, and ensure that the individual measures load onto a single factor with an eigenvalue above 1 (Kaiser Reference Kaiser1960). Preprocessing decisions may also affect the amount of noise present in the measure (Denny and Spirling Reference Denny and Arthur2018). Ensure that the measures chosen specifically match the context being analyzed (Grimmer and Stewart Reference Grimmer and Brandon2013; Benoit, Munger, and Spirling Reference Benoit, Munger and Arthur2017). Move beyond emphasizing statistical significance in reporting results, examining model fit and effect size carefully (Gelman and Stern Reference Gelman and Hal2006). Last, avoid small samples when possible: in this context, noisy measures can lead to substantively large (and statistically significant) apparent effect sizes even when the true effect is tiny (Loken and Gelman Reference Loken and Andrew2017).
Researchers should not necessarily abandon the search to explain variability in judicial writing style. Style may truly be largely stochastic. However, it may also be the case that better measurement of key covariates could reveal a greater systematic component to writing style. For instance, our measurement of legal issue area is admittedly coarse. Future scholars could consider refining this variable: beyond a broad, substantive issue area, how do particular legal topics, doctrinal areas, and case facts relate to writing style? Alternatively, perhaps judicial writing style does emanate from a judge’s innate skill. Better measuring the training, experience, and talent of a judge could potentially explain a great deal more of the variability in style. Future work should entertain these possibilities.
Third, how does the use law clerks affect the measurement of writing style? Many judges use the assistance of law clerks when drafting opinions (Ward and Weiden Reference Ward and David2006, 200–237; Bowie, Songer, and Szmer Reference Bowie, Songer and John2014, 92–93). While the use of law clerks could take a variety of forms, evidence suggests that clerks have played an increasingly prominent role in the actual writing of opinions over time. At the Supreme Court, according to Ward and Weiden (Reference Ward and David2006, 212), “the most common form [of opinion writing process] is delegation, which involves a justice assigning the first draft of an opinion to a clerk and then revising the clerk’s original draft.” A similar pattern appears to occur in the courts of appeals. According to Bowie, Songer, and Szmer (2014, 92), “Many judges ask their clerk to write a first draft of the opinion, after providing guidance about the direction the opinion should take.”
This working arrangement leads judges to cede some control over the style in which the opinion is written to clerks. As the author of the first draft, clerks necessarily play an important role in the selection of language and grammatical structure used in judicial opinions. Even if a judge “heavily edits” the initial draft of opinions (Bowie, Songer, and Szmer 2014, 93), it is important to determine whether those edits are directed toward style, substantive content, or both. Surely these edits must include concerns about substantive content; it is unlikely that the delegation of the early opinion drafting would lead to the dissemination of opinions containing substantive content that is inconsistent with judges’ preferences. But do judges instruct clerks as to writing styles? Are the judges’ edits heavy enough to change not only the substantive content of the opinion but the style in which it was originally drafted? Future examinations of judicial writing style should consider the role that law clerks play in the opinion writing process.
Next, how broadly should scholars conceive of judicial writing style? Several distinct elements of writing style have already emerged in the literature, with more to surely come as the quality of automated text analysis improves. Scholars have already examined affective language (Bryan and Ringsmuth Reference Bryan and Eve2016), certainty (Corley and Wedeking Reference Corley and Justin2014), cognitive complexity (Owens and Wedeking Reference Owens and Justin2011), and reading difficulty (Owens et al. Reference Owens, Justin and Patrick2013; Black et al. Reference Black, Ryan J. Owens and Patrick2016a, Reference Black, Ryan J. Owens and Patrick2016b), the last of which has received a disproportionate share of scholarly attention. While specific hypotheses may dictate which element of writing style warrants investigation in a specific instance, the literature is largely silent in offering a broader justification for prioritizing certain elements of writing style over others. For example, is reading difficulty a more important element of writing style than certainty, affective language, or informality, as the current literature seems to imply? A more complete theoretical development of how these elements fit together, and what the relationship between these elements means for judicial opinion writing, is warranted.
Finally, what are the costs of writing in specific styles? Strategic explanations of judicial opinion writing suggest that judges vary their writing style to achieve specific goals. In many analyses, judges are hypothesized to alter an element of writing style, under certain conditions, to affect the behavior of others. If, for example, decreased reading difficulty increases the likelihood of lower-court compliance with Supreme Court opinions (Black et al. Reference Black, Ryan J. Owens and Patrick2016b, 141–55), why would justices not always minimize the reading difficulty of an opinion to help achieve compliance? The most apparent explanation must be that there is a cost associated with changing writing styles. Currently, this answer is largely assumed rather than tested. Further examinations of writing style should explore the cost of writing in particular styles in greater detail to see whether opinions written in a particular style are in fact more costly.Footnote 11 If changing writing styles is costless, explanations as to why judges do not always maximize the strategic impact of their writing style are needed.
