Judicial Impact and Factual Allegations: How the Supreme Court Changed Civil Procedure through the Plausibility Standard

Morgan L. W. Hazelton

doi:10.1086/712653

Judicial Impact and Factual Allegations: How the Supreme Court Changed Civil Procedure through the Plausibility Standard

Published online by Cambridge University Press: 21 October 2022

Morgan L. W. Hazelton

Show author details

Morgan L. W. Hazelton*: Affiliation:
Saint Louis University, USA
*: Contact the author at morgan.hazelton@slu.edu.

Article contents

Abstract
Introduction
Background: Pleading Standards and Incentive Structures
Literature and Theory
Method
Analyses and Results
Discussion
Footnotes
References

Rights & Permissions

Abstract

Researchers often study the impact of legal change by investigating judges’ decisions and ignoring litigants. Many scholars believe the Supreme Court’s decisions in Twombly and Iqbal increased how specific factual allegations must be to avoid dismissal, but studies generally fail to find an effect. Using text analysis, I find evidence that litigants and their lawyers changed the information they provided after the decisions in certain types of cases. These results call into question prior studies and illustrate the need to consider the behavior of litigants. They also help shed light on issues of access to courts and separation of powers.

Information

Type: Articles
Information: Journal of Law and Courts , Volume 9 , Issue 1 , Spring 2021, pp. 159 - 188
DOI: https://doi.org/10.1086/712653 [Opens in a new window]
Copyright: © 2021 by the Law and Courts Organized Section of the American Political Science Association. All rights reserved.

I. Introduction

Beyond whether law matters, most law and courts scholars agree that the real concern is when, how, and why law matters (see, e.g., Friedman and Martin Reference Friedman, Martin and Geyh2011, 145–49). This interest extends well beyond formal legal decisions and the outcomes of cases for the immediate parties to naturally encompass the consequences of such pronouncements in the world at large (e.g., Rosenberg Reference Rosenberg2008; Hall Reference Hall2011). Unfortunately, for many reasons, researchers struggle to capture the effects of law. For example, scholars often use blunt measures of case disposition as dependent variables (see, e.g., Lax Reference Lax2011). This approach focuses on judges’ actions to the exclusion of litigant behavior (e.g., Baum Reference Baum1988; McGuire et al. Reference McGuire, Vanberg, Smith and Caldeira2009). Additionally, studies tend to focus on the small minority of cases that proceed to high courts after multiple layers of filtration and selection (see, e.g., Friedman and Martin Reference Friedman, Martin and Geyh2011).

It is important to understand how law, including policy announced by the Supreme Court, affects society. Such inquiries require us to carefully theorize how such effects will be felt by individuals and endeavor to expand our tools and approaches to investigate such ramifications. I contribute to a reemerging literature investigating judicial impact by focusing on responses by litigants and their lawyers to changes in federal civil procedure, specifically pleading practice, brought on by two Supreme Court decisions, Bell Atlantic Corp. v. Twombly (550 U.S. 544 [2007]) and Ashcroft v. Iqbal (556 U.S. 662 [2009]) (collectively “Twiqbal”).Footnote ¹ My findings show that litigants in varying circumstances responded to the decisions in ways that have not been fully appreciated. These changes indicate that the Court effectively altered the rules of procedure without the intervention of Congress and in ways that raise questions regarding access to justice. These results also speak to larger issues in the study of law and courts regarding the need to consider the impact of law on the natural consumers of such policies, including litigants and their lawyers in light of changes in vertical precedent.Footnote ² It also highlights the importance of considering variation among legal issues and the content of legal documents.

Legal scholars have long focused on the extent to which law engenders stability and predictability (see, e.g., Schauer Reference Schauer1987). There is a wealth of important theoretical and empirical work that indicates that individuals determine how to act based on their expectations (e.g., Galanter Reference Galanter1974; Priest and Klein Reference Priest and Klein1984; Baum Reference Baum1988). Thus, we would not anticipate that law simply dictates results. Instead, we would expect that nonelites alter their actions in light of legal changes. Despite this, because of the difficulties in capturing such behavior, we know relatively little regarding responses by the targets of policies, including litigants.Footnote ³ Most law and courts research focuses on judicial behavior and appellate outcomes, even though the strongest effects of law likely occur among individuals (see, e.g., Becker and Feeley Reference Becker and Feeley1973; Wasby Reference Wasby1984; Canon and Johnson Reference Canon and Johnson1999; Wheeler Reference Wheeler2010; Hall Reference Hall2011; Boyd Reference Boyd2015).

As part of attempts to understand the implications of law, some researchers have considered the effects of procedural rules (e.g., McCubbins, Noll, and Weingast Reference McCubbins, Noll and Weingast1987; Spiller and Ferejohn Reference Spiller and Ferejohn1992; Kessler Reference Kessler1996; Binder and Smith Reference Binder and Smith1997; Yoon and Baker Reference Yoon and Baker2006; Cox, Thomas, and Bai Reference Cox, Thomas and Bai2008; Kritzer Reference Kritzer2008; Yackee and Yackee Reference Yackee and Yackee2010). Unfortunately, studies have generally focused on differences in outcomes, as has research on the impact of Supreme Court decisions on lower courts (e.g., Canon Reference Canon1973; Johnson Reference Johnson1987; Songer Reference Songer1987; but see, e.g., Stidham and Carp Reference Stidham and Carp1982; Kritzer and Zemans Reference Kritzer and Zemans1993; Spriggs Reference Spriggs1996; Kritzer Reference Kritzer2008, Reference Kritzer2015; Hall Reference Hall2011). We would anticipate that if procedural rules have meaning and Supreme Court decisions are influential, then litigants, especially those represented by attorneys, will adapt their behavior in light of such changes. Such adaptations likely include differences in decisions regarding whether and how to engage the courts. Additionally, while judicial proceedings at the trial court level are exceedingly important because they are the most common and often final engagement that individuals have with courts, they are generally understudied (see Rowland and Carp Reference Rowland and Carp1996; Levin Reference Levin2008; Boyd Reference Boyd2009; Boyd and Hoffman Reference Boyd and Hoffman2013).Footnote ⁴ This is particularly troubling because the actions of litigants in trial courts define the contours of all subsequent legal proceedings, including appeals (Boyd Reference Boyd and Lindquist2017). Considering the behavioral changes of litigants in district courts as the result of procedural changes helps shed light on these areas. It also speaks to a renewed interest in judicial impact (see, e.g., Sweet Reference Sweet2010; Hall Reference Hall2011).

I delve into how represented litigants in federal district courts responded to apparent changes in procedural pleading rules, to help illuminate judicial impact. Such standards dictate the type and amount of information that litigants must include in filings to pursue a claim. Specifically, I consider how changes to the federal pleading standard announced by the Court in Twombly and Iqbal altered how attorneys representing parties provide information about claims in filings. These decisions provide a unique vantage point due to their sweeping nature and legal salience: they apply in nearly every federal civil case and are among the most cited cases of all time (see, e.g., Fitzpatrick Reference Fitzpatrick2011). They are understood by many scholars to have raised the bar to maintain a civil suit in federal court by changing the requirements for how detailed factual allegations must be to avoid dismissal. Because of the nature of different types of claims, there is variation in the amount of information that is available to litigants before they file suit: specifically, in some types of cases, defendants are more likely to have exclusive control over essential evidence (Spencer Reference Spencer2008; Miller Reference Miller2010). To the extent that the Court’s decisions represent a more strenuous standard, we would anticipate that represented litigants would respond to them by stating narrower and more specific claims when possible. I expect this shift will be most pronounced for types of claims where information is readily available to the plaintiff. Such behavior has serious potential implications for subsequent legal proceedings and our understanding of them.

Using a supervised model of allegation specificity, I find evidence that litigants represented by lawyers changed the level of specificity they included in the factual allegations in cases where information is more readily accessible (torts) after the Court made it clear that a new standard was in place. I find no such evidence in cases that are considered hard to plead (civil rights). This change in pleading behavior raises questions regarding findings that these landmark decisions have little to no impact on filings and dismissals and other downstream consequences: to the extent that the bulk of change brought about by the cases occurred in the claim-stating stage, the effect has not been detected or fully appreciated because complaint content is challenging to quantify and, thus, generally ignored. Failing to address such changes has the potential to bias analyses against detecting changes in outcomes. The differences in the changes in pleadings by issue area may also indicate that litigants who need the formal exchange of information allowed in lawsuits the most, such as plaintiffs claiming conspiracy or official misconduct, are less able to adapt. Thus, the Court’s decisions may have altered the pleading standard and, consequently, the allocation of judicial resources without the benefit of congressional action. The measure and related findings provide a strong basis for continuing work in understanding judicial impact and the consequences of litigant responses for our understanding of legal change.

In the next section, I discuss the federal pleading standards, the Twombly and Iqbal decisions, and the incentive structures they created. I then survey the relevant literature and further develop my theory regarding litigant behavior concerning pleading. Next, using a classifier that I developed to measure specificity, I analyze the specificity of factual allegations, in cases that are easy and hard to plead, before and after the decisions. Finally, I discuss the implications and limitations of this research as well as future avenues of inquiry.

II. Background: Pleading Standards and Incentive Structures

An overview of the function of civil complaints in the legal system highlights their importance. A civil case commences when parties (known as plaintiffs or complainants) claiming that an individual or entity has harmed them file a written document outlining their claims in the form of a complaint. A complaint includes many types of information: a caption identifying the relevant parties, a description of the names and address of the parties, an introduction, statements regarding jurisdictional and venue matters, an identification of the causes of action, a listing of the types of damages sought in a prayer, and information regarding the attorneys, such as their addresses (e.g., O’Connor Reference O’Connor2019, form 2B:5). Most importantly, a complaint must include information regarding the factual allegations against the alleged wrongdoer, known as the defendant. According to the relevant procedural rules, complaints “must contain … a short and plain statement of the claim showing that the pleader is entitled to relief” (Federal Rules of Civil Procedure [FRCP] 2018, rule 8[a]).

The purpose of the filing is generally understood as to “give defendants fair notice of its claims and the grounds for them” (Baicker-McKee, Janssen, and Corr Reference Baicker-McKee, Janssen and Corr2020, 165; see Holland Reference Holland1981). In the absence of such notice, complainants are generally not allowed to pursue related claims. For example, courts will generally not allow complainants to carry out discovery regarding claims for which there was not sufficient notice (see FRCP 2018, rule 26[b][1]). The evidence collected in this formal exchange of documents, information, and testimony can often be of central importance to a plaintiff’s ability to prove his or her case (Dodson Reference Dodson2010). Courts can preclude plaintiffs who fail to meet the pleading standard from presenting related evidence at trial or collecting damages based on such claims (see FRCP 2018, rule 15[b], [c]). Such presentations directly shape the official factual record of the case, which is created in the trial court (see Boyd Reference Boyd and Lindquist2017). Thus, the avenues of recovery laid out in the complaint set the stage for all subsequent legal proceedings, including trials and appeals (see Boyd Reference Boyd2015, Reference Boyd and Lindquist2017). Therefore, pleaders have an incentive to cover as many claims as possible to maximize flexibility regarding potential avenues of discovery and recovery.Footnote ⁵

Of course, not all claims are equal. Litigants have varying ability to identify wrongdoing on the part of defendants due to differences in culpability, access to information, and ability to pay for pretrial investigation (Gordon Reference Gordon2011). Some types of claims are “hard to plead,” as discovery is often necessary to allow the plaintiff sufficient information to support his or her claim (see, e.g., Reinert Reference Reinert2010; Gelbach Reference Gelbach2012). Generally, these types of claims involve wrongdoing that was done in secret or acts that are carried out for unlawful reasons that defendants are unlikely to acknowledge. Examples of hard-to-plead claims include “antitrust, conspiracy cases, employment discrimination, medical malpractice, eminent domain challenges, and violations of civil rights” (Curry and Ward Reference Curry and Ward2013, 837; citations omitted).

To illustrate, we would anticipate that someone who is a victim of race- or gender-based discrimination at work (a civil rights claim) would have a harder time obtaining relevant information than someone involved in a car accident (a tort claim). Even before Twiqbal, such claims were hard to prove, and attorneys were wary of taking them (e.g., Clermont and Schwab Reference Clermont and Schwab2009).Footnote ⁶ Overtly discriminatory acts are rare compared to more subtle behavior requiring additional evidence of intent. Generally, access to information regarding systemic behavior, private communications among employers and employees, personnel files, and the like is essential to proving such cases (Miller Reference Miller2010, 45–46). Defendants generally control such evidence (Miller Reference Miller2010). On the other hand, tort claims involve civil wrongs, including assault, personal injury, and product liability, where the plaintiff was often present for the physical acts that are at the heart of the suit. Defendants are less likely to have exclusive control over essential evidence in such cases.

Thus, we would anticipate that the accessibility of information varies by issue area. Additionally, judges may be more wary or hostile to claims that are hard to plead, such as civil rights claims, because of ideological predispositions (see Rowland and Carp Reference Rowland and Carp1996). Despite the variation in accessibility and judicial climate, all claimants have incentives to plead more broadly within a permissive environment (see Schwartz and Appel Reference Schwartz and Appel2010). The larger difference among litigants is their ability to adapt to a stricter setting (Gordon Reference Gordon2011; Wasserman Reference Wasserman2012).

A party defending against a suit can seek to have the court dismiss a claim on the basis that the complaint “fail[s] to state a claim upon which relief can be granted” (FRCP 2018, rule 12). Generally speaking, such motions to dismiss are filed before the parties have conducted discovery (see Gelbach Reference Gelbach2012). For most of the modern legal era, federal courts were guided by the conceivability standard when considering such motions. That standard established that a complaint should only be dismissed for failure to state a claim if no possible set of facts could support it (Conley v. Gibson, 335 U.S. 41 [1957]; e.g., Hatamyar Reference Hatamyar2010; Reinert Reference Reinert2010; Stewart Reference Stewart2010, 169). Conceivability is “a very low hurdle designed essentially to keep lawsuits in court, not out” (Coyle Reference Coyle2013, 223). It was the well-established standard (see, e.g., Hatamyar Reference Hatamyar2010), although some courts may have required more in some cases (Miller Reference Miller2010; Schwartz and Appel Reference Schwartz and Appel2010).

The conceivability benchmark defined the legal standard for 50 years. That changed in 2007, however, when the Supreme Court held in Twombly (an antitrust case) that a complaint must state a plausible, rather than conceivable, claim for it to be sustained. This announcement caught the legal world by surprise (e.g., McMahon Reference McMahon2007; Redish and Epstein Reference Redish and Epstein2008): “Twombly was a bombshell” (Hubbard Reference Hubbard2012); “there was certainly no groundswell to reexamine [the conceivability standard], and no one thought that it was in danger of being altered” (McMahon Reference McMahon2007). After the decision, there was debate within the legal community as to the applicability of the plausibility standard beyond antitrust cases and the exact nature of the standard (see Redish and Epstein Reference Redish and Epstein2008; Hatamyar Reference Hatamyar2010). An empirical study of district courts’ application of the Twombly standard revealed general incoherence (Redish and Epstein Reference Redish and Epstein2008).

Two years later, in Iqbal, the Supreme Court applied the plausibility standard in a civil rights case, thereby dispelling the vast majority of doubt that it was not generally applicable (see Noll Reference Noll2010). The Court also attempted to clarify its rule by stating that plausibility was a “context-specific” determination and that factual allegations were essential in stating claims (15). Thus, the transition from conceivability to a general plausibility standard was complete. While the Court took pains to clarify that the plausibility standard does not require plaintiffs to state all facts supporting their claims, it is clear that the standard requires more factual allegations and specificity than a reading of Conley would imply. With this background in mind, I consider theories of judicial impact and how these decisions provide fertile ground for exploring legal change.

III. Literature and Theory

The importance of understanding how policy makers, such as legislators and judges, make decisions should not be understated. But, it is also of utmost importance that we understand the implications of such decisions. We care about such policies because we believe they are meaningful, and such meaning should also be studied (see, e.g., Rosenberg Reference Rosenberg2008; Hall Reference Hall2011). In conjunction with such inquiry, some scholars in the field have considered the impact of policy in the form of judicial decisions (e.g., Canon and Johnson Reference Canon and Johnson1999; Reed Reference Reed2001; Kritzer Reference Kritzer2008; Keck Reference Keck2009). In fact, after a relative lull in scholarship on the matter (see Baum Reference Baum2011), there has been a revived interest in judicial impact (e.g., Hall Reference Hall2011). There are major theories regarding how populations differ concerning legal change, the role of law in shaping individual behavior, and the conditions under which Supreme Court decisions are most likely to be impactful. These theories indicate that focused studies on individuals who are directly affected by legal changes are particularly helpful in assessing judicial impact. Twiqbal is a ripe area to consider the implication of judicial policy, particularly when considering the behavior of represented litigants.

A. Judicial Impact, Strategic Behavior, and Represented Litigants

When law changes via a judicial decision, the precedent pulls different groups within society into relationship with it (Canon and Johnson Reference Canon and Johnson1999; see Bowen Reference Bowen and Epstein1995). Canon and Johnson (Reference Canon and Johnson1999) theorize there are four central populations in considering judicial impact: interpreting, implementing, consumer, and secondary. Lower courts and other officials charged with the authoritative interpretation of the policy, such as attorneys general, make up the interpreting population (Canon and Johnson Reference Canon and Johnson1999, 18). Judicial impact research often focuses on the response of the interpreting group, such as compliance by lower courts (see Wheeler Reference Wheeler2010; e.g., Benesh and Reddick Reference Benesh and Malia2002). The implementing group generally consists of authorities affected by the decision, such as police officers or prosecutors (Canon and Johnson Reference Canon and Johnson1999, 19). Although implementers are more rarely studied, there are exceptions (e.g., Spriggs Reference Spriggs1996; Hume Reference Hume2009).

The consumer population represents those individuals who are generally targeted and directly affected by the decision, either positively or negatively (Canon and Johnson Reference Canon and Johnson1999, 20).Footnote ⁷ This group is more rarely the target of academic research (but see, e.g., Levine Reference Levine and Feeley1973; Bowen Reference Bowen and Epstein1995; Kritzer Reference Kritzer2008). Despite this general neglect of the consumer population, “knowledge about consumption is crucial to understanding the ultimate impact of judicial politics” (Canon and Johnson Reference Canon and Johnson1999, 95–96). Everyone else is part of the secondary population (22). Research regarding the impact on these individuals generally takes the form of studies of public opinion regarding Supreme Court decisions (see, e.g., Hoekstra Reference Hoekstra2000). This framework indicates that scholars need to be careful when investigating how policies announced by courts affect society.

When we consider judicial impact, we must think through how various populations are likely to react and how those reactions are interconnected (see Canon and Johnson Reference Canon and Johnson1999): “Exactly whose behavior should we expect to change in response to a judicial ruling, and how should we expect their behavior to change?” (Hall Reference Hall, Epstein and Lindquist2017, 454). The precise focus of study depends on the circumstances of the legal change and should include those targeted by the policy (see Canon and Johnson Reference Canon and Johnson1999; Hall Reference Hall2011). The importance of focused studies to our broader understanding of judicial impact should not be underestimated. As Hall (Reference Hall, Epstein and Lindquist2017, 464) notes, “studies that focus narrowly on the causal relationship between judicial rulings and behavior … offer greater conceptual clarity and analytical rigor” than other studies of judicial impact that focus on substantively broader phenomena (see also Barnes and Burke Reference Barnes and Burke2006, 517).

Hall (Reference Hall2011, Reference Hall, Epstein and Lindquist2017) theorizes that due to issues of hierarchical control, the implementation of a decision is more likely when it depends on lower courts (vertical precedent) than when it depends on outside actors (lateral precedent). When the decision relies on federal district courts, the implementation is free from interference from state and local governments or other actors who might frustrate such effects (see Sweet Reference Sweet2010). The impact of vertical precedent is by no means limited to compliance by lower courts (e.g., Becker and Feeley Reference Becker and Feeley1973; Canon and Johnson Reference Canon and Johnson1999).

Scholars have an abundance of reasons to believe that such changes in law influence underlying behavior among litigants, who anticipate compliance by lower-court judges (see, e.g., Galanter Reference Galanter1974; Priest and Klein Reference Priest and Klein1984; Baum Reference Baum1988; McGuire et al. Reference McGuire, Vanberg, Smith and Caldeira2009). We assume that complainants wish to maximize their recovery, while defendants want to limit what they lose (see, e.g., Priest and Klein Reference Priest and Klein1984; Songer, Cameron, and Segal Reference Songer, Cameron and Segal1995).Footnote ⁸ Thus, litigants will behave anticipatorily: they form expectations about the likely outcomes of actions and choose the most advantageous course (Singer Reference Singer2014, 9). Litigants base their beliefs about potential consequences in part on their understanding of the preferences of other parties and judges in suits and how these actors will behave accordingly (see, e.g., Priest and Klein Reference Priest and Klein1984; Fox and Birke Reference Fox and Birke2002; Singer Reference Singer2014).

Represented litigants rely on their attorneys to help them maximize outcomes. Commentators and scholars have identified predicting outcomes and formulating strategy based on those expectations to be among attorneys’ most essential functions for clients (Singer Reference Singer2014, 9, citing Goodman-Delahunty et al. Reference Goodman-Delahunty, Granhag, Hartwig and Loftus2010). Attorneys act based on strategy formed in light of predictions regarding the judges in front of whom they practice (Singer Reference Singer2014). An advocate acting on behalf of his or her clients plays a multifaceted role regarding the impact of precedent (Canon and Johnson Reference Canon and Johnson1999). These varied functions arise from the relationship of the attorney to the client and beyond: lawyers are expert agents for their clients, but at the same time, they are also bound by professional codes of conduct (see Pierce, Cornett, and Long Reference Pierce, Cornett and Long2011). Additionally, plaintiff’s attorneys generally work on a contingency basis: they only recover attorney’s fees if the client wins (see Kritzer Reference Kritzer2002). Thus, they also filter which types of cases to take based on their expectation of recovery (Kritzer Reference Kritzer1997, Reference Kritzer2004, Reference Kritzer2008, Reference Kritzer2015). Thus, Canon and Johnson (Reference Canon and Johnson1999) describe attorneys as acting as quasi consumers of the decisions when they advise clients regarding decisions. Attorneys also work to interpret and implement decisions (Wasby Reference Wasby1984, 23; Canon and Johnson Reference Canon and Johnson1999). Given the multiple roles of attorneys, they often act as linkages between populations (Canon and Johnson Reference Canon and Johnson1999, 24).

The primary consumer population for the Twombly and Iqbal decisions is parties to potential federal civil claims, both plaintiffs and defendants. The majority of federal litigants are represented by attorneys (Levy Reference Levy2018). These attorneys are highly incentivized to be aware of and react to changes in the pleading standard. In regard to the Twiqbal decisions, attorneys representing plaintiffs are intricately involved in a critical act: they draft complaints and determine what level of factual specificity to include (see generally Fox and Birke Reference Fox and Birke2002; Kritzer Reference Kritzer2008, Reference Kritzer2015; Singer Reference Singer2014). Thus, we would anticipate that complaints filed by represented litigants will reflect the expert judgment of their attorneys. Therefore, in considering the impact of the Twombly and Iqbal decisions, theory indicates that we should consider the responses of such litigators to the decisions (see Kritzer Reference Kritzer2008, Reference Kritzer2015). With this in mind, I now turn to existing research on the impact of these cases.

B. Interest and Research Regarding the Plausibility Standard

Unlike policies that affect only specific and discreet areas of law, Twiqbal implicates the vast majority of civil causes of action and cases. This is, in part, because the plausibility standard pertains to initial filings, which are present in almost every federal civil case. Civil cases represent the bulk of the federal docket (see Roberts Reference Roberts2018). Other procedural changes that influence later-stage activities (such as discovery or motions for summary judgment),Footnote ⁹ while important and undoubtedly influential even in cases that never reach those stages, lack Twiqbal’s scope of direct applicability.Footnote ¹⁰

Thus, it should be of no surprise that the legal community and officials have been keenly interested in the ramifications of Twombly and Iqbal (e.g., New York Times 2009; US Congress 2009a; Gordon Reference Gordon2011; Gelbach Reference Gelbach2012). As Marshall and Postman (Reference Marshall and Postman2010) describe: “Few issues are more important in federal litigation than determining whether a case will be dismissed for failure to state a claim or instead slog on into discovery, potential fights over class certification and beyond. And following the Supreme Court’s decisions in Bell Atlantic v. Twombly (2007) and Ashcroft v. Iqbal (2009), few issues have generated as many questions.” Although the decisions’ potential impact is far-reaching, they have garnered much less scholarly attention outside the legal academy (see Yacobucci and McGovern Reference Yacobucci and McGovern2014). The decisions also did not receive much media attention despite their widespread implications (Coyle Reference Coyle2013, 221). However, there has been congressional attention to the decisions (US Congress 2009a, 2009b, 2010, 2011). Moreover, the decisions rank among the top cases for citations of all time (see, e.g., Fitzpatrick Reference Fitzpatrick2011).Footnote ¹¹

A significant portion of the legal academy believes that the decisions represent an upward shift in the level of specificity that litigants are required to provide in pleadings to avoid dismissal (see, e.g., Hatamyar Reference Hatamyar2010; Miller Reference Miller2010; Schwartz and Appel Reference Schwartz and Appel2010; but see Garre Reference Garre2009; Moller Reference Moller2009).Footnote ¹² Such scholars believe that these precedents have a profound effect on the ability of some parties to seek redress via the federal courts, although they disagree regarding the normative implications. For example, Miller (Reference Miller2010, 71) argues that the decisions endanger “access to courts, deterrence, and the private enforcement of public policies” by creating barriers to litigation. Schwartz and Appel (Reference Schwartz and Appel2010, 140–41), on the other hand, assert that Twiqbal represents a positive development that protects defendants from unfair burdens in court via increased judicial gatekeeping. Additionally, there are concerns regarding the separation of powers and the role of the Court, as substantive changes to the FRCP are the province of Congress (Miller Reference Miller2010). Furthermore, some scholars are concerned that the new standard disproportionately impacts hard-to-plead causes of action, including civil rights cases (see, e.g., US Congress 2009a; Hatamyar Reference Hatamyar2010; Schneider Reference Schneider2010; Gelbach Reference Gelbach2012). Such cases primarily consist of statutory discrimination claims (US Courts 2018b). Thus, there are potential ramifications as to redress and redistribution as well as access to courts, which operates as “a basic political resource” (Grossman and Sarat Reference Grossman and Sarat1981).

The Twiqbal decisions have spurred a large number of empirical legal studies aimed at determining their impact (see Hubbard Reference Hubbard2017). The work that has been done in this area helps us consider why and how we should study judicial impact. The first and primary focus of such studies has been on dismissals in the wake of one or both of the decisions (see, e.g., Hannon Reference Hannon2007; Martinez Reference Martinez2009; Seiner Reference Seiner2009, Reference Seiner2010; Hatamyar Reference Hatamyar2010; Cecil et al. Reference Cecil, Cort, Williams and Bataillon2011; Dodson Reference Dodson2012; Gelbach Reference Gelbach2012; Hubbard Reference Hubbard2012; Engstrom Reference Engstrom2013; Epstein, Landes, and Posner Reference Epstein, Landes and Posner2013; Yacobucci and McGovern Reference Yacobucci and McGovern2014).Footnote ¹³ Most of these studies ignored the potential that litigants and their lawyers altered their behavior while also relying on published opinions (see Gelbach Reference Gelbach2012; Hubbard Reference Hubbard2012; Engstrom Reference Engstrom2013), but notable exceptions exist (see, e.g., Cecil et al. Reference Cecil, Cort, Williams and Bataillon2011; Gelbach Reference Gelbach2012; Hubbard Reference Hubbard2012, Reference Hubbard2017; Reinert Reference Reinert2015).Footnote ¹⁴ Generally, the results of this type of study have been mixed, and there is not strong evidence that these cases have had much of an impact (but see Dodson Reference Dodson2012). Attorney surveys on the subject have also yielded a varied picture regarding dismissals (see Hamburg and Koski Reference Hamburg and Koski2010; Lee and Willging Reference Lee and Willging2010a, Reference Lee and Willging2010b; Willging and Lee Reference Willging and Lee2010).

Additionally, a few scholars have considered dismissal rates in light of selection using varying techniques. For example, Hubbard (Reference Hubbard2012, Reference Hubbard2013, Reference Hubbard2017) analyzed cases in which the complainant filed before but a motion to dismiss was decided after each decision. In another instance, Gelbach (Reference Gelbach2012) estimated a lower bound (or low-end estimate) of cases that Twiqbal negatively affected, using a formal economic model of litigant behavior and litigation data from the Federal Judicial Center. The results of these attempts have been varied as well. Other scholars have expressed some concerns regarding the underlying data and assumptions used in Hubbard (Reference Hubbard2013) and Gelbach (Reference Gelbach2012; see Engstrom Reference Engstrom2013; Reinert Reference Reinert2015). Additionally, Hubbard (Reference Hubbard2013, Reference Hubbard2017) inherently assumes that litigants and their lawyers did not and were not allowed to adjust to changes in the legal environment by judges. Furthermore, as pointed out by Engstrom (Reference Engstrom2013), the approach to studying dismissals in Hubbard (Reference Hubbard2013, Reference Hubbard2017) does not and cannot address any chilling effect the cases may have had.

In recognition that Twiqbal may have caused changes that cannot be captured in dismissal rates, scholars have considered other stages of litigation where an effect might be detectable (see Hubbard Reference Hubbard2017).Footnote ¹⁵ For example, Campos, Cotton, and Li (Reference Campos, Cotton and Li2015) provide a formal model that illustrates that heightened pleading standards theoretically could increase illegal activity among potential defendants. A survey of employment attorneys found that a relatively small percentage of attorneys reported that they had changed their case-screening practices because of the Twombly decision (Hamburg and Koski Reference Hamburg and Koski2010). Scholars looking at differences in filing rates have not found significant differences (Gelbach Reference Gelbach2012; Hubbard Reference Hubbard2012).Footnote ¹⁶ Relatedly, Curry and Ward (Reference Curry and Ward2013) considered whether the decisions have changed where plaintiffs are filing suit by studying removals and did not find an effect.Footnote ¹⁷

Some scholars have focused on case terminations. There are studies regarding summary judgment rates before and after Twiqbal (Reinert Reference Reinert2015; Gelbach Reference Gelbach2016; Hubbard Reference Hubbard2017), as a means of determining whether dismissals pursuant to Twiqbal have reduced the number of weak claims in the federal system. These studies do not reveal significant changes, although Gelbach (Reference Gelbach2016) lays out reasons why these findings may not teach us anything about the effect of the decisions. Finally, Sunshine and Pereyra (Reference Sunshine and Pereyra2015) carried out a study to assess the influence of Twiqbal on settlement. They find that fewer cases settled after the change, but the bluntness of their measures coupled with low power and unclear expectations make it hard to interpret the results.

Other research regarding pleading behavior, however, indicates that the decisions may have had a significant impact. In one survey, researchers asked attorneys practicing in the area of employment law about the influence of the decisions on their pleading practices: 70% of attorneys reported that the decisions had influenced how they structured complaints, and 94% of those respondents stated that they had changed approaches to include more factual allegations (Hamburg and Koski Reference Hamburg and Koski2010; Lee and Willging Reference Lee and Willging2010a). Overall, federal judges and attorneys practicing in federal court whom I interviewed report differences in the level of specificity seen in complaints before and after the decisions (Hazelton Reference Hazelton2014). Furthermore, Boyd et al. (Reference Boyd, Hoffman and Ristovski2013) offer one means of measuring the differences in complaints before and after Twombly: they consider the number of distinct causes of action using topic modeling.Footnote ¹⁸ They find some evidence that plaintiffs asserted fewer causes of action after Twombly, although they caution that the results may not be reliable because they used a nonrandom sample based on complaints available via RECAP.Footnote ¹⁹ This study helps further our understanding of the decisions in a way the other studies do not (Engstrom Reference Engstrom2013). Investigations of postpleading effects have inherently assumed that the pleadings have remained constant (see Engstrom Reference Engstrom2013).

There is other cursory work suggesting that federal pleadings may have changed (Cantone et al. Reference Cantone, Cecil and Jani2014; Hubbard Reference Hubbard2017).Footnote ²⁰ First, Cantone et al. (Reference Cantone, Cecil and Jani2014, 66) note preliminary research indicating that pleadings did get longer. Hubbard (Reference Hubbard2017) found no such effect in civil rights cases but noted in passing that he did find that torts pleadings (which were outside of the focus of his article) were longer.Footnote ²¹ Hubbard (Reference Hubbard2017) offers some other measures of specificity but does not integrate them into a model or discuss any of the implications of differences across issue areas.Footnote ²² He concludes that there is little evidence of effect in dispositions (dismissals and summary judgments) but that there is evidence of increased attention to pleadings. He does not, however, account for any changes to the underlying complaints in his analyses of downstream decisions based on his decision to look at cases that straddle the changes.Footnote ²³ These findings support the main conclusions of this article (although not identical in approach) and suggest that further attention is warranted. In the case of a change in pleading standards, we must consider how incentives regarding pleading may have changed and what effect we would expect. We need to consider how the pleadings themselves have changed. Ignoring the content of the filings could potentially obscure the actual effects of the decisions.

C. Hypotheses

Using multiple theories regarding judicial impact and the specific nuances of the Twiqbal decisions, I theorize that represented litigants began filing more detailed complaints after each decision due to an increased likelihood that the judge would apply a stricter pleading standard. The possibility of application of a higher standard incentivized such changes based on the anticipated costs of broadness, including the costs and risks of contending with a motion to dismiss and amending the petition (see, e.g., Hamburg and Koski Reference Hamburg and Koski2010; Lee and Willging Reference Lee and Willging2010a). Furthermore, I hypothesize that Iqbal had a more substantial impact because that decision quelled most doubts as to the general applicability of the plausibility standard. Finally, I anticipate that this increase in specific allegations will be more pronounced within issue areas where facts are more readily available (such as torts cases) compared to those types of cases in which pleading is inherently more challenging (such as civil rights cases).

IV. Method

While the task of measuring specificity is a difficult one, the potential benefits of developing such a measure are vast. It would help us understand the extent to which litigators have altered their behavior in light of the decisions. It may be that the rulings encouraged lawyers representing clients to be more specific in their pleadings, leaving only the weakest cases vulnerable to dismissal. Alternatively, the change may have dissuaded litigators from filing if they could not state their clients’ claims with specificity.Footnote ²⁴ Either effect of this change in law, or a combination thereof, could be very difficult to detect if we fail to consider the actual pleadings. One major concern regarding the current empirical research regarding the impact of Twombly and Iqbal is that complaints likely vary in nature before and after the decisions in ways that are not captured by any covariate. A measure of specificity would also allow us one means of comparing rulings before and after the decisions, even where selection bias may have changed the mix of complaints. It is with these goals in mind that I developed a measure of specificity.

Text analytical tools are offering new avenues of research in the social sciences (e.g., Grimmer and Stewart Reference Grimmer and Stewart2013). Such tools allow for the “systematic analysis of large-scale text collections without massive funding support” (Grimmer and Stewart Reference Grimmer and Stewart2013, 2). I use natural language processing tools to consider the specificity of factual allegations in complaints to learn about the impact of the decisions. In the following subsections, I detail the data that I collected, the classifier that I developed, and the analyses and results.

A. Data

The unit of analysis is the factual allegations section in complaints filed in federal district courts by attorneys.Footnote ²⁵ I focused on the specific sections of the complaint that are dedicated to recitation of the factual allegations. Such statements are at the heart of any inquiry as to the specificity of a complaint (see, e.g., Sherwin Reference Sherwin2008, 73–74; Pritchard Reference Pritchard2011, 766). Complaints include many different types of information, much of which does not pertain to the level of detailed factual allegations provided in the document: for example, complaints contain the caption, names and address of the parties, introduction, jurisdictional and venue statements, the prayer, and attorney information (e.g., O’Connor Reference O’Connor2019, form 2B:5). Inclusion of these additional sections would increase the potential noise in the measure while also being theoretically unrelated to the key concept, specificity of the allegations underlying the claims. The analyses reported in the next section were run on the factual allegations. This excludes the text of the caption, jurisdictional and venue statements, party information, prayer for relief, and signature block. Luckily, most complaints contain fairly consistent headings that allowed the process of trimming the complaints to be mostly automated. Where automated tagging was not possible, research assistants tagged the factual allegations by hand and made other necessary changes.Footnote ²⁶

The factual allegations are from complaints filed 6 months before and after each decision in non–Americans with Disabilities Act civil rights cases (hard to plead) and torts cases involving injury (easy to plead) from 11 federal district courts.Footnote ²⁷ Because of the large potential cost of accessing the relevant documents via the PACER system, I requested academic fee exemptions from all federal district courts. A list of granted exemptions can be found in appendix A.1.Footnote ²⁸ I randomly selected one district court from each geographic circuit from the district courts for which I had exemptions. Information regarding the sampled courts can be found in appendix A.2. I chose to sample one court from each circuit to provide representation from each circuit based on potential differences in precedent, culture, and norms (e.g., Kritzer and Zemans Reference Kritzer and Zemans1993).

Using the Search Dockets function in the Bloomberg Law database, I identified all relevant civil rights and torts cases filed in each sampled district between 6 months before and after Twombly and Iqbal, respectively. A list of the Nature of Suit codes used to identify the relevant cases is available in appendix A.3.Footnote ²⁹ The stratified sampling of individual cases from each court was carried out via the sampling plan detailed in appendix A.4 in order to ensure sufficient coverage across courts, period, and issue area. Ultimately, I assembled 3,181 eligible cases from the 11 district courts.

My choice to focus on distinct causes of action enables me to narrow the moving parts for these analyses and avoid common types of omitted variable bias (see Friedman and Martin Reference Friedman, Martin and Geyh2011). I chose civil rights cases since much of the intellectual debate regarding the effect of the decisions has centered on these types of cases. I also looked at torts cases involving injuries because it is an area for which scholars have not voiced concerns regarding a disproportionate impact from the decisions and, thus, allows for comparison. Furthermore, civil rights and torts are areas of law where heightened pleading standards generally do not apply.Footnote ³⁰

Only factual allegations from original complaints were considered. These original complaints represent the litigants’ initial statements of their claims. They encapsulate the lawyers’ first calculated statements aimed at leaving as many causes of action open as possible while still meeting a minimum threshold for those claims. Pleaders make such statements before they know which judge will preside over the case and see what the defendant has stated in its answer, both of which could influence the content. Therefore, it is the appropriate group of complaints to consider when examining the influence of the decisions. I only include allegations from complaints that were initially filed in federal court.Footnote ³¹ Using PACER to access the Case Management/Electronic Case Files (CM/ECF) systems for each court in the sample, the eligible original complaints in the actions were identified from the docket sheets.

B. Classifier

For this project, I developed a logistic regression classifier to measure the specificity of the factual allegations in the complaints. The classifier allows me to compare the relative specificity of complaints over time for a much larger number of observations than would have been possible based on hand coding. Such a classifier is simply a model that helps one classify text into categories (Bird, Klein, and Loper Reference Bird, Klein and Loper2009). Specifically, I wish to classify factual allegations with regard to how specific they are. To do so, I took advantage of a supervised model based in large part on one developed by Louis and Nenkova (Reference Louis and Nenkova2011) to classify the specificity of news summaries. My classifier was trained on a set of factual allegations from the complaints that I collected (see generally Genkin, Lewis, and Madigan Reference Genkin, Lewis and Madigan2007; Hopkins and King Reference Hopkins and King2010; Louis and Nenkova Reference Louis and Nenkova2011). Approximately 17% of eligible civil rights and torts-injury cases ( $n = 552$ ) were hand coded by the research team, consisting of myself and law student research assistants, regarding the specificity of the factual allegations (details of the coding can be found in app. B.1 along with information regarding the resulting binary measure of specificity in app. B.2).Footnote ³² These factual allegations were used to train the classifier: specifically, the text of the allegations was used to estimate a logistic regression with the binary hand-coded measure of specificity as the dependent variable and linguistic features of the texts (such as types and rarity of words), described in table 1, as independent variables.

Table 1.

Features of the Classifier

Note. LIWC = Linguistic Inquiry and Word Count.

Further details regarding the features of the classifier can be found in appendix C.1. The model underlying the classifier was estimated using two-thirds of the hand-coded data that were randomly selected ( $n = 362$ ). (The results can be found in appendix C.2.) I used the estimates from the logistic regression to generate predicted probabilities regarding specificity of the remaining complaints ( $n = 2,629$ ) that the team collected but did not hand code as to specificity: in other words, I estimated the specificity of the unseen complaints using parameters estimates for the relevant linguistic features obtained through the analysis of the hand-coded allegations. These continuous fitted values (bounded between 0 and 1) serve as the measure of specificity in the larger data set that I use to test my hypotheses (see generally Genkin et al. Reference Genkin, Lewis and Madigan2007; Hopkins and King Reference Hopkins and King2010; Louis and Nenkova Reference Louis and Nenkova2011).

The remaining one-third of the hand-coded data ( $n = 190$ ) were held in reserve to test and validate the model. Comparing the fitted values for the unseen data to the hand-coded values reveals a recall (correct prediction) of a little over 83%. This represents an approximately 60% proportional reduction in error. Furthermore, the classifier produced facially valid results. For example, the classifier identified the following set of factual allegations as being very likely to be broad: “Each of the Defendants owed one or more common law duties of care to Plaintiffs. Defendants breached those duties, proximately causing damages to Plaintiffs.” These allegations are a textbook example of doing nothing more than stating the elements of a cause. On the other end of the spectrum, the classifier identified the complaint that the following extract is from as being very likely to be specific:

On March 9 and March 10, 2008, notwithstanding the duties and obligations described above, the Defendant, Office of the Sheriff of Crawford County, by and through its agents and employees, committed one or more of the following acts of utter indifference or conscious disregard: A. Had no policies or procedures in place to ensure that [Plaintiff’s Name] was closely monitored for signs of alcohol or drug overdose; or B. Failed to provide or intentionally deprived [Plaintiff] of the medicine that he needed upon his arrival when it first knew, or should have known, that he had ingested alcohol and drugs; or C. Failed to provide or intentionally deprive [Plaintiff] of necessary medical attention throughout his incarceration when it knew, or should have known, that he was suffering from cocaine intoxication.

These allegations provide much richer factual allegations regarding the claims being presented.

Having measured the specificity of the factual allegations in complaints using the classifier, I now turn to assess the impact of Twombly and Iqbal on the specificity. The analysis was run on the 2,629 observations (1,348 civil rights cases; 1,281 torts cases).Footnote ³³ I estimate the differences in specificity using a fractional logit because the dependent variable, the fitted values for specificity, is bounded between 0 and 1. The primary independent variables of interest are the period, issue area, and the interaction of the two. The period is defined by when the complaint was filed: pre-Twombly (6 months before), post-Twombly (6 months after), pre-Iqbal (6 months before), and post-Iqbal (6 months after). Next, I include a variable for whether the case was a civil rights case (with torts cases as the baseline category). I anticipate that there is likely to be variation by issue area regarding the detail required to state a claim even under the most lenient standards. This is particularly important in light of the theory that the change in the pleading standard matters more in some issue areas, such as civil rights, than others. Thus, I include the interactions between each period and issue area.

I also include important control variables. My interviews with judges and attorneys indicated that practices vary across the federal judiciary and that local legal culture, such as state court practices, had an influence on judges and attorneys (Hazelton Reference Hazelton2014). Additionally, prior research indicates that aspects of local legal culture can influence the interpretation of procedural rules (e.g., Kritzer and Zemans Reference Kritzer and Zemans1993). Thus, fixed effects for courts were included to deal with variation in circuit precedents as well as local rules and cultures. Additionally, while litigants and their lawyers do not know the identity of the judge they will be in front of before they file (Federal Judicial Center 2014b), they do have information about the district’s general ideological makeup. The ideology of a district likely influences the extent to which litigators feel the need to justify their claims and protect themselves from dismissal under any standard (see, e.g., Carp et al. Reference Carp, Songer, Rowland and Stidham1992; Rowland and Carp Reference Rowland and Carp1996). Specifically, I hypothesize that attorneys will file more specific complaints when they anticipate appearing before conservative judges, who are generally considered to be disposed toward defendants, who tend to be businesses and their insurers (see, e.g., Carp et al. Reference Carp, Songer, Rowland and Stidham1992; Spaeth et al. Reference Spaeth, Epstein, Ruger, Whittington, Segal and Martin2011). Thus, I calculated and included the median Judicial Common Space score for Article III judges in the district at the time that the complaint was filed.Footnote ³⁴

V. Analyses and Results

The results of the analysis are reported in table 2.Footnote ³⁵ To aid interpretation of the estimates in light of the interaction, I have included figure 1, which illustrates the mean estimated probability of specificity for the complaints in each period by issue area.

Table 2.Fractional Logit—Specificity of Classified Observations

Note. n = 2,629; χ² = 286.76(18), p≥ .01; Akaike information criterion = 30,641.91; Bayesian information criterion = 30,753.52.

* p < .05, two-tailed test.

Figure 1.

Predicted probabilities of specificity across periods, by issue area: a, civil rights; b, torts. Estimates of complaint specificity in each period by issue area while holding the district at its modal category with the corresponding median ideology and 95% confidence intervals.

Figure 1 includes the statistical estimates and corresponding confidence intervals by period and issue area but does not provide statistical evaluations regarding the differences between these estimates. To provide readers with statistical estimates of the marginal effects of the difference across periods, I have also included figure 2, which illustrates the differences in the mean estimated specificity of complaints for each of the first three periods (pre-Twombly, post-Twombly, and pre-Iqbal) as compared to the post-Iqbal period. Figure 2 provides the precise statistical estimates and corresponding confidence intervals of the differences in the estimates by period and issue area found in figure 1.

Figure 2.

Marginal effects: differences in predicted probabilities of specificity across periods, by issue area: a, civil rights; b, torts. Estimates of change in complaint specificity in each period compared to the post-Iqbal period by issue area while holding the district at its modal category with the corresponding median ideology and 95% confidence intervals.

A comparison of the estimates over the periods reveals significant differences related to the periods and issue areas.Footnote ³⁶ The level of specificity seen in factual allegations was significantly higher in torts cases after the Iqbal decision than before Twombly, whereas for civil rights cases the level of specificity does not vary significantly over the periods. The differences between the pre-Twombly and post-Iqbal eras across the issue areas are not significantly different from each other at the 95% confidence level ( $p = .09$ ). The differences between torts complaints filed after Iqbal are also significantly more specific than the periods immediately after Twombly and before Iqbal. No such differences are apparent in civil rights cases. These differences between torts and civil rights cases between these periods are significant ( $p = .02$ and $p < .01$ , respectively): the difference in specificity in torts pleadings after Iqbal and the two middle periods were significantly larger than those in civil rights cases.

Overall, there is evidence of a post-Iqbal effect in torts cases but not so for Twombly. This is likely a result of uncertainty surrounding the applicability of the pleading standard after Twombly. Thus, there is evidence that Iqbal had an impact on the specificity found in torts complaints. Additionally, there is an indication that this effect was significantly different in torts cases than in civil rights cases, as hypothesized. This indicates that concerns regarding the relative ability of complainants to adapt to a plausibility standard may be warranted. Furthermore, variation in specificity in factual allegations over time indicates that comparisons of outcomes before and after Iqbal are not necessarily reasonable: the specificity of the underlying allegations is an omitted variable in such calculations. Pleadings set the stage not just for motions to dismiss and rulings on such motions but also the scope of claims and recovery and, thus, settlement verdicts and appeals. Such analyses need to account for adjustments in pleadings.

I also find evidence that control variables are significantly related to complaint specificity. First, the district median ideology has the hypothesized effect: when litigators anticipate being in front of a conservative judge, they file more specific complaints. Also, even while controlling for district median ideology, we see evidence that the specific court in which the complaint was filed can lead to different levels of specificity. For example, complainants tended to file more specific factual allegations in the Eastern District of California, the Southern District of Illinois, the Western District of New York, the District of Rhode Island, and the Southern District of West Virginia compared to the District of New Jersey (the baseline category). This helps illustrate the importance of controlling for such factors, as aspects of court and circuit rules and culture may influence local practice.

Overall, the results indicate that some plaintiffs, those in torts cases involving injuries, did change their pleading practices and became more specific after the Court made clear in Iqbal that the plausibility standard applies in all cases. There is not, however, significant evidence of such changes in civil rights cases. These differences between periods are generally statistically significant across issue areas. Such effects are consistent with commentators and scholars who have expressed concerns that plaintiffs in some issue areas may struggle or be unable to adapt to a higher pleading standard. Such differences raise issues regarding the ability of plaintiffs to vindicate important rights regarding areas such as discrimination (Miller Reference Miller2010) as well as concerns as to the proper costs of litigation (Schwartz and Appel Reference Schwartz and Appel2010). Finally, there is evidence that represented litigants file more specific complaints where they are likely to be before conservative judges and depending on court-specific factors, indicating that the application of procedural rules varies for litigants on the basis of the courts in which they find themselves.

VI. Discussion

The findings of this project have important implications regarding the impact of Twiqbal specifically and the study of legal change generally. Regarding the impact of the plausibility standard, this study provides direct, focused, and rigorous evidence that complaints in easy-to-plead cases became more specific as a result of the announcement of the general applicability standard by the Supreme Court in Iqbal. While there were previous suggestions of such changes, I am the first to provide an explicit standard for measuring specificity and evidence that litigants responded to changes in the pleading standard in varying ways. Such changes call into question studies that do not account for complaint specificity due to the risk of omitted variable bias.

Also, the variation by issue area is important: the apparent inability of plaintiffs in civil rights cases to increase the level of specificity with which they plead is in keeping with the concerns of many legal scholars. Hard-to-plead issue areas include core constitutional issues such as civil rights, and, thus, these results indicate that governmental abuses may be less likely to be checked under the new standard. It also indicates that studies of the impact of Twiqbal that fail to account for differences across issue areas are potentially missing an important effect: although the FRCP applies to all civil matters, the impact of the same rule can vary across issue areas due to differences in the nature of the claims and the available information. Additionally, the change in specificity found in torts complaints indicates that litigants have understood these decisions to represent a change in the applicable standards for pleadings. Generally, such changes to the FRCP must be made by Congress, as they are legislative in nature. These findings can help inform the ongoing debate among legal scholars as to whether the rules have been amended in effect and the normative implications of any such amendment in terms of the value and costs of litigation.

This research also has broad implications. This project illustrates the importance of carefully theorizing and studying the impact of judicial decisions and law. Such inquiries are vital to our understanding of how law functions in society and the implications of our models of judicial decision-making. Especially when we are considering policies directed at lower courts, scholars should consider and account for the possibility of litigant change. This article illustrates how litigants and lawyers adapt to changes in law in subtle and complex ways that may confound our understanding of courts if scholars do not account for such behavior. Changes to vertical precedent can occur before litigants make it to the courthouse and interact with judges. This project provides further evidence that litigants and lawyers do adapt in light of legal change. Such adaptation means special care must be taken when considering cases before and after a change in law. It also illustrates the importance of looking at context: the effect of this general pleading rule varies by issue area in ways that are consistent with the litigants’ differing circumstances. Furthermore, it relates to greater issues of information sharing and certainty. These results also help highlight the potential pitfalls of conducting research without special attention to possible variation across issue areas.

We need to use new approaches and tools to help understand consumer behavior in the legal system. Because of the importance of writings in the study of law and courts, text analysis tools are a natural expansion point for scholars. Approaches like the one I used in this project can help open the door to understanding the implications of legal policy more generally. Tests regarding the measure indicate that it is effective in capturing specificity in complaints. This classifier could be used in other studies implicating specificity in filings. It also could easily be adapted to consider other legal and political texts. For example, it could be used with minor adaptations to examine the effect of specificity on the use of Supreme Court precedent by the lower courts.

The results of this study are promising, but remaining questions should be noted. The results offer some evidence that a lack of information keeps civil rights claimants from being able to adapt to the new pleading standard in comparison with torts cases. Further research should be done, however, to verify that differences in specificity in complaints are the result of differences in the available information. A more detailed study could be done by looking at the specificity of claims within issue areas and comparing the pleadings with variables capturing the likelihood that such information is available. Such factors include whether the plaintiff inherently was present for the wrongful acts and the claim pertains to covert actions. Additionally, complainants in such cases may have found means to adapt over time. Researchers should also investigate the long-term implications of the plausibility standard. Furthermore, additional research can expand the number of courts to verify these effects and study nuances between specificity and court characteristics. Likewise, these findings call into question the assumption that the Supreme Court’s decisions did not influence specificity among complaints and that such changes are consistent across issue areas. The results indicate that at least in some areas, the majority of existing studies likely underestimate the impact of the decisions. Further consideration of such dismissals in light complaint specificity, made possible by the development of the measure and this study, will help shed light on these issues. Answering such questions is more possible now than before due to renewed interest in how legal decisions affect society and the advent of new and exciting tools for measuring aspects of legal documents and decisions. This study helps illustrate the potential benefits of such approaches.

Footnotes

The judges who granted my requests for exemptions from Public Access to Court Electronic Records (PACER) fees and to release the data, and the judges and attorneys who allowed me to interview them, have my deepest-felt gratitude. I am also grateful to Scott Hendrickson, William Hubbard, Michael Nelson, Keith Schnakenberg, Amy L. Steigerwalt, Joe Cecil, and participants in the Fall 2012 Northwestern Law School Political Economy Seminar, 2012 Midwest Law and Economics Association annual meeting, 2013 American Political Science Association annual meeting, 2014 Midwest Political Science Association annual meeting, 2015 Southern Political Science Association annual meeting, and 2018 Civil Justice Research Initiative Meeting for helpful comments regarding this project. I also thank my research assistants, especially Raymond Flores, Shengron Liu, and Jeffrey Zhao, for all their hard work in collecting, coding, and processing data and technical assistance. This project would not have been possible without support from the Center for Empirical Research in the Law.

1. Pleading practice refers to the portion of an attorney’s work that is dedicated to writing “document[s] that each party is required to communicate to the opponent before trial” (Garner Reference Garner1996).

2. Vertical precedent refers to Supreme Court cases where implementation can be carried out by lower-court judges, as opposed to lateral issues where the Court must rely on the decisions being carried out by other governmental actors (Hall Reference Hall2011).

3. I use the term “litigant” to refer to the unit formed by the represented party and its lawyer(s) (see, e.g., Wedeking Reference Wedeking2010).

4. If citizens engage the courts, trial courts are almost always the only courts with which they deal (Boyd Reference Boyd and Lindquist2017). In the federal system, e.g., in the 2018 fiscal year, the district courts dealt with seven times as many cases as the circuit courts and 57 times more than the Supreme Court (Roberts Reference Roberts2018; US Courts 2018a).

5. This does not mean that attorneys never include specific facts in a permissive environment (see Willging and Lee Reference Willging and Lee2010; Cantone, Cecil, and Jani Reference Cantone, Cecil and Jani2014). There can be reasons to do so, including positioning for settlement (Hubbard Reference Hubbard2013). However, in the absence of a requirement of specificity, an attorney will generally include broader claims (see, e.g., Lee and Willging Reference Lee and Willging2010b; Willging and Lee Reference Willging and Lee2010). Complainants have an incentive to include all claims in this first filing because the rules create a serious risk that claims not brought at the outset may be disallowed (see FRCP 2018, rule 15). Any time after the 21 days following filing, the party must receive the consent (at least implicitly) of the opposing party or the court in order to amend his or her pleadings (see FRCP 2018, rule 15[a][2]). Furthermore, later amendments may fall outside the statute of limitations unless they are found to relate back to the subject matter of the original filing (see FRCP 2018, rule 15[c]). Thus, there is a strong incentive to assert as many claims as possible from the outset or, at the very least, during the period of amendment without leave.

6. The difficult nature of the cases has incentivized attorneys taking these cases on a contingency fee basis to screen the cases with readily available information since before Twiqbal (e.g., Eisenberg and Lanvers Reference Eisenberg and Lanvers2009; see, e.g., Kritzer Reference Kritzer1997, Reference Kritzer2004).

7. This impact can be potential, rather than actual, in nature (Canon and Johnson Reference Canon and Johnson1999, 93).

8. Recovery may include monetary damages, equitable relief, vindication, publicity, etc.

9. In motions for summary judgment, the moving party asks the court to find that there is “no genuine dispute as to any material fact and the movant is entitled to judgment as a matter of law” based on the existing factual record (FRCP 2018, rule 56).

10. Of course, the impact within a given issue area may vary greatly due to variation in the need for information. As discussed in Sec. II, the issue areas vary in terms of the extent to which defendants control important evidence. Where plaintiffs can only access essential information through discovery (which is often the case in civil rights cases), a change in pleading standards should have a profound impact. When information is more readily available before a case is filed (as it is in many torts cases), such a change will matter less.

11. The decisions are among the top-cited Supreme Court cases by federal courts (Steinman Reference Steinman2016). Since their announcements, Twombly has been cited in 239,830 judicial decisions and Iqbal has been cited in 214,826 (Shepard’s Citations Service, December 2, 2019). The decisions surpass many seminal decisions, such as Miranda v. Arizona (1966; 69,499) or Chevron v. NRDC (1984; 17,655). They also appear to be on course to eclipse the total citations to important procedural cases such as Anderson v. Liberty Lobby, Inc. (1986; 280,035) and Celotex Corp. v. Catrett (1986; 265,094).

12. It is possible that some courts were already applying a plausibility standard. If so, the impact of the decisions was likely to create more consistency across lower courts, which is a vital function of the Supreme Court (US Supreme Court 2010, 10[b]).

13. The studies vary regarding the extent to which they consider motions to dismiss where plaintiffs may have the chance to replead. When a court has granted a motion to dismiss, the primary means of repleading is a court-issued leave to amend. According to rule 15(a)(2) of the FRCP, a judge has the discretion to grant leave to amend and should do so “when justice so requires.”

14. Concerns regarding bias that could be caused by using published cases are well documented (e.g., Swenson Reference Swenson2004).

15. Michalski and Wood (Reference Michalski and Wood2017) investigated the impact of Twombly and Iqbal in the states by comparing various litigant behaviors in Nebraska, which adopted the plausibility standard, with other states that did not. Their study offers an interesting perspective but is not generally applicable to this study, as it looks to the impact in the states where there are several confounding factors when considering the effect of the federal pleading standard.

16. Of course, the lack of an apparent decrease in filings does not preclude the possibility that plaintiffs have chosen not to file due to the change in pleading standards (see Gelbach Reference Gelbach2012), especially if Campos et al.’s (Reference Campos, Cotton and Li2015) model is correct.

17. They hypothesized that a heightened pleading standard in federal courts would encourage plaintiffs to file in state courts that apply a more lenient notice standard. Defendants, however, would be encouraged to invoke federal jurisdiction and have the case heard in a federal district court instead (otherwise known as “removing” the case to federal court) to take advantage of the federal pleading standard.

18. Topic modeling is a technique for grouping texts together based on similarity in the underlying information. Specifically, Boyd et al. (Reference Boyd, Hoffman and Ristovski2013) used spectral cluster analysis to quantify the topics found in the complaints.

19. The RECAP system allows individuals to upload documents that they obtained from PACER for free use by others. Boyd et al. (Reference Boyd, Hoffman and Ristovski2013, 257) pulled complaints from RECAP, which at the time had 1% of cases in the PACER system.

20. Among their analyses, Michalski and Wood (Reference Michalski and Wood2017) considered whether the adoption of the higher standard in Nebraska resulted in changes in various types of torts-based complaints in regard to length, number of claims, or amendments. They did not find significant differences. The independence of state legal systems from the federal system, even where they borrow from the federal standards, in addition to important institutional differences such as judicial selection and hierarchy, means that state systems do not provide a good proxy for the federal system in terms of considering changes in pleading standards. Furthermore, Michalski and Wood (Reference Michalski and Wood2017) limit their investigation of complaints to a few blunt measures that do not isolate factual allegations within the complaints.

21. I also specified a regression model with word count as the dependent variable in keeping with my full model. For torts cases, the differences between post-Iqbal and all other periods are statistically significant.

22. Among his measures, he did have research assistants hand code the detail in the complaints. He provided no guidance to the research assistants as to coding, including to undergraduates (who had no experience with complaints; Hubbard Reference Hubbard2017, 485). There were no significant differences, but we have no indications regarding the validity of such coding.

23. As noted above, Hubbard (Reference Hubbard2017), as well as Hubbard (Reference Hubbard2012), does not contend with the fact that judges likely treated cases that straddled the changes differently from those that were filed after the changes.

24. However, the available empirical evidence indicates that changes in the way in which complainants present claims, as opposed to filings, represent the bulk of any effect (Hamburg and Koski Reference Hamburg and Koski2010; Lee and Willging Reference Lee and Willging2010a).

25. The reasons for focusing on represented parties to the exclusion of pro se litigants were multifold, including differences in the applicable pleading standard, awareness of the decisions, and the nature of claims. This is not to imply that pro se cases are not of interest. Such cases are, however, outside the scope of my theory and study.

26. For example, some attorneys list a separate prayer for relief after each cause of action. In such cases, the separate prayers were stripped from the text to allow for consistent information to be compared. Additionally, some documents incorporated exhibits or other complaints by reference or attachment. The text of those documents is not included.

27. I excluded Americans with Disabilities Act cases because of an expansion of the definition of disability that occurred in 2008 that could bias my results (see Gelbach Reference Gelbach2012). I chose the 6 months before and after the decisions for two primary reasons. First, it allowed for the most reasonable comparison of the impact of each of the cases. The Court decided Twombly and Iqbal almost exactly 2 years apart: Twombly was announced on May 21, 2007, and Iqbal was announced on May 18, 2009. Thus, using the periods before and after each decision decreased the possibility of omitted variable bias regarding seasonal changes and societal changes over time. Second, these time periods allow for the clearest measurement of differences in the ability to meet the changing standard before increased selection and adaptation further clouded the picture (see Gelbach Reference Gelbach2012; Hubbard Reference Hubbard2012). Specifically, it allows us to see the extent to which there are differences in the ability of litigants to increase specificity that will either be insurmountable without discovery or require costly adjustment in terms of investigation (see, e.g., Gordon Reference Gordon2011). Both possibilities would decrease the chances of those types of cases being filed and successfully pursued.

28. The chief judge of the district court has discretion with regard to granting such requests (https://www.pacer.gov/documents/epa_feesched.pdf).

29. There are known limitations associated with using the Nature of Suit codes (Boyd et al. Reference Boyd, Hoffman and Ristovski2013). It remains, however, the only feasible means of a priori identifying a type of case within the federal filing system.

30. Some causes of action, such as fraud, have always required more detailed allegations (see FRCP 2018, rule 9).

31. Complaints initially filed in state court in cases that were removed to federal court were outside the scope of my consideration, as they were drafted to comply with state, rather than federal, rules of civil procedure. Furthermore, such complaints are not readily accessible.

32. As described in app. B.2, the main results do not depend on using the binary version of the coding.

33. The distribution of complaints by period was. pre-Twombly, 542; post-Twombly, 576; pre-Iqbal, 712; and post-Iqbal, 799.

34. Like Boyd (Reference Boyd2010), I calculated Judicial Common Space scores for district judges using the approach created by Giles, Hettinger, and Peppers (Reference Giles, Hettinger and Peppers2001) and extended by Epstein et al. (Reference Epstein, Martin and Westerland2007). The underlying data for these scores were obtained from Poole (Reference Poole2014) and Federal Judicial Center (2014a).

35. Additionally, for interested readers, I have included the same analysis run only on the hand-coded factual allegations using the binary dependent variable. The results of this analysis can be found in app. D.1.

36. Results using a binary version of the dependent variable in a logistic regression, the approach taken with the classifier, are consistent and can be found in app. D.2.

References

Baicker-McKee, Steven, Janssen, William M., and Corr, John Bernard. 2020. A Student’s Guide to the Federal Rules of Civil Procedure. St. Paul, MN: West.Google Scholar

Barnes, Jeb, and Burke, Thomas F. 2006. “The Diffusion of Rights: From Law on the Books to Organizational Rights Practices.” Law and Society Review 40 (3): 493–524.CrossRef Google Scholar

Baum, Lawrence. 1988. “Measuring Policy Change in the U.S. Supreme Court.” American Political Science Review 82 (3): 905–12.CrossRef Google Scholar

Baum, Lawrence. 2011. “Probing the Power of the Supreme Court, Reviewing Tom S. Clark, The Limits of Judicial Independence and Matthew E. K. Hall, The Nature of Supreme Court Power.” Tulsa Law Review 48 (2): 203–13.Google Scholar

Becker, Theodore Lewis, and Feeley, Malcom eds. 1973. The Impact of Supreme Court Decisions: Empirical Studies. 2nd ed. New York: Oxford University Press.Google Scholar

Benesh, Sara C., and Malia, Reddick. 2002. “Overruled: An Event History Analysis of Lower Court Reaction to Supreme Court Alteration of Precedent.” Journal of Politics 64 (2): 534–50.CrossRef Google Scholar

Binder, Sarah A., and Smith, Steven S. 1997. Politics or Principle? Filibustering in the United States Senate. Washington, DC: Brookings.Google Scholar

Bird, S., Klein, E., and Loper, E.. 2009. Natural Language Processing with Python. Sebastopol, CA: O’Reilly.Google Scholar

Bowen, Lauren. 1995. “Do Court Decisions Matter?” In Contemplating Courts, ed. Epstein, Lee 376–89. Washington, DC: CQ Press.CrossRef Google Scholar

Boyd, Christina L. 2009. “Placing Federal District Courts in the Judicial Hierarchy.” PhD diss., Washington University in St. Louis.Google Scholar

Boyd, Christina L. 2010. “Federal District Court Judge Ideology Data.” http://clboyd.net/ideology.html.Google Scholar

Boyd, Christina L. 2015. “Litigant Status and Trial Court Appeal Mobilization.” Law and Policy 37 (4): 294–323.CrossRef Google Scholar

Boyd, Christina L. 2017. “Gatekeeping and Filtering in Trial Courts.” In The Oxford Handbook of US Judicial Behavior, ed. Lee Epstein and Lindquist, Stefanie A. New York: Oxford University Press.Google Scholar

Boyd, Christina L., and Hoffman, D. A.. 2013. “Litigating toward Settlement.” Journal of Law Economics, and Organization 29 (4): 898–929.CrossRef Google Scholar

Boyd, Christina L., Hoffman, David. A., Zoran Obradovic, and Ristovski, Kosta. 2013. “Building a Taxonomy of Litigation: Clusters of Causes of Action in Federal Complaints.” Journal of Empirical Legal Studies 10 (2): 253–87.CrossRef Google Scholar

Campos, Sergio J., Cotton, Christopher. S., and Li, Cheng. 2015. “Deterrence Effect under Twombly: On the Costs of Increasing Pleading Standards in Litigation.” International Review of Law and Economics 44:61–71.CrossRef Google Scholar

Canon, Bradley C. 1973. “Reactions of State Supreme Courts to a U.S. Supreme Court Civil Liberties Decision.” Law and Society Review 8 (1): 109–34.CrossRef Google Scholar

Canon, Bradley C., and Johnson, Charles A. 1999. Judicial Policies: Implementation and Impact. Washington, DC: CQ Press.CrossRef Google Scholar

Cantone, Jason A., Cecil, Joe. S., and Jani, Dhairya. 2014. “Whither Notice Pleading: Pleading Practice in the Days before Twombly.” Southern Illinois University Law Journal 39 (1): 23–70.Google Scholar

Carp, Robert A., Songer, Donald, Rowland, C. K., and Stidham, Ronald. 1992. “Voting Behavior of Judges Appointed by President Bush.” Judicature 76 (6): 298–303.Google Scholar

Cecil, Joe S., Cort, George. W., Williams, Margaret. S., and Bataillon, Jared J. 2011. “Motions to Dismiss for Failure to State a Claim after Iqbal: Report to the Judicial Conference Advisory Committee on Civil Rules.” Federal Judicial Center, Washington, DC. https://www.uscourts.gov/sites/default/files/motioniqbal_1.pdf.CrossRef Google Scholar

Clermont, Kevin M., and Schwab, Stewart J. 2009. “Employment Discrimination Plaintiffs in Federal Court: From Bad to Worse.” Harvard Law and Policy Review 3 (1): 103–32.Google Scholar

Cox, James D., Thomas, Randall. S., and Bai, Lynn. 2008. “There Are Plaintiffs and … There Are Plaintiffs: An Empirical Analysis of Securities Class Action Settlements.” Vanderbilt Law Review 61 (2): 355–86.Google Scholar

Coyle, Marcia. 2013. The Roberts Court: The Struggle for the Constitution. New York: Simon & Schuster.Google Scholar

Curry, Jill L., and Ward, Matthew A. 2013. “Are Twombly and Iqbal Affecting Where Plaintiffs File? A Study Comparing Removal Rates by State.” Texas Tech Law Review 45 (4): 827–75.Google Scholar

Dodson, Scott. 2010. “New Pleading, New Discovery.” Michigan Law Review 109 (1): 53–89.Google Scholar

Dodson, Scott. 2012. “A New Look at Dismissal Rates in Federal Civil Cases.” Judicature 96 (3): 127–35.Google Scholar

Eisenberg, Theodore, and Lanvers, Charlotte. 2009. “What Is the Settlement Rate and Why Should We Care?” Journal of Empirical Legal Studies 6 (1): 111–46.CrossRef Google Scholar

Engstrom, David Freeman. 2013. “The Twiqbal Puzzle and Empirical Study of Civil Procedure.” Stanford Law Review 65 (6): 1203–371.Google Scholar

Epstein, Lee, Landes, William. M., and Posner, Richard A. 2013. The Behavior of Federal Judges. Cambridge, MA: Harvard University Press.Google Scholar

Epstein, Lee, Andrew D. Martin, Jeffrey A. Segal, and Westerland, Chad. 2007. “The Judicial Common Space.” Journal of Law, Economics, and Organization 23 (2): 303–25.CrossRef Google Scholar

Federal Judicial Center. 2014a. “History of the Federal Judiciary.” http://www.fjc.gov.Google Scholar

Federal Judicial Center. 2014b. “How the Federal Courts Are Organized: Federal Judges and How They Get Appointed.” http://www.fjc.gov.Google Scholar

Fitzpatrick, Brian T. 2011. “Twombly and Iqbal Reconsidered.” Notre Dame Law Review 87 (4): 1621–46.Google Scholar

Fox, Craig R., and Birke, Richard. 2002. “Forecasting Trial Outcomes: Lawyers Assign Higher Probability to Possibilities That Are Described in Greater Detail.” Law and Human Behavior 26 (2): 159–73.CrossRef Google Scholar PubMed

Friedman, B., and Martin, A. D.. 2011. “Looking for Law in All the Wrong Places.” In What’s Law Got to Do with It? What Judges Do, Why They Do It, and What’s at Stake, ed. Geyh, Charles, 143–72. Redwood City, CA: Stanford University Press.Google Scholar

Galanter, Marc. 1974. “Why the ‘Haves’ Come Out Ahead: Speculations on the Limits of Legal Change.” Law and Society Review 9 (1): 95–160.CrossRef Google Scholar

Garner, Bryan A. 1996. Black’s Law Dictionary. Pocket ed. Eagan, MA: West Group.Google Scholar

Garre, Gregory. 2009. Testimony in Has the Supreme Court Limited Americans’ Access to Court: Hearing Before the Committee on the Judiciary. 111th Cong., S. Doc. No. J-111-64. Washington, DC: US Government Printing Office.Google Scholar

Gelbach, Jonah B. 2012. “Locking the Door to Discovery—Assessing the Effects of Twombly and Iqbal on Access to Discovery.” Yale Law Journal 121 (8): 270–345.Google Scholar

Gelbach, Jonah B. 2016. “Material Facts in the Debate over Twombly and Iqbal.” Stanford Law Review 68 (2): 369–425.Google Scholar

Genkin, Alexander, Lewis, David. D., and Madigan, David. 2007. “Large-Scale Bayesian Logistic Regression for Text Categorization.” Technometrics 49 (3): 291–304.CrossRef Google Scholar

Giles, Micheal W., Hettinger, Virginia. A., and Peppers, Todd. 2001. “Picking Federal Judges: A Note on Policy and Partisan Selection Agendas.” Political Research Quarterly 54 (3): 623–41.CrossRef Google Scholar

Goodman-Delahunty, Jane, Granhag, Pär Anders, Hartwig, Maria, and Loftus, Elizabeth F. 2010. “Insightful or Wishful: Lawyers’ Ability to Predict Case Outcomes.” Psychology, Public Policy, and Law 16 (2): 133–57.CrossRef Google Scholar

Gordon, Leslie A. 2011. “Convoluted in Court: For Federal Plaintiffs, Twombly and Iqbal Still Present a Catch-22.” ABA Journal Online, January 1.Google Scholar

Grimmer, Justin, and Stewart, Brandon. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21 (3): 1–31.CrossRef Google Scholar

Grossman, Joel B., and Sarat, Austin. 1981. “Access to Justice and the Limits of Law.” Law and Policy 3 (2): 125–40.CrossRef Google Scholar

Hall, Matthew E. K. 2011. The Nature of Supreme Court Power. New York: Cambridge University Press.Google Scholar

Hall, Matthew E. K. 2017. “Judicial Impact.” In The Oxford Handbook of US Judicial Behavior, ed. Epstein, Lee and Lindquist, Stefanie. A., 460–82. New York: Oxford University Press.Google Scholar

Hamburg, Rebecca M., and Koski, Matthew C. 2010. “Summary of Results of Federal Judicial Center Survey of NELA Members.” Report by the National Employment Lawyers Association. https://www.uscourts.gov/sites/default/files/nela_summary_of_results_of_fjc_survey_of_nela_members.pdf Google Scholar

Hannon, Kendall W. 2007. “Much Ado about Twombly—a Study on the Impact of Bell Atlantic Corp. v. Twombly on 12(b)(6) Motions.” Notre Dame Law Review 83 (4): 1811–46.Google Scholar

Hatamyar, Patricia W. 2010. “The Tao of Pleading: Do Twombly and Iqbal Matter Empirically?” American University Law Review 59 (3): 553–731.Google Scholar

Hazelton, Morgan L. W. 2014. “Procedural Postures: The Influence of Legal Change on Strategic Litigants and Judges.” PhD diss., Washington University in St. Louis.Google Scholar

Hoekstra, Valerie J. 2000. “The Supreme Court and Local Public Opinion.” American Political Science Review 94 (1): 89–100.CrossRef Google Scholar

Holland, Kenneth M. 1981. “The Federal Rules of Civil Procedure—a Policy Evaluation.” Law and Policy 3 (2): 209–24.CrossRef Google Scholar

Hopkins, Daniel J., and King, Gary. 2010. “A Method of Automated Nonparametric Content Analysis for Social Science.” American Journal of Political Science 54 (1): 229–47.CrossRef Google Scholar

Hubbard, William H. J. 2012. “Testing for Change in Procedural Standards, with Application to Bell Atlantic v. Twombly.” Journal of Legal Studies 42 (1): 35–68.CrossRef Google Scholar

Hubbard, William H. J. 2013. “A Theory of Pleading.” Working Paper no. 466, University of Chicago Public Law and Legal Theory. https://chicagounbound.uchicago.edu/public_law_and_legal_theory/432/.Google Scholar

Hubbard, William H. J. 2017. “The Effects of Twombly and Iqbal.” Journal of Empirical Legal Studies 14 (3): 474–526.CrossRef Google Scholar

Hume, Robert J. 2009. How Courts Impact Federal Administrative Behavior. New York: Taylor & Francis.CrossRef Google Scholar

Johnson, Charles A. 1987. “Law, Politics, and Judicial Decision Making: Lower Federal Court Uses of Supreme Court Decisions.” Law and Society Review 21 (2): 325–40.CrossRef Google Scholar

Keck, Thomas M. 2009. “Beyond Backlash: Assessing the Impact of Judicial Decisions on LGBT Rights.” Law and Society Review 43 (1): 151–86.CrossRef Google Scholar

Kessler, Daniel. 1996. “Institutional Causes of Delay in the Settlement of Legal Disputes.” Journal of Law, Economics, and Organization 12 (2): 432–60.CrossRef Google Scholar

Kritzer, Herbert M. 1997. “Contingency Fee Lawyers as Gatekeepers in the Civil Justice System.” Judicature 81 (1): 22–29.Google Scholar

Kritzer, Herbert M. 2002. “Seven Dogged Myths Concerning Contingency Fees.” Washington University Law Quarterly 80 (3): 739–94.Google Scholar

Kritzer, Herbert M. 2004. Risks, Reputations, and Rewards: Contingency Fee Legal Practice in the United States. Redwood City, CA: Stanford University Press.CrossRef Google Scholar

Kritzer, Herbert M. 2008. “Daubert in the Law Office: Routinizing Procedural Change.” Journal of Empirical Legal Studies 5 (1): 109–42.CrossRef Google Scholar

Kritzer, Herbert M. 2015. Lawyers at Work. New Orleans: Quid Pro.Google Scholar

Kritzer, Herbert M., and Zemans, Frances Kahn. 1993. “Local Legal Culture and the Control of Litigation.” Law and Society Review 27 (3): 535–58.CrossRef Google Scholar

Lax, Jeffrey R. 2011. “The New Judicial Politics of Legal Doctrine.” Annual Review of Political Science 14:131–57.CrossRef Google Scholar

Lee, Emery G., and Willging, Thomas E. 2010a. “Attorney Satisfaction with the Federal Rules of Civil Procedure: Report to the Judicial Conference Advisory Committee on Civil Rules.” Federal Judicial Center, Washington, DC. http://www.fjc.org.CrossRef Google Scholar

Lee, Emery G., and Willging, Thomas E. 2010b. “Defining the Problem of Cost in Federal Civil Litigation.” Duke Law Journal 60 (3): 765–88.Google Scholar

Levin, Hillel Y. 2008. “Making the Law: Unpublication in the District Courts.” Villanova Law Review 53 (5): 973–1001.Google Scholar

Levine, James P. 1973. “Constitutional Law and Obscene Literature: An Investigation of Bookseller Censorship Practices.” In The Impact of Supreme Court Decisions: Empirical Studies, ed. Theodore Lewis Becker and Feeley, Malcolm, 119–38. New York: Oxford University Press.Google Scholar

Levy, Mitchell. 2018. “Empirical Patterns of Pro Se Litigation in Federal District Courts.” University of Chicago Law Review 85 (7): 1819–67.Google Scholar

Louis, Annie, and Nenkova, Ani. 2011. “General versus Specific Sentences: Automatic Identification and Application to Analysis of News Summaries.” Technical Report no. MS-CIA-11-07, Department of Computer and Informational Science, University of Pennsylvania.Google Scholar

Marshall, C. Kevin, and Postman, Warren. 2010. “Pleading Facts and Arguing Plausibility: Federal Pleading Standards a Year after Iqbal.” Jones Day, June. http://www.jonesday.com/.Google Scholar

Martinez, Anthony. 2009. “Plausibility among the Circuits: An Empirical Survey of Bell Atlantic Corp. v. Twombly.” Arkansas Law Review 61 (4): 763–86.Google Scholar

McCubbins, Mathew D., Noll, Roger. G., and Weingast, Barry R. 1987. “Administrative Procedures as Instruments of Political Control.” Journal of Law, Economics, and Organization 3 (2): 243–77.Google Scholar

McGuire, Kevin T., Vanberg, Georg, Smith, Charles E. Jr, and Caldeira, Gregory A. 2009. “Measuring Policy Content on the US Supreme Court.” Journal of Politics 71 (4): 1305–21.CrossRef Google Scholar

McMahon, Colleen. 2007. “The Law of Unintended Consequences: Shockwaves in the Lower Courts after Bell Atlantic Corp. v. Twombly.” Suffolk University Law Review 41 (4): 851–71.Google Scholar

Michalski, Roger, and Wood, Abby K. 2017. “Twombly and Iqbal at the State Level.” Journal of Empirical Legal Studies 14 (2): 424–69.CrossRef Google Scholar

Miller, Arthur R. 2010. “From Conley to Twombly to Iqbal: A Double Play on the Federal Rules of Civil Procedure.” Duke Law Journal 60 (1): 1–130.Google Scholar

Moller, Mark. 2009. “Twombly and Iqbal: Reality Check.” Cato at Liberty, October 13. http://www.cato-at-liberty.org.Google Scholar

New York Times. 2009. “Restoring Access to the Courts.” December 22.Google Scholar

Noll, David L. 2010. “The Indeterminacy of Iqbal.” Georgetown Law Journal 99 (1): 117–49.Google Scholar

O’Connor, Michol. 2019. O’Connor’s Federal Civil Forms. Albuquerque, NM: Thomson Reuters.Google Scholar

Pierce, Carl A., Cornett, Judy. M., and Long, Alex B. 2011. Professional Responsibility in the Life of the Lawyer. Eagan, MN: West Group.Google Scholar

Poole, Keith T. 2014. “Common Space Scores, Congresses 1st–112th (June 2014).” Voteview. http://voteview.com/basic.htm.Google Scholar

Priest, George L., and Klein, Benjamin. 1984. “The Selection of Disputes for Litigation.” Journal of Legal Studies 13 (1): 1–55.CrossRef Google Scholar

Pritchard, J. Scott. 2011. “The Hidden Costs of Pleading Plausibility: Examining the Impact of Twombly and Iqbal on Employment Discrimination Complaints and the EEOC’s Litigation and Mediation Efforts.” Temple Law Review 83 (3): 757–92.Google Scholar

Redish, Martin H., and Epstein, Lee. 2008. “Bell Atlantic v. Twombly and the Future of Pleading in the Federal Courts: A Normative and Empirical Analysis.” Working paper, SSRN eLibrary.Google Scholar

Reed, Douglas S. 2001. “Not in My Schoolyard: Localism and Public Opposition to Funding Schools Equally.” Social Science Quarterly 82 (1): 34–50.CrossRef Google Scholar

Reinert, Alexander A. 2010. “The Costs of Heightened Pleading.” Indiana Law Journal 86 (1): 119–335.Google Scholar

Reinert, Alexander A. 2015. “Measuring the Impact of Plausibility Pleading.” Virginia Law Review 101 (8): 2117–83.Google Scholar

Roberts, John. 2018. “2018 Year-End Report on the Federal Judiciary.” US Supreme Court, Washington, DC. https://www.supremecourt.gov/publicinfo/year-end/2018year-endreport.pdf.Google Scholar

Rosenberg, Gerald N. 2008. The Hollow Hope: Can Courts Bring about Social Change? Chicago: University of Chicago Press.CrossRef Google Scholar

Rowland, Charles K., and Carp, Robert A. 1996. Politics and Judgment in Federal District Courts. Lawrence: University Press of Kansas.Google Scholar

Schauer, Fred. 1987. “Precedent.” Stanford Law Review 39 (3): 571–605.CrossRef Google Scholar

Schneider, Elizabeth M. 2010. “The Changing Shape of Federal Civil Pretrial Practice: The Disparate Impact on Civil Rights and Employment Discrimination Cases.” University of Pennsylvania Law Review 158 (2): 517–70.Google Scholar

Schwartz, Victor E., and Appel, Christopher E. 2010. “Rational Pleading in the Modern World of Civil Litigation: The Lessons and Public Policy Benefits of Twombly and Iqbal.” Harvard Journal of Law and Public Policy 33 (3): 1107–50.Google Scholar

Seiner, Joseph A. 2009. “The Trouble with Twombly: A Proposed Pleading Standard for Employment Discrimination Cases.” University of Illinois Law Review 2009 (4): 1011–60.Google Scholar

Seiner, Joseph A. 2010. “Pleading Disability.” Boston College Law Review 51 (1): 95–149.Google Scholar

Sherwin, Emily. 2008. “The Jurisprudence of Pleading: Rights, Rules, and Conley v. Gibson.” Howard Law Journal 52 (1): 73–98.Google Scholar

Singer, Jordan M. 2014. “Gossiping about Judges.” Florida State University Law Review 42 (2): 427–78.Google Scholar

Songer, Donald R. 1987. “The Impact of the Supreme Court on Trends in Economic Policy Making in the United States Courts of Appeals.” Journal of Politics 49 (3): 830–41.CrossRef Google Scholar

Songer, Donald R., Cameron, Charles. M., and Segal, Jeffrey A. 1995. “An Empirical Test of the Rational-Actor Theory of Litigation.” Journal of Politics 57 (4): 1119–29.CrossRef Google Scholar

Spaeth, Harold, Epstein, Lee, Ruger, Theodore, Whittington, Keith, Segal, Jeffrey. A., and Martin, Andrew D. 2011. “Supreme Court Database Code Book 2011 Release 03.” Supreme Court Database, Washington University in St. Louis. http://scdb.wustl.edu/.Google Scholar

Spencer, A. Benjamin. 2008. “Plausibility Pleading.” Boston College Law Review 49 (2): 431–94.Google Scholar

Spiller, Pablo T., and Ferejohn, John. 1992. “The Economics and Politics of Administrative Law and Procedures: An Introduction.” Journal of Law, Economics, and Organization 8 (1): 1–7.Google Scholar

Spriggs, James F. 1996. “The Supreme Court and Federal Administrative Agencies: A Resource-Based Theory and Analysis of Judicial Impact.” American Journal of Political Science 40 (4): 1122–51.CrossRef Google Scholar

Steinman, Adam. 2016. “Ever Wonder Which SCOTUS Cases Have Been Cited the Most?” Civil Procedure and Federal Courts Blog, September 21. https://lawprofessors.typepad.com/civpro/2016/09/ever-wonder-which-scotus-cases-have-been-cited-the-most.html Google Scholar

Stewart, Devon J. 2010. “Take Me Home to Conley v. Gibson, Country Roads: An Analysis of the Effect of Bell Atlantic Corp. v. Twombly and Ashcroft v. Iqbal on West Virginia’s Pleading Doctrine.” West Virginia Law Review 113 (1): 167–208.Google Scholar

Stidham, Ronald, and Carp, Robert A. 1982. “Trial Courts’ Responses to Supreme Court Policy Changes—Three Case Studies.” Law and Policy Quarterly 4 (2): 215–34.CrossRef Google Scholar

Sunshine, Benjamin, and Pereyra, Victor Abel. 2015. “Access-to-Justice v. Efficiency: An Empirical Study of Settlement Rates after Twombly and Iqbal.” University of Illinois Law Review 2015 (1): 357–400.Google Scholar

Sweet, Martin J. 2010. Merely Judgment: Ignoring, Evading, and Trumping the Supreme Court. Charlottesville: University of Virginia Press.Google Scholar

Swenson, Karen. 2004. “Federal District Court Judges and the Decision to Publish.” Justice System Journal 25 (2): 121–42.CrossRef Google Scholar

US Congress. 2009a. Access to Justice Denied: Ashcroft v. Iqbal. Hearing before the Subcommittee on the Constitution, Civil Rights, and Civil Liberties of the Committee on the Judiciary. 111th Cong., H.R. Doc. No. 111–36. Washington, DC: US Government Printing Office.Google Scholar

US Congress. 2009b. “Notice Pleading Restoration Act of 2009.” 111th Cong., S. 1504.Google Scholar

US Congress. 2010. “Notice Pleading Restoration Act of 2010.” 111th Cong., S. 4054.Google Scholar

US Congress. 2011. Barriers to Justice and Accountability: How the Supreme Court’s Recent Rulings Will Affect Corporate Behavior. Hearing before the Committee on the Judiciary. 112th Cong., S. Doc. No. J-112-31. Washington, DC: US Government Printing Office.Google Scholar

US Courts. 2018a. “Judicial Business of 2018.” http://www.uscourts.gov/.Google Scholar

US Congress. 2018b. “Table C-2. U.S. District Courts––Civil Cases Commenced, by Basis of Jurisdiction and Nature of Suit, During the 12-Month Periods Ending March 31, 2017 and 2018.” https://www.uscourts.gov/statistics/table/c-2/federal-judicial-caseload-statistics/2018/03/31s.Google Scholar

US Supreme Court. 2010. “Rules of the Supreme Court.” http://www.supremecourt.gov.Google Scholar

Wasby, Stephen L. 1984. The Supreme Court in the Federal Judicial System. Belmont, CA: Wadsworth.Google Scholar

Wasserman, Howard. 2012. “Overcoming Iqbal.” PrawfsBlawg, June 26. http://prawfsblawg.blogs.com.Google Scholar

Wedeking, Justin. 2010. “Supreme Court Litigants and Strategic Framing.” American Journal of Political Science 54 (3): 617–31.CrossRef Google Scholar

Wheeler, Darren A. 2010. “Methodological, Theoretical and Topical Challenges in Judicial Implementation and Impact Studies.” Indiana Journal of Political Science 13:18–26.Google Scholar

Willging, Thomas E., and Lee, Emery G. 2010. “In Their Words: Attorney Views about Costs and Procedures in Federal Civil Litigation.” Federal Judicial Center, Washington, DC. http://www.fjc.gov.CrossRef Google Scholar

Yackee, Jason Webb, and Yackee, Susan Webb. 2010. “Administrative Procedures and Bureaucratic Performance: Is Federal Rule-Making ‘Ossified’?” Journal of Public Administration Research and Theory 20 (2): 261–82.CrossRef Google Scholar

Yacobucci, Peter, and McGovern, Patrick. 2014. “The Missed Revolution: How Twombly and Iqbal Tilted the Legal Playing Field While Political Science Remained Silent.” Politics, Bureaucracy and Justice 4 (1): 42–49.Google Scholar

Yoon, Albert, and Baker, Tom. 2006. “Offer-of-Judgment Rules and Civil Litigation: An Empirical Study of Automobile Insurance Litigation in the East.” Vanderbilt Law Review 59 (1): 153–96.Google Scholar

Table 1.Features of the Classifier

Table 2.Fractional Logit—Specificity of Classified Observations

Figure 1.Predicted probabilities of specificity across periods, by issue area: a, civil rights; b, torts. Estimates of complaint specificity in each period by issue area while holding the district at its modal category with the corresponding median ideology and 95% confidence intervals.

Figure 2.Marginal effects: differences in predicted probabilities of specificity across periods, by issue area: a, civil rights; b, torts. Estimates of change in complaint specificity in each period compared to the post-Iqbal period by issue area while holding the district at its modal category with the corresponding median ideology and 95% confidence intervals.

Hazelton Supplementary material

File 549.7 KB

Article contents

Judicial Impact and Factual Allegations: How the Supreme Court Changed Civil Procedure through the Plausibility Standard

Abstract

Information

I. Introduction

II. Background: Pleading Standards and Incentive Structures

III. Literature and Theory

A. Judicial Impact, Strategic Behavior, and Represented Litigants

B. Interest and Research Regarding the Plausibility Standard

C. Hypotheses

IV. Method

A. Data

B. Classifier

V. Analyses and Results

VI. Discussion

Footnotes

References

Save article to Kindle

Save article to Dropbox

Save article to Google Drive

Reply to: Submit a response

Your details

You have entered the maximum number of contributors

Conflicting interests