A. Rationality, Formal and Substantive

At the height of the modern era, Max Weber’s findings and methodology were omnipresent in various areas of social scientific research. However, the reliance on his conclusions and even his methodology in organizational research eventually dwindled owing to the changing landscape of business organizations from the industrial to the postindustrial (Lounsbury and Carberry Reference Lounsbury and Carberry.2005). Despite this, Weber’s explanation of organizations remains pertinent to the discussion of organizations, including the possibility of state courts developing in this way (Howard Reference Howard2011).

Rationality has been recognized as the major theme in Weber’s works on organizations (Kalberg Reference Kalberg1980). Despite the widespread use of the term, Weber was ambiguous at best in his description of rationality (Levine Reference Levine1981). He described 16 different versions of the term (Brubaker Reference Brubaker1984). This ambiguity has led to multiple interpretations, from which the most appreciable and consistent theme has been laid forth by Wallace (Reference Wallace1990). Wallace describes rationality as a “consciously rule bound comparison and choice among alternative means to a given end” (206). Plainly stated, although more than one route to achieve a goal may exist, there is a prescribed method.

Criminal justice as well as law and society authors have suggested that Weber’s most applicable heuristic is the neo-Weberian constructed dialectic of formal and substantive rationality (Kronman Reference Kronman1983; Savelsberg Reference Savelsberg1992; Ulmer and Kramer Reference Ulmer and Kramer.1996). Formal rationality, the highest form of rationality in legal thought (Ewing Reference Ewing1987), ultimately legitimates a similar means-end rational calculation by referencing universally applied rules, laws, or regulations (Weber Reference Weber1978, 975, 226). Weber states that “legal formal rationality exists when formally trained jurists carry out laws that apply to all citizens of the state in a manner such that only unambiguous general characteristics of the case are taken into account in terms of purely processual and legal factors” (656–57).

Substantive rationality is useful in describing the considerations of a judge that are aside and apart from the process required by consideration of technical law. It conveys only one element common to all “substantive” analyses: namely, that actors do not restrict themselves to note the purely formal and relatively unambiguous fact that an action is based on “goal-oriented” rational calculation with the most technically efficient methods. They apply certain criteria of ultimate ends (moral, political, etc.) against these scales of “value rationality” or “substantive goal rationality” (Weber Reference Weber1921/1978, Reference Weber1978) to determine the best course of action. Put more succinctly, substantive rationality is holistic and adaptive thinking. It is the form of rationality that focuses upon problem solving within a system of more generalized beliefs and considerations as opposed to the narrow, technocratic characteristic of the formal rational (Kalberg Reference Kalberg1980). As protectors of the formal requirements of due process and providers of equity and justice, judges must use both formal and substantive rationality (Ulmer and Kramer Reference Ulmer and Kramer.1996). They are inherently aware of and balance the tension between the substantive, critical analysis and the formal, technocratic analysis.

Judges use substantive rationality when engaged in critical thinking and analysis to ensure the overarching values of the judicial system (i.e., equity, equality, justice; Savelsberg Reference Savelsberg1992). As protectors of due process and managers who process a case docket, they are also very much concerned with formal rationality. Weber predicted that this formal rationality would increase with the size of the bureaucracy, most assuredly conflicting with the larger morals of society, thereby replacing substantive and all other forms of rationality (Blau Reference Blau1958).

This suggests, and other authors concur (e.g., Savelsberg Reference Savelsberg1992; Ulmer and Kramer Reference Ulmer and Kramer.1996), that an imbalance between substantive and formal rationality may result in inefficiency in the legal system. Such may encourage overstandardization to the point of failure to be properly sensitive to the particulars of a case or, on the other end, encourage prejudice in judicial decision making, among other problems. Bureaucratic size may encourage inefficiency once these scales have been tipped. Considering the converse (although Weber did not directly comment on this), it is possible that bureaucracy may be too small and thereby too abundant in substantive rationality to be effective. Literature regarding other institutions, such as schools, has suggested that a “Goldilocks principle” may be applied to the sizing of institutions (Lee and Smith Reference Lee and Smith.1997).

Recent theoretical development of the concept of organizational size and its effect on substantive rationality is limited (Jaskyte Reference Jaskyte2012). Supplementary empirical work, wherein we may begin to standardize the unit of analysis and control for the storm of affectors when considering judicial action, is needed.

B. Rationality and Size Applied to State Criminal Courts

Since Weber and the seminal works of authors such as Blau (Reference Blau1958), Scott and Blau (Reference Scott and Blau.1962), and Stinchcombe (Reference Stinchcombe1965), researchers have explored the effects of both substantive and formal rationality on courts and found high degrees of rationality in courts (Horwitz and Wasserman Reference Horwitz and Michael1980). Drawing from Weber, Savelsberg (Reference Savelsberg1992) suggests that as our criminal-law-making mechanisms modernized, they constructed a formally rational system that controlled judicial behavior in order to minimize disparity of treatment (looking at sentencing). Other research supports this claim (Ewing Reference Ewing1987; Westby Reference Westby1991). He then describes the process by which the criminal court substantivized as well as the resulting neoclassical backlash (increasing formal rationality).

State criminal courts are arenas in which two sets of considerations apply (Hall Reference Hall1995). In the state criminal courts, the substantively rational (i.e., the criminal case’s relevant and particular circumstances, needs, or characteristics) is considered by the court only through a formally rational lens: mandated criminal court procedure (decisions of law; Ewing Reference Ewing1987). Formal rationality remains, despite its conflict with substantive rationality (Thibaut et al. Reference Thibaut, Walker, LaTour and Pauline1973). In a study of local courts, Tepperman (Reference Tepperman1973) found that bureaucratic size affects decision making. Individualized decision making (presumably based in substantive rationality) is unlikely to thrive with large increases in the scale of organizational and systematic operations. The technocratic thinking and hyperrationalized decision making that thwarts substantive rationality may develop as an unanticipated consequence of bureaucratic growth.

Paralleling these findings with Weber, we question whether an increase in a court’s size equates to formal rational social action and a decrease of substantive social action within the courtroom. The research suggests that extraneous factors can sway highly consequential decisions of expert decision makers (Danziger, Levav, and Avnaim-Pesso Reference Danziger and Liora2011). Research on judicial behavior in the courtroom suggests that the gender and ethnicity of the judge or defendant may play a role in decision making (Abrams, Bertrand, and Mullainathan Reference Abrams, Marianne and Sendhil2012). Further research suggests that making repeated judgments or decisions is taxing; it depletes individuals’ executive function and mental resources, which can, in turn, influence their subsequent decisions (Muraven and Baumeister Reference Muraven and Roy2000). There is a likelihood that workload and time of day can affect decision making by judges. To date, no study has quantitatively operationalized substantive rationality and formal rationality in any proceeding other than sentencing.

A. State Criminal Court Is Optimal for Testing Propositions

The data collected relate to judicial performance within state courts, as the standardized nature of these courts provide optimal control in research design for testing these propositions. All courthouses are required to follow the same rules and procedures as required by the Florida Rules of Criminal Procedure, the Rules of Judicial Administration, and other statewide standards. In addition, all Local Rules of Court for each courthouse in the study were roughly equivalent, with no real variance in each set’s requirements. Criminal courts within Florida provide a greater degree of consistency in the types of laws, rules, and subject matter between districts compared to myriad possibilities existing in federal and state civil courts.

Additionally, state criminal courts have more formalized and perfunctory types of hearings that provide increased consistency of action. This creates a simplified environment that can be more easily observed and interpreted. Interviews with court administration from all courthouses in the study and pilot observations of all courthouse arraignments were used to assure that courthouse design and administration of the courthouses used were very similar (and in most cases, identical) and that there were few impediments to standardization across courthouses created by differing policies or the Local Rules of Court.

B. State Misdemeanor Division Arraignments Ideal for Comparative Analysis

Criminal courts use an initial proceeding called an arraignment. At any given misdemeanor arraignment, the judge will hear cases of a similar nature, each with a low and relatively comparable level of severity. This standardization reduces the variance in case severity and stimulus as much as possible. There is high uniformity. Pilot observations conducted of all courts used in this data set as well as a review of the charges of the recorded data revealed a low level of variance in the way arraignments were conducted. Likewise, if any part of a given arraignment process varied greatly (e.g., through showing a video to explain a defendant’s rights instead of allowing the judge to explain them), then that arraignment was removed from consideration.

A. Data Collection

After pilot data were collected from October 2012 to July 2013, the data collection process was improved, and training of new researchers ensued. The data collection process for this data set began in October 2015 and ended in April 2017. It consisted of checklists of judge performance in the brief time of each arraignment wherein defendants are going to plead guilty or no contest to a charge without assistance of an attorney (pro se; n = 481). We conducted direct observations of pro se arraignments at selected courthouses in eight judicial circuits; the chosen courthouses varied in size, from smallest (5 judges) to largest (43 judges). We organized and labeled them as follows: small (5), small (6), small (5), small (8), small (10); medium (17), medium (17); and large (43). Each district was chosen to ensure that each judge would have roughly the same number of defendants per arraignment and that each judge tested would have roughly the same job expectations (same organizational structure). Each observed judge was chosen, because they were professionally required to perform the same type of tasks. Also, each judge was graded performing only the tasks involved with taking guilty or no contest pleas from pro se defendants.

B. Misdemeanor Arraignments Are Ideal for Evaluating Judge Performance

Arraignments, with their high attention to process and administration, are an ideal choice for measuring formal and substantive rationality. In the arraignment setting, the court has a routine but essential gatekeeping procedure. All criminal cases that move toward adjudication must proceed through arraignment. In other words, all cases that have not received a No Information (or pursued under the Nolle Prosequi doctrine that gives prosecutors wide discretion over which cases they pursue) will have an arraignment hearing. Under Rothgery v. Gillespie County (554 U.S. 191 [2008]), arraignment is a “critical stage” of the criminal justice processing (and hence the right to representation by counsel applies); however, it is not a highly adversarial proceeding in which the adversaries dominate the “action.” Little in the way of evidence is introduced; the arguments that are presented to the judge may be requests for hearings or the introduction of extenuating or mitigating circumstances relevant to sentencing. These standardized processes, which all the judges must cover during the proceeding (the “boilerplate” requirements), were recorded as indicators of formal rationality.

The role of the judge is front and center; his or her performance can be directly observed or assessed without many extraneous or complicating factors entering the proceeding—unlike a trial or a suppression hearing. It is ideal to measure whether the judge has carried all formally rational requirements required of the judge’s performance by the Florida Rules of Criminal Procedure (e.g., ensuring that the defendant is aware of her charge, ensuring that the defendant knows of her right to an attorney, ensuring that the defendant is freely and knowingly entering her plea). These provide an opportunity to examine the consistency of formal rational behavior across judges and courthouses. However, arraignments provide an opportunity to measure substantive rationality as well.

C. Measures: Substantive Rationality

One rule of law, Florida Rule of Criminal Procedure Rule 3.111 (2012), offers a strategic way to examine a more complete quality-driven, individualized, and substantive approach in judging. Although criminal defendants have the right to an attorney at their initial appearance and beyond, very few take advantage of this opportunity and appear pro se (Rothgery). Rule 3.111 requires judges to investigate the status of a defendant who has waived the right to counsel. Rule 3.111 (d)(2) (2012) requires, in part, that for a defendant to properly advance at any crucial stage of a proceeding without an attorney, the judge must ensure that the defendant has been subjected to a prescribed process in which counsel is offered to the defendant. The process of offering counsel must ensure that “a thorough inquiry has been made into both the defendant’s comprehension of that offer and the defendant’s capacity to make a knowing and intelligent waiver. Before determining whether the waiver has been done knowingly and intelligently, the court shall advise the defendant of the disadvantages and dangers of self-representation.” The Florida Supreme Court has determined that such a process, a.k.a. a “Faretta hearing” (Faretta v. California, 422 U.S. 806 [1975]), will inquire into (1) the defendant’s understanding of the nature and complexity of the case and (2) whether she understands the dangers of representing herself (Faretta; State v. Young, 626 So. 2d 658 [Fla. 1993]).

Rule 3.111 is a rule of black-letter law that is rare in that it is designed to militate against a mechanized, formalistic approach and encourage creativity. It lacks a clear-cut explanation of the degree of development of the rapport requisite between judge and defendant (Jones v. State, 584 So. 2d 120 [Fla. 4th DCA 1991]; Brown v. State, 830 So. 2d 203 [Fla. 5th DCA 2002]; Wilson v. State, 947 So. 2d 1225 [Fla. 1st DCA 2007]). The Florida Supreme Court has interpreted this rule to mean that there are “no magic words” that can determine whether a defendant has knowingly and intelligently waived her rights; rather, a full and careful rapport between judge and defendant must be made (Potts v. State, 718 So. 2d 757, 760 [Fla. 1998]). It greatly encourages creative and critical thinking on the part of the judge, a.k.a. substantive rationality.

Research instruments were brought back to the field to gather systematic information from courthouse observations and the judges within the selected courthouses. Standardizing a way in which quality of interaction between the defendant and the judge could be recorded was a difficult task. Evaluators recorded whether the judge followed Rule 3.111. Although there are no magic words for fulfilling the requirements of 3.111, there are requirements nonetheless. All of the components of the defendant’s mental state that 3.111 requires the judge to creatively and adaptively check were broken down into six different areas that could be assessed. Researchers were trained on this evaluation. Practiced, evaluated, and modified through pilot observations and interrater reliability checks, this method proved the most efficient way for the researchers to consistently collect data regarding the judge’s attention to 3.111. If researchers were unsure how to score a part of a judge’s interaction, they then recorded the entire question and requested a review and score from the lead researcher. This intricacy of recording was possible via handwritten notes for researchers, because only a small percentage of defendants at each arraignment were pro se and pleading guilty or no contest at that early phase. While we feel that this was the most efficient way to record the data, we regretted not being able to include a temporal coding component regarding judge-defendant interaction during the Faretta inquiry, as such would be another indicator of substantive rationality. Following is the six-part checklist:

1. 1. Whether the court inquired into the defendant’s legal experience (history).

2. 2. Whether the court inquired into defendant’s understanding of the nature of the case.

3. 3. Whether the court inquired into the defendant’s understanding of the complexity of the case.

4. 4. Whether the court inquired into the defendant’s ability to represent self.

5. 5. Whether the court inquired into the defendant’s competency to make the decision to represent self.

6. 6. Whether the defendant appeared, at the conclusion of the interaction, to understand the dangers and disadvantages of self-representation to the rater.

D. Measures: Formal Rationality

Within larger organizations, the greater focus on standards may also equate to greater standardized performance. A greater focus on formality within an organization may improve how well its internal actors follow its standards. Rogers, Roux, and Biggs (Reference Rogers, Roux and Biggs.2000) suggest that greater size leads to greater standardized performance and to internal actors meeting standards with greater precision. Others agree (Barclay and Birkland Reference Barclay and Thomas1998). Our addition to this conversation begins with a novel and more quantitatively oriented method. Using our checklist measures, we seek to test this by recording how well judges perform all required tasks that may be standardized tasks in an arraignment. Boykin v. Alabama (395 U.S. 238 [1969]) and its progeny explain that judges are required to ensure that a defendant who enters a guilty plea does so lawfully, without undue coercion, knowingly, and intelligently.

Using ethnographic methods and interviews, we saw that every courthouse we visited required that the judge inquire as to numbers 2–7 in the list a few lines below. Stemming from the intent explained in Boykin, the wording for these requirements are unlike the requirements of 3.111 in that they are expressed to the judge in exactly the format in which they are to be used. In other words, they are communicated directly to the judge via the relevant Rule of Criminal Procedure, and they elicit very simple, one-word responses and do not require engaged interaction as Rule 3.111 requires. These are items that a judge can inquire about without any (or very little) creativity or critical thinking. They are the exhaustive list of the common items that court administration explained a judge was required to inquire about when taking a plea and indicate how well a judge can follow the exact lettering of the law. In addition, starting the arraignment on time is another formal rational requirement that requires simple rule following and very little creative thinking. Therefore, we included starting the arraignment on time as part of the exhaustive list of formal rational requirements of a judge’s professional duties when taking a plea. We collected data consisting of binary responses to the following indicator prompts:

1. 1. Whether the judge started on time (within 1 minute) of the formal start time of arraignment.

2. 2. Whether the judge inquired into the defendant’s current mental condition (drugs/alcohol).

3. 3. Whether the judge inquired into the defendant’s education level.

4. 4. Whether the judge mentioned the public defender.

5. 5. Whether the judge asked if the defendant understood his right to trial.

6. 6. Whether the judge asked the defendant if he knowingly entered the plea.

7. 7. Whether the judge inquired into the defendant’s age.

E. Arraignment Docket Size

Workload can affect judicial performance (Glendon Reference Glendon1994), as well as rationality. Therefore, it is important to control for docket size as much as possible in order to accurately gauge the effect of courthouse size. Before any data were collected in any of the three prongs, 23 Florida courthouses were visited to search for courthouses wherein the number of cases handled by each criminal court was similar. Researchers consulted courthouse administration at all 23 courthouses in order to approximate the workload of each criminal judge. This was determined by interviews and reviewing the daily incoming criminal caseload and finding the mean assigned to each judge. In every courthouse studied as a part of this research (and almost every courthouse in general), the judges were assigned the cases on a perpetual rotation. If a courthouse systematically assigned a greater number of cases to one judge over another, the courthouse was omitted from the study. In every courthouse reviewed, there was an articulable, systematic way in which new criminal cases were assigned to judges that ensured each judge carried the same caseload.

We sought to perform an in-depth study at each courthouse and control for judge arraignment workload by design. To accomplish this, we reduced the number of courthouses in the sample to those comparable in arraignment size. From the 23 courthouses considered, this reduced the applicable number to 8. Among these 8, there were courthouses of various sizes (as measured by judge number), ranging from 5 to 40 judges. Further, we took the mean daily arraignment docket for the previous 3 months. We chose only those 8 courthouses wherein the daily arraignment dockets that remained were close to 80 cases (±10). Arraignment dockets were measured from the respective clerk’s daily docket.

F. Instrument

Observational checklists (n = 481) were used to record the behaviors of judges and to capture some of the performance indicators we propose for rationality. One set of those behavioral indicators concerned some formal rational action features of the judicial role at arraignments. Another set of indicators focused on substantive rational action, or the manner in which Rule 3.111 was implemented. The advantage of this two-part instrument design is one of triangulation. Different research methodologies were used to see whether patterns that emerge from each instrument will eventually converge. If the previous literature and propositions are correct, indicators of substantive formality should decrease with size just as formal rationality increases. This is exploratory research; convergence among methods will indicate whether the theoretical development is pointed in the right direction. The disadvantage of this method is the degree of complication of the two-part data collection. The two-pronged quantitative approach design makes it more difficult to keep the abstract theoretical constructs sorted out as they are woven through different methodological adjustments of each of the approaches. To manage the complexity, the design components are referenced in terms of substantive rationality and formal rationality.

G. Interrater Reliability

In preparation for data collection with checklists, the 11 research assistants that assisted with completing the checklists were trained in three different 2-hour seminars teaching them exactly what to look for when conducting the observations in arraignment. Each was instructed regarding the formal rational requirements of a judge. They were trained regarding Florida Criminal Rules of Procedure Rule 3.111 as well as the accompanying relevant case law. It was important that they understood, in a similar fashion, exactly what the case law requires regarding Rule 3.111 (indicators of substantive rationality). They were also instructed on what to look for regarding the performance of the judges (indicators of formal rationality), including start time, the offering of the public defender, whether the defendants were offered an affidavit of indigency to gain a public defender, and the delivery of a copy of the information to the defendant. They were instructed on where to sit in the arraignment to gain the best vantage point. They were retrained and quizzed individually to ensure their knowledge was consistent and accurate.

Once engaged in completing observational checklists, nine of the 11 conducted practice checklists in one chosen courthouse (to ensure efficiency in reliability analysis), whereupon two would attend the same arraignment. They were instructed to sit across the courtroom from each other, to not travel together, and to not discuss the checklists or the events of each observed arraignment. Their results were then cross-checked. The research assistants then tallied each arraignment day and returned the results to the lead author of this research. The results of that day were verified with the research assistant who observed with them. Once each of these reached better than 74% consistency with one another, the project administrator attended the arraignments and performed the grading as well, following the protocol mentioned above. In other words, once interrater reliability reached 75% consistency (or higher), the project administrator joined the rating sessions. A lower bound of .70 is considered “acceptable” in psychometric assessments of reliability, so 75% consistency seemed a reasonable threshold to proceed with additional ratings. If unable to reach this rate of consistency in recording, research assistants were dropped from this portion of the data collection. As a result, two research assistants ceased observational checklist performance and continued work with the project in other capacities. This training period, including all observational checklist consistency matters, took a month to complete. Special care was taken to ensure that all assistants were unfamiliar with the supporting theory upon which this work is based and were never informed regarding the theoretical leanings until after their data collection duties were complete.

H. Statistical Analysis Strategy

Descriptive statistics were calculated for each courthouse in terms of the average percentage of cases per arraignment (i.e., number of completed tasks per arraignment / number of pro se guilty pleas per arraignment) in which a formal or substantive rationality task was completed. Because the large majority of judges completed all formal and substantive rationality tasks, the data were highly skewed. To facilitate statistical testing, these data were discretized into binary categories representing “full” formal or substantive rationality (i.e., the formal or substantive rationality task was completed in 100% of cases) or not “full” formal or substantive rationality (i.e., the task was carried out in less than 100% of cases). Next, courthouses were grouped based on size into small (i.e., 10 or fewer judges), medium (11–20 judges), and large (more than 20 judges) courthouses to facilitate meaningful comparisons of odds ratios among courthouses by size. It is important to note that this sample had only two courthouses that qualified as medium in size and one courthouse that qualified as large. Because the large courthouse perfectly predicted completion of the full rational tasks, odds ratios (ORs) were calculated for comparisons of only small versus medium courthouses. However, there was sufficient variation among the large group with respect to substantive rationality tasks to allow separate ORs to be calculated comparing small versus medium and small versus large courthouses. Because the data were nested (i.e., arraignments within judges and judges within courthouses), a multilevel logistic regression model was employed to facilitate accurate statistical testing that accounted for the within-judge and within-courthouse correlational structure. Moreover, each multilevel model controlled for docket size, time of day, and duration (in minutes); all multilevel effects on formal and substantive rationality are explained in table 1. All data were prepared and analyzed using Stata/IC 12.1 for Mac.

Table 1.Multilevel Effects on Formal and Substantive Rationality

I. Descriptive Statistics as Controls

As discussed, additional factors have been found to sway consequential decisions of third-party decision makers (Danziger et al. Reference Danziger and Liora2011). Some research suggests gender and ethnicity of judge and defendant play a role in decision making, although others have found they do not (Steffensmeier and Hebert Reference Steffensmeier and Hebert.1999; Abrams et al. Reference Abrams, Marianne and Sendhil2012). We recorded the ethnicity and gender of the judge in all arraignments. Further research suggests that making repeated judgments or decisions depletes individuals’ executive function and mental resources, which can, in turn, influence their subsequent decisions (Muraven and Baumeister Reference Muraven and Roy2000). We sought to measure and control for this potential effect. The basic unit of analysis was arraignments nested within courthouses. Of the 481 arraignments in the database, 244 were at the 8:30 or 9:00 a.m. docket and 237 were at the 1–2 p.m. docket. See table 2.

Table 2.

Characteristics of Judges and Arraignments, by Courthouse Size

Despite several attempts at data collection design, records of the ethnicity and sex of the defendants were unable to be reliably recorded given the pace at which other variables had to be recorded. Ethnicity and sex of the judge at each (as recorded in the checklists) arraignment were recorded, returning a sample that was mostly white and male. From the 481 arraignments, 52 arraignments were presided over by eight judges observed to be black/nonwhite by the task evaluator. One hundred twenty-one arraignments were presided over by 18 judges observed to be female by the task evaluator. Unfortunately, only three or eight courthouses had more than one female judge, and only three courthouses had any nonwhite judges; this lack of variability in judges’ gender and ethnicity across courthouses prohibited their inclusion in the multilevel model owing to the bias expected when estimating their effects. See table 2.