INTRODUCTION
Before the global pandemic, there was already a move toward online court programs as governments struggled with restricted budgets and the need to increase access to justice. The public health challenges of the pandemic has accelerated this move, creating the possibility of a fundamentally reshaped justice system (Freeman Engstrom Reference Freeman2020). One aspect of the online court movement is developing systems that laypeople can access without the assistance of a lawyer. These systems respond to each of the three primary factors pushing governments to adopt online courts: saving costs (by reducing the need for state-sponsored subsidies to lawyers); increasing access to justice (by reducing barriers to court including cost, inconvenience, and fear); and protecting public health (by enabling remote filing and processing of court files).
Reforms were underway before the pandemic. The most ambitious example—with a budget of over £1 billion—is the reform of the courts of England and Wales (Hodges Reference Hodges2016). This includes reform to all areas of the court system from the operation of criminal and family courts to redesigning the infrastructure (technological and physical) that underpins them (Rozenberg Reference Rozenberg2020). An example of a more incremental approach has been the Civil Resolution Tribunal (CRT) of British Columbia, which features a front-end system to help laypeople resolve disputes without resort to lawyers (Salter and Thompson Reference Salter and Darin2016). Less ambitious programs for online forms or filing systems designed with laypeople in mind were also underway before the pandemic (for overviews of online dispute resolution and online courts see commentary by leaders in the field including: Loebl Reference Loebl2019; Susskind Reference Susskind2019; and Katsh and Rule Reference Katsh and Colin2016).
These systems are promoted as tools for increasing access to justice, but what this means in practice is not always clearly articulated by those driving the reforms. The benefits of these systems include improving the “completeness and legibility” of claims filed by laypeople (Sela Reference Sela2016, 359), increased user satisfaction, cost reduction (for the state and the disputant), reduction in time to resolution, reduced need for hearings, increased rates of settlement, and increased litigant engagement (Byrom Reference Byrom2019). While these are benefits, they do not themselves equate to improved “access to justice.” The concept of access to justice is “admittedly not easily defined” but includes access to dispute resolution under the auspices of the state and a result that is “individually and socially just” (Cappelletti and Garth Reference Cappelletti and Bryant1978, 182). Access to justice, as Rhode observed, is not just access to formal court process (Rhode Reference Rhode2004) but the courts are of course an important component. For courts to deliver just results, a number of elements are required, including—importantly for our purposes—an effective hearing and a decision in accordance with substantive law (Byrom Reference Byrom2019). These elements of access to justice require the decisionmaker and the opposing party to be in possession of all the law-relevant facts; without knowledge of all the law-relevant facts, an effective hearing and a decision in accordance with substantive law cannot be achieved.
This, therefore, raises a question: how adept are laypeople at providing law-relevant facts when they are elicited through an online portal? Without the assistance of lawyers (who are a significant cost in disputes), how do laypeople undertake the task of entering information about themselves, the person or entity they are in dispute with, the details of that dispute from their perspective, and the remedy they are seeking? This is a difficult question to answer. The facts of real-world disputes cannot be omnisciently known. Furthermore, “facts” in the real world are not fixed events simply waiting to be inputted into a court portal. Real-world events are transformed into legal facts through retelling, labelling, blaming, interacting with others, and participation in the legal process (Matoesian Reference Matoesian2001; Felstiner, Abel, and Sarat Reference Felstiner, Abel and Austin1980). Given these difficulties—and the fact these systems are new innovations—testing of online court portals has tended to focus on measuring subjective user experiences of these systems (Cambridge Pro Bono Project 2019, 96; Sykes, Dickson, and Ewart Reference Sykes, Rebecca and Sarah2020), rather than examining how accurate and complete laypeople’s narratives are. It is possible, however, to gain some insight into laypeople’s claiming by taking the dispute out of the real world setting and into the laboratory, where the ground truth about the facts is already known. That way, the important facts are known, and it is possible to test whether laypeople select the law-relevant information to input into an online portal. The experimental setting is the “best candidate when one wants to maximize control and make causal claims about relationships” (van Boom, Desmet, and Mascini Reference van Boom, Pieter, Peter, Willem, van Boom and Mascini2018). This is what we did in this study. We created a justiciable dispute and a mock online court (MOC). We then recruited laypeople and lawyers, gave them the same information about the dispute and access to our MOC, and asked them to enter a claim. In doing so, we seek to provide further insights into how laypeople can be effectively supported to explain a dispute narrative to a court.
In the first section of this article, we canvas the substantial literature on the interaction of laypeople with the justice system for insights on laypeople’s ability to engage in legal claiming. Given that online court development has been significantly influenced and tested through “design thinking” or “human-centered design,” we also consider this literature. We argue that a design-based approach has been helpful to reorient court systems to meet the needs of laypeople but needs to be complemented by other research.
In the second part of the article, we turn to explaining the experiment we conducted to examine how laypeople, compared to lawyers, perform the task of entering a claim in an online court. This is the type of complementary research we envisage as being important to ensure online court systems deliver genuine access to justice. Although this paper focuses only on the accuracy of the law-relevant facts which were entered into our MOC, the data produced also provides potential insights into discourse and other aspects of lay claiming strategies.
Laypeople and the Courts
The research on lay interaction with the civil justice system comes from the extensive work on those variously referred to as pro se, self-represented litigants, or litigants in person. This research examines how self-represented litigants engage with the courts, including the deficits in how self-represented litigants explain their cases to the court. These studies have found that a critical obstacle for these litigants is their “inability to state clearly the issues in dispute” (Dewar, Smith, and Banks Reference Dewar, Barry and Cate2000, 56). Moorhead and Sefton’s study found that self-represented litigants “were more likely [than represented litigants] to put forward cases that were entirely misconceived,” a problem that stemmed from, amongst other things, not understanding the purpose of litigation and struggling to identify legally relevant matters (2005, 176; see also Trinder, et al. Reference Trinder, Rosemary, Emma, Joanna, Richard, Leanne, Mark, Victoria and Julia2014).
Failing to understand what should be included in a claim creates difficulties for self-represented litigants. McKeever, et al. identified in their Northern Ireland study that self-represented litigants were unsure about how to prepare documents and what to include: “A recurring comment was the ‘belt and braces’ approach to statements and affidavits where [self-represented litigants] were unsure or were unfocussed about what to include, so included everything for completeness, and so produced over-long written submissions,” (2018, 105). Trinder, et al. conversely observed that self-represented litigants “often filed papers … with too little … information in them” (2014, 42). They speculated that an explanation for very brief documents was that self-represented litigants did not want to engage in pre-disclosure—a requirement of court procedure—and preferred to try and ambush the opposing party (ibid.). Keeping details to a minimum may be part of this strategy or may stem from not understanding what information must be disclosed and therefore erring on the side of brevity.
Self-represented litigants are also thought to struggle with the limits of what the judicial system can offer by way of remedies. As Conley and O’Barr concluded: “the law has come to define the problems of ordinary people in ways that may have little meaning for them, and to offer remedies that are unresponsive to their needs as they see them” (1990, 177). Moorhead and Sefton note that self-represented litigants did not understand the nature of remedies sought against them (2005, 155).
The difficulties that self-represented litigants encounter in preparing court documents should not be surprising. Lawyers who are well-versed in advocacy know that selecting details is intimately connected to the applicable law and the lawyer’s theory of the case. As Rose’s Pleading without Tears advises, “there is no substitute for a full-understanding of the case itself, and the relevant law. In the absence of either of the above, the drafting is bound to be second-rate” (Young and Selby Reference Young and Hugh2017, 11). This ability to select law-relevant facts is a skill that is carefully honed in legal education. In her linguistic study of legal education, Mertz explains how law students are trained in a specific storytelling style where facts selected “are centered on warrants derived from layers of legal authority” (2007, 67).
What is relevant in a story depends on what the legal authority deems relevant. For example, a legal narrative of an incident in a carpark might begin like this:
On or about 25 January 2020, I drove to Target at 67 Main Road, Fairview. On entering the carpark from the Main Road entrance, I searched for an empty space and saw one in the third row. I proceeded towards the carpark but as I approached, a black Toyota Hilux utility (plate unknown) reversed suddenly.
This story includes particular details that are focused on being able to lay blame on a particular party. Precise details about time, location, and vehicle appearance are all included. There is only one perspective (the storyteller’s) speaking as one witness. This is legal storytelling, which Matoesian has referred to in the context of an adversary criminal trial as a type of “regulated storytelling contest between champions of competing, interpretative stories” in which “facts are proven through a complex process of persuasion” (2001, 5, quoting Gary Goodpaster).
In contrast, Mertz explains that laypeople tell “conflict stories” which use “linguistic markers designed to assure listeners of the epistemological strength of speaker’s claim” as well as drawing on a “wealth of cultural warrants through which blame and responsibility are allocated, which include moral claims and emotional contexts” (2007, 67). Let us take the carpark story once again, but this time told as an everyday conflict narrative:
People are so inconsiderate. A couple of weeks ago I was at Target and just as I was pulling into a park some [expletive] guy came roaring out of a park—without even looking—and cut me off. There was a woman standing waiting with her trolley and she was really angry too cos he could have hit her kid.
This story uses various mechanisms to convince the listener that the storyteller has been wronged. It uses a general scene setting statement about “people” being “inconsiderate” and it uses colorful adjectives (an expletive and “roared”) for emphasis. It also adopts multiple perspectives—the storyteller’s and the bystander’s—to increase credibility. None of these devices, however, are part of legal storytelling.
Mertz’s observations of the difference between lay and legal storytelling would predict that laypeople will omit pleading details that support legal warrants and instead focus on details that draw on cultural warrants. In contrast, lawyers have come to possess (via their training) knowledge of “how to construct versions of conflict stories that can be understood by legal authorities and given legal effect” (ibid., 67). This is not a general skill, available to all who are sufficiently educated or “sharp” in their reasoning, but instead “a very particular, culturally laden kind of thinking” in which lawyers are the “experts in one profession’s specialized way of processing relevant information” (ibid., 98).
While lawyers are trained in this form of reasoning, it does not mean that their skill in deploying it is consistently of a high standard. The problems that plague self-represented litigants’ prepared work can feature in lawyer-prepared work. The problems with lawyer-prepared work may be under-reported as, typically, both qualitative and experimental studies have focused on investigating the bias against self-represented litigants (Quintanilla, Allen, and Hirt Reference Quintanilla, Allen and Hirt2017; Macfarlane Reference Macfarlane2013; Toy-Cronin Reference Toy-Cronin2016). However, some qualitative studies do record the problems with lawyer-prepared claims. Studies in England have noted that there were “criticisms of the quality of solicitors’ paperwork” (Moorhead and Sefton 2005, 193) and that “on the whole, initiating documents in most fully represented cases was thorough and complete” but an analysis of files found that solicitors’ work on these tasks was “not always done well” (Trinder, et al. Reference Trinder, Rosemary, Emma, Joanna, Richard, Leanne, Mark, Victoria and Julia2014, 39). Similarly, in another study about self-represented litigants, a judge observed, “you’d be surprised how terrible some lawyers’ submissions are” (Toy-Cronin Reference Toy-Cronin2016, 736).
Redesigning the System
One response to the difficulties that laypeople encounter engaging with the courts is to reorient the system to make it more user-friendly. Court systems have long been regarded as “inaccessible and incomprehensible to ordinary people” (Woolf Reference Woolf1995, 119). What is required is not an upskilling of laypeople, but a court that is simplified so that it responds to the people “for whom the system of civil justice exists” (ibid.). To develop these simplified courts, the legal field has increasingly looked toward “design thinking” or “human-centered design” (Shanahan and Carpenter Reference Shanahan and Anna2019, 113; Hagan Reference Hagan2018), methods now being applied across private and public organizations. As Kimbell explains:
[Designers] are seen as using an iterative process that moves from generating insights about end users, to idea generation and testing, to implementation. Their visual artefacts and prototypes help multidisciplinary teams work together. They ask “what if?” questions to imagine future scenarios rather than accepting the way things are done now. With their creative ways of solving problems, the argument goes, designers can turn their hands to nearly anything. Design is now central to innovation and since organizations … in the public sector … are under pressure to increase user satisfaction and effectiveness, then designers and their thinking have something important to offer (2011, 287).
A system-design approach begins with a prototype and then tests it with the intended users. User-testing triggers a continuous cycle of design iteration, which continues throughout the life of the new system. For those involved in the development of the major online court systems, this approach is becoming a “new orthodoxy” (Tomlinson Reference Tomlinson2019, 71–72). For example, Susan Acland-Hood, the Chief Executive Officer of HM Courts & Tribunals Service, has referred to needing to “keep on testing, learning and refining rapidly” (Acland-Hood Reference Acland-Hood2019), to following the “principle of testing and trying everything with real users to help us get it right” (Acland-Hood Reference Acland-Hood2018a), and to developing new technologies “in partnership with and around the people who use them” (Acland-Hood Reference Acland-Hood2018b).
This design-centered approach has much to recommend it in that it reorients thinking about the system to those it is intended to serve. However, it also has limitations. The emphasis on user experience means an emphasis on subjective assessments of procedural justice and satisfaction. These are not unimportant. Procedural justice contributes to citizens perceiving the legal process as legitimate and obeying its authority (Tyler Reference Tyler2006). But, people may subjectively assess a procedure as fair, even though it may not produce a substantively just outcome (Byrom Reference Byrom2019). Similarly, user satisfaction is a useful metric but “is only one of several values which are important to a properly functioning court and tribunal system.” The “pursuit of satisfaction” needs to be balanced “with the other desirable needs of the justice system such as predictability, efficiency and the delivery of just outcomes” (Moorhead, Sefton, and Scanlan 2008, 2).
The field of dispute system design, which began from examining dispute resolution systems within organizations (Costantino and Sickles Merchant Reference Costantino and Christina1996), encourages careful articulation of the goals of the system being developed (Amsler, Martinez, and Smith Reference Amsler, Janet and Smith2020). The goals of an online court must go beyond subjective experiences of the users and focus on whether the design can achieve an effective hearing and a decision in accordance with substantive law. A key element of reaching these goals is that the system must elicit the law-relevant facts. All parties must “present the information necessary to enable a decision maker to make a determination based on applying the law to the facts of the case and that the decision maker is able to comprehend this information” (Byrom Reference Byrom2019, [9.1]).
In addition to the claim informing the decision maker of the relevant facts, it should also persuade the opposing party. As Rose’s Pleading without Tears advises, the claim is a chance to stamp your “authority” on the case. When confronted with a well-drawn claim opposing counsel might reconsider “the wisdom of proceeding with the action” and “see if a ‘deal’ could be done” (Young and Selby Reference Young and Hugh2017, 9).
The complexity and importance of accurately entering a claim is, however, often brushed over in discussions about online courts. It is assumed that with a plain language, user-friendly interface, disputants can perform this task online, even though many laypeople struggle to do this on paper. It is also assumed that the facts are simply sitting there, waiting to be entered, and that writing them into an online court portal is a neutral, objective activity. This is contrary to the nature of a conflict story, however, which as Conley and O’Barr have observed, “does not exist fully developed on its own, but only emerges through a collaboration between the teller and a particular audience” (Conley and O’Barr Reference Conley, William, O’Barr and Conley1990). In the case of an online portal, the teller interacts with the interface, and the portal therefore acts as a collaborator. Inherent in any interface are various features that allow, encourage, and constrain user actions to various degree, together referred to as the “choice architecture” (Thaler, Sunstein, and Balz Reference Thaler, Sunstein, John, Shafir and Princeton2013). For example, some actions are allowed but require time and effort (such as filling in a text box) and others are easy but necessary (making a selection from a pre-populated list before being able to move to the next page). The interface will, therefore, have an important role in shaping the narrative that is ultimately told.
At this point, some online courts proponents will point out that the integration of dispute resolution processes into the online court interface means that the original framing of the dispute is unimportant. Some online court designs have an intermediary who will assist the parties in clarifying the facts and negotiating a resolution. If it is all going to be negotiated and discussed, then why bother trying to convert a lay dispute narrative into a legal one? Why not just let people talk? The problem is that the quality of the initial information is important; there is a real risk that if a claim is poorly drafted it will be dismissed (Sela Reference Sela2016, 357–58). The court—whether a judge or someone employed to assist with a resolution process—has no omniscient standpoint. They can only know what they are told and, therefore, the information inputted matters. The quality of what is inputted also matters for more pragmatic, fiscal reasons. If the court has to engage in refining a lay conflict narrative into a legal one, the promised cost savings start to look elusive. This is illustrated by Trinder, et al.’s discussion of the role of “third party (quasi) lawyers” in family court disputes. These third parties included people acting in facilitative roles, such as children’s lawyers, who worked with the parties out of court in attempting to reach an agreement, as well as identifying the parties’ positions and presenting those to the judge (2014, 118). This is a real-world analogue to an online court facilitator. The authors noted that while this use of third parties “could be a very effective model … if pursued as a formal strategy” it would require more resources to support the appointment of these third parties (ibid.). The role of a facilitator in an online court could be an effective way to refine claims, but it is resource intensive. The goal of an online court should still be to, as much as possible, help people to explain their claim accurately. “It is a false economy for cases to begin and proceed with pleadings mispremised” (Kós Reference Kós2016, [49]).
Therefore, in addition to design-thinking, user-testing orientations, we need a method to see how laypeople select facts and prepare a claim. This can help designers create court portals that maximize law-relevant information so that both the opposing party and decision-maker are well informed and that realize the promise of cost saving for the user and state provider.
METHOD
We designed and implemented a laboratory study to test how people interact with an online court portal. It is only through a laboratory study that we can gain the omniscient view of the facts. In the real world, the true facts cannot be known, but in a laboratory study we control the facts and can therefore measure which facts people select for inclusion in a claim, how they discuss them, and what they omit. We recruited both laypeople and lawyers so we could compare the performance of the two groups. This also enables us to consider the strength of lawyers’ claims as being a necessary assistant to court proceedings.
We developed a mock online court portal (MOC) based on British Columbia’s CRT Loans and Debts Dispute Resolution Portal, as it was at the time when we began the research in May 2018 (it has since been altered). Using Qualtrics software, we mimicked the same questions and branching logic to develop a replica.Footnote 1
The question about how well laypeople versus lawyers perform this task requires both quantitative and qualitative analysis. Some aspects can be measured quantitatively—the presence or absence of certain elements in the claim—and for these aspects we developed three hypotheses which we tested through quantitative analysis. Other aspects of the task—the way in which elements of the justiciable problem are explained in the portal—are only identifiable through qualitative analysis. We thus use both quantitative and qualitative analysis to answer our central research questions: Do lawyers perform the task of claiming better than laypeople? If so, with the user-friendly MOC’s assistance, will laypeople be able to perform the task as well as lawyers?
The three hypotheses that we tested through quantitative analysis are as follows:
-
1. That laypeople will name the same number of parties as lawyers. The fact pattern has three possible parties but there is only a strong legal claim against one. For the other two parties to be liable there would need to be a more complex argument involving agency and/or a constructive trust and there are not enough facts to support this. As laypeople will not understand the limits of these arguments—but would know from the fact pattern that they are part of the story—we might expect that they will name more parties than lawyers. However, laypeople may not understand that they can name multiple parties in a claim. We have therefore predicted there will be no difference between the two groups.
-
2. That lawyers will include more legally relevant details than laypeople when drafting their claims. The review of the literature suggests that laypeople will select different facts for inclusion in a claim, omitting legally relevant facts, and selecting facts that support a lay conflict narrative.
-
3. That there will be a bimodal distribution in claim length among laypeople, who will write claims that are both much shorter and much longer than claims prepared by lawyers. This is also based on the literature that suggests laypeople are either overly brief or excessively inclusive in drafting their claims.
In addition to quantitatively testing these three hypotheses, we also engaged in a qualitative analysis of the participants’ clarity of reporting detail and the use and purpose of certain types of detail.
The Justiciable Problem
With the help of two civil litigators, we designed a fictional dispute with the aim of conveying a realistic legal problem that was also suitable for the laboratory. The dispute centered on a loan dispute between twin siblings, Jake and Amy. Jake advanced two loans to Amy, neither of which she repaid. The first was $20,000 to establish a food truck company (“Loan One”). The food truck company (“the Company”) had the potentially offensive name “AfroEatz,” a fact of some importance to a potential counterclaim by Amy against Jake. The terms of Loan One were set out in a written contract, witnessed by a neighbor, and Loan One was repayable within a year. Amy also agreed to repay the bank fees of $850 that Jake incurred in breaking his term deposit to access the funds. The second loan was advanced several months after Loan One, and was for $5,500 to purchase goods for the Company (“Loan Two”). Loan Two was repayable in three months along with $1,000 interest and 25 percent of the shareholding in the Company. The date for repayment of both loans had passed and no repayment had been received, despite Jake’s demands for payment. Amy claimed that she had no money, that Jake had caused reputational harm to the Company, and that Loan Two was a gift, not a loan. Jake had discovered that Amy had incorporated the Company but the registered shareholders were Amy (50 percent) and her boyfriend Mark (50 percent). Jake’s relationship with his sister had completely broken down and Jake was asking for help filing his claim to recover what he was owed.
We conveyed information about the dispute to the participants through a written bundle of information and through an interview with “Jake,” played by a confederate. The written bundle of information was designed to communicate the dispute in a realistic way and included emails with a narrative of the dispute, Facebook Messenger and text messages, a dated written contract between the siblings, a bank statement, and a Companies Office record. During the interview with the confederate, further detail was disclosed, including dates and further documentation. We refer to this information collectively as the “fact pattern.” All participants received all the information in the fact pattern.
Participants
We recruited sixty-seven participants from two populations: laypeople (members of the community with no legal training) and qualified New Zealand lawyers. A summary of the profile of the two groups appears in Table 1. We recruited thirty-eight lay participants from the Dunedin community, a medium-sized New Zealand city with a university. We did not advertise on campus, however, as we wanted a representative lay population and we therefore recruited using Facebook advertisements. We excluded all volunteers who were past or present law students and anyone with any legal qualifications. Seventeen lay participants had had some previous involvement with the New Zealand justice system (e.g., as disputants or witnesses in Family Court proceedings, employment mediations, tenancy disputes), but no participants had any involvement with a case similar to the factual pattern, so were all included in the final sample.
Summary of Participant Demographics
The twenty-nine lawyer participants were recruited by way of advertisements and approaches to personal contacts in the legal profession. To participate, the lawyer must have been admitted to practice law in New Zealand and have three-years post qualification experience (PQE) in any type of litigation. We strictly adhered to these criteria which made recruitment more difficult but protected the findings against interpreting any deficiencies in the lawyers’ claims as caused by inexperience. The participants had a range of experience, with 55 percent of the lawyers having more than ten years PQE and two lawyers carrying the senior rank of Queen’s Counsel.
Experimental Procedure for Lay Participants
Lay participants completed the procedure during a laboratory session that lasted for approximately three hours and was audio and video recorded. The laboratory session included a meet and greet with an explanation of how the session would proceed. Participants were told we were interested in how laypeople could help other individuals to file a claim in court; specifically, we were asking them to help Jake who had a dispute with his sister, and he wanted help to prepare his claim. The participants were deceived to increase the likelihood that they would be more engaged and invested in the experiment. The session then proceeded as follows: participants filled out a questionnaire to collect demographic information; participants then reviewed the bundle of written materials for up to thirty minutes (as needed; all participants finished their review before the thirty minutes expired); interviewed the confederate; and finally, entered the claim into the MOC. After they completed all the steps the participant was fully debriefed and told the true purpose of the experiment, that the dispute was fictional, and the confederate was playing the role of a “mock client.” The experimenter explained to the participant the aims of the study and why deception was necessary. The participant was invited to ask any follow up questions about the process and aims of the research generally. At the conclusion of the debrief procedure, the experimenter gave the participant NZD $60 as reimbursement for expenses incurred.
Experimental Procedure for Lawyer Participants
The procedure for lawyer participants differed in two main ways to the lay participant procedure. First, we did not attempt to deceive the lawyers about the true nature of the study as lawyers are required to identify clients and enter into a written retainer before advising them. Second, lawyers did not attend in-person but participated remotely. This was because we would not have been able to recruit enough participants locally or to secure so much in-person time.
The procedure was otherwise the same as for lay participants but modified to be delivered remotely. As with the lay participants, the time allocated was three hours and there was a demographic questionnaire. However, the questionnaire was administered at the time the lawyer gave their consent to participate (electronically) rather than at the beginning of the experiment. The experiment began, as it had with lay participants, with the lawyer receiving the bundle of written information but this was provided via email. Once the lawyer had reviewed this material, they then called the confederate via video conferencing (Zoom) and conducted the interview. At the conclusion of the interview, the lawyer received a link to the MOC and filled out the claim. After the lawyer had completed the claim, they were automatically redirected to another screen. That screen contained further information to debrief the lawyer about the purpose of the project. Our contact details were provided and the lawyer was invited to arrange a time to ask any follow-up questions about the procedure and aims of the research generally. Finally, the lawyer was invited to provide their contact details so that we could send to them an NZD $50 book voucher as reimbursement for expenses incurred.
Using Judges to Assess Claims
To supplement our own analysis of the data, we recruited four retired members of the New Zealand judiciary who had previously presided over civil court matters to assess a set of claims. The judges were not provided with any compensation for their participation. We sought both qualitative comments and reactions to the claims and asked them to rate each claim’s quality using a Likert scale.
We compiled a workbook of twelve of the claims generated in the MOC to send to them. We were only able to provide this smaller sample of claims as the judges were volunteers and there was a significant amount of reading in the small sample; we needed to keep the request reasonable to secure their participation. To select the lay claims, one author read through the “what happened” response and grouped the responses by length and by appearance of legal style (use of headings, prominent use of dates, particular legal terminology, defined terms). The other author then took one file from each of the categories to include in the sample, i.e., long and not legal style; medium and not legal style. To select lawyer claims, a sample of five with differing styles in terms of length and approach were selected. Once in the workbook, the claims did not state whether they were by a lawyer or a lay person. The workbook had a wide right-hand margin for comments and at the end of each claim there was a box for further comments and assessment (see Figure 1).
Comment box for Judicial Participants.
ANALYSIS
The quantitative and qualitative analyses were undertaken separately but in dialogue, with preliminary findings of one type then investigated with the other type. The combination of both quantitative and qualitative content analysis was designed to provide greater insight than what could be derived from only one form of analysis (Oleinik Reference Oleinik2011).
Quantitative Content Analysis
The quantitative coding procedure was modeled on research that uses controlled laboratory studies to investigate the accuracy of memory reports (see for example Zajac and Hayne Reference Zajac and Harlene2003). The participants’ responses (their “claims”) were extracted from the MOC, and then coded by two experimenters. For the lay participant data, the full dataset was independently coded by two experimenters. Intercoder agreement was 95 percent. For the lawyer participant data, a subset of the sample was independently coded by two experimenters (17 percent). Intercoder agreement was 92 percent. We computed a Cohen’s kappa coefficient using IBM SPSS Statistics for Mac, Version 26.0 to assess the reliability of the coding. For the details reported by laypeople, the kappa statistic ranged from moderate (0.50) to almost perfect (1.00) with the majority being almost perfect (1.00). For the details reported by lawyer participants, the kappa statistic ranged from moderate (0.44) to almost perfect (1.00), again with the majority being almost perfect.Footnote 2 Any discrepancies in the coding were discussed by the experimenters and resolved through consensus.
To determine whether participants understood the dispute generally, and the broader legal issues, we coded the details they provided using a checklist of narrative details. There were nine items on the check list, so the participant could receive a maximum score of nine. The experimenter reviewed the entire claim for evidence of the nine dispute narrative details, and coded the details using a binary scheme. If the detail was reported, they were awarded one point. If the detail was omitted, they were not awarded the point. For example, if the participant explained that Jake gave Amy some money to start a company but never specified the sum (i.e., $20,000, one of the nine key details), the participant would not be awarded the one point for having reported that key narrative detail.
In addition to coding according to the narrative checklist, we coded the party or parties the participant elected to bring the claim against: the sibling (Amy); the Company; the partner (Mark); or any combination of all three.
We also coded for inclusion of seventy details regarding Loan One and Loan Two (see Table 2 for summary). Law-relevant dispute details were identified by the two civil litigators who had helped construct the claim. This was coded using a ternary coding scheme to take into account ambiguous reporting of a details. If the detail was explicitly reported, the participant was given two points. If the detail was unclearly reported, the participant was given one point. If the detail was not reported at all, the participant was given no point.
Summary of Loan One and Two Dispute Details
Qualitative Content Analysis
The qualitative analysis was conducted using NVivo and was a form of qualitative content analysis (Graneheim, Lindgren, and Lundman Reference Graneheim, Britt-Marie and Berit2017, 29). The procedure compared the responses to each question in the MOC, looking for similarities and differences in the mode of expression and information between participants, and between participant groups (lawyer versus lay). For each question, a series of codes was developed to reflect any observations about style (syntax, tense, direct or indirect demand) as well as the content. For example, the question which required a one sentence summary of the dispute was coded for syntax (sentence fragment, simple sentence, complex sentence, more than one sentence) and for content (mentions family relationship, mentions loans, no mention of loan). Linked to the code was a memorandum to record observations and insights about the data and how these linked to the literature.
The judicial evaluation data on the twelve claims was collated in a spreadsheet. This enabled identification of similarities and differences in their reactions to the claims.
RESULTS
Hypothesis 1 - Selecting Parties
The first hypothesis concerned selecting the defendant or defendants to the proceeding. Identifying the parties is the critical first step in filing any claim in court. If the claimant does not correctly identify the person(s) or organization(s) that owes the legal duty, the proceeding will not succeed; conversely, incorrectly identified parties will have to bear the cost of responding to the proceedings, even though they have no liability.
The hypothesis was that there would be no difference in the number of parties laypeople and lawyers named in the proceeding. As the data was not normally distributed, we have conducted nonparametric analyses. A Mann-Whitney U test supported our hypothesis, and demonstrated that there was no significant difference between the number of parties the lawyers (Mdn = 2.00) and laypeople (Mdn = 1.00) named; U = 450.0, z = −1.40, p = .161, r = −.171. As with any small sample it is possible that the difference between the medians would become statistically significant if the sample was bigger.
Although there was no difference in the number of parties named, we conducted exploratory analyses to investigate whether there was any difference as to who the participants named. In the fact pattern, Amy was the borrower so she should have been named as the first respondent. However, there were two other possible parties: the Company, and Mark (co-director of the Company and 50 percent shareholder in the Company). Amy told Jake that Loan One was to enable her to set up and run the Company and Loan Two was for expenses related to this business. It was possible to make an argument that Amy was acting as an agent for the Company when she accepted the loans, or at least when she accepted Loan Two. This would be a difficult legal argument, however, as there was nothing in the facts that strongly supported her acting as an agent for the Company, and the Company was not incorporated until after Loan One was advanced. The Company could be included for strategic reasons (e.g., to encourage settlement by drawing the company in to the proceeding), but there was no strong legal foundation for the claim. Similarly, there was nothing in the fact pattern that suggested Mark was liable, or at what point he was even aware of the loans. The only conversation Jake had with Mark was after Amy failed to repay the loans. In that conversation, Mark said the loans were to Amy, not the Company. There was nothing in the fact pattern to establish a claim against Mark in his personal capacity and he should not, therefore, be named. Therefore, who is named as a party has important legal implications.
All of the participants named Amy as a party to the proceeding. However, 65.5 percent of lawyers and 44.7 percent of laypeople also named Mark or the Company as a party to the proceeding. We investigated whether there were any differences across groups in naming Mark or the Company. As we have a small sample, we have reported the Fisher exact test. These analyses indicated that a small number of participants in both groups named Mark as a party (p = .25); whereas a larger number—and significantly more so for lawyers—named the Company as a party (p = .007, ϕ = .337) (see Table 3).
Percentage of Laypeople and Lawyers who named Amy, Mark and the Company as a party to the proceeding
By looking at which parties were included in the claim, we have developed hypotheses that warrant further investigation in subsequent research. A small percentage of laypeople and lawyers brought Mark into the proceeding. It is possible that these participants did not understand that Mark could not be personally liable; or his liability clearly had to be linked to the company. This suggests limited understanding of corporate law, an issue we discuss further below. Lawyers more often brought the Company into the proceeding. A plausible explanation for the lawyers’ claiming behavior was that these participants were seeking a strategic advantage by involving a company, or adopting a strategy that would spread the liability across multiple parties.
The judicial evaluators were critical of both the laypeople and lawyers for naming the Company and Mark but then failing to plead how they were liable. Where a connection was created to plead liability of the Company or Mark, the judicial evaluators were still critical. In the MOC section for remedies, one lawyer said, “I want the court to find there is a constructive trust over the profits.” One judicial evaluator noted “no legal basis for pleading this claim” and another said “for a claim of this size, an unnecessary complication.” The preferred approach, from the judicial evaluator’s perspective, was to only claim against Amy.
Hypothesis 2 - Detail Selection
The second hypothesis was that lawyers would include more law-relevant detail than laypeople. To assess the content that lawyers and laypeople included in the claim, we first examined the dispute narrative details. As the data was not normally distributed, we have conducted non-parametric analyses. A Mann-Whitney U test demonstrated that there was a significant difference between the groups, with lawyers (Mdn = 9.00) reporting a higher number of narrative details relative to laypeople (Mdn = 7.00), U = 150.50, z = −5.34, p < .001, r = −.653. Similar findings were observed between the dispute details, with lawyers (Mdn = 44.00) again including a higher number of relevant details in the claim compared to laypeople (Mdn = 27.00), U = 207.00, z = −4.356, p < .001, r = −.532. This finding supported our hypothesis that lawyers would include more law-relevant details than laypeople.
The fictional dispute included varying levels of legal complexity. Most notably, Loan One was set out in a written contract, whereas Loan Two was a verbal contract. It is possible that the difference in reporting of dispute details emerged because laypeople did not think that the verbal contract was enforceable and, therefore, did not plead it. To see whether that could explain the difference, we conducted exploratory analyses to look at the loans separately. Wilcoxon signed-rank tests indicate that there was a difference in the inclusion of law-relevant details across the loans. Both laypeople: W = 79.00, z = −3.744, p < .001, r = −.607 and lawyers: W = 39.50, z = −3.733, p < .001, r = −.693 reported a higher number of law-relevant details for Loan One, as compared to Loan Two. Both groups were better at identifying the legally relevant details from Loan One (see Table 4).
The Median Number (95% CI) of Legally-Relevant Details Reported for Loan One and Two, for Laypeople and Lawyers
However, lawyers still outperformed laypeople in including law-relevant details. A Mann-Whitney U test demonstrated that lawyers reported a higher number of law-relevant details, relative to lay people, for both Loan One: U = 252.00, z = −3.790, p < .001, r = −.463; and Loan Two: U = 188.00, z = −4.600, p < .001, r = −.562 (see Table 4). These analyses confirmed that the overall differences did not simply emerge because there were two different loan types. Although laypeople did report significantly less of the Loan Two details compared to Loan One, so too did the lawyers.
Hypothesis 3 - Length of Claim
Our third hypothesis was that there would be a bimodal distribution for claim length among laypeople who will prepare very long or very short claims. Laypeople used a median of 142 words (range = 11–945 words) in the descriptive textbox to describe the narrative; lawyers used a median of 235 words (range = 73–528 words). This was a statistically significant difference, U = 353.00, z = −2.506, p = .012, r = −.306, but with only a medium effect.
However, this difference could not tell us whether laypeople behaved as we hypothesized (i.e., significantly shorter and longer claims), so next we investigated if the distributions differed. The lawyers’ length of claim was normally distributed: W(29) = .948, p = .161; but this was not the case for the laypeople: W(38) = .712, p < .001. Closer inspection revealed that laypeople’s claims were positively skewed—that is, more laypeople prepared much shorter claims. With this sample, the hypothesis that there would be a bimodal distribution in laypeople’s claim length was not supported.
We conducted exploratory analyses to investigate whether the length of claim had any relationship with the completeness of the claim. We used a nonparametric correlational test for the laypeople as the length of claim was not normally distributed. Kendall’s Tau-b correlation revealed that there was a moderate positive correlation for the laypeople: Tau-b (36) = .475, p < .001. Whereas, there was a strong positive (Pearson) correlation for the lawyers: r(27) = .787, p < .001 (see Figure 2). Increasing words did not increase inclusion of relevant detail at the same rate for laypeople, relative to lawyers. Laypeople tended to “cap out” in the amount of relevant detail they included; there was a tipping point where more words did not increase substantive detail.
Number of words participants require to articulate claim as a function of the proportion of relevant details included.
This finding tends to support the idea that some law-relevant facts are simply not salient to laypeople. Allowing more space will not increase the number of details they provide.
Brevity is generally prized in legal writing. It has institutional benefits in terms of time and, therefore, cost savings when filling out court forms. The judicial evaluators were, unsurprisingly, critical of pleadings that were long but did not use the word count to include legally relevant details. For example, judicial evaluators commenting on LAY14’s claim said: “The pleading is a general ramble that fails to limit itself to pertinent facts” and “Overloaded with totally irrelevant material which is quite unnecessary. Words for the sake of words. Tighten, cull and focus discussing.” Lawyer-pleaded claims were not immune from this criticism. For example, one judicial evaluator assessing LAW28’s claim said it was “full of unnecessary and unhelpful details of no probative value.” However, the judicial evaluators were similarly critical of the very brief pleading included in the subset: “this claim identifies existence of loans and failure to repay but fails to give essential particulars” and “while the brevity is to be also applauded it is too brief; no references to written agreements of texts.” We did not include in the sample of claims sent to the judges the extremely brief pleadings that some laypeople made, so these comments did not relate to the very shortest claims. The judicial evaluators’ comments illustrate that length is a relevant factor for the fact-finder, and they are rightly reluctant to try to draw out, or construct, the necessary details from a claim. This suggests the importance of presenting a well-constructed claim to the fact-finder.
FURTHER ExPLORATORY ANALYSIS
The hypotheses are oriented to looking at the inclusion and omission of information and the number of words used, information that most easily lends itself to quantitative analysis. It is not only the presence or absence of information that is important, however. So too is the clarity with which the claim is expressed. In this section, we look more closely at the ways different types of detail were reported. This does not go as far as analyzing the different discourses participants used, but we did use exploratory quantitative analyses, as well as qualitative content analysis, to tease out the way participants deployed details.
Clarity of Reported Details
Recall that the participants’ claims were coded depending on whether the details were explicitly reported, unclearly reported, or omitted entirely. We used this coding to focus on clarity, and investigated whether there were differences in the number of explicitly and unclearly reported facts in the participants’ claims. We again used nonparametric tests. A Mann-Whitney U test indicated that there was a significant difference between the groups, with lawyers explicitly reporting a greater number of facts (Mdn = 23.00) relative to laypeople (Mdn = 11.00), U = 130.00, z = −5.34, p < .001, r = −.652. Whereas, laypeople unclearly reported a greater number of facts (Mdn = 5.00), relative to laypeople (Mdn = 2.00), U = 191.00, z = −4.594, p < .001, r = −.561.
When we compared the claims that had high numbers of unclearly reported facts against those with high numbers of explicitly reported facts, the difference in clarity for the reader was stark. Comparing LAY22 and LAW22 provides an illustrative example. LAY22 discusses both loans together, making no distinction between the terms and consideration for Loan One and those for Loan Two. The claim includes the statement: “She agreed to give me 25% of the shares” but there is no mention of what entity the shares are in and there is no link between the promise of shares and Loan Two. In contrast, LAW22 is explicit about all these facts: “My sister made other promises to me that caused me to make the second loan, including making a commitment to me to give me 25% of the shares in the company she was establishing for her food truck business.”
This has a significant impact on the overall comprehensibility of the claim. In contrast, the unclear reporting of details alerts the other party and adjudicator to their existence, which is preferable to complete omission, but leaves questions that need to be answered before the claim could be considered complete. As one of the judicial reviewers noted “the adjudicator is left to fill in the gaps.”
Use of Chronological Detail
One type of detail—chronological detail—is worth further exploration. Chronological information—what Tillers and Schum (Reference Tillers and David1991, 955–56) decribe as “the temporal locus of evidence in time”—is essential to legal reasoning, and litigants who fail to discuss facts with reference to time will produce flawed claims. We looked in closer detail at each group’s use of dates in their claims. In the fact pattern, there were eight important dates. The participants use of dates were coded as being clearly described (e.g., on 3 March 2017…), unclearly reported (e.g., in March…), or omitted entirely. Mann-Whitney U tests demonstrated that lawyers (Mdn = 5.00) explicitly reported a higher number of dates compared to laypeople (Mdn = 1.00; U = 156.50, z = −4.788, p < .001, r = −.585); whereas, laypeople (Mdn = 5.00) omitted a higher number of dates compared to lawyers (Mdn = 3.00; U = 225.50, z = −4.155, p < .001, r = −.508). There was no group difference for unclearly reported dates (Lawyers Mdn = 0.00; Laypeople Mdn = 0.50; p = .124).
It is unsurprising that laypeople would be less likely to include dates in a claim, as they might be considered clutter in ordinary conversation. Sequencing and precise timing is often important in legal argument, however. Take, for example, the chronology for events relating to Loan Two. Amy claims that Loan Two was a gift, not a loan (as discussed below). The chronology, however, shows that the terms of Loan Two were discussed before the money was advanced, supporting the argument it was a loan. These chronological details are included to strengthen Jake’s position and minimize the force of Amy’s defense.
Deploying Detail with Purpose
Including dispute details is important in that it notifies the fact finder and opposing party of the relevant facts. However, as discussed at the outset, a good pleading deploys detail strategically and advocates for the party’s position. Strategic use of detail is not suitable for quantitative analysis as it requires interpretation through the lens of legal strategy. This aspect of the investigation is exploratory and uses only qualitative analysis.
There were distinct differences between the way lawyers and laypeople deployed detail in three parts of the claims: how the participants raised the possible defenses, how the participants talked about prior efforts to resolve the dispute, and what the participants hoped to achieve if the claim was successful. We discuss these in turn.
Raising the Defense and Counterclaim
Within the fact pattern there were details about Amy’s likely responses to any claim that she was obliged to repay the loans. Inclusion of these details were not essential to create a valid claim, but they could be strategically deployed. In response to Loan One, there was a possible counterclaim that Jake had damaged the Company, named “AfroEatz” (“the Counterclaim”). This was explained in the fact pattern as follows:
“It turns out that someone had complained about the name of the food truck (hardly surprising…) to a reporter, who had then used a photo of the truck as an example of cultural appropriation at music festivals. Amy thinks that this was bad publicity and caused her to lose business…. But I would never make a complaint.”
In response to the claim for repayment of Loan Two, the fact pattern included a statement that Amy told their mother Loan Two was a gift, something that Jake “absolutely denies” (“the Defense”). It is not strictly necessary to raise the possibility of the Counterclaim or the Defense. However, it is potentially strategically useful to raise and refute possible defenses and counterclaims when filing a claim. As one of the judicial evaluators noted, raising the defense was “irrelevant as to liability but it does identify nature of Amy’s likely defense.”
Twenty percent (n = 6) of the lawyers used this strategy of identifying both the counterclaim and the defense. Having raised the counterclaim and defense, the lawyers then denied the facts underlying both. There was only one slip, where a lawyer raised the defense that Loan Two was a gift, but did not deny it, replying only to the counterclaim. In all other cases, the lawyers either raised the counterclaim and defense and gave a bare denial, or raised them and denied by giving a simple statement of facts. Examples of each are as follows:
Bare denial: “Amy has said she won’t pay the $20,850.00 because she claims I have caused her damage to the business through some adverse publicity. I deny that I have done anything to damage her business. Amy says she won’t pay the $6,500.00 because it was a gift from me which I deny” (LAW3).
Denial with a simple statement of the facts supporting the denial: “The First Respondent has advised (through our mother) that she agrees the first loan was for the company, but that I have caused damages as she believes I made a complaint about the name of AfroEatz. I never made this complaint. The First Respondent has advised me (again through our mother) that the second loan was a gift. I deny this. I am able to show text messages asking for interest on the loan and a bank statement showing I only paid the principal, not the whole amount” (LAW13).
In comparison, laypeople more often raised Amy’s expected responses to the claim: 24 percent (n = 9) identified both the Counterclaim and the Defense in their claim; 21 percent (n = 8) raised only the Defense. The Counterclaim (that a media statement damaged the Company) is more complex than the Defense (that Loan Two is a gift), which probably explains why so many laypeople only raised the Defense. What was also notable was that having raised the Counterclaim and/or Defense, most (though not all) offered no reply or denial. For example:
“When they were in contact Amy said that she could not pay back $20,850.00 as she did not have the money and that the $5,500.00 was a gift. The $20,000.00 Amy refused to pay back because of supposed damages that I had caused. This is because she claims that I complained about the name of the food truck to a reporter and caused them to lose business.” (LAY7)
To raise a possible defense and counterclaim in a notice of claim, but to offer no response to it, might embolden the defendant to defend the case, as well as leaving doubt in the mind of an adjudicator. For these reasons, it is strategically important that if a claimant raises a defense or counterclaim it should also offer a reply, otherwise it might be better to omit any mention at all (as many of the claimants did). Those that raised the Defense and Counterclaim, without the reply, lost a valuable advocacy opportunity.
Discussing the Failed Attempt at Mediation
The MOC contained a question that related to resolution of a dispute that is not traditionally part of pleading a court claim: “what have you done so far to try and resolve this dispute?” The fact pattern contained a number of different pieces of information that could have been used to answer the question. One that most participants mentioned was Jake’s attempt to have his mother mediate between him and his sister. The information in the fact pattern about the mother’s intervention was as follows:
“Now mum becomes involved in the story. She agreed to be an ‘intervenor’ or ‘mediator’ or something. After mum talked to Amy and Mark, she then reported back to me. Apparently—Amy agreed she had borrowed the $20,000 for business but was now refusing to repay the loan because of the “damages” that I caused. … Amy went on to tell mum that the $6500 I said she owed was actually a gift! Absolute rubbish! … Since mum got involved, I am not speaking to either Amy or Mark at all. I’m also pretty mad with mum, I don’t think she really helped things.”
Some of the lawyers (n = 12, 40 percent) and a few of the laypeople (n = 7, 18 percent) reported this as a failed attempt at mediation. For example, “My mother attempted to mediate a solution, but that did not succeed” (LAW10). Strategically, this is a sound approach as it is a statement to the court that the informal attempts at resolution have failed, and the court’s intervention is now warranted. The rest of the lawyers who mentioned the mediation (n = 14, 47 percent), and some of the laypeople (n = 13, 34 percent), claimed it occurred but were not explicit about the outcome, for example “My mother has attempted to mediate” (LAW10) or “I have attempted to resolve this dispute by way of family intervention and informal mediation” (LAW19). The failure of the mediation is implied by the language of “attempt” and by the fact that the dispute has proceeded to court, so this is not overly problematic.
There was another cohort of laypeople who mentioned the mediation (n = 14, 37 percent), but did so in a way that created confusion about the event. This included creating doubt about whether the mediation had occurred at all (e.g., “I have also contacted my parents with the story in the effort to get them to talk to her” (LAY10)), and a lack of clarity about the nature of the intervention (e.g., “Mum and Dad are involved” (LAY23)). In another instance, the participant made an oblique reference to the Counterclaim, but provided no further explanation:
“Asked mum to talk to Amy and Mark as a mediator to see what could be done, instead was told none of the money would be coming back because of apparent “damages” to give the food truck a bad name.” (LAY4)
This participant made no other mention of the facts related to the Counterclaim, so the adjudicator would be left wondering what this refers to.
The way in which laypeople discussed the mediation information points to an important difference in how laypeople, as opposed to lawyers, report information. The basic details are reported (there is some form of intervention from the parents), but how this expressed is very uncertain. The details are not being reported with a clear purpose (to inform the court that intervention is necessary because all informal strategies are foreclosed) but floated as part of a general narrative.
Raising the Assignment of Shares as a Remedy
One of the key differences observed in the qualitative analysis was how lawyers and laypeople discussed the assignment of shares. In the fact pattern, Amy promised Jake a 25 percent shareholding in the Company when Jake advanced Loan Two. The Companies Office record, included in the written materials, listed Amy and Mark as equal shareholders, omitting Jake. 65.5 percent of the lawyers (n = 19) requested transfer of shares as a remedy. This example is typical: “I want judgment that [the Company] must transfer to me 25 percent of its ordinary shares” (LAW9). A further two lawyers did not request the shares as a remedy but discussed the shares in the “any other comments” box. One lawyer suggested it as an interim remedy while finance was arranged to repay the debt and the other lawyer said that the shares would not be claimed because “Accepting shares in lieu of debt would be a recipe for disaster in view of the personal dynamics.”
In contrast, only 18.4 percent (n = 7) of laypeople made a request for shares as a remedy. Of those seven, only three made a direct request for a share transfer, for example: “I want to be listed as 25 percent shareholder in [the Company] as agreed” (LAY25). A further four laypeople made some reference to the shares but expressed ambivalence about whether or not they wanted a share transfer. Their responses disclosed uncertainty about the implications of being a shareholder:
“I want the 25% shares in Amy’s company that she promised me, but only if it will make me some money and not cause me any risk.” (LAY2)
“I have no interest in being part of the business as I do not understand how business or shareholder status works.” (LAY24)
“I want a 25% shareholding in the company as agreed. I do not want to be involved in the day to day running.” (LAY31)
Six other laypeople mentioned the shares in the final screen of the MOC which asked for “any other comments.” These participants did not ask for the shares to be transferred but noted the failure to transfer the shares as promised. We can infer that they thought the information was important enough to mention but were not sure what action could, or should, attach. This example is typical:
“I was told that I would be a shareholder in Amy’s company when it was registered as a business. Instead, Amy and Mark have split the shares 50/50 and I am not listed anywhere as a shareholder as I was told.” (LAY4)
This analysis, as well as the analysis for selecting parties, suggests laypeople are uncertain around issues of corporate identity and liability. There is a significant difference in the understanding between laypeople and lawyers about how corporations operate, the role of shareholders, and remedies against a company. This suggests that a large number of claimants (in this study, a quarter of the laypeople) would need some support and explanation to be able to claim a remedy of this nature against a company (see also Moorhead and Sefton 2005, 155).
Overall Quality of Claims – Lay versus Lawyer
We asked the judicial reviewers to grade a subset of claims on a five-point Likert scale from “Poor” (1) to “Excellent” (5). There were twelve claims in the subset, five filed by lawyers and seven by laypeople, but we did not tell the reviewers that. The literature tells us that judges are likely to be biased against laypeople’s claims, so it was important they were blinded (Quintanilla, Allen, and Hirt Reference Quintanilla, Allen and Hirt2017).
We averaged the scores given for each claim by the four different judges to arrive at an overall score for each claim. The average scores for each claim assessed and the judges’ assessment as to whether it was prepared by a layperson or lawyer are presented in Table 5. The claims actually completed by lawyers gained a higher average score (3.55/5; range of averaged scores 3–4) than those actually completed by laypeople (3.05/5; range of averaged scores 2.1–3.75). This again supports the finding that there is an overlap at the middle where laypeople’s claims are of similar quality to lawyers. In general, however, lawyers have somewhat higher overall performance and laypeople have somewhat lower overall performance.
Results of the Judges’ Assessments
The review by the judges, however, also highlighted a point which often confounds general discussions about the relative skill of laypeople compared to lawyers: there are some laypeople who can claim very proficiently and there are some lawyers, despite being qualified and experienced, who are inept. We asked the judges to say whether they thought each claim was filed by a lawyer or a layperson. Notably, the layperson with the highest score (3.75) was misidentified by all of the judges as being a lawyer. The judges commented that this claim was “free of irrelevancy,” “direct sufficient and to the point,” and “succinct pleading which includes all the elements necessary to establish legal liability.” The lawyer with the lowest score (3) was misidentified by all but one judge as being a layperson. The judge who did identify the person as a lawyer referred to the claimant’s use of legal terminology such as “respondent” and “writ” to make the identification, not any qualities in the pleading. The judges were all critical of the pleading calling it “full of unnecessary and unhelpful details of no probative value,” “too long,” “full of irrelevant material,” and using “unnecessarily emotive language.” One of the judges noted “I hope it was not a lawyer.” It was, in fact, a lawyer who had more than ten years’ experience.
While laypeople’s work is often critiqued as defective, this research provides evidence that while most lawyers perform their work well, and some exceptionally well, there is a tail-end who perform poorly.
General Discussion
Competent legal storytelling requires mastery of an entirely different register from lay storytelling. Lawyers are trained to tell legal stories, but as the costs of participating in litigation have continued to increase, governments and the judiciary have sought more cost-effective alternatives, such as user-friendly online portals, where self-represented litigants can prepare and file their own dispute. This project used a laboratory experiment to test the assumption that laypeople can select out the law-relevant information and input it into an online portal.
Although a small number of laypeople perform the task very well, most were not as proficient at legal storytelling as lawyers, even with the support of a user-friendly portal. Laypeople omitted more legally salient details than lawyers. Some laypeople filed claims that were too brief to provide sufficient detail. Yet greater word count did not necessarily correlate to an increase in relevant detail. The concept of corporate liability also caused confusion with a group of the laypeople.
Where laypeople did provide the legally relevant information, it was not always clearly articulated. The way some claims were explained was confusing and gave rise to other questions that would need to be answered. Where these defects occur, it falls to someone else to ask the right follow-up questions or to draw inferences from the information provided. Sometimes the information was clear on its face, but was not strategically deployed. This meant that litigants lost an opportunity to strongly advocate their position.
Possible Solutions
For those who use online court system to have access to substantive justice, most will need help to transform their problem into a legal narrative that can be adjudicated. How best to provide that assistance? We consider four possibilities: more lawyers, more inquisitorial judges, help from an intermediary, and improving the portal interface.
More Lawyers and More Inquisitorial Judges
The major problem with the solution of using more lawyers is the expense. Legal fees are an enormous barrier to access the courts and self-help solutions are specifically designed to avoid this cost. Similarly, governments who see online solutions for their cost saving benefits are unlikely to be interested in paying or more judges, one of the most expensive parts of a court system. Judges performing the role of helping litigants shape their dispute narrative is also challenging in countries with an adversarial model (Moorhead Reference Moorhead2007; Zuckerman Reference Zuckerman2014).
Apart from expense, our research suggests lawyers may not always be the best solution. We found evidence that some lawyers perform well below the level expected of a trained professional; in fact, some lawyers were outperformed by laypeople. Furthermore, the research also provides support for the idea that some lawyers’ approach to dispute resolution can complicate cases. Lawyers claiming against the Company or suggesting a constructive trust argument, showed a willingness to engage in a complex argument for a case where the maximum recovery was approximately USD $17,500. Simply providing lawyers is not necessarily the answer to assisting access to justice. It depends on the quality of the lawyer and on their approach to claiming.
Help from an Intermediary
Online forms tend to provide structure to elicit details about the parties, but they offer only vague encouragement along the lines of “describe the dispute” to elicit a narrative of the dispute. We found not all laypeople could perform this task, with participants providing incomplete and unclear information. An intermediary who could elicit further information and clarify unclear statements could be provided to read the dispute description and interview the litigant (see Zorza Reference Zorza2002 who made suggestions along similar lines although perhaps overestimated the ability of people to narrate a dispute). This would be, however, very resource intensive and it would require a new class of legal actor who would need to have their own set of ethics and skills. While it might ultimately save both on the provision of legal services and on the need for judicial intervention, governments will be reluctant to invest in a class of intermediaries to provide the assistance laypeople need to shape a lay narrative into a legal one. We think it is much more likely that the provision of user-friendly portals will be seen as providing access to justice, where access to justice is synonymous with direct access without the expense of a lawyer. Attention, therefore, needs to be directed at improving the interface so that it not only creates a satisfying user-experience, but it also elicits law-relevant facts.
Improving the Technology
One way to achieve this could be through greater use of “nudges,” which Cass Sunstein defines as “liberty-preserving approaches that steer people in particular directions, but that also allow them to go their own way” (2014). Within a court portal, it is possible to intentionally (as well as unintentionally) build in nudges to encourage claimants to provide the type of information a court needs. The interface could, for example, encourage the claimant to not be overly brief by saying “claimants usually write at least two hundred words in this section.” This might encourage those claimants who were excessively brief to write more and therefore include more legally salient facts.
The architecture of the portal could also provide more structure to elicit legally salient information. Since chronology is important in legal reasoning, providing the functionality to arrange facts that have been inputted by date order could help a claimant restructure their free narrative into a more law-friendly format. This would then allow the claimant, opposing party, and adjudicator, to draw inferences from the ordering.
Improving the technology also means using methods that go beyond the user-needs (the concern of design-thinking and user-testing methods). Consideration of how the portal supports the inputting of accurately stated, law-relevant facts is also important so that we can understand if the technology supports this aspect of access to justice. Future testing of online courts needs to use a variety of methods to ensure that there is robust evidence that it can genuinely achieve access to justice.
Limitations and Future Directions
A number of factors could limit the validity of our findings in the civil justice setting. For example, it was not possible to create a laboratory scenario that would allow the participants to have a first-person experience of the dispute. Instead, the participants were required to play the role of the “lay assistant.” There is a risk that we have not captured the way laypeople would explain a first-person narrative; for example, we might have underestimated the use of prolix or discursive writing techniques, because the lay participants were not emotionally connected with the dispute. Further, people who volunteered for a study advertised as an “online court” were likely to be people who are comfortable using a computer and communicating in English. In reality, people will have additional barriers to being able to file a case online including access to technology and English literacy.
It is possible that we have overestimated inclusion of relevant information in the claim, because participants presumed they needed to include as much of the provided information as possible primarily because it had been given to them; not because they deemed it relevant. This finding has been observed in other experimental studies, which has shown that participants include irrelevant information solely because they assume it has been given with good cause (Hornikx and Hahn Reference Hornikx and Ulrike2012). This effect is likely to be more profound with the laypeople who are less likely to be familiar with using fact patterns as a basis from which to construct and communicate an argument, as opposed to lawyers who were trained and assessed at law school in this way. It is reasonable to assume that the differences between laypeople and lawyers will be even more pronounced in real world disputes.
We expect that reasons will be sought to disbelieve the findings that some of the lawyers filed poor claims. We were careful to include only experienced lawyers so that the results could not be explained by differences in experience. It is possible that lawyers took less care than they would with a real claim because they knew they were in a laboratory setting. We did, however, tell people that the purpose of the experiment was to compare lay and legal claiming.
The care we took in ensuring only experienced lawyers took part compounded the difficulty of recruiting lawyers (many offered up their juniors, an offer we refused) and this limited our sample size. We hope other authors will build on the method we have developed here and that with larger sample sizes, more detailed analysis of a number of variables can be carried out. This could include developing further aspects to the experiment such as asking laypeople and lawyers for their reasons about what they included or omitted and testing problems with different levels of complexity. It could also include testing whether or not giving people general or situation specific legal information about their problem (such as the “Solutions Explorer” provides in the CRT) improves the quality of the claim that is subsequently entered.
Notwithstanding the necessary caveats inherent to laboratory-based research on real-world issues, the findings raise a number of lessons for the design of online court forms, and for thinking about laypeople’s engagement with the justice system.
Conclusion
This study used a novel method of court research to explore the extent to which a user-friendly portal supports litigants to input law relevant facts in a dispute. We suggest that using this method could compliment research on online courts. In particular, it is a means of looking beyond subjective user experiences to whether a portal is enabling access to an effective hearing and to a decision in accordance with substantive law. As the global pandemic accelerates the move to online courts, we need to continue to develop robust methods to ensure that this alluring form of court participation also meets the core principles of access to justice.
