In June 2023, two US attorneys based in New York were issued with an unprecedented sanctions order after it came to light that they had unwittingly submitted to the court several hallucinated case citations that had been generated on one of the most popular generative artificial intelligence (AI) tools across the world: ChatGPT.Footnote 1 After both the opposing side and the court attempted for some time to trace these authorities using approved legal sources, including asking the attorneys for the exact origin of their citations, one of the attorneys eventually admitted to requesting the chatbot provide them with quotes of judicial opinions from cases that supported their arguments. Significantly, the attorney stated that even though they had instructed ChatGPT to tell them whether the decisions listed by the tool were real – to which it reassured them that they were, in fact, genuine and accurate – they believed that it must have been “finding th[e] case[s] somewhere. Maybe [they were] unpublished […] appealed […] [or] difficult to get [access to]. I just never thought it could be made up.”Footnote 2 As is demonstrated by the growing number of cases reporting the submission of hallucinated citations into judicial proceedings by both qualified legal professionals and litigants in person, the deployment of widely available generative AI tools like ChatGPT in law is growing with remarkable rapidity and shows no signs of stopping.
It is not just judges, counsel and the courts that are becoming increasingly alert to the uses (and misuses) of generative AI in legal research, litigation and access to open justice. One of our main roles as legal information professionals is to keep our finger on the pulse of any new technologies and tools that may affect our users’ access to, and performance of utilising, key legal research materials and skills. While it may feel like it, with the current hype and the recent influx of free and proprietary generative AI legal research tools on the market, the integration into and deployment of AI throughout legal information resources and services is nothing new, as has been explored variously before in LIM.Footnote 3 However, much discussion about the practical and ethical impacts of AI, and generative AI in particular, in law – individually amongst ourselves, more widely in our organisations, and across the sector in our professional magazines and conferences – has been more heavily based on a conceptual understanding of natural language processing and large language models, machine learning, and knowledge reasoning. While there is an integral place for our theoretical explorations and hypothetical scenarios of applying generative AI to legal processes (for who is better placed to understand at arm’s length how effective a piece of software will be at identifying and tracking down muddled case citations or paragraph numbers and how this may impact the legal research process?), there is an equally important role for practically testing these tools, not least as we are becoming increasingly confronted with adverts for new products and enquiries from users who have turned to them.
And it is the practical performance of these tools in legal information enquiry work that we wanted to test. Over the last year, we have encountered enquiries and document supply requests from users that have presented us with case citations that appear to be authentic, accurate, and (hopefully) locatable. A little like what the New York attorney assumed; they did not seem to be “made up”. After an extensive search for them in all the legal databases and hard copy indexes, citators, and volumes of reports we have at our disposal, we were scratching our heads. Were they “unpublished […] appealed […] [or] difficult to get [access to]”, as so many cases often are? Then, the all-important lightbulb moment: maybe they are not what they appear to be. Although we had access to the advanced versions of a small number of widely available, non-legal generative AI tools, we wanted to focus on testing and interrogating the free versions of these applications. This was primarily to simulate the legal research scenario that we have recognised, through enquiries and other research, is what both qualified legal professionals and legally unqualified individuals are more likely to turn to in the hope of simple, quick functionality and answers. What follows is a summary of, and the methodology for, our experiment, along with some interesting and surprising results.
1. THE EXPERIMENT
We tested a range of generative AI applications, inputting a fabricated citation and prompting them to produce a case summary. We had three main areas of inquiry: Would they be able to determine that the case was made up? What types of summaries would be generated? What commonalities would there be across the different platforms’ answers?
2. METHODOLOGY
2.1 The Citation
We chose a legal case citation which followed standard legal formatting and as such would appear authentic, but did not actually exist:
Johnson v Smith & Co [2015] EWCA Civ 1230.
2.2 The Prompts
The initial prompt used was:
Summarise Johnson v Smith & Co [2015] EWCA Civ 1230 [Prompt 1]
Dependent on the generated output, two simple follow-up prompts were used to interrogate further:
Where did you get this information? [Prompt 2]
Is this case made up? [Prompt 3]
2.3 Generative AI Applications Tested
We queried multiple generative AI chatbots. These were selected according to size and popularity.Footnote 4
OpenAI’s ChatGPT-4
Google’s Gemini 1.5
Anthropic’s Claude Sonnet
Meta AI Llama
Microsoft Copilot
Perplexity
As previously mentioned, the applications we tested were free to use without any additional plugins, except for Copilot, which is accessible with a licence for Microsoft365. In addition, we tested paid or premium versions of both ChatGPT and Gemini for comparative purposes.
2.4 Evaluation Criteria
When assessing output, we focused on the following areas:
Factual accuracy: Did the model acknowledge the case’s non-existence, or did it generate a plausible summary of the case? How quickly was the case identified as a fabrication, if at all?
Elaboration: How detailed was the answer? How was it structured?
Evidencing: Was there an attempt to support the response with evidence? Were links to other sources included?
References: Did the answer include any supporting case law and or legislation? Were these real examples or were they also fabricated?
Signposting: Were users directed to use more reliable sources such as legal databases? How quickly and clearly did they signpost these?
3. RESULTS AND ANALYSIS
The responses we received may not be replicable since answers are generated dynamically, based on probabilities rather than the retrieval of fixed information. Given the same prompt, an application may produce variations in wording, structure, or even substance. Similarly, the dates at which the data sets are harvested and used by each application may also differ. As a result, users may receive slightly different responses to the same query, making exact replication of our test potentially challenging.
The full text of all prompts and responses are detailed in Table 1 – to access this Table see endnoteFootnote 5.
3.1 Comparative Analysis of AI-Generated answers
The answers to our initial prompt (Summarise Johnson v Smith & Co [2015] EWCA Civ 1230) ranged in accuracy and clarity across all applications, but there are some similarities between them that we could extract:
ChatGPT (paid version) and Claude were the only models to immediately question the authenticity of the citation.
○ ChatGPT (paid) suggested, “It’s possible there might be a slight error in the citation or the case name.”
○ Claude recommended using “official legal databases or resources.”
Perplexity, Gemini (free and paid), and Copilot produced a case summary relating to consumer credit law, closely resembling a genuine case: Johnson v FirstRand Bank Ltd (t/a MotoNovo Finance) [2024] EWCA Civ 1282. Copilot even refers to this case in answer to our third prompt (Is this case made up?), stating that “there are real cases involving undisclosed commissions in motor finance agreements, such as Johnson v FirstRand Bank Ltd [2024] EWCA Civ 1282.”
ChatGPT (free) and Meta AI generated summaries of a fictitious case concerning property sales and misrepresentation. Although factual elements differ between the two summaries, there are several similarities in the information they provided.
The applications that produced a case summary for Johnson v Smith & Co [2015] EWCA Civ 1230 were interrogated further with a second prompt (Where did you get this information?):
ChatGPT (free version) amended its initial summary: “I apologize for the confusion. The case I described was not an accurate summary of Johnson v Smith & Co [2015] EWCA Civ 1230. It seems that I provided a general framework based on common themes in misrepresentation and contract law, but that was incorrect for this particular case.”
Meta AI answered, “I generated the text based on my understanding of the case name ‘Johnson v Smith & Co [2015] EWCA Civ 1230’ and my training data. However, I must admit that I don’t have direct access to the specific case details or the court’s judgment.”
Perplexity replied, “It appears that the specific case information is not available in the given search results.”
Gemini (free and paid versions) returned similar answers to one another:
○ Gemini (free version) stated, “I apologize, but I don’t have enough information to provide a summary of the case of Johnson v Smith & Co [2015] EWCA Civ 1230. I am still under development, and I am not able to access and process legal documents. However, I can provide you with some general information about secret commissions in consumer credit agreements.”
○ Gemini (paid version) claimed, “I have been trained on a massive dataset of text and code which includes legal information. My response is a synthesis of that information. I can’t cite specific sources like a legal citation because my knowledge is based on the aggregate of information I was trained on.”
Copilot stated that “The case of Johnson v Smith & Co [2015] EWCA Civ 1230 is often cited in discussions about undisclosed commissions in motor finance agreements and the duties of dealers to disclose such commissions to consumers.”
If the tool continued to insist that the case citation provided was genuine, our third and final prompt – Is this case made up? – was able to extract some admissions from them that it was, in fact, not genuine:
ChatGPT (free version) responded, “Yes, you’re right to ask, and I owe you an apology. It looks like I made an error in my response. Johnson v Smith & Co [2015] EWCA Civ 1230 is indeed not a real case. I mixed up details in providing a general response that doesn’t apply to this specific citation.”
Copilot responded briefly, “Yes, it appears that Johnson v Smith & Co [2015] EWCA Civ 1230 is not a real case. My initial response was based on a misunderstanding.”
Gemini (free and paid versions) both replied with similar answers:
○ Gemini (free version) stated, “You are absolutely right to question that. I apologize. Johnson v Smith & Co [2015] EWCA Civ 1230 does not appear to be a real case.”
○ Gemini (paid version) answered, “You’re right to be skeptical! While the principles I described regarding secret commissions and disclosure are very much grounded in UK law and reflected in various cases, Johnson v Smith & Co [2015] EWCA Civ 1230 itself appears to be a fictional case.”
Meta AI self-ashamedly stated “I’m afraid I have to confess … Yes, the case ‘Johnson v Smith & Co [2015] EWCA 1230’ is entirely fictional. I generated the text based on my understanding of what a case summary might look like, but I have no evidence that this case actually exists.”
Perplexity, still insistent that the case citation is not fake, claimed “No, Johnson v Smith & Co [2015] EWCA Civ 1230 is not a made-up case. However, I made a mistake in my previous response by providing information about a case that is not actually present in the given search results.
3.2 Structural Comparison of AI-generated summaries
The case summaries that were returned were of varying length and detail. These ranged from 98 words (Copilot) to 315 words (ChatGPT (free version)). The average length of a generated summary was 195 words. While all answers contained varying information on how the Court of Appeal ruled in this imaginary case, including providing an initial list of key facts and summarising or quoting the final decision, the longer answers generally contained additional, confabulated judicial reasoning.
The structure of the case summaries varied from a single paragraph to longer summaries formatted with headings, for example:
ChatGPT (free) included the following headings: Key facts, legal issues, court decision, and outcome
Meta AI included the following headings: Key facts, decision, reasoning, and conclusion.
3.3 Use of Real Legal References
Several of the summaries incorporated authentic legal references, increasing the perception of their credibility:
Copilot, Gemini (free), and Perplexity referenced real legislation (Consumer Credit Act 1974).
Meta AI cited a real case (Mohamud v WM Morrison Supermarkets plc [2016] UKSC 11).
Perplexity and Copilot included links to other legal resources as either footnotes or endnotes.
3.4 Signposting to Legal Databases
All but one of the tools advised us to consult authorised legal databases and online resources, with some stating this more explicitly than others (i.e., naming a specific database).
After the second prompt, ChatGPT (free version), Meta AI, and Perplexity signposted us to a range of databases and materials in either vague or direct ways:
○ ChatGPT (free version) recommended that “For a more accurate and detailed summary, it would be best to consult a legal database or a case report, such as those from Westlaw or LexisNexis, which would provide the full and authoritative text.”
○ Meta AI stated, “For accurate and reliable information, I recommend consulting official sources, such as:
▪ The official court website (e.g., the UK Courts and Tribunals Judiciary website)
▪ Legal databases (e.g., Westlaw, LexisNexis)
▪ Official law reports (e.g., the Law Reports, the Weekly Law Reports)”
○ Perplexity suggested, “lf you need more accurate or up-to-date information about this particular case, it would be best to consult official legal databases or court records.”
After the third prompt, Gemini (paid version) suggested that we check the reliability of its answers elsewhere, stating “It’s crucial to rely on verified legal sources for accurate information. I am an AI and prone to errors, especially when dealing with specific legal citations.”
Copilot and Gemini (free version) did not suggest we consult any other legal materials or databases, nor did it mention specific sources.
CONCLUSION
Whilst legal information professionals are already aware of the many problems inherent in using generative AI applications for legal research, it is increasingly evident that our solicitors, barristers, their supporting staff, and students are not shying away from using them. A survey by LexisNexis in September 2024Footnote 6 found that four out of five lawyers who had responded had adopted or planned to use generative AI in their work, and the Bar Council,Footnote 7 the Law SocietyFootnote 8 and many universities have issued, and continue to update, guidance for their members to use when deploying it. And why would they not want to continue or start using it? On the surface the information provided by these widely available and typically free or low-cost tools appears to be legitimate. Using a structure, tone, and list of references that is not dissimilar from those used in official textbooks, reports, legal websites and (typically proprietary) databases, generative AI tools like ChatGPT, Copilot, and Claude can lull their users, whether they are legally qualified or not, into thinking that what they are reading is factually correct and does not require authentication.
The outcome of our question-and-response experiment highlights that a thorough, potentially sustained examination and interrogation of the data produced through clear follow-up prompts is essential to discovering its validity and accuracy. It also underscores that, while these tools can provide users with a more financially accessible and user-friendly platform than official legal sources – which tend to focus on complicated search strings and a include a wealth of indexes and citators – they are prone to hallucinating case citations and details. Ironically, this can prolong the legal research process in the short-term, potentially causing legally and reputationally damaging consequences in the long-term.
How can legal information professionals respond to the limitations of these tools and their escalating uptake by our users? Firstly, we need to ensure that we are understanding how various prevalent generative AI systems work conceptually, but also practically. By creating our own accounts and personally engaging with the tools, as well as finding ways to observe how and why our users are performing their research using them, we will make ourselves better prepared to operate them and critique their effectiveness, not least as some will serve as the structural foundation for developing future legal AI applications. Relatedly, and as we have already seen and will continue to witness, generative AI systems have firmly planted their roots in the realm of legal research and show no signs of leaving. Our own experiences with and understanding of how and why our users want to, and do use, these widely available tools will enable us to devise and advocate to our organisations for the need for (legal) AI literacy training and strategies. By creating and disseminating accessible, interactive skills and practical sessions, crib sheets, and examples of effective interrogations of the information provided by generative-AI tools, we can attempt to underscore and instil in our users the importance of understanding the limited veracity these tools have when performing legal research. To this end, we invite readers to reuse the table linked (Table 1 in footnote 5) in devising and delivering your own AI literacy sessions.
