I have been critical of lawyers and judges who cite hallucinations in their writing. I used to say "hallucinated cases." Attorney Middlemier* admonished me lately for referring to "hallucinated cases" and "hallucinated authority" from time to time. The point is somewhat obvious: such references are neither "cases" nor "authority," they are merely hallucinations. Let's all get that part straight going forward. Some might argue they are "fraud," but that is a bit judgmental.
Attorney Middlemier also criticized me for my statements that every lawyer now knows better and that avoidance of hallucinations is easy. Middlemier's contention here is that no one has ever provided simple, easy-to-follow advice on avoiding hallucinations. I stand contrite and throw myself on the mercy of the reader. Apologies (see below for a more thorough apology).
Lawyers, judges, "lend me your ears." I come to save your name and reputation, not to bury you. (adapted from Julius Caesar, Billy Shakespeare, 1599).
I strive today, instead, to provide a working suggestion for the legal professional on avoiding the citation of hallucinations in legal writing. It is offered with the best of intentions and hopes. The reader will perhaps forgive some reference to ancient pop culture, but there is sound advice on this point from A Fish Called Wanda (1988 Prominent Features), more on that follows.
How to Avoid Hallucination in Legal Writing:
Step One - do not use artificial intelligence (AI) large language models (LLM) such as ChatGPT, ClaudeAI, Gemini, and similar for legal research or writing. These are not search engines that scrub the World Wide Web for data, information, and knowledge. They are assimilators that strive to assemble information in a way that is pleasing. Some compare them to golden retrievers who wish to please you. If they cannot find the stick you seek, they will bring you something else instead. Unlike the gentle and kind golden, they will lie to you about the something else and try to convince you it is indeed a stick.
LLMs are not built to do research. And, they are offered free of charge. Hint to the masses: if you are not paying for a service, you are not the customer; you are the product. The LLM is trading your information for its own. It is learning from you as you try to learn from it. Caution to lawyers: anything you put into an LLM, particularly a free one, becomes usable by that LLM. Your client's confidences and privacy may be at risk from such use. You could be liable for that in legal settings or Bar disciplinary proceedings.
If you use an LLM and it produces a polished, convincing argument with multiple citations, and the result is the absolute best you could hope for in terms of simplicity and completeness ... this is a red flag. Be wary of golden retrievers bearing gifts.
Step Two - (for when step one does not work and you use an LLM instead of doing lawyer/judge work), verify the results. As Wanda (Jaime Lee Curtis) did in the epic film in her name, verify:
"Now let me correct you on a couple of things, OK? Aristotle was not Belgian. The central message of Buddhism is not 'Every man for himself.' And the London Underground is not a political movement. Those are all mistakes, Otto. I looked them up."
The point here is that Otto (Kevin Klein) hallucinated (made things up). Wanda is not arguing, implicating, or officiating. She is simply doing the legwork. She found the fallacies in a straightforward and simple manner. She "looked them up." Ingenious! (She is the protagonist after all). "Look them up," "look them up," "look them up." Step Two is simply "look them up."
How does one effectively look them up? Know that even paid services like Westlaw and Lexis AI can and do hallucinate. See the Stanford Study, May 2024.
So, for statutes, rules, or regulations, it is easiest to use a search engine (Google, Edge, Safari, etc., also called a "browser") and search for the statute and rule citations you have selected. This should lead you to government websites that publish statutes and rules. Strive in Florida to find results on the sites of the Legislature (https://www.leg.state.fl.us/) or the Senate (https://www.flsenate.gov/). If a statute exists, your search by number (e.g., for 440.25) should yield a result on both of these sites. If you do not find it on these sites, that is a red flag.
Use a trusted subscription service for your case law citations. In Lexis, Westlaw, Vlex, or a similar database, search for the case by name (use the non-AI tool on those services; see Stanford Study above). If that search yields a positive result, verify your citation matches the one in the database. If it does not yield a positive response, then search the citation you have.
Negative results from either such attempt should be a red flag (you should likely delete that citation as untrustworthy and return to the drawing board). If you unequivocally love the case you are searching to authenticate, perhaps keep striving. But it is highly recommended that you abandon the case or authority if you cannot find it on these subscription services.
Having failed to verify this way, you could turn next to the internet. If you turn to the internet, use the same search engines (browsers) to search by case name and citation. It is possible you may yet find an actual copy of an opinion. However, there will remain doubt due to its absence from the commercial databases (Lexis, Vlex, Westlaw). That does not mean the authority you are striving to verify cannot be real, but you should be harboring suspicions and reservations by that point in your process.
For example, if you have cited Gusmorino v. T.G.I. Friday’s, 928 So. 2d 446 (Fla. 1st DCA 2006) or Specialty Emp. Leasing v. Davis, 596 So. 2d 695, 696 (Fla. 1st DCA 1992), paste those names into a browser and evaluate the results. Be careful - some browsers provide an AI LLM response; e.g., Chrome may provide a Gemini analysis that is as much hallucination as the original LLM provided creation.
You may see search results for your case in databases like the University of Florida Repository (https://scholarship.law.ufl.edu), the Florida State University collection (https://ir.law.fsu.edu), University of Miami (https://repository.law.miami.edu), The Myfloridacfo site, and other public sites like Vlex, Justia, Digital Commons, Findlaw, and more. If your case is not generating such confirmatory results, this should be a red flag. I implore you at this point to abandon that LLM-generated authority, no matter how much you may love it.
Step Three - this one is seemingly the toughest part. Having verified the existence of the authority, now you must actually read it. If you find that Fla. Stat. §440.38(7) and Gusmorino exist (it doesn't), that does not conclude the analysis. As a lawyer, you are seeking relief, making an argument. You are asserting, when you cite authority, that the authority stands for something in particular. You cannot know that unless you read it.
Not to put too fine a point on things, the lawyer or judge receiving a pleading, or hearing an argument, with such citations should also engage in both Step Two and Step Three. Trust, but verify is the critical point.
This is a good moment for the lawyer to pause and review Rules Regulating The Florida Bar, Rule 4-3.3. This is about honesty-in-fact. Lawyers are supposed to tell the truth when speaking or writing to a tribunal. When you cite or argue hallucination, that is not the truth; it is fantasy. With all you have invested in becoming a lawyer, is that job, reputation, license, and more worth keeping? The lawyer may certainly decide, but "choose wisely" (Indiana Jones and the Last Crusade, 1989).
As you read the authority, the analysis is whether it says what you think it says. For a comedic example, refer to Inigo Montoya's reference to "inconceivable." Princess Bride (1987)("you keep using that word. I do not think it means what you think it means"). You have to read the case, statute, or rule. You have to decide what it means. You have to articulate its meaning, holding, relevance, and applicability.
You cannot merely rely blindly on what your clerk, paralegal, partner, associate, priest, doctor, or spouse says. Yes, those are all trustworthy people. Yes, if they tell you the best pie in town is at Pizza Planet, try one (you won't lose your license for trying a pizza). No, Virginia, lawyers cannot rely blindly on what those people, or any people, say about legal authority. Lawyers cannot blatantly make up authority or facts (that is lying) and cannot rely on what others make up either.
Lawyers have to do the work, read the authority, make judgments, and arrive at their own conclusions. It is singularly the role and responsibility of the lawyer signing the document or uttering the argument. If a computer could do it, then we would have no need for lawyers, law schools, professionalism, and so much more. But no computer can do it, so you must.
Hint: In case you missed it, lawyers have to analyze and make judgments. They cannot blindly rely on their most trusted human compatriots. Similarly, you cannot blindly rely on a computer program (AI LLM). When the lawyer signs their name, the tribunal will rely. We count on you to be honest, accurate, and forthcoming. Reread Rule 4-3.3.
To simplify and reiterate:
- Do not use LLMs for legal research and drafting (use them to check grammar, structure, spelling, etc.).
- If you do use them to research and draft, verify every authority with a subscription, search engine, or both (do this if your associate, paralegal, or partner is used to research and draft).
- Read every authority you cite, regardless of whether you found it, your paralegal found it, or some LLM found it.
- Remember that it is your name, your reputation, and your license on the line if you are less than complete and honest in your statements.
Epilogue
An apology was mentioned briefly above. Perhaps a longer apology like that offered by John Cleese (playing a lawyer) in A Fish Called Wanda (1988 Prominent Features)(perhaps one of the most outlandish, ridiculous, and humorous films of all time):
“I offer a complete and utter retraction. The imputation was totally without basis in fact, and was in no way fair comment, and was motivated purely by malice, and I deeply regret any distress that my comments may have caused you, or your family, and I hereby undertake not to repeat any such slander at any time in the future.”
As a matter of full disclosure, I have not used any LLM in the production of any of my blog posts. All of the authorities and references cited herein have been personally reviewed by the author. Do we need or want some rule requiring a disclosure like that on every filing? Likely, Rule 4-3.3 is sufficient.
*Horace Middlemier is not a real person. He is a figment of the author's imagination, a foil, an "everyman," used purely as an illustration and to protect the identity of many who provide fodder for these pages. Any similarity to any real person, living or dead, is pure coincidence and not intended.
