This blog has repeatedly returned to artificial intelligence. A sound theme that is highly recommended is Trust but Verify (August 2025). A list of AI posts is here. Artificial intelligence tools, particularly large language models (LLM) like Chat, Claude, and Gemini, are prone to hallucination. The causes behind that are reasonably simple.
Most resonantly, I have heard that LLMs are like golden retrievers. They want to please the user. Sent to the woods for a particular ball to retrieve, they will strive to achieve that. If they cannot find that toy, they will find something they view as comparable, like a stick, and bring it instead. They see value in the retrieving as much as in the ball itself.
The potential for hallucination is real, pernicious, and persistent. It is increasingly incredible that any lawyer or judge could claim to not know that LLMs are not search engines, and that they do hallucinate. Thus, the Trust but Verify post concludes:
The only hope we have for minimizing their errors is in our human, patient, and careful verification. This means reviewing what is generated by staff, by lawyers, by judges, and by AI. Review and verify.
That admonition could be critical to the legal profession, but more important still to others. A recent news story on Daily Dot documents the allegations of one patient whose Tick Tock video has amassed tens of thousands of hits. She complains that her physician ordered an EKG test and then relied on an AI interpretation of the results.
The patient says that she was told "the AI ... decided she’d had a heart attack," and she was referred to a specialist. It took about a month for her to be seen by a cardiologist, and she says “I passively, for one month, thought I was going to die.”
The good news came from the cardiologist who concluded she did not have a heart attack. She says that "the specialist explained that her primary care doctor had signed off on the AI’s reading without even looking at her chart." The original physician, allegedly, trusted without bothering to verify.
The harm in the alleged situation is readily seen. The patient was given an unnecessary scare and suffered for weeks under the impression and fear of an incorrect diagnosis. But, turn the table. What if the AI had made no diagnosis, went unverified by the physician, and some important risk or danger had been overlooked?
In that instance, the referral to a cardiologist would perhaps not occur. The primary care physician who is too lazy to check the test results, to verify, might send a patient home without a critical diagnosis and specialist referral. This could leave a condition untreated and might result in unfortunate or even fatal outcomes.
Is the physician so different from the lawyers and judges? The LLMs are a tool. All AI programs are tools. They are subject to misuse and mistakes. The critical element in delivering professional services is the human element. If the human fails, the results are in peril. If untoward outcomes result, it is the human's fault, not the computer program's.
Everyone using AI needs to remember the admonition of Bob Wilson memorialized in an earlier post, "A fool with a tool is still a fool." The user can leverage tools. A chainsaw will make you faster and more efficient at dropping trees. However, if you are not careful with a chainsaw, you could be badly hurt.
