All over the country, lawyers are using artificial intelligence to write briefs and help them prepare for court. It is not going well.
A family in Alabama lost a trust dispute last month because their lawyer filed citations to cases that do not exist. The Alabama Supreme Court dismissed their appeal, calling the conduct egregious, and barred the lawyer from filing in that court again without co-counsel sign-off.
In the same month, a federal judge in Oregon sanctioned two lawyers $110,000, the largest AI hallucination penalty in American legal history, after they submitted 23 fabricated citations and eight invented quotations. The case was subsequently dismissed.
In Manhattan, a judge ruled recently that a defendant who used a general-purpose AI chatbot to help prepare his case had waived attorney-client privilege. If you type your defense strategy into a chatbot, the government can subpoena it, read it, and use it against you.
According to a database compiled by lawyer and data scientist Damien Charlotin, there have now been more than 1,300 cases globally where a court or tribunal has commented on AI-generated hallucinations in legal filings. Behind each of those cases is a client who paid a lawyer and trusted the system. Behind each, more often than not, sits a lawyer who placed blind trust in a technology that generates text with complete confidence and no capacity for self-verification.
Not all AI is created equal. There is a real difference between general-purpose large language models like ChatGPT and Claude that have been trained on the open web, and industry-specific legal AI tools that are plugged into the same databases lawyers have been using for decades. Unfortunately, Wall Street has struggled to tell the difference.
When Anthropic released a legal plug-in for Claude recently, it contributed to a roughly $285 billion selloff in technology stocks. The chaos in courtrooms around the world tells a different story. Solving legal AI is harder than tweaking a standard large language model.
I have practiced law across three jurisdictions and now serve as General Counsel of one of the world’s largest legal technology companies, LexisNexis.
The question I am asked most often right now is, “Which AI is most capable?” My view is that this is the wrong question. The right one, “Which AI can be trusted in a courtroom?” is a different question. In law, those are not the same thing.
The obligation runs to the client and to the court simultaneously. The American Bar Association has identified how five of its Model Rules of Professional Conduct are directly impacted by AI use. They are competence, confidentiality, candor toward the tribunal, and supervisory responsibility.
When a lawyer submits a hallucinated citation, they fail their client. It is also a failure to the court. In fact, it corrupts the record the entire system depends on.
The structural problem runs deeper than which model a lawyer uses. General-purpose AI is designed to produce text that looks like the right answer, which in most domains is most of the job. In law it is the wrong job. The model cannot verify that the case it cited exists, that the case says what the brief claims, or that the case remains controlling authority. The gap is architectural, not a capability problem to be solved by the next training run. The consequences are concrete. Lawyers get sanctioned, claims get dismissed, defendants get handed to the prosecution.
The question to ask of any legal AI tool is not how it performs on a benchmark, but what it is built on, and whether a lawyer can trace, verify, and stand behind the output in open court.
There are two ways AI changes the practice of law. The first is compression, the same work, faster. The second is expansion, work that was never possible before. AI’s expansion potential in law is enormous, but it can only rest on a foundation that does not fabricate the underlying law. Litigation strategy built on decades of judge-specific outcome data is not a faster version of an existing task. It is work that did not previously exist. The same is true of regulatory monitoring built into deal documents that update the moment the law changes. There are many other examples, and the list grows weekly.
The market will eventually price what the profession has always known. In law, the cost of a wrong answer is paid in someone’s freedom, their assets, or their family’s future.
The opinions expressed in Fortune.com commentary pieces are solely the views of their authors and do not necessarily reflect the opinions and beliefs of Fortune.
In recent months, the integration of artificial intelligence (AI) in the legal field has encountered significant challenges, raising serious concerns about the reliability and integrity of AI-generated legal documents. Instances of malpractice due to AI errors have emerged, most notably in Alabama and Oregon, where lawyers faced severe repercussions for submitting fabricated case citations and quotations. These failures highlight a broader issue: the legal profession’s reliance on AI that lacks the capability for self-verification and accuracy.
One striking example involved a family in Alabama whose lawyer’s submission of nonexistent case citations led to the dismissal of their trust dispute appeal by the Alabama Supreme Court. The court deemed the lawyer’s conduct « egregious, » resulting in a ban on the lawyer from filing in that court without co-counsel approval. In another instance in Oregon, a federal judge levied a $110,000 sanction on two lawyers for submitting 23 fabricated citations and eight invented quotations, marking the largest penalty for AI-related malpractice in U.S. legal history. These cases underscore the risks that clients face when their lawyers misuse AI technology.
The implications of using AI in legal contexts extend beyond mere errors; they also affect fundamental legal principles. A recent ruling in Manhattan highlighted the potential loss of attorney-client privilege when a defendant utilized a general-purpose AI chatbot to prepare his defense strategy. This ruling established that communications entered into a chatbot could be subpoenaed by the government, potentially compromising a defendant’s case.
According to a database compiled by lawyer and data scientist Damien Charlotin, there have been over 1,300 instances worldwide where courts have commented on AI-generated inaccuracies in legal filings. Each case represents a trusting client and a lawyer who has placed faith in AI technology that generates plausible-sounding but potentially false information.
It is important to distinguish between different types of AI tools. General-purpose large language models like ChatGPT and Claude are trained on a wide array of internet data, while industry-specific legal AI tools are designed to operate within established legal databases. Unfortunately, investors and stakeholders have struggled to discern these differences, leading to instability in tech markets, especially following the release of a legal plug-in for Claude.
As a practicing lawyer and General Counsel for LexisNexis, a leading legal technology company, I often receive inquiries about which AI tool is most capable. However, I believe the more critical question is: « Which AI can be trusted in a courtroom? » In legal practice, the distinction is essential, as the responsibility of lawyers extends to both their clients and the court system.
The American Bar Association has identified five Model Rules of Professional Conduct that are significantly impacted by AI usage: competence, confidentiality, candor toward the tribunal, and supervisory responsibility. Lawyers who submit erroneous AI-generated citations not only fail their clients but also undermine the integrity of the judicial system, as such errors corrupt the court record that is foundational to legal proceedings.
This issue transcends the capabilities of specific AI models. General-purpose AI systems are designed to generate text that appears correct, which is acceptable in many fields but is problematic in law where precision and accuracy are paramount. These models cannot verify the existence of cited cases or the validity of the claims made. The gap in architecture between AI’s design and the needs of legal practice is fundamentally problematic. The consequences of these errors can be severe, leading to sanctions against lawyers and dismissed claims that ultimately harm defendants.
When evaluating any legal AI tool, the focus should not solely be on its performance metrics but rather on the underlying data it uses and the ability of lawyers to trace, verify, and support its outputs in a court setting. There are two primary ways that AI can transform legal practice: through compression, which enables faster completion of existing tasks, and expansion, which allows for new types of work that were previously impossible. The potential for AI to expand legal capabilities is vast, but it must be rooted in a foundation of accurate and verifiable legal information.
As the legal profession continues to explore the integration of AI technologies, it is critical to recognize that the cost of incorrect legal answers can have profound implications, impacting individuals’ freedoms, assets, and family futures. The legal community must tread carefully, ensuring that the tools used meet the standards of reliability and integrity that clients deserve.
In conclusion, the current landscape of AI in law raises significant questions about reliance on technology that lacks the ability to verify its outputs. The profession must prioritize trustworthiness in AI applications, recognizing the unique demands of the legal field and the potential consequences of AI-generated errors. As the market adjusts to these realities, the importance of accuracy and accountability in legal practice will remain paramount.
