In a stark warning about the risks of AI-generated legal research, a California attorney recently learned an expensive lesson after submitting a court appeal filled with fabricated case citations generated by ChatGPT.

Amir Mostafavi, the Los Angeles-area lawyer at the center of this cautionary tale, used the AI tool to “improve” his appeal without verifying the generated content. The result? A blistering court opinion and a $10,000 fine from a three-judge panel who found that 21 of 23 case quotes were completely invented.

This incident highlights a growing concern in the legal profession about AI’s tendency to “hallucinate” or generate convincing but entirely fictional information. A Stanford University analysis revealed that some AI models generate false information in one out of three queries, a statistic that should make any professional pause.

The court’s opinion was unequivocal: attorneys must personally read and verify every citation, regardless of its source. This isn’t just about one lawyer’s mistake, it’s a systemic issue that’s rapidly spreading. Experts tracking AI-related legal errors have seen cases jump from a few per month to several per day.

Mostafavi acknowledges the inevitability of AI in legal work, comparing it to how online databases replaced law libraries. However, he warns that “we’re going to have some victims, we’re going to have some wreckages” as the legal system adapts to this technological disruption.

California’s judicial system is already scrambling to address these challenges. The Judicial Council has mandated guidelines requiring courts to either ban generative AI or develop strict usage policies. Meanwhile, the California Bar Association is considering updates to its professional conduct standards.

As AI continues to evolve faster than professional training can keep up, the message is clear: technological shortcuts are no substitute for professional diligence and personal verification. Lawyers and professionals across industries must approach AI tools with critical awareness and rigorous fact-checking.