In a cautionary tale that’s sending shockwaves through the legal tech world, a California attorney learned the hard way that artificial intelligence isn’t always a lawyer’s best friend. Amir Mostafavi was slapped with a historic $10,000 fine after submitting a court appeal riddled with completely fabricated legal citations generated by ChatGPT.

The California 2nd District Court of Appeal didn’t hold back in its scathing opinion, revealing that 21 out of 23 case quotes in Mostafavi’s brief were entirely made up. This landmark decision serves as a stark warning to lawyers everywhere about the dangers of blindly trusting AI-generated content.

Mostafavi admitted he used ChatGPT to “improve” his appeal, claiming he was unaware the AI would invent case citations. His case highlights a growing concern in the legal community: AI’s tendency to confidently generate completely fictional information.

This isn’t an isolated incident. Experts tracking AI legal hallucinations have seen cases of fake citations skyrocket from a few per month to several daily. A Stanford University analysis found that some AI models generate fabricated information in one out of three queries, a sobering statistic for professionals relying on these tools.

The California Judicial Council is now scrambling to regulate AI use, requiring courts to either ban generative AI or develop strict usage policies by December. The California Bar Association is also considering updates to its professional conduct guidelines to address these emerging technological challenges.

Legal professionals like Mark McKenna from UCLA’s Technology, Law & Policy Institute warn that the problem will likely worsen before it improves. Many law schools and firms are rushing to adopt AI without fully understanding its limitations.

Mostafavi’s experience serves as a critical reminder: technology is a tool, not a replacement for professional diligence. As AI continues to evolve, lawyers must approach these technologies with caution, skepticism, and a commitment to verifying every single piece of information.

For now, the legal world is learning a expensive lesson - artificial intelligence might be intelligent, but it’s far from infallible.