It’s 2023. A New York attorney is building a legal brief for an aviation injury suit against Avianca Airlines. He has a deadline, a client to serve, and a new AI tool to help. He asks ChatGPT to find supporting case law.
ChatGPT returns six cases — complete with case names, court designations, and years. They look exactly right. He includes them in the brief and files it in the Southern District of New York.
Opposing counsel cannot locate any of the cited cases. The judge demands copies. The attorney asks ChatGPT for the source documents. ChatGPT produces more text — none of it corresponds to anything real.
Every case was fabricated: Varghese v. China Southern Airlines. Martinez v. Delta Airlines. Zicherman v. Korean Air. The names, docket numbers, courts, and legal reasoning were invented. The attorney had no idea.
The firm was sanctioned $5,000. Judges whose names had been falsely cited had to be personally notified.
This is misinformation — an LLM generates output that is factually false but delivered with complete confidence. No attacker. No exploit. No jailbreak. The model simply made things up, and the user trusted it.
Misinformation is when an LLM’s output is factually wrong, stated confidently, and trusted enough to cause real harm.