Slide 16 · Scenario 3
Medical FAQ Chatbot Gives Dangerous Guidance.
OWASP Scenario 3, grounded in the Air Canada ruling.
SCENARIO 3
Medical FAQ Chatbot Gives Dangerous Guidance
A healthcare company deploys a general-purpose LLM as a patient-facing FAQ chatbot, handling questions about medications, symptoms, and care instructions. A patient asks about combining two prescription medications they have been prescribed. The chatbot responds with a confident, clinically-toned answer that the combination is safe. It is not. The patient follows the chatbot’s guidance. The Air Canada ruling establishes precedent: the company is liable for what its deployed LLM says.
Why it matters: General-purpose LLMs are not domain-validated for clinical accuracy. The same model that writes marketing copy will answer drug interaction questions with identical confidence — regardless of whether it is right.
The OWASP Language
OWASP warns that a company providing a chatbot for medical use “without ensuring sufficient accuracy” creates liability. The Air Canada ruling confirms that “the AI said it” is not a viable legal defense.
What Would Have Prevented It
Domain-specific validation before deployment: RAG grounded in clinical databases, mandatory clinical review of chatbot responses, refusal to answer drug interaction questions directly, and clear UX disclaimers that the chatbot is not a medical provider.