Hallucination mitigations for LLM apps

You cannot stop LLMs from hallucinating. That is worth internalising before you spend a quarter trying to. What you can do is stop hallucinations from reaching users, and stop the ones that leak through from causing real damage. This post is about the layered defence stack that actually works in production, and where hallucination risk belongs in your governance framework.

Why hallucinations happen

LLMs are next-token predictors trained on a huge corpus of text. When asked something that lies outside what they have reliable patterns for, they do not know they do not know — they generate the most statistically plausible continuation. That is not a bug you can fix with better prompting. It is how the underlying mechanism works.

The result is confident, fluent, well-formed output that happens to be wrong. Users trust it because it reads like knowledge. That combination — high confidence and low accuracy — is what makes hallucination dangerous.

Why "just tell it not to" does not work

A common first attempt is to add "Only answer if you are certain" or "Do not make up facts" to the system prompt. This helps a little on the margins and fails on the cases that matter. The model does not have introspective access to whether a given generation is grounded. Instructions in the prompt do not create knowledge the model does not have.

Better prompting is worth doing, but treat it as a small tuning dial, not as your defence.

The defence layers

Five layers, each mitigating a different part of the problem. You do not need all five for every use case, but you need to know which ones apply to yours.

Layer 1 — RAG grounding

Retrieve authoritative documents at query time and instruct the model to answer only from the retrieved content. For factual questions where you own the source of truth (documentation, product data, customer records), this is the highest-leverage single control. A grounded model still hallucinates when the retrieval is weak, but the failure mode moves from "invented facts" to "irrelevant answers" — which is a much easier problem to detect.

Layer 2 — Citation requirement

Require the model to cite the specific source passage for every factual claim, and validate the citation post-hoc. A claim without a matching source is dropped or flagged. Citation is a structural constraint that punishes ungrounded output automatically, and it gives users a way to verify the claims themselves.

Layer 3 — Factuality post-check with a second model

Route the generated answer through a second model whose only job is to compare the answer against the retrieved sources and flag statements that are not supported. This costs an extra inference call but catches a meaningful percentage of the hallucinations that slip past the first layers. Do not use the same model as judge of itself — cross-model checking is more effective.

Layer 4 — Confidence thresholds

For classifier-style outputs, expose the model's confidence score and route low-confidence responses to a different path — either a human reviewer or a refuse-to-answer fallback. Below a defined threshold, the system does not guess; it stops.

Layer 5 — Refuse-to-answer patterns

Design the product so that "I do not know" is a first-class answer, not a failure. A system that refuses to answer 5% of questions with grace is far more trustworthy than one that answers 100% of questions with 15% hallucinated content. Refusal is a feature, not a defect.

Evaluation with a golden dataset

You cannot manage what you do not measure. Build a small golden dataset — 100 to 500 examples with expected answers — that represents your real use cases. Run it against every model version and every prompt change. Track hallucination rate as a first-class metric alongside latency and cost. Any regression on the golden set blocks deployment.

Do not skip this step because it feels tedious. The teams that ship reliable LLM apps are the ones with a maintained eval set. The ones that ship unreliable apps skipped it.

Where hallucination risk lives in the risk register

Hallucination should be a named risk in every LLM-powered system in your inventory, with the severity graded by consequence. A hallucinated marketing tagline is different from a hallucinated dosage or a hallucinated legal citation. Grade the severity by what happens when the model is wrong, not by how often it is wrong. High-consequence use cases need more layers of the defence stack, and the risk register should record which layers are in place for each system.

How Zilonex Govern helps

Zilonex Govern ships a hallucination risk template that maps the five defence layers to controls, tracks which layers are enabled for each AI system in your inventory, and stores golden-set evaluation results as audit evidence over time. You get an at-a-glance view of which systems are protected by which layers, and a trail of how those numbers have changed release over release.