Prevent AI hallucinations: answers need evidence
By Philipp Kant 4 min read
An AI hallucination is an answer from a language model that sounds convincing but is wrong: an invented value, a false source, a guessed number. In a company you do not prevent it with better wording in the prompt but with a rule we call the evidence requirement: the agent answers only with evidence from your data, and without evidence it asks.
Why language models hallucinate
A language model writes the most likely next text. Ask it for the weight of a product it does not know, and a plausible number is more likely than the sentence “I don’t know.” The model is not lying; it is filling a gap. That becomes dangerous when the answer goes to a customer or gets written into a system.
Three situations cause most hallucinations in day-to-day business:
- The value is missing from the data, and the model fills it in from general knowledge.
- Two sources contradict each other, and the model picks one without saying so.
- The question is narrower than the data, for example about a variant that appears only in a data sheet.
The evidence requirement in three parts
- Answers only from retrieved data. Before every answer the agent looks things up in the PIM, the ERP, or the documents. General knowledge helps it understand the question; it is not a source for values.
- Every statement with a source. Each value comes with where it was found: item number and attribute, document and page. A person can check any answer in seconds.
- No evidence, no number. When the evidence is missing, the agent asks or passes the question to the responsible person. An estimate is not an allowed answer.
What an answer with evidence looks like
A customer asks about an LED panel (invented example). On the left, a model without data access; on the right, an agent with the evidence requirement:
| Question | Without the requirement | With the requirement |
|---|---|---|
| Does it fit 6 cm of installation depth? | “Yes, most panels are slim enough.” | “Yes, it needs 48 mm.” Source: PIM, installation depth |
| Is it DALI-dimmable? | “Yes.” | “Only the variant ending in D.” Source: data sheet p. 1 |
| How long does it last? | “About 50,000 hours.” | “There is no verified value for that. I am asking product management.” |
The third answer matters most. It sounds less competent and is the only one that cannot do damage. You can watch the workflow in the live demo on our home page, workflow 3.
How it is built
- Read access to the sources: through interfaces the agent reaches the PIM, the ERP, and the document store, read-only and only for what the workflow needs.
- Retrieval before the answer: for questions about text, retrieval (RAG) finds the right passages; for questions about values, the agent queries the system directly through a tool. How RAG works is covered in What is RAG?.
- A check before output: a second step checks that every statement has a source. Statements without one are removed or marked as an open question.
- Measurement: an evaluation set built from real questions shows how often the agent answers correctly, correctly says “I don’t know,” and still makes something up.
What happens with contradictions
When two sources contradict each other, such as 36 watts in the PIM and 40 watts in the data sheet, the agent picks neither. It puts the contradiction in front of the person who owns the data, with both sources. Only after that decision does a value count.
That sounds like extra work and is the opposite. The contradictions are already in the data today; nobody sees them until a customer complains. An agent with the evidence requirement makes them visible, sorted by product group and revenue. Which errors are typical is shown in 9 product data errors AI agents find.
How to recognize an agent you can rely on
| Check | Reliable | Not reliable |
|---|---|---|
| Source | names where every value came from | answers without a source |
| Knowledge gap | says “not available” and asks | always answers |
| Contradiction | shows both values, does not decide | silently picks one |
| Action | writes only after approval | writes immediately |
| Measurement | error rate from an evaluation set | “works in the demo” |
FAQ
Can hallucinations be prevented completely? Not completely, but they can be contained and caught: answers only from retrieved data, a source check before output, and approval before any action that changes something.
Does RAG stop hallucinations? RAG gives the model the right passages, which lowers the rate a lot. Without the rule “no statement without a source,” the model still fills in missing values. RAG is the foundation; the evidence requirement is the rule on top.
Which model hallucinates least? The model matters less than data access and the rule. Which model fits your questions best is shown by an evaluation set built from your real cases, not by a general ranking.
What is an answer that says “I don’t know” good for? It keeps a wrong value away from the customer and shows a gap in your data. Collected, those answers become a list product management can work through.
How answers from your own content feel is shown by the free Chat With Your Website demo. AI agents for business with the evidence requirement, connected to your PIM, ERP, and documents, are what we build, at a fixed price.
Ask an AI about this post
