What is RAG? Retrieval in plain terms
By Philipp Kant 4 min read
Retrieval-augmented generation (RAG) is a way to make a language model answer from a specific body of documents: before the model writes, a search step finds the passages that matter, and the model answers from those passages and cites them. The model’s general knowledge supplies the language; your documents supply the facts.
That one design choice is why RAG is the default for company knowledge: manuals, policies, product data, contracts, support history.
RAG works in two phases: indexing and answering
Indexing happens ahead of time, and again whenever documents change:
- Collect the sources: pages, PDFs, tickets, wiki articles.
- Chunk them into passages of a few hundred words, each with its title, URL, and date attached.
- Embed every chunk: a model turns it into a vector, a list of numbers that places similar meanings close together.
- Store the vectors in a vector index.
Answering happens per question:
- Embed the question the same way.
- Retrieve the closest chunks, usually the top 4 to 10.
- Generate: the model gets the question plus those chunks, with the instruction to answer only from them and cite which one it used.
Our Chat With Your Website tool runs exactly this loop live: it crawls a site, chunks and embeds it, and answers with sources.
Retrieval decides the answer quality
When a RAG system gives a bad answer, the model is rarely the cause. The retrieval step handed it the wrong passages, or none. The usual failure modes:
| Symptom | Cause | Fix |
|---|---|---|
| Answer is vague or generic | Chunks too large, the relevant sentence drowns | Smaller chunks, split along headings |
| Answer misses context | Chunks too small, a table cut in half | Overlap, keep tables whole |
| Answer is outdated | Index not refreshed | Re-index on change, store the date |
| Right document, wrong version | No metadata filter | Filter by product, language, validity |
| “I don’t know” although it is there | Pure vector search misses exact terms | Hybrid search: vectors plus keywords |
Clean source data matters as much as any of these. A RAG system over contradictory documents answers contradictorily, which is the same problem MDM and PIM exist to solve for product data.
RAG, fine-tuning, or a long context window
These three get confused. They solve different problems:
| Approach | What it changes | Use it when |
|---|---|---|
| RAG | What the model knows at answer time | Facts change, sources must be cited, documents are many |
| Fine-tuning | How the model behaves | You need a fixed format, tone, or task skill, not new facts |
| Long context | How much fits in one prompt | A few documents, one-off analysis, cost per call is acceptable |
For company knowledge that changes and has to be traceable, RAG is the answer. Fine-tuning a model on documents to teach it facts is slow to update and gives no citations.
A production RAG system needs more than the demo
A demo answers five prepared questions well. A system people rely on needs:
- An evaluation set: 50 to 200 real questions with known answers, run on every change.
- Permissions: users only retrieve what they may read.
- Freshness: re-indexing when sources change, with dates in answers.
- Citations people can click, so every answer can be checked.
- Refusals: “not in the documents” beats a confident invention.
- Monitoring and cost tracking per question.
These are the parts that take the weeks in a real project. We describe the full shape in AI integration for German SMBs.
FAQ
What is retrieval-augmented generation? A method where a search step retrieves relevant passages from your own documents before a language model answers, so the answer is grounded in those passages and can cite them.
How does RAG work step by step? Documents are chunked, embedded as vectors, and stored. A question is embedded the same way, the closest chunks are retrieved, and the model answers from them with citations.
When should a company use RAG instead of fine-tuning? When the knowledge changes, when answers must be traceable to a source, or when there are many documents. Fine-tuning changes behavior and format, not reliable facts.
Does RAG stop hallucinations? It reduces them sharply when retrieval works and the model is told to answer only from the retrieved passages and to refuse otherwise. An evaluation set shows whether it does.
If your company knowledge lives in documents nobody can search, that is a RAG system we build.
