RAG explained: why AI answers go wrong and how retrieval fixes it

Short answer: AI answers go wrong when the model does not have the right facts, so it falls back on what it learned in training, which may be outdated, generic or simply invented. Retrieval-augmented generation (RAG) fixes this by searching your own documents first and handing the most relevant passages to the model to answer from, with citations. Most RAG failures are retrieval failures, so the quality comes from how you chunk, search, rerank, refresh and evaluate, not just from the model.

Why do AI models give wrong answers?

A language model stores what it learned during training in its weights. That knowledge has three problems for business use:

• It is out of date. It stops at the training cut-off.
• It is not yours. It has never seen your contracts, policies, tickets or product specs.
• It is not traceable. The model cannot point to where a "fact" came from.

When asked something it does not know, a model often produces a fluent, confident answer anyway. OWASP lists this as "Misinformation" in its 2025 Top 10 for LLM applications.

The 2020 paper that named RAG, by Lewis and colleagues, described the same gap: models' ability to access and precisely use the knowledge in their weights is limited, and providing provenance and updating knowledge remain open problems. Their answer was to pair the model with a searchable index of documents, a "non-parametric memory" it can look things up in.

What is RAG, in simple terms?

RAG is an open-book exam for the model. Instead of answering from memory, it gets a few relevant pages from your documents and answers from those.

There are two halves:

1. Indexing (ahead of time). Split documents into chunks, turn each chunk into a searchable form, and store it in an index.
2. Answering (per question). Search the index for the best chunks, pass them to the model with the question, and ask it to answer only from them and cite them.

The answer is only as good as the passages that reach the model. If the right passage is never retrieved, no model can answer correctly from it. That is why the rest of this article is mostly about retrieval.

How should you chunk documents for RAG?

Chunking means splitting documents into pieces small enough to search and to fit in a prompt. It sounds mechanical, but it decides what the search can find.

Two things go wrong:

• Chunks too large. One chunk covers five topics, so it matches many questions weakly and wastes prompt space.
• Chunks without context. A chunk is cut off from the heading or document it belongs to.

Anthropic's write-up on contextual retrieval gives a good example of the second problem. A chunk reading "The company's revenue grew by 3% over the previous quarter" is nearly useless alone: it does not say which company or which quarter.

Practical rules:

1. Split on structure, not character counts. Use headings, sections, paragraphs, table rows and list items.
2. Keep context with each chunk. Prepend the document title, section heading, date and owner. Anthropic's approach goes further and adds a short, generated summary of where the chunk sits in the document.
3. Keep tables and code intact. Half a table is worse than none.
4. Store metadata. Source, date, version, department and access group are what make filtering, freshness and citations possible.

What are embeddings, and why are they not enough?

An embedding turns a chunk of text into a list of numbers that captures its meaning. Chunks about similar ideas end up close together, so a question about "cancelling my plan" can find a passage about "terminating a subscription" even with no shared words. This is vector search.

Its weakness is exact terms. Product codes, error numbers, part IDs, people's names and legal clause numbers often matter more than meaning, and embeddings can blur them.

Keyword search, such as the long-standing BM25 method, is the opposite: it is literal. Anthropic's article notes BM25 "is particularly effective for queries that include unique identifiers or technical terms", using the example of searching for "Error code TS-999".

What is hybrid search, and when should you rerank?

Hybrid search runs vector search and keyword search together and merges the results. You get paraphrase matching and exact matching in one step. For most business documents, it is a sensible default.

Reranking is a second pass. Search pulls back a generous set of candidates, then a reranker, a model that scores each passage against the question more carefully, picks the best few to send to the model.

Anthropic published measurements of these steps on its own test sets, using the rate at which a relevant chunk was missing from the top 20 results:

In that setup they retrieved 150 candidates, reranked them, and passed the top 20 to the model. Your numbers will differ by data and domain, but the pattern, that combining methods beats any single one, is worth testing on your own documents.

Why does the number and order of passages matter?

It is tempting to send the model as many passages as fit. Research suggests caution. The 2023 "Lost in the Middle" study by Liu and colleagues found that performance is often highest when relevant information is at the beginning or end of the input, and degrades significantly when the model must use information in the middle of a long context.

So:

• send fewer, better passages rather than many weak ones;
• put the strongest passages first;
• remove near-duplicates so one fact does not crowd out others.

A related point: if your entire knowledge base is small, Anthropic notes that below about 200,000 tokens you may be able to put the whole thing in the prompt and skip retrieval. RAG earns its keep as content grows, changes often, or needs per-user access rules.

How do you keep RAG answers fresh?

RAG's big promise is current answers, but only if the index is current. A policy updated last week is no help if the index still holds last year's version.

1. Re-index on change, not on a fixed yearly clean-up. Track each source's last update and re-process what changed.
2. Delete as well as add. Removed documents must leave the index, or the model will keep citing them.
3. Store dates and versions in chunk metadata, and prefer newer versions when two conflict.
4. Show the date in citations so users can judge freshness themselves.
5. Monitor index lag: the time between a source changing and the index reflecting it.

How should the model be told to use retrieved passages?

Good retrieval can still be wasted by a vague instruction. The prompt that wraps the passages is part of the system and deserves the same care as the search.

A solid answering prompt does four things:

1. Limits the source. Tell the model to answer only from the passages provided, not from general knowledge.
2. Allows "I don't know". Make it explicit that saying the answer is not in the documents is a good outcome, not a failure. Without this, models tend to fill gaps.
3. Requires citations. Ask for the source of each claim, using the document titles or IDs you passed in, so users can check.
4. Separates data from instructions. Mark the passages clearly as reference material, and tell the model that any instructions inside them are content to report, not commands to follow.

Then check the output in code where you can. If an answer cites a document that was not in the retrieved set, flag it. If it has no citations at all, treat it as unverified.

When is RAG the wrong tool?

RAG is built for questions answered by text: policies, manuals, contracts, tickets, notes. It is weaker for questions that need calculation across many records, such as "what was revenue by region last quarter?" Retrieving a handful of passages will not add up thousands of rows.

For those, let the AI query structured data directly, through a governed query tool or a semantic layer with agreed metric definitions, and use RAG for the explanation around the number. Many useful assistants combine both.

What are the common RAG failure modes?

Two of these are security problems, not quality problems. OWASP's 2025 list includes "Vector and Embedding Weaknesses", and its prompt injection guidance covers indirect injection, where instructions hidden in external content, such as a retrieved document, change the model's behaviour. The fix for leaks is to filter by the user's permissions before retrieval, so restricted chunks never reach the model at all.

How do you evaluate a RAG system?

Evaluate retrieval and answers separately. Otherwise, when quality drops, you cannot tell which half broke.

The Ragas paper (Es and colleagues, 2023) frames RAG evaluation along three lines: whether retrieval finds relevant, focused context; whether the model uses that context faithfully; and the quality of the answer itself. It also shows these can be scored without hand-written reference answers for every question.

A practical loop:

1. Build a question set of 50 to 200 real questions, with the document or passage that should answer each.
2. Measure retrieval: how often is the right passage in the top results?
3. Measure faithfulness: does every claim in the answer appear in the retrieved passages?
4. Measure usefulness: does the answer actually address the question?
5. Re-run on every change to chunking, search settings, prompts or models.
6. Add real failures from user feedback to the set every week.

How CloudOEngine helps

CloudOEngine is one platform and one console for connecting data, including databases, files and camera streams, running pipelines, and running AI. Its AI runtime includes an LLM gateway, agents with tool access, and RAG, so retrieval sits next to the data it searches rather than in a separate stack.

Data, pipelines and AI share one permission model and one audit trail, so you manage access and review activity in one place instead of per tool. Governance is included, not upsold: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets.

Agents can combine RAG with real tools, such as filing a ticket, and every step and tool call is traced and inspectable. When an answer looks wrong, you can open the run and see what was retrieved and what the model did with it, which is where most RAG debugging starts.

CloudOEngine is built in Bengaluru, India.

Book a demo or see the AI runtime.

Frequently asked questions

What is retrieval-augmented generation (RAG)?

RAG is a way of answering questions with a language model where the system first searches your own documents for relevant passages, then gives those passages to the model to answer from. It lets answers use private and up-to-date information and cite where it came from.

Why does a RAG system still give wrong answers?

Usually because the right passage was never retrieved, or was retrieved but buried among irrelevant ones. Poor chunking, keyword-blind search, stale indexes and missing access filters cause most failures, not the model itself.

What is hybrid search in RAG?

Hybrid search combines meaning-based vector search with keyword search such as BM25, then merges the results. Vector search finds paraphrases; keyword search finds exact terms like product codes and error numbers that embeddings can miss.

Do I need RAG if the model has a long context window?

Not always. If your whole knowledge base is small enough to fit in the prompt, putting it all in can be simpler. Once it grows, changes often or has per-user access rules, retrieval is cheaper and easier to control.

How do you evaluate a RAG system?

Measure retrieval and answers separately. Check whether the right passages appear in the top results for a set of real questions, then check whether answers are faithful to those passages and actually answer the question.

Sources

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
2. Introducing Contextual Retrieval (Anthropic)
3. Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
4. Ragas: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
5. OWASP Top 10 for LLM Applications 2025
6. LLM01:2025 Prompt Injection (OWASP GenAI Security Project)

#rag #retrieval augmented generation #hybrid search #embeddings #llm evaluation