Chunking strategies for RAG: size, overlap and structure
Good RAG chunking splits documents on their own structure, keeps chunks a few hundred tokens with context, and tests size and overlap on real questions.
The best chunking strategy for RAG splits each document along its own structure (headings, sections, paragraphs, table boundaries) and only falls back to cutting by size when a section is too long. Chunks of a few hundred tokens, each carrying its document title and section path, are a sound starting point, with little or no overlap. Then test two or three sizes on real questions, because published evaluations disagree on the winner and your documents decide it.
Chunking gets less attention than model choice, and it shouldn't. A chunk is the unit your search can return. If the answer to a question is split across two chunks, or buried in a chunk that also covers four other topics, retrieval struggles and no model downstream can recover what it never saw. Our RAG explainer covers the whole pipeline; this article stays on the one step that quietly decides most of it.
A chunk has two jobs that pull against each other
Every chunk has to be findable and it has to be useful once found.
Findable means the chunk's embedding sits close to the questions it can answer. Small, focused chunks do this well, because an embedding of 150 tokens about one leave rule is about that rule, while an embedding of 2,000 tokens about a whole HR policy is a blurred average of everything in it.
Useful means the chunk, handed to the model, carries enough to answer. Here small chunks lose. "Unused days lapse after 31 March" means little if the chunk doesn't say which kind of leave, which company or which year.
So chunk size is a trade, and the measurements show it. Chroma's 2024 technical report scored retrieval at the token level across five corpora (about 328,000 tokens in total). Small chunks gave far better precision: a recursive splitter at 200 tokens with no overlap reached 7.0% precision, against 1.5% for the same splitter at 800 tokens with 400 overlap. Recall barely moved between them (88.1% against 85.4%). Precision looks low in absolute terms because it counts every retrieved token that wasn't part of the answer, which is exactly the padding that fills your prompt and your bill.
The main ways to split a document
There are five families of chunking, and most real systems combine two of them.
| Strategy | How it splits | Good at | Watch out for |
|---|---|---|---|
| Fixed-size | Every N tokens or characters, optionally overlapping | Simple, predictable, fast | Cuts sentences, tables and lists mid-way |
| Recursive | Tries paragraph breaks, then lines, then spaces, until pieces fit a size limit | Keeps paragraphs together with no parsing work | Ignores headings; sizes are often in characters, not tokens |
| Structure-aware | Headings, sections, pages, table and code boundaries from the parsed document | Chunks match how the author organised the content | Needs good parsing; very long sections still need a size cap |
| Semantic | Starts a new chunk where the meaning of adjacent sentences shifts, using embeddings | Topic changes without headings, such as transcripts | Extra compute; results vary by method |
| Proposition / LLM-based | A model rewrites or splits text into small self-contained statements | Fact lookups | Slow and costly at index time; the model can drop or alter content |
Recursive splitting is the default in many frameworks. LangChain's documentation describes it as trying a list of separators in order (by default a blank line, a newline, a space, then nothing) "until the chunks are small enough", which keeps paragraphs, then sentences, then words together as long as possible. Note one detail from the same page: chunk size is measured in characters by default. If your embedding model limit or your mental model is in tokens, set the length function to count tokens, or your "500" means something different from what you think.
What the published tests say
Three sources are worth reading in full, and they don't agree, which is itself the lesson.
The Chroma report found that strategy choice moved recall by up to 9%. A plain recursive splitter at 200 tokens with no overlap was consistently strong across its metrics. The 800-token, 400-overlap setting, which the report describes as the default of OpenAI's Assistants at the time, had slightly below-average recall and the lowest precision and IoU in its main table. A popular semantic chunker at its default settings scored slightly below average on every metric, while an LLM-based chunker reached the highest recall (91.9%). The authors are clear that their dataset is small, and that the LLM chunker's runtime, from near-instant to tens of minutes, isn't reflected in the scores.
NVIDIA's June 2025 post tested end-to-end answer accuracy on five document sets, mostly PDFs such as financial filings and reports. One chunk per page had the highest average accuracy (0.648) and the most consistent results across datasets. Fixed token chunks from 128 to 2,048 tokens averaged between 0.603 and 0.645. The extremes hurt: 128-token chunks scored worst on one long-document set, and 2,048 tokens did worse than 1,024 on two others. Factoid questions did well with smaller chunks (the post suggests 256 to 512 tokens), while analytical questions needed larger chunks or whole pages.
Then the semantic question. Qu, Tu and Bao asked directly whether semantic chunking is worth its computational cost and concluded that the extra cost does not buy consistent performance gains over fixed-size chunking.
My reading: the recommended sizes span 200 tokens to a full page because the documents and questions differ, not because someone made a mistake. A product FAQ answered by single sentences and an annual report answered by comparing three tables are different problems. Start simple (structure first, a size cap of a few hundred tokens) and spend your effort on evaluation before you spend it on a cleverer splitter.
Overlap: a patch for boundaries, not a default
Overlap repeats the end of one chunk at the start of the next, so a fact that falls on a boundary appears whole in at least one chunk. It exists because fixed-size splitting cuts blindly.
The evidence is mixed. In the Chroma tests, removing overlap improved IoU, since overlapping chunks retrieve duplicate tokens. But with a smaller embedding model and 250-token chunks, recall fell from 82.4% with 125 tokens of overlap to 77.1% with none. NVIDIA used 15% overlap after trying 10%, 15% and 20% on one dataset, and notes that 10 to 20% is common practice.
If you split on paragraphs and headings, boundaries mostly fall where the author put them, and you need far less overlap than a blind fixed-size split. I'd start at zero for structure-aware chunks, try a modest overlap for fixed-size chunks, and keep whichever scores better on your question set. Overlap also inflates index size, and it can put two near-identical chunks into the top results, crowding out a different passage that would have helped.
What a structure-aware chunk looks like
The diagram shows one policy document split along its headings, as an illustration. Each section becomes a chunk, the table stays whole inside the section it belongs to, and every chunk is stored with its context and metadata.
The prefix is cheap and does a lot of work. Without it, chunk 2 might read "Up to 15 days may be carried forward; the remainder lapses" (an invented line), which could match a question about any leave type in any policy you hold.
Giving chunks back the context they lost
Splitting always throws away some context, so several techniques try to put it back.
The simplest is the breadcrumb prefix above: document title, section path, and perhaps the date, prepended before the chunk is embedded. Anthropic's contextual retrieval goes further and has a model write a short note, usually 50 to 100 tokens, explaining where each chunk sits in its document. Their example turns "The company's revenue grew by 3% over the previous quarter" into a chunk that also says it comes from an SEC filing on ACME Corp's Q2 2023 performance. Contextual embeddings alone cut their top-20 retrieval failure rate by 35%. With prompt caching they estimate the one-time cost at $1.02 per million document tokens, under the assumptions stated in their post.
Late chunking, proposed by Günther and colleagues in 2024, works inside the embedding model instead. The whole document (up to the model's context length) passes through a long-context embedding model first, and the token outputs are split into chunks just before pooling, so each chunk's vector reflects its surroundings. The paper reports better results across retrieval tasks with no extra training.
At the other extreme, the Dense X Retrieval paper indexes propositions: small, self-contained statements of a single fact, rewritten by a model. It found that indexing at this fine granularity beat passage-level indexing on retrieval and improved question answering within a fixed computation budget.
A practical middle path is to search small and answer large: index short chunks for precise matching, but when one matches, send the model the full parent section it came from. That gets you the findability of small chunks with the completeness of larger ones, at the cost of storing the link between them.
What I would do first: add the breadcrumb prefix, because it costs nothing at query time. Try generated context only if your evaluation shows chunks being retrieved for the wrong entity, year or product.
Tables, code and messy PDFs
Most chunking failures I'd expect in a business corpus are not about size. They come from parsing.
A table split by rows loses its header, so row 14 becomes a string of numbers nobody can interpret. Keep tables whole where they fit; where they don't, repeat the header row and caption in every piece. NVIDIA's tests extracted tables and charts as separate items rather than letting the chunker split them, which is a sensible default.
Code should be split on functions or classes, never mid-block. Slides usually work best as one chunk per slide with the slide title kept. Scanned PDFs need OCR before any of this, and headers, footers and page numbers repeated on every page should be stripped, or they end up in every chunk and make unrelated chunks look similar to each other.
Check the parsed text before you tune anything. Open twenty chunks at random and read them. If a person can't tell what a chunk is about, an embedding won't either.
Metadata travels with the chunk
A chunk without metadata can be found but not filtered, dated or cited. Store at least the source file or record, the section path, a version or modified date, and the access group of the original document.
Access is the one that bites. If a salary band table from a restricted HR folder is chunked into the same index as the public handbook, the only thing that keeps it out of an answer is a filter applied at search time. In CloudOEngine, files and databases connect on the same platform as the AI runtime and its RAG, under one permission model, so access rules aren't maintained separately for the index. The same metadata also drives freshness: when a document changes, you need to find and replace every chunk that came from it, which is much easier when each chunk records its source and version. Our data readiness checklist covers ownership and lineage for the sources themselves.
How to choose a chunking strategy for your documents
This is the order I'd follow on a new corpus.
- Collect 50 to 100 real questions, and for each, mark the passage that answers it. Without this set, every chunking change is a guess.
- Parse the documents properly: headings, tables, code blocks, slide titles. Read a sample of the output.
- Start with structure-aware splitting plus a size cap of a few hundred tokens, measured in tokens, with no overlap and a breadcrumb prefix on every chunk.
- Measure retrieval on its own: is the marked passage in the top results, and how much irrelevant text comes with it?
- Try one or two alternatives that match your failures: smaller chunks if answers are buried, larger chunks or whole pages if answers need surrounding context, overlap if facts are being cut at boundaries, generated context if chunks lose their subject.
- Keep the winner, record the settings next to the index, and re-run the set whenever documents, parsing or the embedding model change.
The step people skip is the fourth. When a RAG answer is wrong, you need to see which chunks were retrieved before you can tell whether chunking, search or the prompt is at fault. CloudOEngine traces every agent step and tool call, so a bad answer can be opened and its retrieval inspected rather than guessed at.
How CloudOEngine helps
CloudOEngine is one platform and one console for connecting data (databases, files and camera streams), running pipelines, and running AI. Its AI runtime includes an LLM gateway, agents with tool access, and RAG, so the documents you chunk and the assistant that answers from them live in the same place.
Data, pipelines and AI share one permission model and one audit trail. That matters for chunking because the access rules on a source document are what must keep its chunks out of the wrong person's answer, and you manage them once instead of per tool. Governance is included, not upsold: tenant isolation, role-based access, audit logs and per-tenant encrypted secrets.
Agents can combine retrieval with real tools, such as filing a ticket, and every step and tool call is traced and inspectable. When you change chunk size, overlap or prefixes, you can open individual runs and see what was retrieved for a question and what the model did with it, which is how you tell a chunking problem from a prompt problem.
CloudOEngine is built in Bengaluru, India.
Book a demo or see the AI runtime.
Frequently asked questions
What is chunking in RAG?
Chunking is splitting documents into smaller pieces before they are indexed for search. Each piece is embedded and stored on its own, so the chunk is the unit the system can find and hand to the model.
What is a good chunk size for RAG?
There is no universal number. Published tests have found good results anywhere from about 200 tokens to whole pages, depending on the documents and the questions, so start with a few hundred tokens or one chunk per section and measure on your own questions.
Should RAG chunks overlap?
A small overlap can help recall when chunks are short and a fact might fall on a boundary, but it also stores and retrieves duplicate text. If you split on headings and paragraphs, you often need little or no overlap; test both settings rather than copying a default.
Is semantic chunking better than fixed-size chunking?
Not reliably. A 2024 study found its extra computing cost did not bring consistent gains, and another evaluation found one popular semantic chunker scored below simple recursive splitting. Some newer semantic methods did score well, so treat it as something to test, not a default.
How do I chunk tables and code for RAG?
Keep each table or code block whole where possible, and repeat the table's header row and caption if you must split it by rows. Half a table, or a function cut in two, rarely answers anything.
Why do chunks need metadata?
Metadata such as source, section path, version, date and access group lets the system filter by permission and freshness before searching, and lets the answer cite exactly where it came from.
Sources
- Evaluating Chunking Strategies for Retrieval (Smith and Troynikov, Chroma Technical Report, 2024)
- Finding the Best Chunking Strategy for Accurate AI Responses (NVIDIA Technical Blog, 2025)
- Is Semantic Chunking Worth the Computational Cost? (Qu, Tu and Bao, 2024)
- Introducing Contextual Retrieval (Anthropic)
- Dense X Retrieval: What Retrieval Granularity Should We Use? (Chen et al., 2023)
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (Günther et al., 2024)
- Recursive text splitter (LangChain documentation)