What is the best chunk size for RAG?

There's no universal answer, but a good starting point is 400 to 512 tokens with 10 to 20% overlap. Then tune to your query type: 128 to 256 tokens for precise fact lookup, and 512 to 1024 tokens for analytical or summarization queries. Benchmark a couple of sizes against your own questions rather than trusting a single default.

How much overlap should chunks have?

Ten to twenty percent of the chunk size — roughly 50 to 100 tokens on a 500-token chunk. Overlap stops meaning from being cut at a boundary; too little fragments context across the split, and too much inflates your index and causes near-duplicate retrievals. NVIDIA found about 15% optimal on financial documents.

Is semantic chunking better than fixed-size?

Not reliably. Semantic chunking groups sentences by topic, which sounds better, but benchmarks show its gains over recursive or fixed-size chunking are inconsistent and often don't justify the extra embedding compute. Fixed and recursive chunking remain strong, cheap baselines — start there and only move to semantic if your evaluation shows a real lift.

What is late chunking and when should I use it?

Late chunking embeds the entire document first with a long-context embedding model, then pools the token embeddings into chunks — so each chunk's vector carries context from the whole document, not just its own text. Use it for long documents with long-range dependencies, where a passage only makes sense given earlier context.

Does contextual retrieval actually improve accuracy?

Yes, measurably. Anthropic reported that prepending an LLM-generated context note to each chunk cut top-20 retrieval failures by 35% with contextual embeddings, 49% when combined with contextual BM25, and 67% when reranking was added — at roughly $1.02 per million document tokens using prompt caching. It's most worth it for high-stakes corpora where a missed passage is costly.

What is parent-document (small-to-big) retrieval?

It decouples how you match from what you return. You index small child chunks for precise matching, but when one matches you hand the larger parent section to the model so it has surrounding context. This gives you the recall of small chunks and the coherence of large ones without compromising either.

Which chunking strategy should I start with?

Recursive chunking at about 500 tokens with 15% overlap. It respects natural boundaries, is cheap, and is a strong baseline for most content. Add structure-aware splitting for PDFs and manuals, parent-document retrieval when answers need more context, and contextual retrieval only where the stakes justify the extra cost.

← Resources

By Priya Shah — Senior Engineer, RAG + Knowledge · 2026-05-22 · TUTORIAL

Chunking Strategies for RAG: Sizes, Overlap, and What Works

Chunking sets the ceiling on RAG quality before retrieval even runs. This guide compares the six strategies that dominate practice, gives concrete size and overlap numbers, and explains which one fits which kind of query.

Why chunking decides RAG quality

The chunk is the unit your retriever actually fetches, so chunk boundaries set the ceiling on everything downstream. Make chunks too small and each embedding lacks enough signal to be found reliably; make them too large and the relevant passage's signal is averaged away, diluting the vector so it no longer matches the query. Get chunking wrong and no reranker or bigger model rescues recall — the right passage was never retrieved. That's why chunking, not model choice, is usually the first thing to tune.

The six strategies that dominate practice

Current practice clusters into six approaches:

Fixed-size: equal token/character spans with overlap. Oldest, cheapest, reproducible — and a surprisingly strong baseline.
Recursive: split on a priority of separators (paragraph → line → sentence) to stay under a size limit while respecting boundaries. The recommended default.
Semantic: group sentences by embedding similarity so each chunk is one topic. Gains over recursive are inconsistent and cost extra embedding compute.
Structure-aware: split on document structure (headings, sections, pages, markdown). Best for PDFs, manuals, and financial docs.
Late chunking: embed the whole document first with a long-context model, then pool token embeddings into chunks — so each chunk carries global context.
Contextual retrieval: prepend an LLM-generated context blurb to each chunk before indexing.

Start with recursive; reach for the others when your documents or query types demand it.

Chunk size and overlap: concrete numbers

There is no universal best size, but there are good starting points. Begin around 400–512 tokens with 10–20% overlap (roughly 50–100 tokens on a 500-token chunk). Then match size to query type: small chunks (128–256 tokens) favour precise fact and keyword lookup, while larger chunks (512–1024 tokens) favour analytical and summarization queries where narrative flow matters.

Overlap prevents meaning from being severed at a boundary — too little fragments context, too much inflates the index and duplicates retrieval. NVIDIA's benchmark found about 15% overlap optimal on financial documents, and that page-level chunking gave the highest, most consistent accuracy across mixed corpora, with factoid queries peaking at 256–512 tokens and analytical queries at 1024+.

Advanced: contextual retrieval and late chunking

Two newer techniques attack the same problem — a chunk that reads "revenue increased 15%" is useless if you can't tell which company or quarter.

Anthropic's Contextual Retrieval prepends a 50–100-token, LLM-generated context note to each chunk before both embedding and BM25 indexing. Measured against a 5.7% baseline top-20 failure rate, contextual embeddings cut failures 35%, adding contextual BM25 cut them 49%, and adding reranking cut them 67% — at roughly $1.02 per million document tokens with prompt caching.

Jina's late chunking inverts the pipeline: embed the full document with a long-context model (up to ~8,192 tokens), then pool the contextualized token embeddings into chunks, so each chunk embedding already carries cross-chunk context. Use it for long documents with long-range dependencies.

Parent-document retrieval and metadata

Two refinements pay off almost everywhere. Parent-document ("small-to-big") retrieval decouples retrieval granularity from generation: index small child chunks (≈100–500 tokens) for precise matching, but return the larger parent section (≈500–2,000 tokens) to the model so it has surrounding context. And metadata enrichment — attaching source, section heading, page number, and timestamps to each chunk — improves filtering, enables citations, and lifts retrieval precision independent of chunk size.

Choosing — and tuning — in osFoundry

A quick decision guide: recursive is the safe default; structure-aware or page-level for structured docs; semantic for topically dense prose; late chunking for long documents with a long-context embedder; contextual retrieval for high-stakes corpora where context loss is costly; parent-document when you need both precision and context.

osFoundry treats all of this as configuration, not code. Auto-chunking ships a sane recursive default, and the custom RAG pipeline lets you tune size, overlap, and strategy per knowledge base, with reranking and parent-document retrieval as pipeline-stage toggles. Because the stages are configurable, you can A/B chunking settings against your own query distribution — factoid versus analytical — and keep the rest of the stack fixed, which is exactly the tunable-parameter workflow the research recommends.

Frequently asked questions

What is the best chunk size for RAG?: There's no universal answer, but a good starting point is 400 to 512 tokens with 10 to 20% overlap. Then tune to your query type: 128 to 256 tokens for precise fact lookup, and 512 to 1024 tokens for analytical or summarization queries. Benchmark a couple of sizes against your own questions rather than trusting a single default.
How much overlap should chunks have?: Ten to twenty percent of the chunk size — roughly 50 to 100 tokens on a 500-token chunk. Overlap stops meaning from being cut at a boundary; too little fragments context across the split, and too much inflates your index and causes near-duplicate retrievals. NVIDIA found about 15% optimal on financial documents.
Is semantic chunking better than fixed-size?: Not reliably. Semantic chunking groups sentences by topic, which sounds better, but benchmarks show its gains over recursive or fixed-size chunking are inconsistent and often don't justify the extra embedding compute. Fixed and recursive chunking remain strong, cheap baselines — start there and only move to semantic if your evaluation shows a real lift.
What is late chunking and when should I use it?: Late chunking embeds the entire document first with a long-context embedding model, then pools the token embeddings into chunks — so each chunk's vector carries context from the whole document, not just its own text. Use it for long documents with long-range dependencies, where a passage only makes sense given earlier context.
Does contextual retrieval actually improve accuracy?: Yes, measurably. Anthropic reported that prepending an LLM-generated context note to each chunk cut top-20 retrieval failures by 35% with contextual embeddings, 49% when combined with contextual BM25, and 67% when reranking was added — at roughly $1.02 per million document tokens using prompt caching. It's most worth it for high-stakes corpora where a missed passage is costly.
What is parent-document (small-to-big) retrieval?: It decouples how you match from what you return. You index small child chunks for precise matching, but when one matches you hand the larger parent section to the model so it has surrounding context. This gives you the recall of small chunks and the coherence of large ones without compromising either.
Which chunking strategy should I start with?: Recursive chunking at about 500 tokens with 15% overlap. It respects natural boundaries, is cheap, and is a strong baseline for most content. Add structure-aware splitting for PDFs and manuals, parent-document retrieval when answers need more context, and contextual retrieval only where the stakes justify the extra cost.