RAG chunking determines what a retrieval system can find, how much context reaches the language model, and whether answers remain precise or become vague. This guide provides a reusable checklist for choosing chunk size, overlap, metadata, and reranking across FAQs, manuals, code, legal material, and technical documentation.
Overview
In a retrieval-augmented generation (RAG) chatbot, source documents are divided into smaller records called chunks. Those chunks are typically embedded, stored in a vector database, retrieved for a user question, and passed to an LLM as context. Chunking is therefore not a cosmetic preprocessing step. It affects retrieval relevance, citation quality, latency, storage requirements, and the model's ability to understand a complete idea.
There is no universally correct chunk size for RAG. A useful starting point is to preserve one coherent unit of meaning per chunk while keeping enough surrounding information to answer a realistic question. The right choice depends on document structure, query length, embedding behavior, context limits, and the cost of retrieving several neighboring chunks.
Before changing a splitter, define a small evaluation set of real questions. Include questions that require a definition, a specific procedure, a comparison, and information spread across headings or pages. Measure whether the correct passage is retrieved and whether the final answer is supported by it. The RAG chatbot evaluation guide can help organize this testing.
Core decisions
- Chunking method: Decide whether boundaries should follow characters, tokens, sentences, semantic shifts, or document structure.
- Chunk size: Set a target that fits the information density and typical question scope.
- Overlap: Repeat a controlled amount of context at boundaries where meaning may continue.
- Metadata: Preserve enough source and structural information to filter, cite, and debug results.
- Reranking: Add a second-stage relevance check when vector similarity alone returns plausible but poorly ordered results.
Checklist by scenario
Use the following comparison as a starting point, then validate it against your own questions and documents. A hybrid approach is often more reliable than applying one splitter to every file.
| Scenario | Recommended starting strategy | Chunking guidance | Key metadata |
|---|---|---|---|
| FAQs and short support articles | Structure-aware or recursive splitting | Keep each question and answer together. Avoid splitting an answer merely to reach a fixed character target. | Product, category, article ID, version, locale, update date |
| Product manuals and procedures | Heading-aware recursive splitting | Preserve the heading, prerequisites, numbered steps, warnings, and relevant tables with the procedure they describe. | Document title, section path, product model, revision, page, procedure type |
| Code and technical references | Structure-aware splitting by file, class, function, or symbol | Keep signatures, imports where useful, docstrings, and adjacent implementation context. Avoid arbitrary cuts through a function. | Repository, branch or release, file path, language, symbol, line range |
| Legal and policy documents | Structure-aware or semantic splitting | Keep clauses, definitions, exceptions, and conditions intact. Retain section numbering and document version. | Jurisdiction, policy type, effective date, clause, version, access scope |
| Long technical or research documents | Recursive or semantic splitting with parent context | Use headings and paragraph boundaries first, then split unusually long sections. Store the parent heading separately. | Author or owner, title, section path, page, revision, publication date |
Choosing a splitting method
Fixed-size splitting is simple and predictable. It can be useful for an initial baseline or highly uniform text, but it may cut through a definition, list, or argument. Use it as a comparison point rather than assuming it is the final design.
Recursive splitting attempts larger boundaries first, such as headings or paragraphs, before falling back to sentences or smaller units. It is a practical general-purpose option for mixed documentation.
Semantic splitting groups text according to changes in meaning. It can preserve conceptual units better, but it adds processing complexity and still requires safeguards for very long sections.
Structure-aware splitting uses the source format: FAQ pairs, Markdown headings, HTML sections, PDF page and heading relationships, or code symbols. It is usually the strongest starting point when reliable document structure is available.
Setting chunk size and overlap
Choose chunk size based on the smallest useful answer unit, not the maximum context window of the LLM. A chunk should usually answer a focused question or provide a meaningful part of a procedure without requiring excessive neighboring text. Larger chunks may retain more context but can dilute similarity signals and consume more prompt space. Smaller chunks improve focus but may omit definitions, qualifiers, or prerequisites.
Overlap helps when a sentence, list, or explanation crosses a boundary. Start with modest overlap and increase it only when evaluation shows that answers lose necessary context at splits. Excessive overlap duplicates content, increases storage and retrieval noise, and can cause the same passage to occupy too many result positions. For structure-aware documents, preserving the heading and parent section may be more valuable than adding large token overlap.
What to double-check
Metadata schema
Metadata should support three jobs: retrieval filters, source display, and troubleshooting. A practical record might include:
{
"source_id": "support-manual-042",
"document_title": "Device Setup Manual",
"section_path": ["Installation", "Network setup"],
"chunk_index": 7,
"version": "current",
"updated_at": "YYYY-MM-DD",
"page": 18,
"product": "device-family-a",
"access_scope": "internal"
}Use stable identifiers so a document can be replaced without creating confusing duplicates. Keep the original text location, including page, heading, URL, or line range where available. For sensitive corpora, include access-control metadata and enforce it during retrieval rather than relying on the prompt to hide restricted content. See the chatbot security and privacy checklist for related controls.
Retrieval and reranking
Vector search is good at finding semantically related text, but the highest-similarity chunks are not always the best answer sources. Consider reranking when documents use similar language, when users ask precise questions, or when the initial result set contains several near-duplicates. A practical pipeline is to retrieve a wider candidate set, apply metadata filters, rerank the candidates using the question and passage together, then send only the most useful results to the LLM.
Do not use reranking to compensate for broken parsing or poor chunk boundaries. First verify that the needed passage exists in the indexed text and that its metadata is correct. Then compare retrieval with and without reranking using the same evaluation questions.
Evaluation checklist
- Can the system retrieve the exact chunk containing the answer?
- Does it also retrieve definitions, exceptions, or prerequisites?
- Are neighboring chunks useful, or are they mostly duplicates?
- Can the chatbot cite the correct document and location?
- Does retrieval respect product, version, tenant, and access filters?
- Does the answer remain grounded when the question is ambiguous?
- What happens when the answer is absent from the knowledge base?
Common mistakes
- Using one global chunk size: FAQs, source code, contracts, and manuals have different natural boundaries.
- Splitting without preserving headings: A paragraph may be technically intact but meaningless without its section title.
- Adding excessive overlap: More repeated text does not guarantee more context and can crowd out distinct results.
- Ignoring tables and lists: Flattened tables can lose the relationship between labels and values. Lists can lose sequence or conditions.
- Indexing stale versions together: Retrieval may surface an obsolete procedure unless version and effective-date handling is explicit.
- Optimizing only for retrieval scores: A retrieved passage can be relevant while still failing to support a complete, accurate answer.
- Skipping raw-text inspection: PDF extraction, OCR, code parsing, and HTML cleanup can damage content before chunking begins.
- Changing several variables at once: If chunk size, embedding model, top-k, and reranking all change together, it becomes difficult to identify what improved results.
When to revisit
Review your RAG chunking strategy before a major seasonal planning cycle, after a substantial documentation update, or whenever the chatbot's questions change. Revisit it when you add PDFs, tables, code, scanned files, multilingual content, or new access-control requirements. A change in embedding model, reranker, vector database, context budget, or document parser is also a reason to rerun evaluation rather than assuming the previous settings remain appropriate.
Use this practical maintenance loop:
- Collect failed answers, unanswered questions, and cases with incorrect citations.
- Classify each failure as a parsing, chunking, metadata, retrieval, reranking, or generation problem.
- Inspect the original document and the exact indexed chunks for representative failures.
- Change one variable, such as boundary logic, chunk target, overlap, metadata filtering, or reranking.
- Rerun a fixed evaluation set and compare answer support, retrieval relevance, latency, and prompt size.
- Record the chosen configuration by corpus or document type, then monitor new failures after deployment.
The strongest default is not the most elaborate splitter. It is a tested, observable pipeline that preserves document structure, carries useful metadata, retrieves focused evidence, and makes failures easy to diagnose. For the broader architecture and deployment context, review the production-ready RAG chatbot guide.
