Chunking: Fixed, Semantic, Recursive, Agentic

MLOps Series RAG Systems

Why Chunking Matters

Before any document enters a vector database, it must be split into chunks — contiguous text segments small enough to embed meaningfully and fit within the LLM's context window at retrieval time. Chunking quality has a direct, measurable impact on retrieval recall and downstream answer accuracy. A chunk that is too large dilutes the embedding with off-topic sentences, lowering cosine similarity with a focused query. A chunk that is too small strips away the context needed for the LLM to generate a coherent answer.

Research from the LlamaIndex team shows that moving from naive fixed-size 1024-token chunks to semantic-boundary-aware splitting can improve retrieval hit rate by 15–25% on domain-specific corpora, with no change to the embedding model or vector database. Chunking is often the highest-leverage optimisation in a RAG pipeline.

Overlap ensures that sentences near chunk boundaries are not orphaned. A 10–15% overlap relative to chunk size is a common default (e.g., 50 tokens for a 512-token chunk).

Fixed-Size Chunking

The simplest strategy: split the text every N characters or tokens, with an optional overlap of M tokens. This is fast, deterministic, and requires no model inference. It works well when your documents are relatively homogeneous (e.g., structured reports where every paragraph is roughly the same length) and when you need predictable chunk counts for cost estimation.

The downside is that fixed boundaries cut through sentences, paragraphs, and even words. A split in the middle of a key sentence can render a chunk uninformative — the embedding captures noise, and the retriever misses the relevant passage.

def fixed_size_chunk(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]: """Split text into fixed-size character chunks with overlap.""" chunks = [] start = 0 while start < len(text): end = start + chunk_size chunks.append(text[start:end]) start += chunk_size - overlap return chunks # Token-aware version using tiktoken import tiktoken def fixed_token_chunk(text: str, max_tokens: int = 256, overlap: int = 30) -> list[str]: """Split by token count for precise context-window budgeting.""" enc = tiktoken.encoding_for_model("gpt-4o") tokens = enc.encode(text) chunks = [] start = 0 while start < len(tokens): end = min(start + max_tokens, len(tokens)) chunk_tokens = tokens[start:end] chunks.append(enc.decode(chunk_tokens)) start += max_tokens - overlap return chunks # Usage document = open("report.txt").read() chunks = fixed_token_chunk(document, max_tokens=256, overlap=30) print(f"Produced {len(chunks)} chunks")

Recursive Splitting

Recursive character text splitting, popularized by LangChain, tries a hierarchy of separators in order: \n\n (paragraphs), \n (lines), . (sentences), (words). At each level, if the resulting segment exceeds chunk_size, it recurses into the next separator. This naturally preserves paragraph and sentence boundaries when possible, falling back to word-level splits only for very long sentences.

The algorithm is greedy — it fills each chunk as close to chunk_size as possible before starting a new one. Overlap is applied at the character level. This approach is a strong default for unstructured text (articles, documentation, transcripts) and has almost zero computational cost.

from langchain_text_splitters import RecursiveCharacterTextSplitter def recursive_chunk(text: str, chunk_size=512, overlap=50) -> list[str]: """LangChain recursive splitter — respects paragraph/sentence boundaries.""" splitter = RecursiveCharacterTextSplitter( chunk_size=chunk_size, chunk_overlap=overlap, separators=["\n\n", "\n", ". ", " ", ""], length_function=len, ) return splitter.split_text(text) # Markdown-aware variant for docs with headers from langchain_text_splitters import MarkdownHeaderTextSplitter headers = [ ("#", "h1"), ("##", "h2"), ("###", "h3") ] md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers) md_chunks = md_splitter.split_text(markdown_doc) # Each chunk carries header metadata for context for chunk in md_chunks: print(chunk.metadata, chunk.page_content[:80])

Tip: For code files, use language-specific separators. LangChain provides Language.PYTHON, Language.JAVASCRIPT, etc., that split on function/class boundaries rather than newlines.

Semantic Chunking

Semantic chunking uses an embedding model to decide where to split. The idea: embed each sentence (or sliding window), compute cosine similarity between consecutive embeddings, and place a chunk boundary wherever similarity drops below a threshold. This produces variable-length chunks that correspond to topical shifts in the text — each chunk is semantically cohesive by construction.

The trade-off is cost: you must embed every sentence before chunking, which adds an O(n) embedding pass to the ingestion pipeline. For large corpora, this can be significant — but the retrieval quality gains often justify it, especially for documents with abrupt topic changes (e.g., meeting transcripts, research papers with diverse sections).

import numpy as np from sentence_transformers import SentenceTransformer model = SentenceTransformer("BAAI/bge-small-en-v1.5") def semantic_chunk(text: str, threshold: float = 0.75, min_size: int = 100) -> list[str]: """Split text at points where consecutive-sentence similarity drops.""" sentences = [s.strip() for s in text.split(".") if s.strip()] if len(sentences) < 2: return [text] embeddings = model.encode(sentences, normalize_embeddings=True) # Cosine similarity between consecutive sentences sims = [ float(np.dot(embeddings[i], embeddings[i + 1])) for i in range(len(embeddings) - 1) ] # Find split points where similarity drops below threshold chunks, current = [], [sentences[0]] for i, sim in enumerate(sims): if sim < threshold and len(". ".join(current)) >= min_size: chunks.append(". ".join(current) + ".") current = [] current.append(sentences[i + 1]) if current: chunks.append(". ".join(current) + ".") return chunks # Usage doc = open("paper.txt").read() chunks = semantic_chunk(doc, threshold=0.72) print(f"Semantic split → {len(chunks)} chunks")

Agentic Chunking

Agentic chunking uses an LLM to decide chunk boundaries and even generate chunk summaries or titles. The process feeds the document (or a sliding window) to a language model with instructions like: "Identify logical sections in this text and return the start/end indices." The LLM can also enrich each chunk with a synthesised title and a one-sentence summary, which are stored as metadata and can be used for hierarchical retrieval.

This is the most expensive strategy — each document requires one or more LLM calls during ingestion. But for high-value corpora (legal contracts, medical records, financial filings), the precision of LLM-driven segmentation can dramatically improve retrieval relevance and reduce downstream hallucination.

import openai import json client = openai.OpenAI() CHUNK_PROMPT = """You are a document segmentation expert. Given the following text, identify logical sections. Return a JSON array where each element has: "title" (short), "summary" (1 sentence), "start" (char index), "end" (char index). TEXT: {text} """ def agentic_chunk(text: str, model: str = "gpt-4o-mini") -> list[dict]: """Use an LLM to identify logical chunk boundaries.""" resp = client.chat.completions.create( model=model, messages=[{"role": "user", "content": CHUNK_PROMPT.format(text=text)}], response_format={"type": "json_object"}, temperature=0, ) sections = json.loads(resp.choices[0].message.content)["sections"] chunks = [] for sec in sections: chunks.append({ "title": sec["title"], "summary": sec["summary"], "content": text[sec["start"]:sec["end"]], }) return chunks # Usage doc = open("contract.txt").read() chunks = agentic_chunk(doc) for c in chunks: print(f"[{c['title']}] {c['summary']}") print(c["content"][:120], "...\n")

Cost warning: At $0.15/1M input tokens (gpt-4o-mini), agentic chunking a 100-page PDF (~75K tokens) costs ~$0.01 per document. This is negligible for small corpora but adds up when processing millions of documents.

Best Practices

After working with chunking strategies across dozens of production RAG systems, here are the patterns that consistently yield the best results:

Do

Start with recursive splitting — it is the best default
Tune chunk size empirically: run your eval set at 256, 512, 1024 tokens
Use 10–15% overlap to avoid orphaning boundary sentences
Preserve document metadata (source, page, section header) per chunk
Add a "parent chunk" reference for small chunks so the LLM can fetch broader context
Benchmark retrieval recall at each chunk-size setting

Don't

Don't use fixed-size chunking with zero overlap on long documents
Don't chunk below 100 tokens — embeddings lose semantic signal
Don't use agentic chunking as a first pass — reserve it for high-value docs
Don't ignore chunk-size interaction with your embedding model's max input
Don't mix different chunk sizes in the same collection without normalisation
Don't skip evaluation — subjective "looks right" checks are insufficient

Chunk-Size Selection Pipeline

from dataclasses import dataclass @dataclass class ChunkConfig: strategy: str # "fixed", "recursive", "semantic" chunk_size: int # in tokens overlap: int # in tokens threshold: float | None = None # semantic only CONFIGS = [ ChunkConfig("recursive", 256, 30), ChunkConfig("recursive", 512, 50), ChunkConfig("recursive", 1024, 100), ChunkConfig("semantic", 512, 50, threshold=0.72), ] def evaluate_config(config: ChunkConfig, corpus, eval_queries) -> float: """Chunk corpus, index, run eval queries, return recall@10.""" chunks = chunk_corpus(corpus, config) index = build_index(chunks) recall = compute_recall(index, eval_queries, k=10) print(f"{config} → recall@10 = {recall:.3f}") return recall best = max(CONFIGS, key=lambda c: evaluate_config(c, corpus, queries)) print(f"Best config: {best}")

Summary: Chunking is the foundation of retrieval quality. Start with recursive splitting at 512 tokens / 50-token overlap. Evaluate against your domain. Upgrade to semantic or agentic chunking when the data justifies the cost.