In plain words
When an AI answers from your documents, it does not read every page each time. A search step first picks a few relevant passages and hands only those to the AI. For that to work, the documents must already be cut into pieces. Those pieces are called chunks.
How you cut matters. Pieces that are too big bury the useful sentence in noise. Pieces that are too small lose their meaning: a line saying "it rose 3%" is useless if you can't tell what "it" is.
Good chunking follows the shape of the document, keeping a heading with its paragraphs or a question with its answer, so each piece makes sense on its own.
Everyday example
Cutting a recipe book into cards
Imagine turning your family's recipe book into index cards so anyone can grab the right one fast. Cut one recipe per card and it works. Cut every ten lines regardless of where recipes start, and the card for dal ends halfway through, while the next card begins with "add the tempering" with no dish named. Chunking is choosing where to cut.
Where the example breaks: A person can flip back a page to see what a card belongs to. A chunk is retrieved alone, so any context left behind on the cutting floor is simply gone unless it is copied in.
How it actually works
Chunking is the preparation step of retrieval-augmented generation. It happens once, before any question is asked.
Fixed size
Cut every so many tokens. Pinecone calls it "the most common and straightforward approach" and a good place to start. Chunks are commonly a few hundred tokens.
Follow the structure
Split at paragraphs first, then sentences, then words, only going smaller when a piece is still too long. Or cut at headings in Markdown or HTML, so each section stays whole.
Overlap the edges
Repeat a little text at the end of one chunk and the start of the next, so a sentence that straddles a cut still appears whole somewhere. Libraries expose this as a setting such as
chunk_overlap.Add back context
Anthropic's Contextual Retrieval prepends a short note to each chunk, such as which company and quarter it is about, before turning it into an embedding. With keyword search and reranking added, failed retrievals fell by 67% in its tests.
- Rule of thumb
- Pinecone: if a chunk "makes sense without the surrounding context to a human, it will make sense to the language model"
- Classic failure
- "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? (Anthropic's own example)
- When to skip it
- Anthropic suggests that a knowledge base under about 200,000 tokens can simply go whole into the prompt
- Fancier option
- Semantic chunking: compare sentence embeddings and cut where the topic shifts
Questions people ask
What is the best chunk size for RAG?
There is no single best size. It depends on your documents, your embedding model and your questions. A few hundred tokens with a small overlap is a common starting point. The honest answer is to test: ask real questions and check whether the right chunk comes back.
What is chunk overlap?
Overlap repeats a little text at the boundary between neighbouring chunks. If a key sentence falls across a cut, it still appears complete in at least one chunk. Too much overlap wastes storage and returns near duplicates.
Why not just give the AI the whole document?
Sometimes you can. Modern context windows are large. But a big library will not fit, and sending everything every time is slower and costs more. Chunking plus search sends only the few relevant pieces.
How does chunking affect hallucinations?
Badly cut chunks can cause wrong answers. If a chunk loses its context, search may fetch it for the wrong question, or the model may misread it. Clean, self-contained chunks make the retrieved text more reliable.