Chunking and indexing strategies

0:00 Chunking decides what can be found

Imagine you tear a policy manual into strips and hand one strip to a colleague. The strip says: it must be submitted within fourteen days. What must? Travel claims? Complaints? Your colleague has no idea, and neither does your AI system. That's chunking, and it quietly decides what your AI can and can't find. In this lesson you'll learn the trade-offs, the strategies that work, and a hands-on way to give every chunk the context it needs.

0:33 Why it matters

Why does this matter so much? Because chunking happens once, quietly, at indexing time, and then shapes every answer your system ever gives. Think of it like cutting a pizza. Cut it into tiny crumbs and nobody gets a proper bite. Leave it whole and nobody can share it. The right cut depends on the pizza and the people eating it. Documents are the same: the right chunk size depends on the document's structure and the questions people ask.

1:07 The size trade-off

Here's the trade-off. Small chunks, a sentence or a short paragraph, are precise. Each vector represents one idea. But they lose context, like our mysterious it. Large chunks, several pages, keep the context but blur the meaning. The vector becomes an average of many topics, so it matches no specific question strongly, and you stuff lots of irrelevant text into the model's prompt. There's no universal best size. A sensible starting point is a few hundred words with a little overlap, then you tune on your own test questions.

1:46 Pick a strategy per document type

Match the strategy to the document. For well-structured content like policies, manuals and web pages, split on the structure: headings, sections, list items, table rows. For long unstructured text, like call transcripts, semantic chunking splits where the topic shifts. And some documents need their own rules. Split code by function, spreadsheets by row groups with headers repeated, email threads by message, and FAQs by question and answer pair. A fixed-size splitter with overlap is a fine baseline, but structure-aware splitting usually wins on business documents.

2:23 Simple example: a returns page

Here's a simple example. A returns policy page has three headings: eligibility, how to return, and refunds. A fixed-size splitter might cut straight through the middle of the refunds section, so the refund amount ends up in one chunk and the refund timing in another. Split on the headings instead, and each chunk answers one question cleanly: can I return this, how do I send it back, and when do I get my money? Same page, far better retrieval.

2:57 Make chunks self-describing

Now the single most useful trick: make every chunk self-describing. Before you embed a chunk, prefix it with where it came from. Employee Handbook 2026, Expenses, Travel reimbursement. Suddenly our it-must-be-submitted strip is obviously about travel claims, and a question about travel expenses finds it. You embed the header plus the text, but you still show users the clean text with a citation. Store metadata too, such as office, effective date and access level, so you can filter.

3:31 Contextual retrieval

For messy documents with weak headings, you can go further with what Anthropic published as contextual retrieval. At indexing time, you show a model the whole document and one chunk, and ask it to write one or two sentences situating that chunk. You embed that context with the chunk, and index it for keyword search too. It costs extra processing, so use prompt caching to avoid paying for the whole document over and over, and measure recall before and after to prove it's worth it.

4:08 More patterns and hygiene

A few more patterns. Small-to-big retrieval matches on small precise chunks, then sends the model the larger parent section, so you get precision and context. Tables must keep their header row, or be converted to sentences. Scanned PDFs need good text extraction, and it's worth checking a sample by eye before indexing. And keep your index clean: remove superseded versions, date everything, re-index when content changes, and record permissions at index time. Looking ahead, late chunking and image-based page retrieval are maturing fast, and are worth benchmarking on slide decks and scanned forms.

4:48 Business example (illustrative)

Here's a deeper business example, with illustrative numbers. A UK insurance broker indexes policy wordings from twelve insurers. With fixed thousand-word chunks, their test set of sixty questions found the right clause in the top five only about half the time, because exclusions and definitions were mixed together. Splitting on clause headings, adding the insurer and policy name as a header, and keeping definition tables whole raised that to around four in five, without changing the embedding model at all.

5:23 Hands-on in the lesson

In the lesson you'll find a short Python function that splits Markdown on headings, keeps chunks under a word limit with overlap, and adds the section path as a header. You'll also find the prompt pattern for contextual retrieval. Try both on one real document: a policy, a price list, or a product manual. Then ask five real questions and check which version puts the right chunk in the top three.

5:54 Common mistakes

Common mistakes to avoid. Indexing boilerplate like cookie banners, menus and footers, which then match everything weakly. Splitting tables without their header row, so a chunk says fifty, eighty, one hundred and twenty with no idea what those numbers are. Keeping every old version of a document, so retrieval finds contradictions. Never checking OCR output from scanned PDFs. And tuning chunk size by gut feel instead of re-running your recall test after each change.

6:26 How you'll know it's working

How will you know your chunking works? Measure recall at k before and after each change, on the same questions. Watch the average length of passages you send to the model, which should stay lean. And read ten retrieved chunks at random: could a stranger understand each one on its own? If not, add context headers. Chunking is an experiment you run, not a setting you pick once.

6:56 Watch me do it: chunk_markdown()

Watch me do it. I paste the handbook snippet into the chunk markdown function. First, the regular expression splits the text on Markdown headings and keeps the heading text. Next, for each heading, I work out its level from the number of hash marks and trim the section path, so Travel reimbursement sits under Expenses, under Employee Handbook. Then the body is split into windows of two hundred and fifty words, with forty words of overlap. For each window, I build the context header by joining the document title and path with arrows, and I store two versions: embed text, which is header plus body, and text, which is the clean body for display. I run it, and the output reads Employee Handbook twenty twenty-six, four Expenses, four point two Travel reimbursement, followed by the sentence about fourteen days. Finally, I feed embed text into the hybrid search from the last lesson and re-run my five questions.

8:04 Recap

To recap: chunking decides what can be found. Small chunks are precise, large chunks carry context, and you tune the size with evaluation. Split along structure where you can, make every chunk self-describing, consider contextual retrieval for messy documents, and keep your index hygienic. Your next step is to treat chunking as an experiment, not a guess. In the next module, we'll assemble all of this into a full retrieval-augmented generation pipeline.

8:35 Try this now (25 minutes)

Try this now. Take one real document you'd want an AI assistant to know, a policy, a price list or a product guide. Chunk it two ways: a fixed size of about two hundred and fifty words, and by headings with the section path added as a header. Write five questions a customer might ask, and check which version puts the answering chunk in the top three. Keep the winning approach and the five questions: they're the start of your evaluation set.