Chunking contracts, policies and long documents without losing the conditions

How an answer gets separated from its conditions

Business documents put the rule in one place and its limits somewhere else. A payment clause says invoices are due in 30 days, and the next clause says this does not apply to disputed amounts. A leave policy uses the term eligible employee, defined two pages earlier. A price table lists numbers whose currency and validity date sit in a header row on the previous page.

Cut such a document into equal pieces and retrieval may return the rule without the exception. The model then gives a confident answer that is wrong for exactly the case the user asked about. Overlap between chunks helps when a sentence is split at a boundary. It does not bring back a definition from two pages earlier.

Split along the structure the document already has

Microsoft's chunking guidance ties the approach to how structured a document is. Legal texts and specifications have an inferred structure, numbered paragraphs without markup, for which it suggests custom code or custom models; fixed-size chunks with overlap suit unstructured text such as emails and notes. It also calls the chunking approach a semi-permanent choice, because changing it later affects everything downstream.

• Contracts: one chunk per numbered clause, with sub-clauses kept together
• Policies: one chunk per headed section, split further only at subheadings
• Procedures: each procedure whole when it fits, because a step out of sequence is dangerous
• Tables: rows or row groups, with the header row repeated in every chunk

Carry context into each chunk

A clause that reads "the period in 4.2 does not apply" means nothing on its own. Prepend what a reader would need: document title, heading path, version or effective date, and the definitions of terms the chunk uses. Anthropic tested a related idea in 2024, generating a short chunk-specific context with a model and prepending it before embedding and keyword indexing. In its experiments that reduced the top-20 retrieval failure rate by 35 percent with contextual embeddings and by 49 percent when contextual keyword search was added. Those figures describe Anthropic's datasets, not yours.

Cross-references need a second mechanism. When a retrieved clause points to another clause, the application can fetch the referenced one as well, or retrieve small chunks and pass their parent section to the model.

Choosing a strategy by experiment

Collect questions whose correct answer depends on a condition, an exception or a definition, and label both the answer passage and the condition passage. Define two or three candidate strategies and run each against the same questions with the same token budget for retrieved text, adjusting the number of chunks to the chunk size. LlamaIndex notes that smaller chunks may need a larger top-k to retrieve enough material. If you prefer to fix the number of chunks instead, report how much context and cost each strategy then sends to the model, because larger chunks put more text into the prompt.

Chunking experiment: supplier contracts (hypothetical numbers)
40 labelled questions; every run passes about 4,000 tokens of
retrieved text to the prompt, so k is adjusted to the chunk size

Strategy                               k    Answer passage   Answer + its
                                            in prompt        condition in prompt
A  fixed size, small chunks           12    33 / 40          19 / 40
B  one chunk per numbered clause       8    34 / 40          27 / 40
C  clause + heading path + defined     7    36 / 40          33 / 40
   terms prepended to each chunk

Decision: C for contracts. Re-run when the template or embedding model changes.

The figures in the example are invented. Compare the last two columns: all three strategies get the rule into the prompt most of the time, and the difference only appears when the condition is required as well. A test set without condition-dependent questions would have rated them almost equal.

Costs, and where structure-aware chunking struggles

Each document type with its own parser is code to maintain, and Microsoft lists the number of unique chunking implementations as a cost driver. Scanned documents with unreliable headings, clauses longer than the embedding model accepts, and templates that change numbering between versions all weaken the approach. Generating context per chunk with a model adds a processing cost at ingestion. And because chunks and embeddings depend on each other, changing the strategy or the embedding model means reindexing.

A reasonable path is to start with your framework's default splitter as a documented baseline, add structure-aware chunking for the two or three document types that carry the most important questions, and keep the experiment so it can be rerun when templates change.