Build the corpus, then break it on purpose.

Chunking decides what retrieval can ever return whole. You will make a clause disappear, then bring it back.

Recall: Lesson 1 said chunking is where a clause can get cut in half. Today you make that happen with real code, on purpose, then fix it. Deeper chunking theory: KB Lesson 4.

The one idea: A chunk is the largest unit retrieval can ever hand back. If a clause spans two chunks, no retriever, however smart, can return it whole. Chunk size and overlap are the two knobs that decide this, and they are set once, at ingest, long before any customer asks a question.

The demo corpus

DEMO MATERIAL — modeled on Samsung Life's public product categories (term life, critical illness riders). Not real Samsung Life policy text. Never present this to a customer as genuine.

Two short files, saved as plain text:

bright-life-term-summary.txt
------------------------------
SECTION 4: COVERAGE
This policy pays the death benefit stated in the schedule if the
insured dies while the policy is in force.

SECTION 5: EXCLUSIONS
Flood damage to the insured property is covered under the rider,
except in basement units, which are excluded regardless of rider
status.

SECTION 6: PREMIUM TABLE
Age 30-39: 42,000 KRW/month
Age 40-49: 61,000 KRW/month
Age 50-59: 89,000 KRW/month
Policy ID: S-240-113

critical-illness-rider-kr.txt   (demo, Korean clause)
------------------------------------------------------
제9조 (보험금 지급 제외)
회사는 다음 중 어느 한 가지로 보험금 지급사유가 발생한 때에는
보험금을 지급하지 아니합니다.
(Translation: "Article 9 (Exclusions from benefit payment) — the
company will not pay the benefit if the claim arises from any of
the following causes.")

The second file is why embed gets its own lesson next: Samsung Life's real documents are Korean, and an English-only embedder will place this text nowhere near where it belongs in vector space. You will feel that failure directly in Lesson 3.

Chunk it badly, on purpose

Real code, using LangChain's splitter. First pass: a chunk size small enough to slice the exclusion clause in half.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.documents import Document

text = open("bright-life-term-summary.txt").read()
doc = Document(page_content=text, metadata={"source": "bright-life-term-summary", "policy_id": "S-240-113"})

# Deliberately too small: this WILL cut the flood clause in half
bad_splitter = RecursiveCharacterTextSplitter(chunk_size=120, chunk_overlap=0)
bad_chunks = bad_splitter.split_documents([doc])
for c in bad_chunks:
    print(repr(c.page_content))

Run it. Find the chunk boundary that falls inside "Flood damage ... except in basement units". Ask a retriever for "is flood damage covered?" against only that chunk, and it will answer yes, missing the basement exclusion three words later in the next chunk.

Fix it: size and overlap

good_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,      # big enough to hold a whole clause
    chunk_overlap=60,    # copies the tail of one chunk into the next
)
good_chunks = good_splitter.split_documents([doc])

chunk_size sets the ceiling on how big one retrievable unit is. chunk_overlap repeats the last N characters of one chunk at the start of the next, so a clause that starts near a boundary still appears whole in at least one chunk. Bigger overlap costs storage and slightly slower search; it does not cost accuracy. There is no universal correct number, the right size is "big enough to hold your longest clause, small enough to stay specific" — for insurance policy sections, 300-500 characters with 15% overlap is a reasonable starting point, tuned per document once you see real chunk boundaries fail.

Retention practice

Run the bad splitter on critical-illness-rider-kr.txt too. Nothing about chunking cares that the text is Korean, it splits on character count same as English. That is a deliberate setup for Lesson 3: chunking survives the language, embedding does not.

Prove it stuck

A clause keeps starting one character before a chunk boundary. What actually fixes this?

Why does the Korean clause chunk the same way the English clause does?

Why set chunk_size 120 to deliberately break the flood clause?

Read this next (primary source)

★ LangChain's RecursiveCharacterTextSplitter guide. ~10 min. Covers the splitter you just ran plus the separator hierarchy it uses (paragraph, then sentence, then word).

Ask your teacher. Try: "show me a chunk boundary failure in my own corpus text", "what chunk_size fits a 2-page policy document?", or "how does the separator list in RecursiveCharacterTextSplitter actually work?"

← Lesson 1: Pipeline anatomy Next → Lesson 3: Embed, store, retrieve