What Are You Really Embedding?
Notes on chunking before embeddings in Retrieval-Augmented Generation systems
I used to think about chunking as just a pre-processing step in building a RAG system. Literally, a thing you did to break up your data into sizable pieces. You take a document, split it into smaller pieces, embed them, and move on.
But that framing hides what is actually happening.
In retrieval systems like RAG, chunking is not just preparation. It is part of the retrieval design itself.
It decides what the system is even capable of retrieving later.
Chunking changes what “findable” means
When you embed a document, you are not storing the document anymore. You are storing representations of chunks of meaning.
That means the unit you choose becomes the unit your retrieval system understands. If a single idea is split across multiple chunks, retrieval becomes fragmented. If multiple ideas are packed into one chunk, retrieval becomes noisy.
So chunking is not neutral. It defines the shape of the search.
Why equal-sized chunking breaks down
A common approach is to split text by fixed size, characters or tokens. It is simple, predictable, and easy to implement. But it assumes something that is not true.
It assumes meaning is evenly distributed across the text.
In practice:
One idea can span multiple paragraphs
One paragraph can contain multiple ideas
Headings often define boundaries better than length does
So fixed chunking optimises for structure, not meaning. It produces uniform chunks, but not necessarily useful ones.
Semantic chunking is not a technique; it is a constraint
Semantic chunking is often described as splitting by meaning or topic. But the real shift is not how you split text. It is what you refuse to break apart.
You are no longer asking:
“How do I divide this document evenly?”
You are asking:
“What should stay together if I want retrieval to work properly?”
That question changes everything.
The real problem you are solving
Once you treat chunking as part of retrieval design, the problem becomes clearer. You are trying to preserve at least one thing:
A chunk should contain enough complete meaning that retrieval does not depend on reconstructing scattered context later.
That introduces constraints:
How much meaning can fit before a chunk becomes too broad to retrieve precisely
How small can a chunk be before it loses meaning entirely
What happens when meaning conflicts with size limits
What you do when splitting improves compliance but damages interpretability
These are not formatting decisions. They are system design decisions.
What this means in practice
If chunking is wrong, retrieval does not fail loudly.
It fails quietly.
You get:
partially correct answers
missing context
Irrelevant matches that look plausible
useful information that never gets retrieved because it was split incorrectly
And the system still looks like it works. That is what makes chunking a design problem rather than a preprocessing step. It sits upstream of everything else, but its effects only show up downstream.
Closing
Chunking is often treated as a technical detail. But in retrieval systems, it is closer to deciding how knowledge is shaped before the system ever sees a query.
Once you see that, the question becomes: What is the smallest unit of meaning your system can reliably retrieve?