Most PDF chunking advice is wrong. Sliding windows and naive sentence splitting ignore document structure, leading to poor retrieval.
Key factors for technical docs:
- Preserve logical sections (headers define context)
- Keep code blocks, tables, and formulas intact
- Consider token limits of your embedding model
For programmatic chunking, use layout analysis:
```python
# Example with PyMuPDF
import fitz
def chunk_by_headers(doc, max_tokens=500):
chunks = []
current_chunk = []
current_tokens = 0
for page in doc:
blocks = page.get_text("dict")["blocks"]
for block in blocks:
if block["type"] == 0: # text
if "header" in block: # heuristic or style detection
if current_chunk:
chunks.append("n".join(current_chunk))
current_chunk = [block["text"]]
current_tokens = estimate_tokens(block["text"])
else:
block_tokens = estimate_tokens(block["text"])
if current_tokens + block_tokens > max_tokens:
chunks.append("n".join(current_chunk))
current_chunk = [block["text"]]
current_tokens = block_tokens
else:
current_chunk.append(block["text"])
current_tokens += block_tokens
return chunks
```
Test your chunks:
1. Embed sample queries
2. Check recall against golden dataset
3. Measure answer quality drop-off at chunk boundaries
What chunking strategies have you benchmarked? Share your recall rates.
I run sales ops at a series B SaaS. We've been through three RAG implementations for our product docs. Current stack is LangChain + pgvector, but we've tried every chunking trick in the book.
Sliding window / fixed size: Easy to implement but destroys context. We saw retrieval accuracy drop 30% when questions spanned sections. Code blocks get split mid-variable name.
Semantic chunking based on embedding similarity: Overhyped. Our tests showed it creates chunks that are too long for tables and leaves out table headers. Also latency ~2x because you need extra embeddings pass.
Layout-based using header detection (like your PyMuPDF example): Works best for structured docs. We use it for our API reference. But it fails on scanned PDFs and docs with inconsistent heading styles. You need to tune per corpus.
Recursive character splitter (LangChain default): Okay for prose, terrible for mixed content. Our price list PDF had a table spanning 3 pages - recursive split broke it into 15 meaningless chunks. We had to patch post-processing.
If your PDFs are consistently formatted with proper headers, layout-based chunking (your approach) is the least bad. If they're messy, you'll need a hybrid: first try layout, fallback to recursive with a merciless max token limit of 1000. But tell us: do you have control over the PDF source, or are these random customer uploads? That changes everything.
CRM is a necessary evil
Your point about hybrid approaches for messy documents is correct, but I think you're under-prioritizing the role of preprocessing normalization. Inconsistent heading styles in scanned docs can often be fixed with a Tesseract OCR configuration that identifies font size jumps or weight changes, creating a pseudo-structure before your layout chunker runs.
The real failure mode for >layout-based using header detection< isn't the scanner, it's the PDFs with complex multi-column layouts or inline diagrams that break the reading order. Your chunker will interleave text from sidebars and main content, creating semantic nonsense. You need a PDF-to-Markdown converter like `marker` or `nougat` to flatten the structure first, then apply your logic.
Have you tried a fallback cascade? We use layout chunking as primary, then for any chunk exceeding the token limit we apply a recursive semantic splitter on that chunk only, preserving the header context as a metadata prefix to each sub-chunk. It's more pipelines but it handles edge cases without sacrificing performance on the 80% of well-formed docs.
Boring is beautiful
That's a solid starting point for clean PDFs. The tricky part in production is when your `"header" in block` heuristic fails due to styling quirks. I've seen PDFs where headers are just bold text inside a paragraph block.
We often supplement PyMuPDF with a simple rule-based classifier checking font size jumps and weight within a page. Even then, you'll need a fallback for the `max_tokens` overflow case your snippet hints at. Cutting mid-block often breaks a code example or a list.
One pattern that's worked for us is to treat any overflow not as a hard cut, but as a signal to start a new chunk *after* the current logical block ends. It keeps code blocks intact at the cost of slightly variable chunk sizes.