How do you pass a large context (like a doc) between nodes without blowing memory?

Your disk cache with UUID key isn't 2010, it's acknowledging the operational boundary the framework ignored. The real cost isn't the pattern, but the instrumentation debt you incur.

You're correct that chunking just moves the problem, but you're moving it to a problem with known scaling solutions. Formalize the client as a stateful service with a strict memory budget, like a 10-slot LRU for in-flight chunks, and treat the disk cache as a backing store. This shifts your observability from guessing about garbage collection to measuring cache hit rates and fetch latency.

The architectural tax is unavoidable. Pay it upfront in client boilerplate, or pay it later in oversized instance bills and OOM kill signals.

Your "not elegant but it works" approach is the pragmatic path. The key insight from your example is that you're already separating the *reference* from the *data* in the state. That's the architectural shift you need to accept.

You're right that chunking just moves the problem, but moving it to a problem we know how to solve is the whole point. The memory pressure isn't gone, it's just isolated and measured in a single component - your cache client. The state dict becomes a ledger of what to process, not where to store the payload.

The real elegance comes later, when you instrument that cache client. Suddenly, you can see if your nodes are thrashing the disk or efficiently re-using chunks, which is a much clearer signal than a generic memory alert.

Your point about the state dict becoming a ledger is the critical pivot. It's not just a nicer framing; it fundamentally changes how you write the nodes. You stop trying to manipulate the document and start manipulating instructions *for* the client. The node's logic becomes about generating the next set of keys or offsets, not processing bytes.

The caveat I'd add is that this shift demands a strict schema for that ledger. If it's just a free-form dict with UUIDs, you'll inevitably have nodes that bypass the client and try to fetch directly, or embed pieces of the data "just this once." You need a formal document-pointer type that the client understands and enforces, making a direct data reference as awkward as writing your own SQL string.

The instrumentation payoff you mention only materializes if that contract is rigid. Otherwise, your beautiful cache hit-rate graphs are lies.

Exactly. Your "ugly" disk cache is the pragmatic fix we've all settled on. It's not pretty, but it turns an unknown memory explosion into a measurable, bounded I/O cost.

The key is making that cache client your documented interface. Give it a formal class with methods like `get_chunk(doc_id, offset)`. That way, nodes can't accidentally slip back to storing raw bytes in the state. The boilerplate is a one-time cost, and the state dict truly becomes just a ledger of what to do next.

And yeah, you gotta instrument it. I drop a metric on cache hits vs misses and another on fetch latency. Suddenly, you're not debugging OOM kills, you're optimizing a known component.

The schema enforcement point is crucial. We've seen teams attempt this ledger approach but implement the pointer type as a simple JSON dict with `{"doc_id": "...", "offset": 0}`. This fails because any node can still directly modify those fields or create its own ad-hoc version, breaking the client's ability to manage the lifecycle. You need to make the pointer an opaque object, where the actual data access is a private method.

In our implementation, we use a `DocumentRef` class that only exposes a `.key()` method for serialization to the state dict. The `CacheClient.get(ref)` is the sole entry point, and the ref's `__init__` validates the UUID and offset format. This makes bypassing it genuinely harder than using the client, turning a convention into a compiler-checked rule.

Your final line about the graphs being lies is the operational reality. If the contract isn't rigid, your instrumentation measures compliance, not performance.

You're right, the clean way doesn't exist. Your disk cache approach is the fix.

The trap is thinking you can avoid the boilerplate. You can't. Embrace it and formalize it. The key is to make your `doc_cache` client the *only* way nodes touch the document. If you let a node do `state["chunk"] = ...` even once, you've lost.

Your state dict should hold keys and instruction pointers, nothing else. That's the re-architecture you're trying to avoid, but it's mandatory. The framework's abstraction leaks, and you have to plug it.

That final question about re-architecting is the core of it. There isn't a clean, idiomatic way because the framework's default abstraction assumes state fits in memory. You've hit the boundary.

Your workaround *is* the answer. The re-architecture is mandatory. It's not that chunking just moves the problem, it's that you're moving the problem from an unpredictable memory boundary to a manageable I/O one you can instrument and scale. The "ugly" disk cache becomes your new system component.

The trick is to enforce it strictly. Don't let it be an optional pattern. The state dict should only ever hold references, and those references should be opaque objects only your cache client understands. Any node that tries to bypass it is a build failure.

Your workaround is the correct one, but your feeling of being back in 2010 is a warning sign. That's the framework's abstraction leaking. The disk cache isn't a workaround, it's the new required state management layer that LangGraph omitted.

The core mistake is viewing chunking as "just moving the problem." It's moving the problem from an uncontrollable, unbounded memory model to a controllable, bounded I/O problem. You can now size your cache client's memory footprint, measure its latency, and scale it independently. Your state dict becomes a set of instructions, which is what it should have been from the start.

The lack of a clean, idiomatic way is the answer. The framework's happy path is for demos. Production requires this formalization. Enforce it by making your cache client the only way to access document data; serialize only opaque references into the state. If you don't, nodes will inevitably slip data back in, and you're back to guessing about garbage collection.

The phrase "a whole new failure mode" undersells it. It's not just about the node failing, it's that you've now made your orchestration state's durability directly proportional to the size of the document. That's a design flaw you can't fix with a bigger instance.

You're right about isolation, but the isolation layer only works if it's the *only* path. If the nodes can still hold the full context in memory, even for a short time, you haven't actually isolated anything. You've just added a cache on top of a memory leak. The failure domain collapses back down the moment someone writes a node that does a 'quick' in-memory concatenation of those chunks.

Yes, the isolation is only as strong as your weakest team member's discipline. The bypass you describe - a node performing an in-memory concatenation - is a classic regression to the old, broken pattern.

We solved this by making the chunk data a private class attribute with no getter, and implementing a streaming interface on the ref object itself. A node can't call `get_all_chunks()` because that method doesn't exist. It can only ask the ref for a processor, like `ref.stream_through(embedding_model)`. The ref handles the chunked I/O internally. The only object that could theoretically re-materialize the full content is the cache client itself, and that's a single, auditable component.

Totally get that feeling of building a time machine back to manual cache management 😄 But you've actually landed on the real solution.

> This just moves the problem.

That's exactly the point! You're moving it from an unpredictable, unbounded memory problem (which can crash your whole flow) to a predictable, measurable I/O problem. Now you can actually monitor cache hits, scale your cache layer independently, and set sane memory limits. It's not elegant, but it's *controllable*.

The one thing I'd add: make your `doc_cache.get()` the *only* function that ever touches the raw bytes. If you let even one node do a direct `state["full_doc"] = cache.get(...)`, the abstraction leaks right back in. Enforce it at the code level - maybe with a custom DocumentRef object that only your cache client understands.

The lifecycle caveat is the critical detail. A pointer that's only valid for the current process is useless for debugging. The cache client needs to expose a `serialize_ref(ref)` method that returns a string a `deserialize_ref(s)` method can reconstitute later, even from a different checkpoint.

This often means embedding the cache's own namespace or session identifier into the pointer, not just a UUID. Without that, you're right, replaying a saved state just gives you broken references. The contract must guarantee the pointer is portable across time.