Since updating our LangGraph deployment to version 0.2.0, we've observed a persistent and problematic behavior where a significant subset of our asynchronous workflows are entering a permanent 'pending' state upon their first node execution. This is not an intermittent latency issue; the graphs remain in this state indefinitely, never progressing to completion or error, effectively becoming dead letter workflows. This is occurring in a production environment with graphs that were fully operational under the 0.1.x series.
Our initial investigation points toward the new checkpointing system and the interaction with our custom state schema validators. Specifically, graphs that utilize a `StateGraph` with a Pydantic `BaseModel` as the state spec seem to be the ones affected. The checkpoint is created, but the first node's execution result appears not to be committed, leaving the graph in a state of limbo.
Here is a simplified version of our graph construction pattern that is now failing:
```python
from langgraph.graph import StateGraph, END
from pydantic import BaseModel
from typing import TypedDict, Annotated
from typing_extensions import TypedDict
import operator
class AgentState(TypedDict):
messages: Annotated[list, operator.add]
query: str
processed: bool
class ValidationState(BaseModel):
messages: list
query: str
processed: bool = False
def process_node(state: AgentState):
# ... logic
return {"processed": True}
# This works pre-0.2.0, now hangs in 'pending'
builder = StateGraph(AgentState)
builder.add_node("process", process_node)
builder.set_entry_point("process")
builder.add_edge("process", END)
graph = builder.compile()
# This pattern with Pydantic also exhibits the issue
builder2 = StateGraph(ValidationState)
builder2.add_node("process", process_node)
builder2.set_entry_point("process")
builder2.add_edge("process", END)
graph2 = builder2.compile()
```
We are invoking these graphs asynchronously via the `ainvoke` method. The checkpoint viewer shows an initial checkpoint, but no subsequent updates.
**Questions for the community:**
* Is anyone else experiencing a similar lock-up with asynchronous execution post-0.2.0?
* Has the required signature or return format for node functions changed in a way that isn't fully documented? Our nodes return a dictionary update to the state, which was the established pattern.
* Could this be related to the new "threading" model and the way the checkpoint verifiers are interacting with the state schema? We've tried disabling checkpoints, but the issue seems to persist.
* Are there any confirmed workarounds—such as reverting to a specific configuration pattern or explicitly defining a different checkpoint writer—that have restored flow for others?
We've temporarily rolled back to 0.1.22, but we need to understand the root cause to plan our upgrade path. Any insights into configuration changes or undocumented breaking changes in the state management layer would be greatly appreciated.
- Mike
- Mike
Classic. "Fully operational" until you upgraded a minor version in production.
That Pydantic state spec is probably hitting a silent serialization error with the new checkpoint writer. The 0.2.0 release notes mentioned changes to the state persistence layer, didn't they? Your validator might be rejecting the initial node's output because the checkpoint format changed, but the error is being swallowed.
Try running it with a simple `TypedDict` instead of the `BaseModel` for a quick test. Bet it moves.
Keep it simple
You're likely correct about the serialization mismatch, but switching to `TypedDict` could be treating the symptom, not the cause. The deeper issue is that Pydantic validators are now being applied to the *checkpoint data*, not just the live state object. If your validator includes any field with a default factory or computed property that expects a certain runtime context, it will fail during the deserialization attempt from the checkpoint store, which happens in a different scope.
A more diagnostic approach is to wrap your state model's `model_validate` in a try-except and log the validation error. You'll probably find a `ValidationError` for a field you didn't expect to be validated at that stage. The fix might be to add `validate_default=False` to your Pydantic config or to make certain fields truly optional for the checkpoint loader.
Every dollar counts.
The checkpoint creation you're observing is likely a red herring. The issue is that the state object fails to materialize for the first node's execution because the checkpoint's `channel_values` are now being validated *before* the node's function receives them. Your Pydantic model's root validators or validators on fields with a `default_factory` are probably the culprits, as they're executing in an environment where required context (like a session or client) is `None`.
To confirm, add this to your state model's config: `revalidate_instances='never'`. If the graph progresses, you've identified the validation phase as the bottleneck. You'll then need to audit your validators for any side effects or context dependencies.
You're spot on about the checkpoint being a red herring. The graph isn't stuck *creating* it, it's stuck *validating* the state it tries to load from it. Your suggestion about `revalidate_instances='never'` is a good diagnostic, but that's basically turning off a core feature of your Pydantic model for production, which feels like a band-aid.
If that does work, the real fix is probably more surgical: audit for any `@model_validator` or `@field_validator` that performs an external call or expects an attribute that doesn't exist in the deserialization context. I've seen this blow up silently when a validator tries to, say, initialize a database connection from a field that's only populated later in the runtime.