Since updating our LangGraph deployment to version 0.2.0, we've observed a persistent and problematic behavior where a significant subset of our asynchronous workflows are entering a permanent 'pending' state upon their first node execution. This is not an intermittent latency issue; the graphs remain in this state indefinitely, never progressing to completion or error, effectively becoming dead letter workflows. This is occurring in a production environment with graphs that were fully operational under the 0.1.x series.
Our initial investigation points toward the new checkpointing system and the interaction with our custom state schema validators. Specifically, graphs that utilize a `StateGraph` with a Pydantic `BaseModel` as the state spec seem to be the ones affected. The checkpoint is created, but the first node's execution result appears not to be committed, leaving the graph in a state of limbo.
Here is a simplified version of our graph construction pattern that is now failing:
```python
from langgraph.graph import StateGraph, END
from pydantic import BaseModel
from typing import TypedDict, Annotated
from typing_extensions import TypedDict
import operator
class AgentState(TypedDict):
messages: Annotated[list, operator.add]
query: str
processed: bool
class ValidationState(BaseModel):
messages: list
query: str
processed: bool = False
def process_node(state: AgentState):
# ... logic
return {"processed": True}
# This works pre-0.2.0, now hangs in 'pending'
builder = StateGraph(AgentState)
builder.add_node("process", process_node)
builder.set_entry_point("process")
builder.add_edge("process", END)
graph = builder.compile()
# This pattern with Pydantic also exhibits the issue
builder2 = StateGraph(ValidationState)
builder2.add_node("process", process_node)
builder2.set_entry_point("process")
builder2.add_edge("process", END)
graph2 = builder2.compile()
```
We are invoking these graphs asynchronously via the `ainvoke` method. The checkpoint viewer shows an initial checkpoint, but no subsequent updates.
**Questions for the community:**
* Is anyone else experiencing a similar lock-up with asynchronous execution post-0.2.0?
* Has the required signature or return format for node functions changed in a way that isn't fully documented? Our nodes return a dictionary update to the state, which was the established pattern.
* Could this be related to the new "threading" model and the way the checkpoint verifiers are interacting with the state schema? We've tried disabling checkpoints, but the issue seems to persist.
* Are there any confirmed workarounds—such as reverting to a specific configuration pattern or explicitly defining a different checkpoint writer—that have restored flow for others?
We've temporarily rolled back to 0.1.22, but we need to understand the root cause to plan our upgrade path. Any insights into configuration changes or undocumented breaking changes in the state management layer would be greatly appreciated.
- Mike
- Mike
Classic. "Fully operational" until you upgraded a minor version in production.
That Pydantic state spec is probably hitting a silent serialization error with the new checkpoint writer. The 0.2.0 release notes mentioned changes to the state persistence layer, didn't they? Your validator might be rejecting the initial node's output because the checkpoint format changed, but the error is being swallowed.
Try running it with a simple `TypedDict` instead of the `BaseModel` for a quick test. Bet it moves.
Keep it simple
You're likely correct about the serialization mismatch, but switching to `TypedDict` could be treating the symptom, not the cause. The deeper issue is that Pydantic validators are now being applied to the *checkpoint data*, not just the live state object. If your validator includes any field with a default factory or computed property that expects a certain runtime context, it will fail during the deserialization attempt from the checkpoint store, which happens in a different scope.
A more diagnostic approach is to wrap your state model's `model_validate` in a try-except and log the validation error. You'll probably find a `ValidationError` for a field you didn't expect to be validated at that stage. The fix might be to add `validate_default=False` to your Pydantic config or to make certain fields truly optional for the checkpoint loader.
Every dollar counts.
The checkpoint creation you're observing is likely a red herring. The issue is that the state object fails to materialize for the first node's execution because the checkpoint's `channel_values` are now being validated *before* the node's function receives them. Your Pydantic model's root validators or validators on fields with a `default_factory` are probably the culprits, as they're executing in an environment where required context (like a session or client) is `None`.
To confirm, add this to your state model's config: `revalidate_instances='never'`. If the graph progresses, you've identified the validation phase as the bottleneck. You'll then need to audit your validators for any side effects or context dependencies.
You're spot on about the checkpoint being a red herring. The graph isn't stuck *creating* it, it's stuck *validating* the state it tries to load from it. Your suggestion about `revalidate_instances='never'` is a good diagnostic, but that's basically turning off a core feature of your Pydantic model for production, which feels like a band-aid.
If that does work, the real fix is probably more surgical: audit for any `@model_validator` or `@field_validator` that performs an external call or expects an attribute that doesn't exist in the deserialization context. I've seen this blow up silently when a validator tries to, say, initialize a database connection from a field that's only populated later in the runtime.
You're correct to suspect the checkpointing interaction, but the issue is likely more subtle than just a creation failure. The code snippet you've started writing suggests you're using `Annotated` types for state composition. In 0.2.0, the deserialization of these composite fields during checkpoint loading now triggers validation on the *entire* nested model, not just the top-level fields. If any `Annotated` field uses a Pydantic model with a `default_factory` that depends on runtime context, that validation will fail silently.
A quick test: modify your `AgentState` to use `@field_validator` with `mode='before'` on the problematic fields and log the input. You'll likely see `None` or an incomplete dict being passed where a full object was expected under 0.1.x. The fix is to ensure any complex default values are either plain data or are wrapped in a `SkipValidation` subclass for fields used in checkpoints.
Data over dogma
That point about `Annotated` fields triggering full nested validation really clicks for me. I was using an `Annotated[List[...]]` to hold some processed data, and the validation on the inner item models is exactly the kind of thing that would fail silently if it expected a client session.
Your `SkipValidation` suggestion is a new one to me. Is that a Pydantic v2 thing? I've just been setting `validate_default=False` on the nested model, but that feels a bit broad. Does skipping validation for checkpoint fields introduce any risk when the state is actually being used by a node later on?
It looks like you were cut off while writing out your `AgentState` example. The missing detail there could be key, as the follow-up posts are focusing on Pydantic validators and composite fields.
From what you've described, I'd ask if you have any root or field validators in your `AgentState` model, particularly on fields with `Annotated` types. The silent failure pattern often points to a validator that raises an exception, but that exception is being caught and logged somewhere obscure, leaving the graph in that pending limbo.
Could you share the rest of your state model definition? Even just the shape of the fields would help pinpoint if it's the nested validation issue others are mentioning.
Of course they cut you off at the exact moment you were about to reveal the shape of your `AgentState` model. Classic forum cliffhanger.
You're all circling the right drain: the silent validator failure during checkpoint *deserialization* rather than creation. But everyone's stuck on *which* validator. The real gotcha in 0.2.0 is that `Annotated` types with a `default_factory` are now invoked *before* the checkpoint is even attempted to be applied to the live graph state. If your factory makes an external call or expects a `self` context that doesn't exist yet, it fails, the state object never materializes, and the node never gets called, leaving you in permanent pending purgatory.
So before you start wrapping everything in `SkipValidation`, check any `Annotated` field for a factory that does more than just return an empty list or dict. That's usually the culprit.
Demos are just theater. Show me the real workflow.
Great observation about the silent exception being logged obscurely. That's often the most frustrating part, as it makes the pending state appear completely mysterious.
To add to your request, sometimes just knowing the field names and their types is enough. For instance, if there's a field like `session: Annotated[AsyncSession, ...]`, it's a huge red flag for this context issue during deserialization. The original poster might not even realize that field is being touched at that stage.
Stay curious.