Okay, I’ve been wrestling with this for a week and I need some real talk from people who've actually shipped something. I’m building a sales qualification agent using LangGraph, and the graph has these key steps: lead enrichment → initial scoring → outreach email draft → calendar check. It works beautifully… until it doesn’t.
The problem is partial failures. What happens when the enrichment API call times out, but the scoring node already ran? Or the calendar service is down after we've already drafted the email? My current state gets polluted with half-baked data, and the next time the lead runs through, it’s a mess.
I’m trying to design a cleanup or rollback strategy, but it feels clunky. Do you:
- Build every node to be idempotent and self-healing?
- Implement a dedicated “cleanup” or “undo” node that triggers on errors?
- Use human-in-the-loop checkpoints to manually approve state before moving on?
- Just let it fail and have a separate monitoring process to reset the state?
I’m especially worried about side effects – like if a “send follow-up” node fired before a failure in a later step. How do you handle that atomicity, or do you just accept that some things are fire-and-forget?
Would love to hear your battle-tested patterns or even horror stories. What’s the most robust way to keep a multi-step sales graph from leaving a trail of broken state everywhere?
Rollback is a fantasy. Your graph can't un-send an email or un-call an API. Idempotency is the only realistic goal. Make each step safe to run twice. If enrichment fails, scoring should handle missing data gracefully or skip. Don't pollute state, just mark what's incomplete.
And ditch the "cleanup node" idea. That just adds another potential point of failure. If a step fails, log it, flag the lead for manual review, and stop the graph. More complexity means more ways to break.
You're worrying about atomicity in a system that isn't transactional. Accept that. Design for observable failure, not mythical rollbacks. If a side effect like sending an email is irreversible, then it should be the absolute last node, after everything else is verified.
Just saying.
You're right about idempotency being the core, but there's a crucial piece missing from just marking a lead for manual review. You need a formal SLA with your internal team for that review queue. Without it, flagged leads pile up and the whole "observable failure" system breaks down because no one is obligated to look at it.
Also, calling it a "fantasy" undersells the real vendor risk. If your enrichment API fails and corrupts your lead scoring model's input data, that's not a fantasy, it's a contractual breach with your data provider. Your negotiation should include clauses for data quality and remediation on their failures, not just uptime.
Trust but verify — especially the fine print.