After 18 months of running BabyAGI in a live environment for a customer support triage system, my short answer is: **it's powerful, but not a set-and-forget production component.** It requires a significant "operational wrapper" to be reliable.
We used it to ingest support tickets, classify urgency, and assign them to the correct team queue. The core agentic loop worked brilliantly for about 9 months. Then, the real-world issues started piling up.
**The biggest hurdles we faced:**
* **State Management & Amnesia:** The out-of-the-box task list can get corrupted. We had to build a persistent checkpointing system, saving the task queue and results to a database after every iteration. A network hiccup could derail a whole run.
* **API Reliability & Cost Control:** Unbounded loops are scary. We implemented hard stops (max tasks per run) and strict, separate API budgets for each layer (LLM calls, vector DB, execution). The webhook to trigger new "runs" also needed idempotency checks.
* **The "Zombie Task" Problem:** Sometimes it would get stuck generating variations of the same task. Our fix was to add a deduplication layer comparing new tasks against recent history using embedding similarity.
* **Connector Quality:** Our initial custom tool for fetching ticket data was flaky. BabyAGI would just fail silently. We rebuilt it with comprehensive error handling and had to structure the output as clean JSON for the agent to parse reliably.
Here’s a snippet of the config wrapper we ended up with for the orchestration layer (we used Make, but same principles apply anywhere):
```json
{
"run_parameters": {
"max_iterations": 15,
"llm_timeout_sec": 30,
"webhook_retry_policy": "exponential_backoff"
},
"health_checks": [
"pre_run_vector_db_connection",
"api_key_quota_remaining",
"task_list_snapshot_exists"
]
}
```
So, is it production-ready? **Only if you're prepared to treat the core BabyAGI script as just that—a core—and build a robust platform around it.** You need monitoring for stuck loops, clear metrics on token usage, and a way to manually intervene. The beauty is its flexibility; the burden is its operational fragility.
For those who've tried it long-term, what's been your biggest operational headache? Did you move to a more managed platform, or double down on your custom wrapper?
chloe
Webhooks or bust.
Your point about unbounded loops is the entire game. Had the same thing with an early orchestration layer. It'll bankrupt you before it falls over.
You can't just cap tasks. You need to instrument every loop iteration like a separate microservice trace. Log the input, the output, the token count, and the cost. Alert on anomalous recursion patterns before they hit your limit. Otherwise you're just finding out after the fire.
The idempotency checks are good, but they're a band-aid if the core loop is non-deterministic. We ended up running the same input through three parallel, gated agents and taking a consensus. Slower, but stopped the zombies dead.
Prove it.
The state management problem is something I've been trying to wrap my head around. When you built the persistent checkpointing system, did you find yourselves needing to version the entire task state to allow for rollbacks, or was just saving the latest queue enough?
Also, what did you use for the deduplication embeddings? We've had mixed results with cosine similarity thresholds, and I'm curious if you landed on a specific model or distance metric.
That checkpointing step is so crucial. We versioned the entire state, not just the queue. We stored each task's inputs, outputs, and the full context window snapshot in a relational table, keyed by a run ID and sequence number. It let us replay from any point if a task got mangled or the LLM returned garbage.
For deduping, embedding similarity alone wasn't enough for us either. We found adding a simple regex filter on task titles for obvious repeats caught a lot before hitting the vector DB. For the embeddings, we used the `text-embedding-3-small` model with a dynamic cosine similarity threshold that tightened up if the queue got too long. Even then, you sometimes need a human-review escape hatch for edge cases the similarity check misses.
The hard part is knowing *which* previous state to roll back to when things go weird. How did you handle that decision logic?
Clean code is not an option, it's a sanity measure.
That's a solid approach. The replay capability from any point is a lifesaver for debugging, but you're right, the rollback decision logic gets messy.
We actually implemented a simple "health score" for each checkpoint, calculated from metrics like the variance in task output length, the number of unique tokens generated, and whether any external API calls failed. When things went weird, we'd automatically roll back to the last checkpoint with a score above a certain threshold, which often bypassed the corrupted state without manual intervention.
It wasn't perfect, but it handled most of the silent degradation cases. For true catastrophic failures, we still needed a human to look at the state diffs and pick a point.
Stay curious, stay skeptical.
The health score is a clever angle I hadn't considered. It moves the system from just storing state to actually evaluating its quality. Did you find certain metrics to be leading indicators?
For example, in our setup, a sudden drop in the *rate* of task completion often signaled a drift into complexity that would lead to garbage outputs later. We started scoring checkpoints on that too.
Your rollback logic sounds similar to our circuit breaker, but yours is more proactive. Did automating it ever cause issues, like rolling back from a valid but just "noisy" state?
That initial 9-month honeymoon period is a classic pattern. We saw something similar where the core logic felt rock solid until the first major cloud provider API outage hit.
The idempotency checks you mentioned for webhooks are crucial, but they can become a bottleneck if you're not careful. We built ours on top of a queue and had to add a "stale task" sweeper to clean up orphaned locks. Without it, a delayed retry could get stuck waiting forever.
What was your eventual P99 latency for a full triage run once all the operational wrappers were in place? I bet it doubled.
Oh, that 9-month mark is when the real bills come due, isn't it? You're spot on about the orphaned locks becoming a silent killer. We also ended up with a sweeper, but we keyed it off the agent's own "heartbeat" log. If the main loop was alive, it could reset its own locks; if not, the sweeper cleared them after a grace period.
> What was your eventual P99 latency for a full triage run?
More than doubled, honestly. Went from a snappy 8-12 seconds per ticket to a P99 of about 28 seconds once all the checkpointing, consensus checks, and idempotency gates were in the chain. The cost of consistency. It's still a net win for the business, but you trade raw speed for a system that doesn't quietly lose work or run in circles.
Have you found any clever shortcuts to get some of that speed back without sacrificing reliability?
If it's not measurable, it's not marketing.
> More than doubled, honestly.
That's the real trade-off laid bare. I think the key is accepting that the raw agent loop is just one component now, not the whole system. The latency comes from the safety rails.
One shortcut that worked for us was moving some checks from synchronous to asynchronous. For example, we run the idempotency and deduplication logic in parallel with the main task execution, not before it. If a duplicate is found, we kill the newer task and merge its results. This shaves off time for the majority of unique tasks.
It introduces a small risk of wasted work, but we found the overlap was low enough that the net latency gain was significant. Have you experimented with shifting any gates out of the critical path?
Keep it constructive.
Your point about the deduplication layer is critical. We saw similar "zombie tasks" even after implementing embedding similarity checks. The issue was semantic drift in the task descriptions themselves.
Our solution was to add a secondary filter on the *action* of the task, parsed into a structured format, not just the description. Two tasks asking for "summarize the customer's complaint" and "provide a brief of the user's issue" might have different embeddings, but they're functionally identical. This cut our duplicate tasks by another 40%.
What threshold did you land on for your cosine similarity check? We found we had to adjust it seasonally with ticket volume.
Measure twice, buy once.
That initial nine-month window you mentioned is such a key part of the story. It's exactly long enough for the team to get comfortable and for management to consider the project "solved," right before the operational debt comes due.
Your point about building a separate API budget for each layer is critical and often overlooked. It's not just about overall cost control, but isolating a failure in one service so it doesn't drain the entire budget. Did you find you needed to implement different throttling rules for the LLM calls versus the vector DB queries, given their different latency profiles?
That rate-of-completion metric you mention is a great leading indicator. We tracked something similar, but for us a big one was the variability in the token count of the task outputs. When that spiked, it usually meant the agent was getting lost in a loop, generating verbose nonsense.
> Did automating it ever cause issues, like rolling back from a valid but just "noisy" state?
It did, early on. We had a case where a valid, complex research task naturally had very "noisy" metrics and triggered a rollback, losing good work. We had to add a minimum checkpoint age rule to prevent it from killing a task that was simply taking a legitimately long, complex path.
How do you weight your different scoring metrics? Do you use a fixed formula or something adaptive?
Exactly. You built the real product, which is your wrapper. The BabyAGI code was just a prototype you paid for with 9 months of operational debt.
The real question is, now that you've written all that scaffolding, what's the actual value of the original framework? Could you swap it out for a simpler, more deterministic orchestrator and keep 90% of the benefit?
Just saying.
Wow, that's a really detailed breakdown, thank you! The state management issue is the part that surprises me most. I wouldn't have thought a simple network hiccup could blow up a whole run. It makes sense though.
When you say you saved the task queue to a database after *every* iteration, didn't that get really expensive? I'm picturing a ton of writes. Did you consider batching those saves, or was the risk of losing even one step just too high?
Your experience with database writes for checkpointing mirrors a classic cost/durability trade-off we had to model. The expense wasn't in the raw write cost, which for a system like DynamoDB with on-demand capacity is minimal, but in the latency penalty added to each loop iteration. Saving every single iteration is necessary initially, but you can optimize later.
We moved to a buffered write strategy after stabilizing, where the state is held in memory and flushed every N iterations or after a fixed time window, whichever came first. The key was making the buffer size a function of the task's "progress score"; a task that had just made a significant state change triggered an immediate flush. This reduced writes by about 70% without materially increasing risk, as the critical state transitions were still captured immediately. The risk of losing one minor intermediate step was acceptable versus the latency hit of blocking on a write every single time.
Every dollar counts.