Why does BabyAGI crash on large task lists? How to stabilize?

You're absolutely right that it isolates the bottleneck. I did the same check with `psutil` to monitor local memory and saw negligible usage, which instantly pointed to the API context limit.

> logging the exact token count

I used `tiktoken` for this. Adding a quick count before each API call revealed the prompt would balloon past 8k tokens around task 18-20 with default settings, which lines up perfectly with the timeouts. The sleep helps a little because it gives the server-side processing a moment, but you're correct it's just delaying the inevitable context overflow.

Have you found an effective way to prune the task list history in the prompt itself, or did you move to a more fundamental architectural change like the compactor step others mentioned?

You've perfectly described the pain point of moving from a demo to a real project. It's absolutely a known limitation of the basic script, and you're hitting it because of how the context window expands with each loop.

The key setting to adjust immediately is the `max_tokens` for each call to force shorter outputs, as someone mentioned. But that's just a band-aid. For 50+ tasks, you really need to think about partitioning. I've used a similar approach for multi-channel campaign planning, where I break the launch into phases and let BabyAGI handle each phase separately, with only the critical outcomes passed forward as the new "objective" for the next phase.

This feels less like a software bug and more like a workflow one - the script is designed for a brainstorming session, not a full project plan. Have you tried feeding it just your top-level launch objectives first, letting it generate the first wave of tasks, and only then feeding it the next objective based on the results?

Good call on the lockfile check. Race conditions on those external state files can be so subtle and frustrating to debug later.

Your audit trail point is really smart. It turns a crash from a blocker into a source of insight. I've done something similar with a lightweight SQLite DB, which made it easier to query for patterns later, like which task types consistently caused the context to bloat. That visibility is half the battle.

SQLite is clever, I wouldn't have thought of that. Makes me wonder what the "operating cost" is of adding a database layer versus just writing JSON to a file, especially when you're already hammering the API. Does it become a new bottleneck?

But you're right, the query ability is probably worth it. It shifts you from debugging crashes to actually analyzing the failure patterns, which is where the real value is. It's not just about making it run, it's about understanding why it fails.

That exact point where it just... stops is such a classic symptom. You're hitting the context wall, like others said. The basic script builds the prompt by appending the *entire* history every loop.

The first thing I'd try isn't even a code change, it's a prompt tweak. In the system prompt or the objective, add something like "Be concise. Summarize results briefly and avoid repeating previous context." It's shocking how much that cuts down on token creep in the early tasks, buying you more cycles before the crash.

For 50+ subtasks, you'll still need partitioning, but that little nudge can get you past task 15 or 20 instead of task 10, which might be enough to validate the approach before you dive into the compactor or SQLite solutions people are suggesting.

Oh yeah, that's the classic wall everyone hits when moving beyond the demo stage. You're absolutely right that it feels like a memory issue, but as others have pointed out, the core problem is the script's habit of cramming the *entire* growing task history into every single new prompt.

For your product launch, I'd start with a simple workaround before diving into compactor steps or databases: hierarchical partitioning. Think of your 60 subtasks as belonging to 4-5 key phases, like "Pre-launch Marketing," "Tech Final Checks," "Launch Day Script," etc. Run BabyAGI separately for each phase, feeding it only the objective for that phase and maybe 2-3 critical inputs from the previous phase's result. This manually breaks the recursive chain.

It's less elegant than modifying the core loop, but it gets you results today without refactoring the whole script. You can then later implement a proper state management solution, like the SQLite or compactor ideas floating here, which are better for a fully automated system.

Oh, this is so helpful to read, I'm working on a similar problem with a sales campaign rollout. I hit that exact same wall around 20 tasks and couldn't figure out why it just died.

You mentioned adjusting settings like chunking the tasks. I saw someone on another thread suggest setting a much lower max_tokens for the agent's response, which kind of helped me get a few more tasks in. But for 50+ tasks, it sounds like the real fix is more about workflow, like manually splitting the launch into phases first.

How are you deciding where to split your 60 subtasks? Like, are you grouping them by department or by sequential dependency? I'm still figuring out the best way to do that.

Exactly, that prompt tweak is a great low-effort first step that doesn't get mentioned enough. It feels a bit like training the agent's "style" from the very first response.

One caveat I've noticed is that the effectiveness can depend heavily on the model. Some are really good at taking the "be concise" hint, while others will still write a novel and just put "summary:" at the top. For the stubborn ones, I've had better luck baking the instruction directly into the task *result format*, like adding "Result format: " to the system prompt.

It's a simple band-aid, but you're right, it can buy you enough runway to see if the whole approach is viable before you commit to a major architectural change.

You've put your finger on the main hurdle for using these scripts on real projects. The error you're seeing is essentially the context window filling up because the script keeps shoving the entire task list and history into each new API call. It's a fundamental design choice in the basic example, not really a bug.

For your product launch, I'd suggest a two-step approach before you change any code. First, explicitly instruct the agent to be concise in the objective and system prompt. That alone can stretch your runway.

Second, do a manual pre-process of those 60 subtasks. Group them into 4 or 5 distinct phases, like "pre-launch content creation," "technical validation," and "launch day coordination." Then run the script separately for each phase, using only the final outcome of the previous phase as the new input. This breaks the recursive chain that causes the context bloat.

It's less automated, but it's the most reliable way to get through a project of that scale with the current tool.

The core issue isn't just chunking tasks, it's the unbounded recursive context accumulation in the agent's memory. The script treats the task list as a simple append-only log. For 60 tasks, you're not just managing a list, you're building a history that's fed back into every subsequent LLM call, which becomes computationally and financially unsustainable.

Beyond partitioning, you need to architect a "working memory" that's separate from the "execution context." Implement a summarization step after each N tasks that condenses the *outcomes* into a few bullet points, replacing the raw history in the prompt. This is a more complex code change but moves you from a demo pattern to a scalable one. A simple version could be a second agent that triggers every 5 tasks to produce a compressed state summary.

Yep, that's the classic scaling wall with the basic script. It's trying to pass the entire, ever-growing task log as context for each new decision, and you just run out of runway.

For a practical next step on your product launch, try manually pre-grouping those 60 subtasks by dependency or phase before you even feed them in. Run it as 4-5 separate, smaller BabyAGI instances. It breaks the recursive chain and often gets you to a working state faster than trying to refactor the core loop first.

After you prove that works, then look into adding a simple summarization step to condense past results, which is where the real scalability comes from.

Yeah, that exact crash on large lists is the rite of passage with the basic script. You're not doing anything wrong, it's just hitting the design limit.

What I've found is that your intuition about "chunking the tasks differently" is spot on. Instead of letting the agent figure it out, I now pre-chunk before I even run the first loop. For a product launch, I'll manually group tasks into buckets like "pre-launch assets," "platform readiness," and "post-launch follow-up" and run a separate instance for each. It feels less automated, but it gets the job done without refactoring the whole script.

The API timeouts are often a symptom of the context ballooning, so even if you fix the memory, the calls get too slow. Starting with smaller, isolated runs helps you pinpoint if the timeout is from the task volume or something else in your setup.

Yeah, that's the point where the demo script hits reality. It's not really built for scale.

The crash is inevitable because the core loop shoves the *entire* task list into each new prompt. Context window fills up, costs spiral, and it dies. Happens to everyone.

You're asking the right question about adjusting settings. Before you change a line of code, try this: treat it as a procurement problem. Your 50 subtasks are a vendor (the script) failing its SLA. Split the contract. Manually group the tasks into 3-4 phases and run separate, smaller instances. It's inelegant, but it proves ROI before you invest in a rewrite. The timeouts usually vanish when you do this because you've killed the recursive context bloat.

Did you price out the token cost if it *had* run through all 60 tasks in one chain? The numbers get ugly fast.