Why does BabyAGI crash on large task lists? How to stabilize?

Yeah, this is a common pain point when scaling the original BabyAGI script. It's not really designed for 50+ tasks out of the box. The main culprits are the agent loops and context window limits.

You'll want to look at two key adjustments. First, reduce the number of tasks returned per "execution" loop. The default often fetches too many, which blows up the context for the LLM call and causes timeouts or memory errors. Try setting it to 3-5 tasks at a time. Second, implement a hard cap on the total number of loops or a timeout to prevent it from running indefinitely on a large list.

Also, pre-process your 60 subtasks into broader parent tasks before feeding them in. Let BabyAGI break those down, rather than giving it the entire granular list from the start. This gives the system a better chance to manage the context.

Totally a known issue. The default script is great for demos but hits walls with real projects.

Two quick fixes that helped me: first, add explicit error handling around the API calls with retries. Those timeouts often come from transient API blips. Second, reduce the `max_tokens` in the completion calls - the default is sometimes too optimistic and causes context length errors as the task list grows.

Also, are you storing the task list in memory? For 60 tasks, I'd start writing results to a simple file or DB after each loop. It's more work but prevents losing everything if it crashes mid-run.

I've hit exactly this when trying to automate our content calendar with 70+ tasks! The core script is a fantastic starting point, but you're right, it struggles with real scale.

Beyond the chunking advice others gave, I'd really look at your initial task prompting. Feeding it "60 subtasks" often overwhelms its reasoning loop. I had better luck providing just 5-7 high-level objectives and letting it recursively break them down. This mimics a more natural project manager thought process and keeps each API call's context manageable.

Also, check if you're using GPT-4? The costs and latency can spiral with that many loops. I wrote a wrapper that logs the state to Airtable after each iteration. It's a bit more work, but it saves the whole run if the API blinks and lets me resume from the last good step. Made a huge difference for reliability. Happy to share a snippet if you're interested!

Ah, the classic "demos beautifully, collapses under real load" vendor archetype, but open-source style. Yes, it's absolutely a known limitation.

Everyone's hit the core issue: the default execution loop is a naive `while True` that assumes infinite stability and context. With 60 subtasks, you're not just making 60 calls - you're making dozens of recursive loops, each call's prompt bloating with the entire growing context of results and tasks. That's a quadratic explosion waiting to happen.

The advice about pre-processing into broader objectives is critical, but let me add a procurement angle: you're essentially doing a total cost of ownership analysis here, just with tokens and time instead of dollars. Running 50+ tasks through the default script, especially on GPT-4, is financially and computationally negligent without guardrails. You need to treat each API call as a line-item with variable cost and failure rate, not a free abstraction. Start by logging the token count and latency of every single call, then you'll see exactly where the wall is.

Exactly this. I've lived that "quadratic explosion" and watched my token budget evaporate.

Your TCO angle hits home. It's why I switched to logging each loop's token count to a simple Google Sheet. Seeing the numbers climb exponentially after just 20 tasks was a real wake-up call. Suddenly, tweaking the chunking size wasn't just about stability, it was about literal cost per loop.

The real fix, in my experience, isn't just adding guardrails to the script. It's accepting that the base version is a brilliant prototype, not a production orchestrator. I ended up wrapping mine in a lightweight FastAPI app that caches the context and enforces a hard stop on total tokens per session. It feels like overkill, but it's the only way I've gotten it to behave predictably on anything resembling a real project.

That point about writing results to a file or DB after each loop is key for resilience. In my setup, I started dumping the task list to a simple JSON file at the end of each agent cycle. This not only saved state on crashes but also gave me a clean audit trail to see where the reasoning was going off the rails with larger task loads.

One caveat I found: if you're using a vector store for the task list, writing to an external file after each loop can introduce a race condition if you're not careful. I had to add a small lockfile check to make sure the script wasn't reading a partially written state on the next iteration.

Yeah, this keeps happening to me too, even with smaller lists of 20-30 tasks. I'm also using the basic repo script.

The API timeouts especially hit me. Do you think it's mostly from the context getting too big, or is there something else like the agent getting stuck in a loop and just timing out? I added a simple sleep(2) between calls, which helped a bit, but not enough for bigger lists.

Also, when you say memory issue, is that on your local machine or from the API side? I'm trying to figure out where my bottleneck really is.

The sleep(2) call is a good instinct, but I think it addresses the symptom, not the root cause. From my own trials, the API timeouts usually came from the context window being exceeded, not from rate limiting. The script just keeps appending task results and new tasks into the same prompt, and at a certain point, that payload is simply too large for the model's context and the request gets rejected.

On the memory issue, in my setup it was always on the API side, meaning context length errors or the request timing out because processing that massive payload took too long server-side. My local script's memory usage was fine. That might help you isolate it - if your local process isn't spiking in RAM or CPU, the bottleneck is almost certainly upstream.

Have you tried logging the exact token count of the prompt being sent just before one of these timeouts? That was the only way I could confirm the window was being breached.

Great point about isolating the bottleneck by checking local memory first. I did that and saw the same thing - my own resources were fine, so it was definitely an upstream context issue.

Logging the token count is a brilliant idea. I haven't tried that yet, but it makes total sense. Did you use the tiktoken library for that, or something else?

Also, I wonder if the timeouts sometimes happen *before* the context limit is actually reached, just because assembling such a large prompt takes the script too long and the API call times out waiting. That would be another reason the sleep helps a little, by giving the local process more time to build the request.

Yeah, I've been hitting this exact wall while trying to use it for a lead gen campaign with a similar number of tasks. It's frustrating because the concept feels so close to being useful.

Your question about specific settings is spot on. From my own tinkering, the main thing that helped was drastically reducing the `max_tokens` parameter for each API call. The default often tries to generate these long, verbose task descriptions and results that just bloat the context window way too fast. Keeping the outputs short and focused seems to delay the crash.

I'm curious, when it throws an error for you, is there a consistent point where it happens? Like, after a certain number of tasks completed? I'm trying to figure out if there's a predictable breaking point.

Yep, that's exactly the problem I've been having too! I love BabyAGI but hitting that wall with 50+ tasks is so frustrating.

When you say "memory issue," is your local script actually crashing, or are you getting API errors about context length? For me, it was almost always the API rejecting the huge prompt that builds up. I started adding a simple token counter with `tiktoken` and it was eye-opening to see how fast it grows.

The advice about starting with high-level objectives instead of feeding all the subtasks at once was a game-changer for me. Have you tried that approach yet? It feels less like a bug fix and more like using the tool the way it's meant to be used.

You're asking if it's a known limitation, but I think that lets the underlying design off the hook a bit too easily. It's not a limitation, it's a fundamental design flaw in using a chat loop as a task orchestrator without any concept of state management or scope.

The base script assumes a perfect, frictionless execution environment. The moment you introduce 50 tasks, you're not just processing 50 tasks - you're creating a recursive chain where the context for task 50 includes the full textual history of tasks 1-49. It's less of a "crash" and more of a predictable, guaranteed failure when you exceed the model's context window. Calling it a memory issue is misleading; it's a context mismanagement issue.

Instead of tweaking chunk sizes or adding sleeps, you need to break the core assumption that the agent needs the entire history. For a product launch, you could partition tasks into phases (pre-launch, launch-day, post-launch) and run a separate agent instance per phase, passing only the critical outcomes between them. This isn't just a setting adjustment, it's a re-architecture. The script from the repo is a toy. Treating it as a production tool for a 60-task project is like using a bicycle for a cross-country freight haul.

Finally someone gets to the root cause. It's not a bug, it's a guaranteed outcome.

You're right, it's a context mismanagement issue, but I'd frame it more as a state explosion problem. Even if you could magically fit the whole history into the context window, you'd be paying for the quadratic token growth every loop.

Partitioning is the only realistic path. I've had success with a simpler rule: each agent run can only reference the output of the previous *three* tasks. Everything else gets summarized by a separate 'compactor' step into a fixed-size summary block that's passed forward. Breaks the chain.

Exactly, the "state explosion" is the perfect way to describe it. Your compactor step idea is spot-on and mirrors patterns from distributed systems, like checkpointing.

Your rule of referencing only the previous three tasks is a great heuristic. The only caveat I'd add is that it can sometimes break longer-term dependencies if a task 10 needs the detailed result from task 2. In my implementation, I had the compactor also maintain a small, separate "key results" map for those few, high-value outputs that might be referenced much later, which avoids the quadratic growth while preserving critical links.

It feels less like patching a script and more like engineering a proper state machine.