Skip to content
Notifications
Clear all

Hot take: The speed claims don't account for LLM thinking time. It's still slow.

2 Posts
2 Users
0 Reactions
0 Views
(@harperk)
Reputable Member
Joined: 3 weeks ago
Posts: 224
Topic starter   [#22926]

Been playing with CrewAI for a few weeks now, building out some multi-agent workflows. The docs and examples make a big deal about the orchestration speed, the "parallel task execution," and how it's faster than doing things manually.

Here's the thing: that speed is purely about the framework's overhead. Which, fine, it's negligible. But the clock everyone actually cares aboutβ€”the wall-clock time from "run" to usable outputβ€”is dominated by the LLM calls. CrewAI doesn't magically make GPT-4 or Claude think faster. If you have a chain of five agents, each making a reasoning call and then a task call, you're still sitting through ten LLM round-trips. The parallelization helps if tasks are truly independent, but so many workflows are sequential because agent B needs agent A's output.

The real bottleneck isn't the framework's coordination; it's the cumulative "thinking" time. You can see this when you run a crew with verbose on. It flies through the system prompts and the hand-offs, then spends 90% of its runtime waiting for API responses. Until that changes, the speed claims feel a bit like boasting about how quickly you can hand off a baton in a relay race where the runners are all moving at the same fixed, slow jog.

I want to see more discussion about optimizing *that* part. Caching, smarter prompt design to reduce token use, or even batching. Otherwise, we're just polishing the hinges on a very slow door.


Data over dogma.


   
Quote
(@briana)
Reputable Member
Joined: 3 weeks ago
Posts: 158
 

Absolutely spot on about the LLM calls being the real bottleneck. I've been building similar multi-step data migration pipelines with these frameworks, and you can watch the logs - it's just sitting there waiting for API responses 80% of the time, even with parallelization.

One thing I've tried that helps a little is aggressively caching common intermediate outputs. If you can structure your workflow so that some of those "thinking" steps are reused across runs, you can cut down on redundant calls. But it's a band-aid, not a solution.

It feels like we're optimizing the 10% of the process we control while the 90% is at the mercy of external API latency. Maybe the next wave of frameworks will focus more on speculative execution or better local model integration to actually tackle that.


Backup first.


   
ReplyQuote