Just finished a pilot where we gave CrewAI access to 200 of our devs and data analysts. We’re a mid-sized SaaS company, so this was a big jump from the small POC.
The main thing that broke was memory. We started with the default local cache, and it just couldn’t handle the concurrent sessions. Tasks would get weird, mixed-up contexts between users. 😅 We switched to using Redis for the `memory` backend, which fixed the chaos. Also, we had to be really careful with LLM costs. Letting everyone use GPT-4 by default would have been a disaster. We created clear agent templates with gpt-3.5-turbo as the standard, and only allowed 4 for specific, approved workflows.
On the infra side, we containerized the CrewAI orchestrator with Docker and ran it on ECS. Biggest lesson? You need to set **very clear** guardrails on what agents can do (like file system access or external API calls) before you let a large group loose with it.
Your point about guardrails is crucial. We saw something similar with tool permissions, where early users gave agents broad access that created unintended API cascades. A preset list of approved tools per agent template saved us.
The memory issue with local cache is a great warning for anyone scaling from a few testers. Redis seems to be the go-to fix for that isolation problem.
Did you implement any specific monitoring for cost or usage patterns after setting the LLM defaults? We found that even with 3.5-turbo as standard, some workflows generated unexpectedly high token counts without careful prompt structuring.
—HR
> We found that even with 3.5-turbo as standard, some workflows generated unexpectedly high token counts
Yes! We saw this too, especially with long-running research tasks. We had to add a simple token estimator check before running some of the heavier agent chains. It's surprising how fast it adds up.
Did you set any hard stops, like automatically killing a process if it exceeded a token threshold? We're still deciding if that's too blunt of an approach.
Redis for memory? Surprised you didn't hit that in the POC. The default cache is fine for a notebook and useless for anything else.
The real guardrail you missed is query timeouts. Containerizing on ECS doesn't stop a runaway agent from hammering your database for ten minutes because someone wrote a vague task. Seen it happen.
Cost controls on the LLM are obvious. The bigger bill is always the compute and API calls from the tools you let it use.
SQL is enough