We're evaluating multi-agent orchestration tools for a new internal platform team. Scale is ~50-100 concurrent workflows, mostly data processing and internal API integrations. Need strong scheduling, error handling, and audit trails.
Key requirements:
* Must be self-hosted or private cloud (AWS).
* Declarative agent/task definitions (Infrastructure as Code).
* Observability: logging, tracing, metrics export.
* Cost: mid-market, not enterprise SaaS pricing.
Evaluating CrewAI, LangGraph, and potentially building on Prefect/Airflow. Initial CrewAI tests show the high-level abstraction is nice, but we hit limits with custom tool integration and scaling.
Has anyone run CrewAI in production at this scale? Specifically:
* How does the `Crew` abstraction hold up with 20+ agents?
* What's the actual resource footprint per running `Kickoff`?
* Is the local vs. cloud (`CrewAICloud`) split a problem for governance?
Our prototype `crew.py`:
```python
from crewai import Agent, Task, Crew
analyst = Agent(
role="Data Analyst",
goal="Clean and validate incoming datasets",
backstory="...",
tools=[csv_validator_tool],
verbose=True
)
# ... more agents
crew = Crew(
agents=[analyst, summarizer, loader],
tasks=[clean_task, summarize_task, load_task],
process="sequential"
)
result = crew.kickoff()
```
—cp
—cp