I've been prototyping with AutoGen for a few months now, building internal tools and workflow assistants. The local setup works, but I'm hitting walls trying to move something to a state where it's reliable and observable for even a small team.
My main concerns are around infrastructure and ops:
* **Hosting:** The multi-agent, stateful nature of group chats doesn't fit a simple Lambda. I'm leaning towards a long-running ECS Fargate task, but the cost profile is completely different from my usual serverless patterns. Has anyone containerized this successfully? Are you using a queue to manage session persistence?
* **Monitoring:** Beyond basic CloudWatch for the container, how are you tracking:
* Token usage per agent/model
* Conversation state errors (e.g., agents getting stuck)
* End-to-end latency for a full group chat resolution
* **Costs:** The ChatCompletion costs are one thing, but I'm also looking at the compute hours for the always-on container. My back-of-the-envelope math shows it could get expensive fast for low-volume use cases.
I'm looking for real numbers and architectures, not theory. What's actually running in your AWS/GCP/Azure account right now, and what would you do differently?
Curious about the same things. I haven't moved to production yet.
But on hosting: I'm also stuck between ECS and a beefy EC2 instance. The stateful session is the problem. How are you handling timeouts or restarts without losing the whole conversation? That's my blocker.
For costs, have you looked at the new OpenAI batch API for non-live parts of the workflow? Might cut some ChatCompletion cost.