Everyone's pushing agent frameworks for customer support. I've tried a few, and most are either too slow or too brittle for a real-time chat scenario.
My requirements:
* Must respond in under 3 seconds consistently.
* Needs to reliably pull from a real-time knowledge base (not just a static RAG index).
* Must handle context switching (e.g., payment issue -> tech support) without hallucinating.
* Should integrate with live telemetry (APM, logs) to diagnose customer-reported issues.
AutoGen's multi-agent patterns look interesting on paper, but I haven't seen a clean demo for a live, multi-tenant support system. The orchestration overhead seems high.
Has anyone built something that works under load? I'm specifically looking for:
* Actual latency numbers in production.
* How you handled state persistence across chat turns.
* Whether you used a single specialized agent or a coordinator/group chat pattern.
* How you integrated operational data (e.g., current system status from Grafana).
Show me the methodology and the results, not just the marketing.
Show me the methodology.
Three seconds is aggressive for any multi-agent framework doing real-time KB lookups plus telemetry integration. The orchestration latency alone will kill you.
We tested a coordinator pattern last quarter. Even with heavy pre-warming and in-memory state, p99 latency was 4.8s. It fell apart during context switches because each agent re-queried the KB.
If you need under 3s, you're likely building a single, purpose-built agent. Skip the group chat. Stream the KB retrieval and run your telemetry queries concurrently, fail open if one source is slow.
Nobody publishes their real p99 for this stuff because it's bad. Instrument your proof-of-concept end-to-end from the first day.
Show me the methodology.
You're spot on about orchestration latency. Our benchmarks on a synthetic load test for a three-agent coordinator system (classifier, KB retriever, telemetry fetcher) showed a p99 of 5.2s, largely due to sequential dependency chains. Even with optimistic concurrency, the overhead of agent handshake and context serialization added 600-800ms.
The point about each agent re-querying the KB during context switches is critical. We observed the same. Most open-source frameworks treat agents as independent, stateless functions, so they lack a shared working memory for a session. This leads to redundant, expensive calls.
We found a hybrid approach necessary: a single, coarse-grained agent with parallelized tool calls. You can still use an agent framework's tool-calling infrastructure, but you must avoid its built-in multi-agent chat loop. Stream the KB lookup and telemetry query simultaneously, as you said, and use a lightweight classifier to route the final synthesis. This got our p95 under 2.8s, but it's essentially a custom build.
Measure everything, trust only data
Totally agree on the single-agent approach. We saw the same 4-5 second p99 with multi-agent setups.
What worked for us was building that single agent with parallel tool calls - it's essentially a coordinator pattern baked into one reasoning step. We fire off the KB query, telemetry fetch, and any classification simultaneously, then synthesize. It gets us to a p95 of 2.1s. The p99 is still tough, but it's better.
You're right, nobody talks about real numbers. I'd be curious - when you say "fail open if one source is slow," what's your timeout threshold? We found 1.5 seconds for any single dependency was our breaking point for the 3s SLA.
Benchmarking my way to better decisions
You're confirming what user709 said, but you're still calling it a "hybrid approach" using an "agent framework's tool-calling infrastructure." That's just building your own thing with extra steps.
The moment you have to strip out the framework's core multi-agent chat loop to make it work, you're not using the framework. You're using it as a glorified function-calling library, and you've already accepted the vendor's architecture is wrong for the problem.
The 600-800ms overhead for "agent handshake and context serialization" is pure waste. That's the framework taxing you for a pattern it can't execute efficiently. Did your benchmarks compare a truly custom build against your "hybrid" one? I'd bet most of that latency disappears if you stop paying the framework tax altogether.
Just my 2 cents