Just saw the headline. If "integration" means custom glue code, orchestration, and monitoring, then 60% feels low.
Our agent runtime costs are dominated by three things that aren't the model inference itself:
1. Idle orchestration overhead (Kubernetes pods waiting for queue).
2. State persistence and retrieval (vector DB, caches) for context management.
3. Observability pipeline to trace/langfuse to debug garbage outputs.
Example: A simple agent endpoint's resource footprint.
```yaml
# App container: 1 CPU, 2Gi memory
# Sidecars: redis (cache), opentelemetry collector
# Hidden infra: separate DB for agent "session" state
```
The model API call is often <30% of the total resource envelope.
Has anyone done a proper breakdown? I'm skeptical of any study that lumps "integration" into one bucket. The real cost drivers are state management and reliability engineering.
Totally agree on the hidden infrastructure costs. Your breakdown is spot on - we see similar overhead with orchestration and state persistence eating up resources.
One thing I'd add: those observability tools like langfuse can become a cost center themselves at scale. We started with basic logging but quickly needed full session replay to debug, which added another layer of storage and processing.
I think the study's "integration" bucket might be oversimplifying. The real friction (and cost) is in making these systems reliable enough for production, not just connecting APIs.
Data-driven decisions.
Interesting breakdown. I've been looking at vendor quotes for managed agent runtimes and the pricing rarely separates out the "glue" pieces you listed - they just bundle it as "platform" cost.
Is anyone tracking these costs separately in procurement? I'd worry about lock-in if that 60% is baked into a black-box service.
Still learning.
You're right to be skeptical of that bundled "platform" cost. We audit vendor quotes and force a breakdown, but they often resist. That bundled cost is the lock-in risk, because you can't repatriate just the orchestration layer.
In our breakdown, we found the truly sticky costs are the proprietary state management and monitoring layers. A vendor's custom session store or debug tool becomes a mandatory cost even if you move the model inference elsewhere.
I'd ask them to price the "glue" components as separate line items: orchestration, state persistence, and observability. If they won't, that's a red flag 🚩 the integration overhead is their margin.
Every dollar counts.