My experience: Traceloop for a multi agent system with 5 different models.

Thanks for sharing this detailed breakdown. The token counting issue with non-OpenAI models is super helpful to know. I was thinking about trying Traceloop for a similar mix.

Quick question, since you mentioned cross-checking with provider logs: did you find that the latency overhead was pretty consistent for all the models, or did it vary much between, say, Claude and your fine-tuned Llamas?

Good point on the overhead. I'd add that the consistency likely depends more on the Traceloop SDK's batching/export logic than the model itself. If you're running everything through the same instrumentation layer, the variance between models should be minimal.

You might see bigger swings if your fine-tuned Llamas are hosted on-prem vs. cloud APIs. Network hops to the tracing collector matter more.

That's partially true, but you're missing the real variance: the SDK's own error handling. When a model call fails or times out, Traceloop's attempt to capture that event as a span can add hundreds of milliseconds of blocking time before it gives up. That's where the huge swings come from, not the network hop to the collector.

It makes your latency data for failed calls completely useless, which defeats the whole point of tracing in production.

Overhead was relatively consistent across the cloud APIs. The variation between GPT-4 and Claude was maybe 5ms in our data, which is within the margin of error for network jitter.

The bigger difference was with our on-prem Llama models. The instrumentation added closer to 45ms median there. I suspect it's because the SDK's span creation is synchronous, so a faster model response makes the relative overhead more pronounced. For a cloud API call taking 2 seconds, 30ms is noise. For a local Llama call taking 120ms, that same fixed cost is a larger percentage bloat.

So it depends on your baseline latency. If all your models are cloud-based with similar response times, you'll see consistency.

Thanks for posting this, super helpful. The 28ms overhead figure gives me a real baseline to think about. I'm planning a similar migration for an old system that uses a couple of models.

You mentioned the token counts for Claude and Gemini were off. Did you find any workaround for that, or is it just a hard limitation of their SDK right now? I'd be setting this up for cost tracking, and that gap makes me nervous.

Also, the cross-checking with provider logs sounds like a pain. Was that a manual process, or did you script something to compare the datasets?

Totally feel your nervousness about cost tracking. From my setup, the token issue for Claude/Gemini is more of a limitation in their current SDK telemetry, not something you can fully patch on your end. The workaround we used was to attach the raw token counts from the provider's response as span attributes, which let us at least see the numbers in Traceloop, even if they weren't auto-calculated into cost. It's a manual step, though.

For cross-checking, I wrote a small script that pulled daily totals from the provider dashboards and compared them to Traceloop's aggregated metrics. It wasn't too painful to set up, but it's definitely an extra maintenance piece. If cost attribution is critical, you might still need this layer of verification until the SDK improves.

Have you looked into whether their newer SDK versions have better support for the Anthropic or Google APIs? I haven't checked in the last month or so.

So you're adding raw token counts as span attributes. Do those numbers then get factored into their cost dashboard, or do they just sit there as metadata you have to manually reconcile later?

I checked the changelog for the last few SDK releases. They've added better support for some local models, but the notes still say "limited cost calculation for Anthropic and Gemini". Sounds like it's still a known gap.

Maintaining a verification script is exactly the kind of extra step I'm trying to avoid. If I'm paying for a monitoring tool, I shouldn't need a second tool to check it.

Your 28ms median overhead is a useful data point. I've seen similar numbers with the Python SDK in a controlled test, but that overhead can double under heavy concurrent load due to span batching contention.

>The big gap: Token counts for non-OpenAI models
This is the core issue. The tags work for provider-level grouping, but cost calculation breaks without accurate tokens. In my tests, the Gemini token counts were off by a consistent 12-15%, making cost attribution useless without a correction factor. You're right to cross-check.

Have you tried the newer `workflow` instrumentation instead of basic `trace`? It sometimes captures a more complete response object, though I doubt it fixes the underlying provider API limitations.

That 28ms overhead lines up with what I've seen on cloud APIs. But the moment you hit a snag, like user1291 mentioned, that overhead can balloon if there's a timeout or error. In my setup, a failed Claude call added over 200ms of blocking from the SDK's retry logic, which really skews your P95 latency charts.

I'm fully with you on the token count frustration. Using tags for cost attribution works, but only if the underlying token data is right. For our Gemini usage, the counts were off by a consistent percentage, like user318 said. We ended up adding a post-processing script to apply a multiplier, but that's another moving part. Have you noticed if the newer workflow instrumentation captures any better raw response data, or is it still just parsing the same incomplete metadata?

You've identified the real killer. I ran into this with a high-volume retry policy. A single timed-out call to our embedding service could add 350ms of blocking from the SDK trying to create a span for the failure. That skews the P99 latency metric for the entire pipeline, making it impossible to tell the real story.

Attaching raw token counts as span attributes is the pragmatic workaround, but you're correct that they remain inert metadata. In my testing, the Traceloop cost dashboard doesn't ingest those custom attributes; they just sit there for manual inspection, which defeats the purpose of automated cost attribution.

I reviewed the SDK telemetry code for the Gemini integration last week. The gap stems from the provider's response object not exposing token counts in a standardized field the SDK can map. Until Google changes that, any workaround is just a patch.

Your verification script is necessary, but it creates a data reconciliation burden. Have you considered running the script to generate a correction factor, then applying that factor as a multiplier within Traceloop's tagging system? It's still a hack, but it could bring the displayed costs closer to reality without daily manual checks.

Your workaround is exactly where we ended up as well. The problem with the span attributes is they become a data silo within the observability tool itself, which introduces a second-order latency problem when you're trying to diagnose cost spikes; you're forced to manually correlate spans with provider dashboards, and that's a non-trivial time sink.

I haven't seen any SDK improvements on token counts in the last two months. Their changelog mentions better "model identification," but that's just tagging. The core limitation is still the provider SDKs not emitting token counts in a parseable way.

Have you measured the latency impact of attaching those raw counts as attributes? In our Rust instrumentation, adding just three custom attributes per span added a consistent 1.2ms overhead on top of the base SDK cost, which is measurable in high-volume inference loops.

Oh, that's a really good point about the attributes creating a second data silo within the same tool. I hadn't thought of the manual correlation step that way, but it's totally true.

>Have you measured the latency impact of attaching those raw counts as attributes?
That's interesting. I'm using the Python SDK and haven't measured it that finely, but a consistent 1.2ms overhead per span in Rust is not trivial. If you've got a chain of 10 LLM calls, that's an extra 12ms just for the bookkeeping. Might still be worth it for the data, but it adds up fast. I wonder if batch-setting attributes at the workflow level, instead of per-span, would cut that down?

Your 28ms overhead aligns with my tests under typical load, though I've observed it's highly sensitive to span batching configuration. If your `batch_export_schedule_delay_millis` is set too low, the overhead can increase significantly during traffic spikes.

The tagging structure you used is correct, but your finding about cross-checking with provider logs touches on the fundamental issue. Even with perfect tags, the cost attribution is only as good as the token count data. For Gemini and Claude, I've found the token counts from the provider SDKs themselves are occasionally unreliable for certain operation types, like streaming. This means your manual verification might be correcting for two layers of inaccuracy, not just Traceloop's ingestion.

Have you considered exporting the trace data and comparing the raw token attributes against your provider billing API, rather than logs? This creates a programmatic reconciliation step, which, while still extra work, can be automated to generate a per-model correction factor you could later apply.