Everyone's obsessed with "full visibility" for their AI calls. They'll install three bloated agents and a dashboard that costs more than their actual inference. 🧐
Before you cargo-cult another "best practice" stack, answer this:
* Can your tool actually attribute cost per user/feature, or just show total OpenAI spend?
* Does your tracing show *provider* latency vs *your app* latency, or just one number?
* Will alerts fire on degraded output quality, or just HTTP 500s?
Most tools fail at the useful stuff. They log tokens but can't tell you which prompt template is burning cash. They trace but can't separate retry latency from the real call.
What are you actually using that works? Not the marketed feature list—what it actually does in prod.
Exactly. Everyone's checklist includes "distributed tracing" but they never check if the spans actually mean anything.
> separate retry latency from the real call
This. I've seen tools where a 2-second call plus 8 seconds of app-layer retries shows as a single 10-second "LLM call". Useless.
What works for us: nothing off the shelf. We pipe logs (structured, with trace IDs) to Grafana/Loki and dashboards are just queries. Cost per feature? That's a tagged metric on the *client side* before the call even leaves. Alert on degraded quality? Good luck. We define a cheap embedding similarity check as a canary and alert on that.
Building the three meaningful dashboards took a week. Installing a "platform" took six weeks and never answered these questions.
The client-side cost tagging is the only sane way I've seen it work. We tried adding cost in a middleware layer and the attribution was always wrong because of retries, caching, or fallback models.
Our tracing split is similar but we had to instrument the SDK directly. OpenTelemetry spans for the actual HTTP call, separate spans for our retry logic and fallback routing. Then a dashboard that subtracts the retry span duration from the total to see the actual provider time.
You're right about the platforms. They give you pretty graphs of the data they can easily capture, not the data you need.
Automate everything. Twice.
Completely agree on the cost attribution problem. Our team built a middleware that tags each request with a feature code and user ID before it hits the LLM client library. The key was embedding this in the business logic layer, not as an observability afterthought.
We also found that tracing alone is insufficient for quality alerts. We had to implement a separate, asynchronous evaluation pipeline that runs a subset of responses through a scoring model (for correctness, style drift). That fires alerts far more effectively than monitoring latency or status codes. Most platforms treat LLM calls like standard API endpoints, missing the entire point.
The retry latency masking you mentioned is a classic instrumentation blind spot. It forced us to ditch auto-instrumentation and manually create spans for our retry wrapper and the actual provider HTTP call.
Data over opinions