I've been using Traceloop for a few months to monitor our LLM calls and costs, and while the built-in dashboards are solid, I kept wanting a clearer, at-a-glance view of our performance outliers. The P95 latency is my key metric for spotting degradation before it becomes a user problem.
So, I built a custom widget that pulls from Traceloop's API to show a rolling 7-day P95 latency for our main pipeline, alongside a cost-per-call average. It's basically a focused status board that lives on our team's internal ops page.
Here’s the core approach:
* The widget uses the Traceloop GraphQL API (the same one their UI uses) to fetch span data.
* I filter for the specific `trace.attributes.service.name` we care about and the `llm.requests` operation.
* Data is aggregated daily, and I calculate the P95 for `llm.operation.duration` over the window.
* A separate query pulls estimated cost data (using `llm.usage.completion_tokens`, etc.) to compute a running average cost per call.
The main benefit? I can now see, in one compact view, if our P95 is creeping up while costs hold steady (an optimization flag) or if both are spiking (a potential vendor/configuration issue). It cost me an afternoon to set up and saves us from logging into the full platform for this single snapshot.
Has anyone else built custom views on top of Traceloop's data? I'm curious if you're calculating different efficiency metrics. I'm thinking of adding a "cost per 1k output tokens" view next.
That's a neat project. I'm curious about the rolling 7-day calculation - are you just taking the last 7 days of aggregated daily P95 values and averaging them, or is it a sliding window over the raw spans? I've been thinking about doing something similar for our own monitoring but I'm not sure how to handle weekends when traffic drops off.
Also, the cost-per-call average sounds like a useful flag but I wonder how you deal with model pricing changes or different model versions in the mix. Does your query handle that at all, or do you just assume a fixed cost per token?
Great questions. The weekend traffic drop-off is a real issue - if you just average the daily P95s, a low-traffic day can skew your trend line significantly. A true sliding window over the raw spans is more accurate but much heavier on the API.
For cost, you raise the critical point. Most DIY dashboards use a fixed token cost, which breaks the moment you switch from GPT-4 to Claude 3.5 Sonnet or when a vendor tweaks prices. Your query would need to join span data with a separate, maintained lookup table of model names and their current per-token rates, which gets messy fast.
That's why for core metrics like these, I usually push for the vendor to build it as a native feature. It keeps everyone on the same, maintained logic.
Keep it real