Alright, so I've been trying to get a handle on LLM call performance in our production setup. We're using a mix of providers (OpenAI, Anthropic, some custom models) and routing requests based on user tier—think basic, pro, enterprise.
I can get average latency for the whole system easy enough, but averages lie, right? Especially when a free-tier user's request to a cheap model gets lumped in with an enterprise user's complex chain to GPT-4. I need to see the spread—the P95, P99—but sliced by who's actually making the call.
My current observability stack (think a standard metrics pipeline) spits out overall histograms. But breaking that down per segment feels... clunky. I've been cobbling together scripts that tag traces with user properties before sending them to my monitoring tool, but the aggregation feels off.
How are you all doing this? I'm looking for the cleanest way to:
* Segment traces/spans by user attributes (plan, region, maybe even feature flag) at the source.
* Calculate latency percentiles (not just averages) for each of those segments in near real-time.
* Avoid a situation where I'm essentially building a custom analytics platform on the side.
I've seen tools like LangSmith, Helicone, and OpenTelemetry with LLM semantic conventions thrown around. From my CRM-hopping days, I know the devil is in the details of how you tag and group data.
* Are you enriching spans before they leave your application, or is there a smarter post-processing step?
* Does your tracing tool have built-in grouping/querying for this, or do you pipe everything to a warehouse and use Looker?
* How are you handling cost attribution alongside this? Same dimensions?
Basically, I want the dashboard that shows "Enterprise EU users have a P99 latency of 2.1s on summarization calls this week" without having to manually query a million log lines. What's the workflow?
Still looking for the perfect one
Yeah, the whole "averages lie" thing hits home for me from the marketing side. Looking at campaign email latency, the average open time across all users is useless if you don't split by segment, like active vs. dormant leads.
You mentioned tagging traces with user properties. I'm curious, when you say the aggregation feels off, is it because the percentiles are skewed by low sample sizes for some segments, like maybe your enterprise tier? That's a problem we run into with A/B testing.
What are you using for your metrics pipeline? I'm wondering if there's a way to push that user tier tag as a first-class dimension when you emit the histogram data point, instead of trying to stitch it together after.