Okay, I need to vent and get some real talk. We're currently using a big-name observability platform for our LLM apps, and their cost tracking is... frustratingly vague. It's basically a giant, opaque bill with some basic filters. Trying to explain to Finance why the Product team's experimental chat feature cost 3x last month is a nightmare.
So we built our own internal dashboard. The goal was dead simple: show actual, attributable LLM spend per team, per project, and even per feature flag. We're pulling raw data from our proxy/logging layer, tagging every call with team IDs and project codes, and rolling it up daily.
The result? We can now see that the Support team's new summary feature is burning cash on GPT-4 when fine-tuned GPT-3.5-turbo would probably suffice. And we caught a bug in our staging environment that was calling the API in a loop. Has anyone else gone down this DIY path? I'm curious about:
- How you're tagging calls (envelope headers? custom metadata?)
- Whether you found a way to get *actual* cost-per-call from vendors, or if you're just applying the public pricing math.
- If this solved political battles over budget, or created new ones 😅
We're wondering if we should open-source our approach or if we're just reinventing a wheel that better tools already have. The commercial platforms we've trialed seem focused on latency and errors, with cost as a second-class citizen. Am I missing something?
The grass is greener? We'll see.
Absolutely, and you've hit on the core reason these DIY tracking systems become essential: attribution. That opaque bill from the platform just becomes noise you can't act on.
We use a similar tagging approach via a proxy, injecting a JSON payload into the request headers. The key for us was making it mandatory and schema-enforced at the proxy level - no team/project code, the call gets logged but flagged as 'unattributed' which creates operational noise they're incentivized to fix. We also tag the environment (prod/staging/dev) and a feature identifier.
On cost, we're calculating it using the public pricing math, as you mentioned. We found vendor APIs don't return a monetary cost, just token counts. The real nuance is in keeping that pricing table - model, input token cost, output token cost - as a versioned reference table in our warehouse, as providers do change rates. This lets us backfill historical costs if a price drops.
It definitely solved some political battles by shifting discussions from "who's to blame" to "here's the cost driver". But it created a new, almost granular, debate about cost allocation rules. For example, if Team A's feature uses a cached response generated by Team B's expensive call last week, who gets charged? We had to establish those ground rules upfront.