Alright, gather 'round the campfire of vendor-induced complexity. While you're all busy evaluating yet another SaaS observability platform that charges per span ingested (and don't forget the premium for "AI-powered" anomaly detection!), I got tired of waiting for our finance team to approve the $15k annual PO.
So I spent an afternoon and cobbled together a simple CLI to query our trace database directly. We're using a popular open-source tracing system, but its UI is a dog for bulk analysis. I needed to answer real questions, like:
* Which of our RAG endpoints is burning the most cash on embedding model calls?
* What's the actual latency distribution for our Llama 3.1 70B calls versus Claude 3.5 Sonnet, after you strip out the queueing time?
* Is anyone actually using that expensive summarization feature we built?
Here's the gist of it. It's just a Python script that hits the trace store's HTTP API and lets me run simple aggregations.
```python
# cost_hawk_query.py
import requests
import sys
TRACE_API = "http://internal-trace-collector:16686/api/traces"
SERVICES = ["llm-gateway", "embedding-service", "rag-orchestrator"]
def query_cost_by_operation(service, days=1):
# Simplified: we assume span duration maps linearly to cost for a given model
params = {
'service': service,
'lookback': f'{days}d',
'limit': 1000
}
resp = requests.get(TRACE_API, params=params)
traces = resp.json().get('data', [])
# This is where you'd map span tags (like 'model') to your internal cost per second
cost_map = {
'gpt-4': 0.03,
'claude-3-opus': 0.045,
'text-embedding-ada-002': 0.0001
}
total_cost = 0
for trace in traces:
for span in trace.get('spans', []):
model = span.get('tags', {}).get('llm.model')
if model and model in cost_map:
duration_sec = (span.get('duration', 0) / 1_000_000)
total_cost += duration_sec * cost_map[model]
return total_cost
if __name__ == '__main__':
for svc in SERVICES:
cost = query_cost_by_operation(svc)
print(f"{svc}: ${cost:.2f} over last 24h")
```
It's ugly, but it works. Ran it this morning and immediately spotted that our "optimized" embedding cache has a 40% miss rate for a specific tenant, which is burning an extra $200 a month on Azure OpenAI embeddings. That's a $2400-a-year bug a shiny dashboard hadn't flagged.
The point is, before you get sold another "comprehensive solution," see if you can just ask your data directly. Half the time, the vendor's magic is just a pretty wrapper around a `GROUP BY` query.
-- cost first
-- cost first