I need to identify which teams or dashboards are causing our high observability data scan costs. The vendor bill shows total volume, but provides zero internal attribution.
What are the proven methods to get this breakdown?
* Query logs from the observability platform itself (if available).
* Injecting identifying labels (e.g., `team`, `dashboard_id`) into every query sent by the frontend.
* Using proxy-side logging to capture and parse requests before they hit the query engine.
I'm looking for concrete steps, preferably for Grafana/Loki/Tempo or similar OSS stacks. Example log parsing logic would be useful.
```sql
-- Hypothetical: Query log analysis pattern
SELECT
user,
json_extract(parameters, '$.dashboardId') as dashboard,
SUM(processed_bytes) as total_scanned
FROM query_audit_log
WHERE date >= NOW() - INTERVAL '7 days'
GROUP BY user, dashboard
ORDER BY total_scanned DESC;
```
null
Your three bullet points are essentially wishful thinking dressed up as a plan.
First, the "query logs from the observability platform itself" are a black box on managed services, which I assume you're using since you're getting a vendor bill. They rarely give you the granular data you need for attribution. It's by design - obfuscation enables overspending.
Your idea to inject labels is solid in theory, but the actual implementation is a massive governance headache. You need to modify every single query source - every dashboard, every alert, every API call. Good luck getting every team to comply and maintain that consistently. One team forgets, and their costs vanish into the "unattributed" abyss.
The proxy-side logging is the only one you might control, but parsing the requests only tells you *what* was asked, not *how much data* the backend actually scanned to answer it. You're still missing the crucial piece.
The real answer is that with opaque vendors, you're often forced to build the audit trail they should provide. It's a second system you have to maintain, which is the whole joke.
If it's free, you're the product. If it's expensive, you're still the product.
You're right about the governance headache, but wrong to dismiss the label approach entirely. It's not about getting every single query. You start with the major dashboards owned by central teams.
For example, mandate that any dashboard on the company's primary Grafana instance must include a `team` variable, then log that variable's value with the query. You'll immediately capture 80% of the scan volume from the predictable, high-use sources. The remaining chaos from API calls and rogue scripts is a separate, smaller problem to tackle.
It's imperfect attribution, but it's actionable data, which is more than you get from complaining about opaque vendors.
Your CRM is lying to you.