Our finance team just forwarded another observability bill with a 40% month-over-month increase. The usual suspect is "data scanned," but the platform's internal cost attribution is about as useful as a screen door on a submarine. "High scan costs" is a label, not a diagnosis.
I need to find the actual source: which specific dashboards, saved queries, or alert rules are executing inefficient queries against massive datasets. The vendor's high-level "by team" breakdown doesn't cut it for an audit. Has anyone reverse-engineered this effectively?
My current approach involves a combination of:
* Enabling query logging (where available) and parsing for patterns like high `GROUP BY` cardinality or full-table scans without time bounds.
* Correlating query patterns with dashboard load times and scheduled report generation.
* Instrumenting the monitoring tool itself via its own API to track who/what is calling it.
The ideal output would be a ranked list: `Dashboard "Executive Overview" → queries prod.events daily without filters → ~$1.2k/month`. Most tools seem allergic to providing this transparency.
Sample script I'm using to fetch query metadata (placeholder API):
```python
# Pseudocode - adapt to your vendor's API
import requests
def get_recent_queries(api_key, timeframe="7d"):
headers = {"Authorization": f"Bearer {api_key}"}
# This endpoint is often buried or enterprise-only
response = requests.get(f"https://api.observability-vendor.com/v1/query_logs?timeframe={timeframe}", headers=headers)
for log in response.json().get('logs', []):
if log.get('bytesScanned', 0) > 1e9: # Flag queries scanning >1GB
print(f"High scan: {log['bytesScanned']} bytesnQuery: {log['query'][:200]}...nInitiator: {log.get('user', 'dashboard')}n---")
```
What's your method? Are you relying on vendor-supplied tools, or have you built your own audit layer? I'm particularly interested in catching those expensive, auto-refreshing dashboards that nobody admits to using.
- Nina
- Nina
Your approach of correlating query patterns with dashboard load times is a solid start. However, I've found that the metadata you can extract from the vendor's query log is often insufficient for true cost attribution. You'll need to instrument the dashboard layer directly.
I've had success by parsing the HTTP request logs from the dashboard/reporting application itself, when possible, to capture the exact query parameters and user context before they're sent to the database. This creates a traceable link between a dashboard load and the resulting scan operation. The script snippet you've started is the right direction, but you'll likely need to augment it with a reverse proxy or middleware logger to capture the full context the tool's internal API obscures.
Without that link, you're still making inferences. For a precise audit, you must tie the financial metric (bytes scanned) directly to the business asset (dashboard or saved query). I can share a more complete pattern for doing this with Grafana and BigQuery if it's relevant to your stack.
Oh, that 40% month-over-month jump is a familiar, painful story. Your approach is spot on, but I've found you often have to go a step further and actually tag the queries at the source. Many platforms let you inject a comment with a dashboard or user ID via a connection parameter or initial query setting. We did this by adding a `/* dashboard_id:123 */` style comment to every query our reporting layer generated. Then, when you pull the query logs, you can grep for that tag and tie the scan cost directly back to the asset. It's a bit of a hack, but it turns that "high scan cost" label into an actual addressable target. The key is making the tagging automatic, otherwise people forget and you're back in the dark.
Test, measure, repeat