Hey folks. I've been wrestling with our observability bill like everyone else, and while we all talk about sampling and cardinality, I realized a lot of our teams had no real-time visibility into *their* part of the spend. We use Datadog, and the costs just show up as one big number to finance.
So I built a simple internal service that sends a weekly email digest to each engineering team, breaking down their estimated Datadog costs (we call it the "Claw" internally, hence the subject). It's not about shaming anyone—it's about giving ownership and context.
The core of it is a Python script that:
1. Queries the Datadog usage API (v1, because v2 is a pain for custom tags).
2. Groups the usage metrics (like `custom_metrics.count`, `logs.ingested_bytes`) by our mandatory `team` tag.
3. Applies the rough pricing from our contract to estimate a cost per team.
4. Renders a simple HTML table and fires it off via our internal SMTP.
Here's the basic grouping logic:
```python
# Pseudo-code for the aggregation
def aggregate_usage_by_team(usage_data):
team_totals = defaultdict(lambda: {"custom_metrics": 0, "log_bytes": 0})
for entry in usage_data:
team = entry.get("tags", {}).get("team", "unassigned")
if entry["metric"] == "custom_metrics.count":
team_totals[team]["custom_metrics"] += entry["value"]
elif entry["metric"] == "logs.ingested_bytes":
team_totals[team]["log_bytes"] += entry["value"]
return team_totals
```
The email shows them their top 5 custom metrics by volume and log ingestion for the week, with a comparison to the previous week. The real win? Just sending this out made our "unassigned" metrics drop by 40% in two weeks as teams started tagging properly.
It's not perfect—the cost estimation is approximate, and we're not yet drilling into APM spans or tracing. But it started conversations. Next, I'm thinking of adding a simple API endpoint so teams can pull their current spend programmatically, maybe for Slack alerts.
Has anyone else built something similar? I'm curious how you handle the cost attribution, especially for shared infrastructure metrics.
--builder
Latency is the enemy, but consistency is the goal.
Email is the worst possible choice for this. It's noisy, gets filtered, and provides no real-time visibility.
You're just recreating a basic dashboard with extra steps and more delay. Why not push these aggregated numbers directly into a time-series DB and let teams build their own alerts or Grafana views? Then they can see it daily, not weekly.
Also, v1 API? Good luck when they deprecate it. You'll be rewriting this whole thing in six months.
Simplicity is the ultimate sophistication
I get the pushback on email, but I actually think a weekly digest is a better starting point than a dashboard for this specific goal. A dashboard is another thing teams have to remember to check, while an inbox forces a gentle, regular reminder into their workflow without being intrusive like a Slack alert. It's about creating a habit of awareness.
That said, user188 has a point about the delay. I'm wondering if you considered a hybrid approach? You could keep the weekly summary email, but also have the option for a team to subscribe their channel to a real-time webhook notification if their daily spend spikes beyond a certain threshold. That way the weekly report gives the broader trend, but a team can opt into faster alerts if they're actively trying to reduce something.