Just wrapped up a project to allocate our Sysdig monitoring costs back to individual teams. Finance was asking for it, and it turns out the data is all there in the API.
The key was using the `group by` on `team_name` in a PromQL query over the `sysdig_container_usage_seconds` metric. We then mapped those team labels to our internal cost centers. The report now breaks down spend by dev, platform, and data engineering, which should make chargeback conversations much clearer.
Has anyone else set up something similar? I'm curious how you handled shared cluster costs or if you built any alerts around budget thresholds. 😊
That's a great start, but you've just built the easy part. The hard part is getting teams to actually pay it.
Finance asking for it is one thing. Getting dev leads to agree those `team_name` labels are accurate and fair for a real invoice is another. I've seen teams suddenly get very creative about what's "shared infrastructure" the moment a bill lands.
What's your plan when platform engineering says the control plane monitoring is a shared cost and shouldn't be allocated? Or when a team claims their spike was due to a platform-wide incident? Your report will start clear conversations, but they won't all be pleasant.
Show me the TCO.
Nice start, but I hope you trust those `team_name` labels more than I would. At my last place, we found teams were labeling things "platform-utils" that were really just their own dev environments. The API gives you data, not truth.
Did you factor in ingestion costs for logs and traces, or just the container runtime? Because the minute you start charging back, someone's going to parse the pricing doc and ask why their five gigs of verbose logging is in the platform bucket.
And wait until you get the first ticket asking to exclude "non-prod" workloads. That's where the real fun begins.
Trust but verify.