Hi everyone! Just finished setting up Sysdig across our multi-tenant SaaS platform and wanted to share what we learned. It was a bit of a journey, but so worth it for the visibility it gives us.
Our main goal was to isolate metrics and alerts per tenant (we call them "accounts") without creating a monitoring nightmare. The key was using Sysdig's `sysdig_customer` field in our instrumentation. We tag every metric and event with the account ID from our app. Then, in Sysdig dashboards and alerts, we can scope everything down to a specific tenant using that label. It's been a game-changer for tracking per-account performance and costs. The team is super happy with the clarity it provides! 😊
One pro-tip: we created a separate Sysdig team for our internal platform monitoring, separate from the per-tenant views. This keeps our own infrastructure alerts clean and separate from customer-specific data. Anyone else doing something similar? Would love to hear how you handle tenant-level alerting thresholds.
Using `sysdig_customer` as the primary tenant dimension is a solid approach. One caveat we've found is that high cardinality from a unique account ID on every metric can become costly, depending on your scale. We mitigate this by applying it selectively to key business metrics and using less granular tags, like pricing tier, for broader infrastructure dashboards.
Your point about separate teams for platform and tenant views is crucial. We extended that by creating alert notification channels routed per tenant tier. This way, a critical alert for a premium tenant can page an engineer directly, while a standard tenant alert goes to a shared dashboard for follow-up during business hours.
Regarding thresholds, we dynamically set them using Sysdig's PromQL integration. We store baseline performance per account tier in a lookup table, and our alert rules reference those values. This avoids manually maintaining hundreds of static thresholds. Have you considered automated baselining for your tenants?
—J
That's a really good point about the cost of high cardinality tags. We're still in the early stages, so we haven't hit scale issues yet, but your strategy of applying the account ID selectively makes a lot of sense. I'll definitely keep that in mind as our tenant count grows.
I'm especially interested in your approach to dynamic thresholds with PromQL and a lookup table. We've been setting static thresholds, which is already feeling unwieldy. Could you share a bit more about how you structure that lookup table? Is it a separate configuration file, or something you manage within Sysdig itself?
Thanks for sharing this detail, it's incredibly helpful.