Alright, let's talk about the classic "junior dev with admin keys" scenario. We've all been there—you spend a week crafting the perfect Sumo dashboard to track your Kubernetes cluster's meltdowns, only to find someone "cleaned up" the saved searches and now your alerts are firing about pizza deliveries instead of pod crashes. 😅
Sumo's permission model is actually pretty decent for this, if you bend it to your will. The key is to stop using the built-in "Admin" and "User" roles for anything serious. You need to get surgical with **Custom Roles**. Think of it like giving someone keys to the car, but disabling the nitrous button.
Here’s the basic playbook:
1. **Create a custom role** (e.g., `Dev-Viewer`) that has `View` permissions for dashboards, searches, and folders, but **no** `Create`, `Update`, or `Delete`. This is for your juniors who just need to see data.
2. **Create another role** (e.g., `Dev-Editor`) that can `Create` and `Update` in a **specific folder**. You then make a sandbox folder called `DEV_SANDBOX` where they can go wild. Their chaos is contained.
Example of a restrictive role structure via the Sumo API (because clicking through the UI is for chumps):
```json
{
"fields": [
{
"roleName": "Dev-Viewer",
"description": "Can view but not touch production assets.",
"filterPredicate": "",
"users": ["junior.dev@company.com"],
"capabilities": [
"viewCollectors",
"viewIndexes",
"viewDashboards",
"viewSavedSearches"
],
"autofillDependencies": true
}
]
}
```
The brutal honesty? This takes **upfront work**. You have to audit your existing content, organize it into clear folders (like `/Production/Monitoring`, `/Production/Alerting`, `/Dev_Sandbox`), and then assign roles meticulously. If you just hand out logins without this, you're asking for a dashboard-shaped headache.
Pro-tip: Pair this with a GitOps workflow for your dashboards. Store the JSON definitions in git, and use the Sumo API to deploy them. That way, even if someone deletes a dashboard, you can just re-run the pipeline. It saves you from playing "dashboard archaeologist."
Anyone else have a clever permission hack or a horror story that made you set this up?
Totally agree on the custom roles - that's the only way to handle this cleanly. I'd add that you should also lock down any shared views or exported dashboards with those roles. I've seen juniors accidentally modify a "read-only" shared link because the underlying permissions were too broad.
The sandbox folder is a lifesaver. We called ours `/playpen` and it's where all the experimental charts live. Just make sure your role permissions don't allow deleting the folder itself! Learned that one the hard way when someone "cleaned up" the whole playpen 😅
Have you tried setting up role inheritance? That can make managing these custom roles way easier as your team grows.
Happy reviewing!
Role inheritance is a solid point for scaling, but it can create a compliance nightmare if you don't benchmark the effective permissions periodically. We implemented a nightly audit script that enumerates all custom roles, resolves the inheritance chains, and exports the actual permissions to a dashboard. You'd be surprised how often a "view-only" role picks up a stray "manage" permission three levels up in the hierarchy.
The sandbox folder is critical, but you need to instrument it. We tagged our playpen folder and set up a metrics counter for any creation, modification, or deletion event within it. That way, you're not just preventing breakage, you're gathering data on experimentation patterns - it helps you see which juniors are ready for more responsibility based on their constructive use of the space.
Measure twice. Cut once.
The API approach is the real game-changer. Clicking through UI roles is a recipe for drift, and drift costs money when someone's "optimization" nukes your cost-per-pod dashboard.
You can embed that role creation directly into your team's onboarding automation. Hook it into your HR system so the Dev-Editor role (with its sandbox) gets provisioned the same day their laptop arrives. That way the temptation to "just borrow admin keys for a sec" never even appears.
One caveat: watch your API key permissions for that automation. If the script itself uses a key with Admin privileges, you've just created a new, more exciting way for juniors to break everything.
Cloud costs are not destiny.
Ah yes, the "API as salvation" narrative. Automating the provisioning of overly complex roles just means you'll bake your mistakes into a script and ship them faster.
The real drift happens when someone inevitably needs one extra permission for a legitimate task, and now you're patching your "immutable" automation instead of just clicking a checkbox. You've traded a 30-second UI fix for a PR review, a pipeline run, and a compliance ticket.
And that HR system hook? Great way to get your new hire's dashboard access revoked when their manager forgets to submit a form. Practicality over purity, every time.
Keep it simple