I've been using Sumo Logic for monitoring large-scale Kubernetes deployments for years, and I only recently fully leveraged dashboard variables for dynamic filtering. This is a fundamental feature for any serious observability platform, and while Sumo's implementation is competent, there are some non-obvious scalability considerations and edge cases that anyone building shared, team-wide dashboards should be aware of.
The basic use case is straightforward: you define a variable that pulls its values from a field in your logs or metrics. For instance, a variable `cluster` sourced from `_sourceHost` or a custom `k8s.cluster` tag. You can then use `{{cluster}}` in your panel queries. The problem most teams hit immediately is performance and cardinality.
* **High-cardinality fields will murder your dashboard load times.** Using `namespace` or `pod_name` as a variable source in a large, multi-tenant cluster is a recipe for a 30-second dashboard load. You must use a filtered query to narrow down the possible values.
* **The "dependent variable" pattern is crucial for usability.** You don't want users selecting a `namespace` from all namespaces across all clusters. You first select a `cluster`, which then populates the `namespace` variable with only namespaces from that cluster. This requires setting the variable's source query to something like:
```sql
_sourceCategory=k8s/containers | where k8s.cluster = "{{cluster}}" | count by k8s.namespace | keys k8s.namespace
```
Note the use of `keys` to extract just the field values.
* **Default values and "All" option:** Always set a sensible default. For a `cluster` variable, defaulting to `*` (All) can be dangerous if your underlying queries aren't optimized to handle wildcards across dozens of clusters. Sometimes it's better to default to your primary production cluster.
The real power, and complexity, comes when you start using these variables in field search and aggregate metrics. For example, creating a templated query for error rates by microservice:
```sql
_sourceCategory=app/logs {{cluster}} {{namespace}}
| parse "service=*," as service
| where level = "ERROR"
| timeslice 1m
| count by _timeslice, service
| compare timeshift 1d
```
This single panel can now show error trends for any combination of cluster and namespace, comparing to the previous day. However, you must ensure your parse operations are efficient and your field extractions are consistent; otherwise, you'll get empty results silently.
The major pitfalls I've seen in production:
* Variables not updating because the source query had a syntax error or returned no data. The UI won't always scream at you; the variable dropdown just stays empty.
* Chain-dependent variables creating circular logic or timing issues on initial load.
* Using variables in `where` clauses with numeric or boolean fields requires careful formatting (no quotes for numbers, obviously).
* Caching behavior can be confusing after you update a variable's definition—sometimes a hard browser refresh is needed.
If you're handing this dashboard off to other teams, document the variable dependencies and expected data shapes. Nothing worse than a SRE trying to debug an incident only to find the "service" variable is empty because the log format changed two deploys ago.
– A
Show me the benchmarks.
Dynamic variables are table stakes, not a "fundamental feature." Grafana had them a decade ago.
The real trick is *not* using them. Every team creates a dozen "dynamic" dashboards, then spends more time waiting for them to load than actually analyzing data. You just traded hard-coded simplicity for configurable bloat.
If you need variable-dependent views, your dashboard is probably doing too much. Split it.
Grafana also has dashboards that load in under a second. The issue isn't variables, it's how people build their queries.
You're right that people overuse them. But the problem is treating them like a filter for everything. I've seen a variable sourced from a 10-million row log field, and they wonder why the dropdown takes 30 seconds.
The real trick is locking them down. Use allow-lists for your variables. Or have a separate, static dashboard for your high-cardinality nightmare like tenant_id. Trying to make one dashboard do both is where you lose performance and your mind.
Don't panic, have a rollback plan.
True, but that 30-second dropdown is a feature. It's a built-in coffee break.
The allow-list trick is solid. We set ours via a scheduled query that populates a lookup file. Means you're not parsing a billion events just to load the variable picker.
But sometimes the real performance sink isn't the variable query, it's the *dependent* panel queries that all re-run with each selection. Seen too many dashboards where every graph fires off 12 heavy queries again because someone used `Auto refresh` on the variable. The dashboard becomes a DDoS attack on your own data source.
Deploy with love