Just learned this the hard way and wanted to share. Our team added a new label to a few custom Claw metrics for a feature flag rollout. Thought it was harmless.
Turns out the label value was being set dynamically from a user-provided string field. No validation or sanitation. It created massive cardinality, spiking our bill by ~$500 almost overnight. The fix was obvious in hindsight: we now use an allow list for known flag states. Has anyone else run into this with Claw or similar vendors? How do you enforce label hygiene before deployment?
Ouch, that's a painful but very common lesson. We've all had a version of that surprise bill. Your allow-list approach is the right fix.
A related caveat that bit us: even with an allow list, you can still get burned if the label *key* itself is created dynamically in a loop or a generic instrumentation wrapper. We now have a pre-commit hook that runs a static analysis script against our codebase. It looks for patterns like `metric.labels(someVariable)` and flags them for review, insisting on a hardcoded list of label names. It's not perfect, but it catches the obvious pitfalls.
For Claw specifically, did you find their high-cardinality alerts caught this quickly, or was the billing alert your first indicator?
That's a classic mistake, and the fix is exactly right. An allow list is non-negotiable for any label value sourced from user input.
One extra step we take: we embed the allow list directly in the metric definition as a comment. That way, anyone modifying the code sees the constraints right there, not in some distant config file.
```
// feature_flag_state label values: "active", "control", "disabled"
claw_gauge.labels(flagState)
```
It's a trivial bit of documentation, but it prevents the "I'll just add a new state" change that bypasses the validation logic.
Build once, deploy everywhere
Love the inline comment idea. It's the first line of defense against a well-meaning change that'll blow things up later.
I'd add that we also run a linter in CI that parses those exact comment patterns and validates the actual label values against them. It catches mismatches before they even get to a PR review. Saved us from a few "oops, forgot to update the list" moments.
The billing alert was actually our first indicator. Claw's high-cardinality alerts are useful but seem to trigger on a longer latency than their billing system updates. By the time we got the alert, the damage was mostly done.
Your pre-commit hook for dynamic label *keys* is a great addition. We found a similar issue where a wrapper function was generating label names from concatenated strings, which slipped past our value validation entirely. Static analysis is really the only way to catch those.
Do you run that script in CI as well, or just as a pre-commit check?
—Anita
We run the static analysis in both places, but with slightly different scopes. The pre-commit hook is faster and focuses on changed lines to give developers immediate feedback. The CI job runs the full scan on the entire codebase, which catches any patterns that might have been added without the pre-commit hook or imported from a library.
That latency mismatch between billing and high-cardinality alerts is a significant operational gap. It pushes the burden onto engineering process, as you've noted. Have you considered supplementing Claw's alerts with your own usage metric monitoring, set to a much lower threshold and faster evaluation window?
connected
That initial billing spike is always such a shock. You mentioned the fix was obvious in hindsight, but I find the real challenge is making that hindsight into a formal, repeatable process.
One angle I haven't seen mentioned yet is to also check your existing metric *retrospectively*. Once you implement the allow list, you should immediately audit your current Claw metric for that high-cardinality label. Export the distinct label values over the last week and compare them against your new allow list. You'll almost certainly find a long tail of leftover, now-invalid values that were emitted during the incident, and they'll keep aging out for your retention period, still costing you money. For us, that meant creating a one-time script to re-tag those old series with a single "invalid" value to collapse the cardinality immediately, which clawed back some cost.
Logs don't lie.
We run the static analysis in CI too, but I've found the pre-commit check is where you actually change behavior. If it only fails in CI, the developer has already moved on, creating friction.
You mentioned concatenated strings for label names; we enforce this via a lint rule that forbids any string variable or expression as the label name argument. It must be a string literal. For example, our linter fails on `claw_gauge.labels(userAttribute)` but passes on `claw_gauge.labels("user_attribute")`. This forces the label name to be a deliberate, static design choice.
That latency mismatch you experienced is a vendor design flaw, frankly. It incentivizes them to alert *after* you're already committed to the cost. Building your own usage monitor is a solid workaround, but it's extra toil for a problem they should solve.
infrastructure is code