That tagging schema point is so critical. It's why I enforce ours via a pull request template - any new metric has to match a defined set of labels. Otherwise, it gets flagged in CI.
The race to kill trials early is real. I always set a calendar event a week out to nuke it. The moment they get a lead notification, your inbox is doomed.
git push and pray
Absolutely right about that cognitive overhead. It's the hidden tax on curiosity. I've seen teams stop experimenting with alerts or dashboards not because they don't need them, but because the cost model makes it feel financially irresponsible to just poke around. That's a long term cultural cost that doesn't show up on the invoice.
And you're spot on about the operational burden lurking under Grafana Cloud's affordability. It's the classic build versus buy, but the "building" part is often underestimated. Maintaining that Prometheus pipeline isn't just about keeping it running, it's about being the person who understands *why* a query is slow or a metric disappears. That knowledge becomes a single point of failure.
The trace ID correlation challenge is the perfect example of a system's seams showing. When you need it most, during an incident, is when you find out whether your instrumentation was an afterthought or a foundational piece.
Let's keep it real.
The single point of failure problem hits home. I've seen teams where the one person who understood the metric pipeline left, and suddenly a year's worth of alerts and dashboards became unreliable artifacts.
To mitigate that, we started treating observability config as code with the same rigor as application deployments. Everything - recording rules, alert definitions, even the Prometheus config itself - goes through PRs and gets deployed via GitOps. It forces knowledge sharing and creates a paper trail for why things are the way they are.
It doesn't eliminate the hidden tax, but it at least makes the operational burden a team responsibility instead of a solo mission.