Step-by-step: How we built a culture of 'observability accountability'.

Hello everyone,

I’ve been reflecting a lot lately on something I see come up repeatedly in our community discussions: the sheer frustration many teams feel when observability bills spiral out of control, especially during unexpected traffic events. It often leads to a reactive, top-down mandate to "just cut costs," which can feel punitive and counterproductive. At my previous company, we faced this exact scenario, and instead of just tightening the budgetary screws, we decided to try something different. We shifted from a model of centralised cost control to one of distributed "observability accountability." The goal wasn't just to reduce spend, but to make smarter, more sustainable decisions about what we instrument and why.

The core idea was to move the conversation away from finance and back to engineering value. We started by making costs completely transparent. Every team got a weekly breakdown of their spend, broken down by signal type (metrics, logs, traces) and tagged by their core services. But raw numbers aren't enough. We paired this with a simple, internal "Observability Value Framework" we built together. For any new instrumentation or retained data, we asked: What critical business or operational question does this answer? Could we answer it with a less granular (and less costly) signal? For example, moving from full-fidelity debug logs in production to sampled traces with specific business attributes, or using derived metrics instead of emitting a custom metric for every single error code.

Crucially, we embedded this thinking into our development lifecycle. During our service design review phase, a new section was added: "Observability Impact." This forced a brief discussion about the cardinality of new attributes, the expected log volume, and the retention needs. It wasn't about gatekeeping, but about conscious choice. Engineers started to see observability as a resource they managed, not a free utility. When a spike happened, they understood the direct link between their architectural choices and the bill.

The results took time, but they were profound. We saw a significant drop in "just-in-case" instrumentation. Teams proactively implemented log filtering at the agent level and adopted thoughtful sampling strategies for their high-volume endpoints. More importantly, the quality of our signals improved because we were collecting data with intent. The occasional cost spike was now a learning opportunity—a chance to analyze a new traffic pattern or an inefficient query—rather than a blame game.

I'm curious if others have taken a similar cultural approach. What mechanisms have you found effective to foster accountability without stifling innovation? How do you balance the need for deep insights with the reality of pricing models that charge per-gigabyte or per-sample? Let's share some stories. 😊

— Alex