Hey everyone, Emma here 👋. I’ve been lurking for a bit, trying to wrap my head around all the options for observability. Our small team's bill was starting to get scary, especially our log ingest—it felt like we were paying for a lot of noise. I wanted to share the concrete, step-by-step process we followed to finally get control over it. It was a bit overwhelming at first, but breaking it down made it manageable.
We started by just asking: "What do we *actually* need to see?" We realized we were sending debug logs from development environments straight to our production observability tool. That was a huge, unnecessary chunk. Our first step was to implement a simple filter in our logging library to drop all logs with a level below "INFO" from non-production environments. That alone cut about 30%.
Next, we tackled high-cardinality culprits. We had a few logging statements that included full request URLs with unique IDs. We wrote a small processor to replace path segments like `/users/12345` with a pattern like `/users/{id}` before the log was ever sent. This dramatically reduced the number of unique log lines and helped a ton with indexing costs.
The biggest win came from looking at our third-party integrations and health checks. We found that verbose, frequent "I'm alive!" pings from load balancers and some chatty external APIs were flooding us. We created an allowlist of log sources we actually cared about (our core application services, auth, and payment processors) and simply stopped ingesting logs from anything else. It felt risky, but we haven't missed them.
Finally, we adopted a sampling rule for our most verbose, but still useful, INFO-level logs from high-traffic endpoints. We sample them at 10%, which is more than enough for trend analysis. For all errors (ERROR/WARN level), we send everything, zero sampling. This way, we never lose signal on actual problems.
The process wasn't one big change, but these four smaller, deliberate steps. We went from feeling helpless about the bill to having a clear map of what we're collecting and why. It’s been a huge relief! I'd love to hear if anyone has taken a similar approach or has other tips for fine-tuning.