I agree that the excitement is justified. Moving from a black box to any instrumentation creates a tangible operational mindset shift. The lack of histograms forces us into the correlative detective work you're describing, but that process itself often uncovers other systemic dependencies you'd otherwise miss.
On the scrape noise, setting a longer interval is a good start, but you also need to be selective with what you actually ingest. The Go runtime metrics are the usual suspects. In our environment, we had to aggressively drop high-cardinality `go_memstats_alloc_bytes` series per pod after the first scrape, as they were bloating our TSDB. The `process_` metrics are also redundant if you're already scraping the node exporter. It's a balancing act to capture the new Boundary signals without the Go telemetry drowning them out.
One subtle quirk we found: the initial scrape after a worker restart can show artificially high rates for the first interval due to counter initialization, which can trip naive alerts. Using `rate(metric[5m])` with an offset of at least two scrape intervals in your queries helps avoid that false signal.
Data over dogma
Totally agree on the scrape interval. We started at 15s and those Go metrics were definitely contributing to noisy graphs. Moving to 30s helped a lot with that baseline chatter.
The correlative approach with active connections and audit logs is clever, but I found the timing is often just a bit too loose for catching brief latency spikes. It's great for long-term trend analysis, but less so for pinpointing a real-time issue.
Durable storage for the counters would be nice for absolute totals, but honestly, I've found that focusing on the rates and that bandwidth limit percent gives me a much clearer picture of current system health. It forces you away from those "lifetime total" vanity metrics.
✌️
You're right about the scrape interval, but the real noise culprit is the default Go metrics. I'd drop them entirely. If you need the data, pull it from the node exporter instead.
Focusing on rates over totals is the only sane way to run Prometheus. "Lifetime totals" are meaningless for anything but a static report. The bandwidth limit percent is the real metric, everything else is just a derivative signal.
Dropping the Go metrics entirely is the pragmatic move, I've done the same in production. The node exporter gives you a cleaner, normalized view of the process anyway. You just have to be careful to keep the `go_threads` metric from Boundary's endpoint if you're not already getting it elsewhere, as it's a decent canary for goroutine leaks in the worker's proxy routines.
On rates being the only sane way, I mostly agree, but that blanket statement has a real cost during post-mortems. When you're trying to reconstruct total data transfer for a specific compliance window or a cost attribution report, you're stuck with log aggregation because the counter resets are lost. It's a trade-off the design makes, but calling lifetime totals meaningless ignores the paperwork part of the job.
The bandwidth limit percent is indeed the primary signal. But in my deployments, it's been a lagging indicator, not a leading one. By the time it climbs, the upstream churn in `active_connections` has already been spiking for minutes. You need to watch the derivative of the rate of connection changes to get ahead of it.
Good point on the paperwork angle. Compliance windows don't care about your elegant rate calculations.
But that lagging indicator problem you mentioned is the real issue. Watching the derivative of connection rates is fine, but it's reactive. If `active_connections` is already spiking, the session setup process is already under stress. The leading indicator I look for is a sustained drop in `go_threads` while `active_connections` climbs - suggests the proxy routines are failing to spawn, a classic pre-churn signal.
You're keeping `go_threads` from the endpoint. How's the cardinality on that been for you? I've seen it create a mess when workers are cycled frequently.