Has anyone else started scraping the new performance metrics from their Boundary workers yet? I've been running the 0.14.x series in a few dev clusters, and the new `/metrics` endpoint on the worker is a game-changer for visibility.
The standard Go metrics are fine, but the new `boundary`-prefixed ones are what I'm excited about. Finally, we can see:
* `boundary_worker_proxy_active_connections`
* `boundary_worker_proxy_bytes_upstream` (total bytes to/from targets)
* `boundary_worker_session_ingress_bandwidth_limit_percent` – super useful for spotting sessions hitting their caps.
I've been building a dashboard to track session establishment latency and worker connection churn. The initial scrape looks something like this in the Prometheus config:
```yaml
scrape_configs:
- job_name: 'boundary-workers'
static_configs:
- targets: ['worker-1:9202', 'worker-2:9202']
metrics_path: /metrics
```
Haven't seen much in the way of histograms or summaries for request duration yet, which is a bit of a miss. I'm also curious if the `bytes_upstream` counters reset on worker restart or if they're persisted anywhere.
What's everyone's experience been? Found any gotchas or particularly useful metric combinations for alerting on worker health? Keen to compare notes and dashboard panels.
- away
Totally agree on the new metrics being a game-changer. The `bytes_upstream` counter definitely resets on a restart, which caught me off guard when I was trying to correlate data over a weekly period that included a deployment. I ended up using `rate()` and `increase()` in Prometheus queries instead of the raw counter value.
I haven't seen any histograms either, which is a shame. I've been approximating session establishment latency using the `boundary_worker_proxy_active_connections` gauge and tracking its changes over short windows. It's not perfect, but it gives you a directional sense.
Have you looked at the memory metrics on the Go side alongside the new boundary ones? I've noticed some interesting patterns where connection churn seems to precede a spike in heap allocations.
✌️
Good catch on the counter reset behavior. Using `rate()` is the correct approach for any counter metric over a volatile time window.
On the histograms, they're a common omission in first-pass metrics implementations. I'd recommend logging an enhancement request. The workaround with the active connections gauge is clever, but you're right that it's only directional. For true latency, you'd need instrumentation at the session handshake level.
I've seen the memory pattern you mentioned. The correlation likely isn't direct causation. High connection churn increases GC pressure on the Go runtime, which can manifest as those heap allocation spikes. It's useful as a leading indicator for potential worker instability, but the root cause is still the churn, not the memory.
independent eye
You're approximating latency with the active connections gauge? That's like measuring network speed by counting blinking lights. It's not even directional, it's just noise.
And yeah, the memory pattern is a red herring. You're seeing a symptom, not a cause. The GC spikes are a distraction from the real issue of why you have connection churn in the first place. Fix the churn, the memory "pattern" disappears.
Counters resetting is Prometheus 101. If that caught you off guard, your baseline monitoring setup is shaky.
Trust but verify.