Your 1-5ms baseline is the best case scenario, and focusing on configuration is the right move. What are your actual sidecar resource requests and limits? Setting them too low can cause GC pressure that spikes latency, but setting them too high can cause node-level contention during deployments.
You also need to define what 'non-essential features' are for your specific use case. For a low-latency stream, features like full request/response body logging are almost always non-essential. Have you benchmarked with everything stripped out except for the core timing and error metrics you actually alert on?
—AF
> you might get more deterministic latency by instrumenting your Go service directly with a minimal SDK
This is the most reliable pattern we've landed on for sub-100ms P99s. The trade-off is now you own the instrumentation code and its load shedding logic. You'll spend less time tuning sidecars but more time maintaining a thin client that batches and asynchronously flushes to your collector.
We use a bounded channel with a non-blocking write. If the channel is full, the metric is dropped and a counter increments. That counter is the only synchronous telemetry call. It adds predictable overhead, usually under 100 microseconds, and you know exactly when you're losing data.
Show me the query.
That bounded channel pattern is solid for controlling overhead. We implemented something similar, but found the "drop counter" itself can become a bottleneck if you're not careful. If the channel fills up during a surge, you're trying to increment a counter synchronously on every single drop. That's still a function call and a memory write on the hot path.
We moved the drop counter increment to the async flusher goroutine instead. The channel pass includes a "dropped" boolean field. The trade-off is you lose real-time drop detection, but you keep the hot path to just a channel send attempt. The flusher aggregates and reports the total drops per batch.
Shifting the counter increment to the async flusher is smart. I've used that same pattern and it kept latency super consistent.
But that "dropped" boolean in the channel means you're now allocating an extra field for every single message, even when nothing is dropped. For a high-throughput stream, that allocation overhead adds up. We ended up using a separate, smaller channel just for a drop ticket, which the flusher counts. It's one more channel to manage, but avoids the per-message allocation.