Best LLM observability for streaming applications with low latency needs

Configuration drift is the silent pipeline killer, and you're right to highlight it. That Helm chart update scenario isn't hypothetical, I've watched it happen because someone added a new `values.yaml` default to "improve debuggability."

You mention a slim SDK for async fire-and-forget. The catch there is you haven't eliminated a network hop, you've just moved it. Now your Go service's goroutine is blocking on a channel send to the telemetry emitter, or dealing with a full channel buffer. You trade sidecar resource contention for application thread contention. For sub-200ms P99, you often need to push the telemetry out-of-process entirely, which brings us right back to a hop of some kind.

The real hybrid approach is using the sidecar, but with its configuration baked into and validated by the service's own integration tests. If a chart update re-enables body logging, your service's latency tests fail before it reaches prod. It's more work, but it's the only way to sleep at night.

You've zeroed in on the core tension with the proxy model: that 1-5ms baseline is the floor, not the average, and it's entirely dependent on a stripped-down configuration. Your point about disabling non-essential features is crucial, but I'd add that "non-essential" needs to be defined by your alerting and debugging requirements, not the vendor's defaults.

Have you measured the delta between having the proxy in the chain versus making a direct call with only application-level timing? That would isolate the proxy's true cost from your own service's overhead. For a sub-200ms P99 target, even a 5ms consistent add might be your entire error budget, and that's before considering the drift others have mentioned.

It sounds like you're weighing if the observability features are worth that guaranteed latency tax on every single request, which is the right question. What's your threshold for acceptable overhead before you'd consider a more integrated, SDK-based approach?

Your collocated sidecar measurement of 1-5ms aligns with our internal benchmarks, but I think the more critical metric for a streaming application is the *jitter* introduced, not just the median. That predictable 5ms might be acceptable, but you need to verify the P99 of the proxy's own processing time, not just its contribution to your end-to-end latency. The buffer-and-forward mechanism for streaming tokens can introduce variable pauses, especially if the sidecar's logging channel is blocked waiting on disk I/O or network flush to its backend.

You mentioned disabling non-essential features. For your latency target, I'd argue the only essential features are token counting and timing, emitted as structured logs from the proxy itself. Every other feature, including request/response body logging (even sampled), adds scheduling complexity on the token stream. Our team moved to a custom-built sidecar that does exactly this: it taps the TCP stream via eBPF to extract token counts and latency histograms, then emits protobufs to a local agent. This cut our proxy overhead P99 from ~7ms to under 1ms.

The real question your breakdown prompts is whether you're measuring the right thing. Is your sub-200ms P99 for the *first token* from the LLM provider, or from the user's perspective through your entire stack? If it's the former, the proxy's overhead is a larger fraction of your budget than you might think. Direct provider calls with client-side instrumentation could shave off that entire variable component, but you lose the unified request view. For us, that trade-off wasn't worth it, but for your target, it might be the only viable path.