Hello everyone,
I've been running Helicone in our staging cluster for a few weeks now to get observability on our LLM API traffic, and overall I'm impressed with the cost tracking and latency dashboards. However, I've hit a snag with what seems to be inconsistent data in the logs, and I suspect it's related to the request sampling configuration.
Our setup uses the Helicone Helm chart behind an Istio ingress gateway, and we're proxying requests to OpenAI and Anthropic. The issue is that when I query the Helicone logs via the UI or the generated metrics in Prometheus, I'm seeing gaps. For a known period of high traffic from our load tests, the request count captured by Helicone appears to be only about 60-70% of what our application logs and the upstream providers' dashboards indicate.
Here's a simplified version of our `values.yaml` for the Helicone proxy:
```yaml
# helicone proxy values
worker:
env:
# ... other vars
SAMPLING_RATE: "1.0" # Intended to be 100%
SAMPLING_TYPE: "request"
cache:
enabled: true
maxSize: 10000
maxEntrySize: 1000
```
My understanding is that with `SAMPLING_RATE: "1.0"`, every request should be logged. I've verified the environment variable is correctly set in the worker pod. The cluster is under moderate load but not throttling on CPU or memory based on the pod requests/limits.
I've started debugging and have a few hypotheses, but I'd appreciate the community's insight:
* Is there a known interaction between the cache settings and request sampling? Could a full cache cause dropped log events before they're flushed to the database?
* Could there be silent failures or rate limiting on the backend (Postgres/ClickHouse) that aren't surfacing as errors in the proxy logs?
* Has anyone else observed this and found a required configuration tweak when running at scale (we're seeing ~50-60 requests per second at peak)?
My next step is to deploy a debug sidecar to tail the proxy's stdout, but I wanted to check here first to see if this is a documented pitfall. Any guidance on validating that Helicone is capturing a complete record, or recommended configurations for high-fidelity logging in a production context, would be invaluable.
kubectl apply -f
yaml is my native language
Ah, the classic "I set it to 1.0, where's my data?" issue. I've been there, and it's almost never the sampling rate itself.
First, double-check your proxy configuration. When you have `cache.enabled: true`, there's an interaction people miss. The cache stores *responses* to deduplicate, but under high load, the cache insertion/write can sometimes be a bottleneck, causing the proxy to prioritize forwarding the request upstream over logging it. Try setting `cache.enabled: false` temporarily and run another quick load test. If your capture rate jumps to ~100%, you've found your culprit.
Also, verify the logs for the proxy workers themselves. If they're being overwhelmed, they might be dropping events before they even get to the sampling logic. The 60-70% figure is a tell - a pure sampling issue would drop a more random/consistent percentage.
What's your `worker.replicas` count? Might need to scale up during those high-traffic periods.
one stack at a time
The cache interaction is a valid point, but the 60-70% capture rate under load points equally to the ingestion pipeline. The proxy logs the request asynchronously after it's forwarded, and if the internal queue to your analytics database (likely ClickHouse) fills up, events are dropped silently. You can confirm this by checking the `helicone_worker_queue_size` metric if you have it exposed.
While you test with the cache disabled, also look at your worker resources. The default Helm chart values for worker replicas and memory can be insufficient for a high-volume staging environment, causing the container to be OOMKilled and restarted during your load test, which creates precisely those gaps.
Spot on about the queue size metric, that's often the silent culprit. I'd add that even if the queue size metric looks normal, you need to check the actual log volume against the ClickHouse `max_insert_block_size` setting. If blocks are too large, the insert pipeline can stall and drop events without a clear queue backlog.
Also, I've seen the OOMKill pattern cause duplicate log entries for the same request after a restart, not just gaps, which can make the capture rate look artificially higher. The original poster should look for that, too.
grep is my friend.
Ah, that config snippet is useful. I had a similar thing happen, but it wasn't the cache or the queue for me.
You've verified the SAMPLING_RATE env var, but have you checked if your `values.yaml` has the `sampling` config under `worker` *and* under `proxy`? The Helm chart can have both, and I think the proxy one takes precedence for the actual proxy behavior. If you have `proxy.sampling` set to a default like 0.1 somewhere else in your values hierarchy, it would override your worker env.
Also, when you say you verified the environment, did you check inside the actual worker pod? I made the mistake of checking the deployment spec once, but a ConfigMap was overriding it at runtime.
Good catch on the precedence! The proxy.sampling setting does override the worker env. I fell into that exact trap a couple versions back.
But there's another layer: if you're using Istio, their sidecar injection can sometimes interfere with the proxy's own sampling logic, especially if you have any mesh-wide telemetry configs. I'd check for any Istio PeerAuthentication or Telemetry resources that might be applying a default sampling rate to outbound traffic, which could intercept the request before Helicone even sees it.
Might be worth doing a `kubectl exec` into a worker pod and echoing the env var, just to be absolutely sure what's in there at runtime.
Istio Telemetry resources are a solid angle, but I'd question whether they'd cause a consistent 30-40% data loss. Mesh sampling usually applies to trace spans, not HTTP request passthrough. The proxy should still see the full request; it just might not get a span attached.
The real test is that `kubectl exec` to check the env. But even if the var is correct, the problem could be in the library parsing it. I've seen floating-point rounding in the sampling logic cause weird skew under high concurrency, not pure drops. What does the sampled data look like? Is it truly random gaps, or are requests from certain model endpoints or users missing entirely?
-- bb