Skip to content
Notifications
Clear all

Anyone else's Helicone cache hit rate suspiciously low?

1 Posts
1 Users
0 Reactions
50 Views
(@data_pipeline_tinker)
Honorable Member
Joined: 7 months ago
Posts: 364
Topic starter   [#17620]

I've been running Helicone in production for several months now to manage costs and latency for a moderate volume of LLM application traffic, primarily directed at OpenAI and Anthropic endpoints. My architecture is fairly standard: the application sends all requests through the Helicone proxy, which is configured to use Cloudflare R2 for caching. The goal, of course, is to serve repeated identical prompts from the cache to reduce cost and improve response times.

However, I've been consistently perplexed by the reported cache hit rate in the Helicone dashboard. It remains stubbornly low, often in the single-digit percentages, despite a high degree of predictability in a significant subset of our user queries. This seems to defy the operational logic of a caching layer. After substantial investigation into our own code and the Helicone documentation, I have a few hypotheses but would appreciate the community's experience.

My configuration for the Helicone proxy is as follows, emphasizing cache parameters:

```yaml
# helm values for helicone
cache:
enabled: true
type: "cloudflare"
cloudflare:
r2BucketName: "my-helicone-cache"
r2AccountId: "${CLOUDFLARE_ACCOUNT_ID}"
r2AccessKeyId: "${CLOUDFLARE_R2_ACCESS_KEY_ID}"
r2SecretAccessKey: "${CLOUDFLARE_R2_SECRET_ACCESS_KEY}"
# Using default TTL of 60 minutes
```

The primary suspect is the cache key generation. According to Helicone, the cache key is derived from a hash of the full request body, headers (including the model), and the target URL path. In practice, I've observed that even minor, non-semantic variations can produce a new key. My questions for the group are:

* Has anyone conducted a deep dive into the *exact* composition of the Helicone cache key? For instance, does the `user` field in the OpenAI request body factor in? If so, this would be disastrous for cache efficiency in multi-user applications, as each user ID would create a unique key even for identical prompt text.
* Are there known issues with non-deterministic elements in requests, such as the `seed` parameter? We do not use `seed` systematically, but I am curious about other potential culprits like timestamps or dynamic metadata injected by other middleware.
* Could the issue be related to the interaction between streaming and caching? A portion of our traffic uses streaming; the documentation is unclear on whether a streamed response is cached in its entirety for a subsequent non-streamed request, or if the modality itself bifurcates the cache.
* Finally, has anyone successfully implemented a custom cache key logic, perhaps by pre-processing requests in a sidecar or at the application layer, to normalize requests before they reach Helicone? For example, stripping out the `user` field or sorting JSON keys in the message array deterministically.

My next step is to set up a controlled experiment: sending a series of identical requests and inspecting the Helicone logs versus R2 object creation events. However, before I embark on that, I wanted to gather anecdotal evidence and potential solutions from others who may have already solved this. A low cache hit rate critically undermines the value proposition of the service, so I am keen to resolve it.


Extract, transform, trust


   
Quote