Skip to content
Notifications
Clear all

Anyone else's Helicone cache hit rate suspiciously low?

2 Posts
2 Users
0 Reactions
70 Views
(@data_pipeline_newbie_42_v2)
Honorable Member
Joined: 6 months ago
Posts: 326
Topic starter   [#28060]

Hey everyone, hoping I can get some advice here. I've been implementing Helicone over the last few weeks to monitor our OpenAI API calls, and overall it's been super helpful for visibility! But I'm hitting a wall with the cache metrics.

My dashboard is showing a cache hit rate of like... 2-3%. That seems *incredibly* low given that we're intentionally trying to cache common, repetitive prompts. I feel like I must be configuring something wrong, but the docs seem straightforward?

Here’s a bit about our setup:
* We're using the Helicone proxy with the `Cache-Control` header set to `max-age=600` for eligible requests.
* Our prompts often have dynamic user IDs or timestamps, but we're using Helicone's custom cache key generator to exclude those fields.
* We're on the Growth plan, so the request limits shouldn't be an issue.

I even built a simple test loop that sent the exact same prompt 10 times in a row, and I only saw one cache hit. 😅 The logs show the `helicone-cache-status` as "miss" almost every time.

Has anyone else run into this? I'm wondering:
* Are there specific header configurations I might have missed?
* Does the cache not work on streaming responses? (We use both)
* Could it be something about how we're generating the auth tokens?

Any screenshots or examples of your working cache config would be a lifesaver. I just want to make sure I'm actually leveraging this feature before our usage scales up further. The cost savings from caching were a big part of why we chose Helicone!


null


   
Quote
(@emmap)
Reputable Member
Joined: 4 months ago
Posts: 240
 

Yeah, that's super low. We hit similar numbers early on. Two quick things to check:

First, your `max-age=600` is great, but make sure you're also setting `s-maxage` if you're using any intermediate proxies? It's a long shot, but we missed that for a week.

Second, and this was our big one, are you using the async or wait-for-response cache mode? If you're on async, the response goes to the cache *after* the first request finishes, so that initial identical test loop might only give you one hit on the very last call if the timing is tight.

Oh, and re: streaming - last I checked, they only cache non-streaming responses by default. There's a beta flag for streaming, but it's finicky.



   
ReplyQuote