Just spent the last two days piping Helicone webhooks into a Grafana dashboard, and the visibility it gives is a game-changer compared to just their built-in UI. While the Helicone dashboard is great for a quick overview, I needed to correlate latency spikes with our own infra metrics and set up custom alerts.
The setup was pretty straightforward:
* Set up a small webhook server (used a Cloudflare Worker) to receive the `request.log` events from Helicone.
* Transformed and forwarded the data into a Prometheus instance (could also use Grafana Cloud's Loki or a time-series DB).
* Built dashboards in Grafana focusing on the metrics I care about most.
Here’s what I’m now tracking on a single pane:
* **P99 latency per model/provider** - Crucial for spotting when a specific provider starts to degrade.
* **Cost per day broken down by project** - More granular than Helicone's billing page.
* **Status code rate over time** - Immediate visual on 429s or 5xx errors from upstream providers.
* **Cache hit rate impact on latency** - Comparing cached vs. uncached P50.
The real benefit is being able to overlay this with our own application metrics. Seeing a correlation between our own service CPU usage and an increase in LLM token latency was an insight we wouldn't have had otherwise.
Has anyone else done something similar? I'm particularly curious about:
* How you handled the webhook volume at scale.
* If you found specific metrics outside of `request.log` that were valuable.
* Whether you're calculating token-based costs directly or relying on Helicone's aggregated cost field.
ms matters