While monitoring LLM costs and performance with Helicone is invaluable, operational stability requires integrating those insights into existing incident management systems. A passive dashboard is insufficient for time-sensitive production issues. This walkthrough details a concrete integration between Helicone's webhooks and PagerDuty to trigger alerts based on specific latency or error rate thresholds.
The core mechanism leverages Helicone's webhook feature to forward specific alerts to a PagerDuty REST API integration endpoint. The configuration involves two primary components: the PagerDuty service configuration and the Helicone alert webhook payload mapping.
First, create a PagerDuty service of type "Generic API". Note the generated integration key. You will then construct a minimal `curl` command to simulate an incident for testing, which informs our webhook payload structure.
```bash
curl -X POST https://events.pagerduty.com/v2/enqueue
-H 'Content-Type: application/json'
-d '{
"routing_key": "YOUR_INTEGRATION_KEY",
"event_action": "trigger",
"payload": {
"summary": "Helicone Alert: High Latency Detected",
"severity": "error",
"source": "helicone-prod",
"custom_details": {
"alert_name": "Latency Spike",
"threshold": "> 5s p95",
"metric_value": "7.2s"
}
}
}'
```
Second, within the Helicone UI, navigate to Alerts -> Webhooks. Create a new webhook targeting the PagerDuty API endpoint. The critical step is transforming Helicone's alert JSON into the PagerDuty v2 event schema. Below is a recommended mapping template for the webhook body:
```json
{
"routing_key": "YOUR_PAGERDUTY_INTEGRATION_KEY",
"event_action": "trigger",
"payload": {
"summary": "Helicone - {{.AlertName}}",
"severity": "{{.Severity}}",
"source": "{{.Provider}}",
"custom_details": {
"alert_name": "{{.AlertName}}",
"metric": "{{.Metric}}",
"threshold": "{{.Threshold}}",
"value": "{{.Value}}",
"time_window": "{{.TimeWindow}}"
}
}
}
```
Key considerations for a robust integration:
* Ensure the `severity` mapping aligns with PagerDuty's expectations (e.g., `critical`, `error`, `warning`). You may need to logic in a downstream middleware to map Helicone's alert levels appropriately.
* The `custom_details` object is crucial for providing on-call engineers with immediate context, reducing mean time to resolution (MTTR).
* Implement a lightweight middleware if you require deduplication logic or need to enrich the alert payload with additional environment data before PagerDuty receives it.
This setup creates a closed feedback loop where anomalies in LLM API consumption—whether cost, latency, or error-based—directly escalate into your engineering team's operational workflow, ensuring proactive response.
--crusader
Commit early, deploy often, but always rollback-ready.
Solid start, but your curl snippet got cut off. The payload mapping is the trickiest part. PagerDuty's custom details field is where you'll want to dump the full Helicone alert JSON, not just a static summary.
Test the escalation policy with a low severity alert first, or you'll wake up the wrong team for a non-critical latency blip.
So we're now paying for two services to tell us a third service is down? The cloud tax is compounding.
This setup will absolutely notify you when latency spikes. The question is whether Pagerduty's response time is better than your own team's. For the price, I hope so.
Beware of free tiers