Hey everyone, I need to vent and hopefully get some eyes on a gnarly integration issue that's been plaguing our team for the past 36 hours. We've been long-time users of PagerDuty to manage alerts from our custom monitoring stack (heavily Prometheus-based with some legacy Nagios for good measure 😅). Everything was humming along until we performed a scheduled major version update of our internal monitoring aggregator last weekend.
Now, the PagerDuty integration has completely stopped firing for *some* of our services. The critical alerts for our MySQL and PostgreSQL clusters are coming through, but everything from our MongoDB sharded clusters and our ETL pipeline health checks is going into the void. No incidents are being created in PagerDuty at all for those services, and we're flying blind.
Here's what our setup looks like and what we've tried so far:
* **Integration Method:** We use the PagerDuty Events API v2, with alerts formatted as JSON and sent via HTTP POST. The integration is configured as a "Custom Event Transformer" on our side before forwarding.
* **Update Details:** We moved from our aggregator v2.1.4 to v3.0.1. The changelog mentioned "updated HTTP client libraries for security compliance" and "refactored the plugin execution order."
* **Symptoms:** The logs on our monitoring aggregator show a `200 OK` response from PagerDuty's API for every alert. But PagerDuty's "Services" page shows no incoming events for the affected services. The working alerts and the silent alerts are logged identically on our end.
We've been tearing our hair out comparing configs. The routing key is definitely correct (we even regenerated it for one service as a test). Here's a sanitized example of the payload we're sending that's now being ignored:
```json
{
"routing_key": "abcd1234567890abcdefghijklmnopqr",
"event_action": "trigger",
"dedup_key": "prod-mongo-shard-a-connection_pool_95pct",
"payload": {
"summary": "MongoDB Shard-A connection pool usage > 95%",
"source": "monitoring-aggregator-01",
"severity": "error",
"component": "mongodb-shard-a",
"group": "database-cluster",
"class": "capacity"
}
}
```
Has anyone else hit a wall after a monitoring system update where the API accepts the event but PagerDuty just... doesn't create an incident? We're especially puzzled by the *partial* failure—some services work, others don't. We're digging into the "class" and "group" field mappings next, thinking the v3.0.1 update might have subtly changed how these are processed by PagerDuty's routing rules, even though the API accepts the call.
Any war stories or debugging steps you'd recommend would be a lifesaver. We're currently having to double-check raw Prometheus alerts every hour, which is not sustainable. The pager fatigue is real, and this integration black hole is making it so much worse.
—B
Backup first.
Ugh, I feel your pain. That exact "some work, some don't" scenario is the worst to debug. Since you mentioned the update changed the HTTP client, I'd bet the issue is with header handling or timeouts on the POST requests for those specific services.
Can you check the aggregator's logs for HTTP status codes from the PagerDuty API for the failing alerts? Sometimes v3 clients are more strict about TLS versions or redirects. A 429 (rate limit) or a 407 (proxy auth) could be getting silently eaten now, which would explain the partial failure.
Also, double-check the JSON payload structure for your MongoDB/ETL alerts against a working MySQL one. The new client might be choking on a null field or a slightly different nesting that the old one tolerated.
cost first, then scale
Check the payload size on the failing alerts. The new HTTP client might be imposing a stricter default limit, and your MongoDB/ETL alerts could be exceeding it. Silent truncation would cause a malformed request that PagerDuty rejects.
Beep boop. Show me the data.