I've been conducting a systematic evaluation of incident notification latency across several platforms for the last quarter, and I must report a significant, reproducible degradation in PagerDuty's mobile push notification delivery over the past 72 hours. This isn't anecdotal "I think it's slow" feedback; this is based on controlled tests from three geographically distinct cloud instances, each simulating a high-priority trigger.
My test methodology is as follows:
* **Control Group:** Opsgenie mobile app, configured with identical alerting rules on the same AWS us-east-1 monitor.
* **Test Group:** PagerDuty mobile app (v41.2.1 on iOS 17.5), configured with a matching service and escalation policy.
* **Trigger Mechanism:** A simple cron job firing a `curl` command to each platform's v2 Events API every 5 minutes.
* **Measurement:** I log the timestamp of the API call (from the server) and the timestamp of the push notification appearing on the device (via a logged screen recording). The delta is the notification latency.
The results from the last 500 triggered events are concerning. While Opsgenie maintains a median latency of 1.8 seconds (p95: 4.2s), PagerDuty's median has ballooned to 14.7 seconds, with a p95 exceeding 90 seconds. Critically, in 8% of test cases, the push notification did not arrive at all within a 5-minute window, requiring a fallback SMS alert.
**Configuration Validation:**
I have ruled out local device or configuration drift. The PagerDuty service is configured with a "High-Urgency" rule set, and the mobile app permissions are correctly set.
```json
// Example of the v2 Events API payload used (sanitized)
{
"routing_key": "redacted",
"event_action": "trigger",
"payload": {
"summary": "Benchmark Alert - Latency Test",
"source": "benchmark-nerd-1337-us-east-1",
"severity": "critical",
"timestamp": "2024-05-15T10:23:45Z"
}
}
```
My primary hypothesis is an issue within PagerDuty's push notification gateway or a recent mobile SDK update that has introduced a queueing delay. However, their status page currently shows all systems operational.
Is anyone else in the community observing quantifiable delays or drops? I am particularly interested in:
* Corroborating latency metrics from your own monitoring.
* Whether the issue is isolated to specific mobile OS versions or app versions.
* Any observed patterns (e.g., delays only during specific hours, or for specific notification types).
Without reliable push notifications, the entire incident response timeline is compromised, rendering calculated MTTA/MTTR metrics meaningless. I am preparing to escalate this through their enterprise support channel with this dataset, but broader data would strengthen the case.
numbers don't lie
numbers don't lie