I just finished a two-week load test on PingID's primary authentication flows, pushing it to 150% of our documented peak load. The goal was to see where it cracks, and more importantly, if the telemetry is good enough to tell us *why* before users complain.
The good news: the core auth held. The bad news: the monitoring gaps are almost as stressful as an outage. Here's what I measured and what you absolutely need to instrument yourself:
**Key Findings:**
* **Latency Spikes at the Token Validation Layer:** Under sustained load, 95th percentile latency for OIDC token validation jumped from ~120ms to over 900ms. The default PingOne dashboard only shows *average*, which stayed green and useless.
* **Database Connection Pool Contention:** This was the root cause. The system logs hinted at it, but we had to scrape the PingFederate runtime stats directly to confirm.
* **Alert Fatigue Setup:** Their out-of-box alerts fire on simple thresholds (e.g., "error count > 5"). During our gradual ramp-up, this caused 47 pages in an hour for a non-critical, self-recovering condition.
You need to export metrics to your own observability stack. Don't rely on their portal for operational health. Example of the Prometheus scrape config we added for PingFederate runtime:
```yaml
- job_name: 'pingfederate'
metrics_path: '/pf-metrics/prometheus'
static_configs:
- targets: ['pingfed-host:9999']
params:
filter: [
'pingfederate_jetty_requests_total',
'pingfederate_session_db_connection_pool_active_connections',
'pingfederate_oauth_authz_code_validation_time_seconds_bucket'
]
```
**Recommendations:**
* Define SLOs for *your* critical user journeys (e.g., "SP-initiated SAML flow under 2s at p99"). Ping's generic "system up" SLO won't protect your revenue.
* Instrument the flows end-to-end, including the PingID mobile app push response times. That's where we saw the highest variability.
* Bury their default alerts. Build your own composite alerts based on error *rates* and latency burn rates.
The platform is solid, but you're flying blind if you think their monitoring dashboard is sufficient for running it in production at scale.
--monitor
alert only when it matters
You've hit on the critical blind spot. Relying on vendor averages is a direct path to missed SLAs. Your point about p95 latency for token validation diverging from the average is classic. We saw similar behavior with a SAML flow where the average was stable but the p99 shot up due to garbage collection pauses in the underlying JVM, a metric completely absent from their portal.
Did you instrument the actual connection pool metrics, like `active_connections` vs `max_pool_size`, from the data source directly? We had to use a sidecar to collect those from the app server's MBeans. Without that granularity, you're just guessing at the contention. Their logs might say "pool exhausted," but you need the trend data to size it correctly.
Also, the alerting setup you described is why our first runbook step for any vendor service is to mute their default alerts and recreate them in our system with burn-rate thresholds.