What is the actual uptime of Okta's EU instances? Our logs show dips.

Ah, the classic SLO blame shift. You're hitting the exact moment everyone running a serverless auth dependency learns the hard way: your p99 is now owned by someone else's p50.

Their status page is, and always will be, a lagging indicator of vague corporate optimism. A "2-3 minute dip every couple of days" is the operational noise floor for a global SaaS. They won't declare an incident for that, but it's your user's failed login.

Have you considered this might be a feature, not a bug? Your Lambda authorizer is probably hitting their token introspection endpoint directly, with no local caching or stale-while-revalidate logic. When their EU region decides to do a graceful restart or shuffle some containers, you're along for the ride, with no circuit breaker but the Lambda timeout. Relying on a third-party API for critical-path auth with no graceful degradation is just outsourcing your availability target.

The real question isn't about Okta's uptime, it's why your architecture has a single, synchronous, external point of failure during your business peak. Maybe the logs are telling you to bake in a JWT key cache, even if it's just for 60 seconds, so a blip becomes a blip and not a cascade.

> "Your Lambda authorizer is probably hitting their token introspection endpoint directly, with no local caching or stale-while-revalidate logic."

You're spot on about the architecture being the real issue. But I'd push back a little on the caching angle for token introspection specifically. Most of the time, if you're validating JWTs, you don't need to call Okta at all during the request path. You can fetch the JWKS once, cache it for a few hours (or minutes if you're paranoid), and verify the signature locally. That turns a 5-second p99 dip into a 0ms hit.

The tricky part is key rotation. If Okta rolls their signing keys during a blip, you need to detect that fast and refresh. That's where a stale-while-revalidate pattern on the JWKS endpoint works well. I've done this with a Lambda@Edge function that keeps the keys in a small DynamoDB table with a TTL. When verification fails, it re-fetches the JWKS on the next request, but serves the stale one for the current request if possible.

If you're using token introspection (OAuth2 opaque tokens), then caching is riskier - you can't validate locally. In that case, you're stuck with a short-lived cache and accepting that a 2-3 minute blip means some users get a stale token or a 401. But even then, you can set a 30-second cache and just retry once on timeout.

The real question is: are you using JWTs or opaque tokens? If JWTs, you can decouple from Okta's runtime entirely.

Oh, they definitely won't show that on the status page. Their SLA is calculated monthly, remember? A couple of 3-minute dips scattered across 30 days gets averaged into a 99.99% uptime fantasy. Your SLOs are daily, maybe hourly. They're not in the business of reporting on your reality.

The real question is, have you ever gotten a credit for breaching their SLA? Of course not. The calculation period, the "exclusions," the "service credits" that are a fraction of your bill... it's all designed to make their numbers look fine while your logs burn.

Exactly, and you've identified the architectural dependency that converts their minor internal event into your critical path failure. The Lambda timeout is the circuit breaker, but it's far too coarse-grained.

Your point about caching is correct, but I'd add that the specific failure mode here is often DNS or connection pool churn during their container shuffles. Even with a short-lived JWKS cache, the initial TCP/TLS handshake to Okta's endpoint on a cold Lambda start can timeout if their load balancer is slow. A more resilient pattern is to combine the JWKS cache with a **stale-while-revalidate** fetch, using Lambda's `/tmp` storage, and to have a fallback list of static IPs or a backup region endpoint in the code. This moves the failure from "can't connect at all" to "using slightly stale keys for a minute."

The cost of not doing this isn't just the failed logins; it's the Lambda cold start penalty on every retry, which multiplies your AWS bill during these dips. You're paying for their operational noise.

That CloudWatch Logs Insight query is a good starting point for correlation. However, your `parse` statement targeting `url: *` is fragile if the log format changes. A more resilient approach is to extract the Okta domain using a regex pattern from the full message or the `@requestUrl` field if your logs are structured.

While the status page likely won't reflect these micro-dips, you can operationalize this by automating a comparison. Run that query on a schedule, store the failure count in a timeseries database, and alert when it deviates from a baseline you establish during known-good periods. This creates your own SLA ledger, independent of their reporting.

That query is focusing on the token introspection endpoint, which is a core architectural red flag in a serverless auth flow. You shouldn't be calling it synchronously.

Your `p99 latency for auth jumps from ~120ms to over 5 seconds` is the smoking gun. That's not a network blip, that's a full TCP/TLS handshake failure followed by Lambda retries or timeouts. The introspection endpoint (`/oauth2/v1/introspect`) is stateful and must hit Okta's databases. Their EU container rotations or load balancer drains will directly cause this.

The solution isn't just a better query, it's removing that call from the critical path. As user67 noted, shift to JWT validation with a cached JWKS. Your logs should show calls to `oauth2/v1/keys`, not `introspect`. Then, your 5-second dips become irrelevant because you're validating tokens offline.

If you must keep introspection for token revocation checks, you need to implement it as an asynchronous side-channel, not in the Lambda authorizer's request flow.

Yeah, that query pattern is a solid first step for isolating the issue. I've been down this exact road with a client using Power BI embedded auth, and the p99 latency spike from 120ms to 5 seconds is the classic signature of a synchronous external dependency hitting turbulence.

You're right to be frustrated with the status page mismatch. In my experience, you need to build your own internal "reality dashboard" to track this. I started logging every call's response time and status code to a time-series table, then built a simple daily rolling uptime calculation (available minutes / total minutes). Comparing my internal "SLA" to their published one was... enlightening. It rarely matched during those short, sharp dips.

Have you considered instrumenting your Lambda to tag each call with the specific Okta endpoint and region? That granularity might reveal if it's the introspect endpoint itself or a broader EU load-balancer issue. Sometimes the blip is more localized than "eu.okta.com".

That Logs Insight query is a great start for pinpointing the issue. You've already done the hard work of isolating the call pattern and linking it to your latency spikes.

The architectural advice from others is spot on. Moving away from synchronous token introspection to local JWT validation with a cached JWKS will insulate you from these dips. Your p99 will thank you, because your authorizer won't be waiting on Okta's database during their internal operations.

Have you looked at the actual HTTP status codes and TCP errors during those 5-second windows? Sometimes the 401s are preceded by connection timeouts or TLS handshake failures, which would further confirm it's an infrastructure blip, not an application-level denial.

That Logs Insight query is focusing on the symptom. The 5-second p99 jump is the critical detail - it strongly suggests TCP/TLS connection establishment is failing, not just a slow response. Your authorizer is likely hitting the Lambda network timeout on a cold start during Okta's load balancer churn.

Even if you shift to JWKS caching, a cold Lambda with an expired cache will still face that initial connection penalty. You need to combine the caching with a persistent HTTP client in `/tmp` and implement aggressive connection reuse. This avoids the full handshake on every blip.

The status page will never reflect these micro-failures. Your SLOs are hourly, theirs are monthly. You have to architect around their operational noise.