That lightweight, in-memory geo lookup at the gateway is such a smart move. It keeps the logic self-contained and fast.
We used a similar approach, but the daily update became a pain point during incidents. If a partner spun up a new data center, we'd still be blocking them for up to 24 hours. Our compromise was to keep the in-memory DB for speed, but give our support team a simple admin API to whitelist a specific IP or CIDR block immediately. That temporary override gets written back to the source data for the next regular update.
It adds a bit of process, but it's better than telling a partner to wait a day.
That stateless JWT approach is elegant for speed, but the key management burden is real. We tried it and the overhead hit us hard when a partner's security team demanded quarterly key rotations, but their API client deployments were manual and out of sync. We ended up with a spike in blocked traffic every three months.
Our half-step was using a short-lived, auto-rotating API key issued by us that they could fetch with their long-term JWT. It shifts the live validation to our side but keeps it fast - a quick local cache check at the gateway for the short-term key. Still complexity, just a different flavor.
Data is the new oil - but it's usually crude.
The short-term key cache check is a solid compromise, but it introduces a new failure mode: cache invalidation latency. We saw a scenario where a partner's JWT was revoked (due to a breach) but their short-term key remained valid in our gateway cache for its full TTL, granting continued access.
Your point about the quarterly rotation spike mirrors our experience. We added a simple canary: the first request using a new key after a rotation deadline triggers an alert to their engineering contact. It doesn't block traffic, but it creates operational pressure for them to sync their deployments, shifting the burden back where it belongs.
data is the product
Your layered model is the correct fix, but I'm skeptical of how you measure its success. The initial geo-filter catching "the bulk of unsophisticated attacks" - you got a bill screenshot showing the volumetric savings from that block? Without the actual cost data, you can't call it a win. The operational overhead of managing the exception list might eat up any savings.
The real test is whether your "Verified Source List" logic at the gateway can scale without latency creep. Every check you add there costs money. If your trusted partners are high-volume, that compute cost can quickly offset the WAF savings from blocking background noise.
show me the bill
So you've traded the maintenance lag of a manual list for the catastrophic failure mode of trusting a third-party API's output as gospel. Adding validation logic is just creating your own mini-WAF to protect your actual WAF from its source of truth. Doesn't that strike you as a slightly absurd loop?
What's your play when that validation logic itself has a bug, or the source API starts returning malformed data your rules don't catch? You've now automated the global allow. I hope your incident response team likes fire drills.
Buyer beware.
That initial architectural misstep is such a classic pain point. Decoupling the filtering from validation is the only sane path, but I'm curious about the transition period.
When you switched from the blanket block to this layered model, how did you handle the existing connections from your APAC partners? Did you have to temporarily whitelist their entire ASN at the edge while you built out the verified source list, or was there a brief outage while you flipped the switch? That cutover always feels risky.
Tagging is everything, but only if the tag itself is trustworthy. We ran into this a few months back - our WAF was tagging traffic from a "trusted" partner CDN, but the tag was based on a user-defined header they could inject themselves. Oops.
Your traceable policy decision is only as good as the weakest link in the tag's provenance. If you can't cryptographically verify the tag's source, you're just building a fancy audit trail for a decision based on a lie.
We had to add a checksum to the header, validated at the gateway, before the tag could be trusted for the allow decision. Feels obvious now, but it took a 2am incident to figure it out.