We've all been there. You configure a rate limit rule at the edge, confident it's solid. Then the pager goes off for a legit user getting blocked, or worse, you see the attack traffic sailing right past it.
The inconsistency is the real headache. Is it the rule logic? The key composition? Or just another "global" setting that isn't actually global?
I'm troubleshooting a case now where a rule based on `http.request.uri` works for `/api/v1/login` but not for `/api/v2/login`. Identical thresholds. The vendor docs just say "it works." It doesn't.
Anyone else find that these rules are more art than science? Where's the observability into the rate limiter's own decision-making? I'd rather see a log of counted requests per key than another marketing dashboard.
More dashboards != better ops
Absolutely feel your pain. That inconsistency between v1 and v2 endpoints is classic, and I've spent too many nights on similar issues.
In my experience, this often comes down to the key composition not being what you think. The rate limiter might be including the HTTP method, or maybe there's a trailing slash difference in how the paths are matched internally. The vendor's "it works" usually means the feature exists, not that your specific logic is sound.
You're spot on about the observability gap. We had to build our own sidecar logger just to count requests per key because the vendor's dashboard only showed blocks, not counts. It turned out our "identical" rule had a different cache duration due to a default profile setting on the v2 path. Maddening.
audit often
Yep, the cache duration profile setting is such a sneaky one. Had a similar headache where a global rate limit config was overridden by a separate "performance" policy attached to a specific API gateway route. The rules looked identical in the UI, but the applied TTLs were different.
Your point about building a sidecar logger resonates. We ended up piping our CDN logs through a quick Lambda to group and count by the rate limit key, just to verify the key composition. Found out it was hashing the full URI *including* query string parameters for one version but not the other. That explained the "inconsistency."
It really does feel like you need a separate monitoring system just to watch your rate limiter.
automate or die