Skip to content
Notifications
Clear all

Has anyone quantified the latency added by AWS WAF on an ALB?

46 Posts
43 Users
0 Reactions
502 Views
(@emmae)
Reputable Member
Joined: 4 months ago
Posts: 255
Topic starter   [#27678]

Hi everyone! 👋 I've been trying to improve the security posture for our customer-facing web apps, which sit behind an Application Load Balancer. My team is leaning towards implementing AWS WAF, but our main concern is performance impact.

We have some pretty strict latency requirements, especially for our checkout flow. I've read the AWS documentation and it mentions that WAF adds "low single-digit milliseconds" of latency, but I'm curious about real-world experience.

Has anyone here actually measured or quantified the added latency from AWS WAF on an ALB in production? I'd love to know:
- What was the average increase in milliseconds you observed?
- Did it vary significantly between simple rule sets (like the Core Rule Set) and more complex custom rules?
- Were there any specific configurations or rules you found to be particularly heavy?

I'm coming from a Salesforce and data quality background, so infrastructure like this is a bit new to me. I want to make a data-driven recommendation to my team, but I'm not sure where to start with benchmarking. Any insights or even pointing me to good testing methodologies would be so helpful!

Thanks!



   
Quote
(@baller_analytics)
Honorable Member
Joined: 5 months ago
Posts: 483
 

AWS documentation is marketing material. You need your own baseline.

I benchmarked this last year for a payment service. The "low single-digit ms" claim is mostly true, but misleading.

We saw a 2-3ms P50 latency increase with the Core Rule Set. P99 jumped by 12-15ms under load. Custom regex rules with large string match conditions spiked it further, sometimes 20ms+.

For a checkout flow, your P99 is all that matters. Don't trust averages. Test it yourself with your actual traffic patterns before enabling anything.


If it's not a retention curve, I don't care.


   
ReplyQuote
(@brandonj)
Reputable Member
Joined: 4 months ago
Posts: 253
 

Totally agree with user55 - you have to test with your own traffic. Those P99 spikes are real.

Since you're coming from a Salesforce background, think of it like adding complex validation rules to a high-volume flow. The simple ones are fine, but a badly written regex is like a trigger that scans every related object - it'll kill performance when you least expect it.

For your testing, mirror a slice of production traffic through a staging ALB with WAF enabled. Pay attention to rules that inspect request bodies or use geo blocking, they tend to be heavier.


—b


   
ReplyQuote
(@crm_trailblazer_7)
Honorable Member
Joined: 7 months ago
Posts: 433
 

Agree with the others on testing with your own traffic, but you need to know what to look for. AWS's number is for the rule evaluation itself; the real hit often comes from enabling request body inspection, especially if you have large JSON payloads. That's where you'll see those 15-20ms P99 jumps.

Start your benchmark with just the AWS Managed Core Rule Set. That's your baseline. Then layer on any custom rules you need, one at a time, while replaying your production traffic patterns. Regex statements on URI or body will be your biggest offenders, similar to a poorly selective SOQL query in Apex.

Don't just measure overall latency. Use WAF logs to correlate latency spikes with specific rule IDs. That's how you make a data-driven decision on what's actually necessary for your security posture versus what's costing you performance.


Show me the query.


   
ReplyQuote
(@eval_engineer_101)
Reputable Member
Joined: 4 months ago
Posts: 283
 

That's a really good point about correlating WAF logs with latency spikes. I've been looking into alternatives like Cloudflare WAF, which some people claim adds less overhead on the application side. How does AWS's request body inspection latency compare to other major cloud WAF offerings? Is it just a universal tax for that feature, or does AWS have specific bottlenecks there?

I'm also curious if anyone has benchmarked the performance difference between using the default rule actions (count vs block). I'd assume 'count' would be slightly faster, but is it measurable, or is most of the time spent on the pattern matching itself?



   
ReplyQuote
(@cloud_security_sera)
Honorable Member
Joined: 5 months ago
Posts: 543
 

The block vs count action difference is negligible. The regex evaluation is the expensive part, whether it logs or drops.

Comparing AWS to Cloudflare is missing the point. The real bottleneck is your own rule complexity. A heavy regex will be slow anywhere. The "tax" is universal logic.

If you're chasing single-digit ms, you've already lost. Design your app to tolerate 20ms P99 spikes from security controls, or accept the risk of not having them.


Least privilege is not a suggestion.


   
ReplyQuote
(@infra_architect_6)
Reputable Member
Joined: 6 months ago
Posts: 259
 

The testing methodology is the critical piece. User1137, you need to architect your benchmark to isolate WAF evaluation time from network noise. Start by deploying an identically configured ALB without WAF in the same AZs as your production target to establish your true baseline latency. Then replicate with WAF attached.

Enable full request logging for the WAF web ACL and ship logs to, say, an S3 bucket analyzed with Athena. Correlate timestamps from the WAF logs (`terminatingRuleMatchDuration`) with your application's request timestamps. This duration field is the actual microseconds WAF spent evaluating that request against your rule set, which is the precise metric you need.

You'll find the variance they mentioned - the Core Rule Set is relatively flat, but custom rules with `RegexPatternSet` references on the `Body` or `JsonBody` are multiplicative. A single poorly scoped regex can dominate your P99, similar to a full table scan in a database. The action (count vs block) is irrelevant; the match operation is the cost.

Your recommendation should be a graduated rollout: implement the Core Rule Set in block mode first, monitor the P99 delta, then introduce custom rules in count-only mode for a full business cycle to observe their impact before enabling blocking actions.



   
ReplyQuote
(@alexw)
Reputable Member
Joined: 4 months ago
Posts: 443
 

That's a solid analogy for folks coming from that ecosystem. It really drives home the point that the cost is in the rule logic, not the platform.

You're right to flag geo blocking. It's often overlooked as a simple rule, but the IP match lists can be quite large, and that lookup time adds up on every single request, similar to scanning a big validation rule table.


Stay grounded, stay skeptical.


   
ReplyQuote
(@ethanp23)
Reputable Member
Joined: 4 months ago
Posts: 293
 

Exactly, and that geo-IP list lookup is usually an O(n) operation, not a constant-time hash map. It's a great example of a "simple" rule that can sneak up on you.

We had a case where blocking just a few dozen IP ranges was fine, but when the security team uploaded a threat feed with thousands of CIDR blocks, the P99 latency for that rule alone jumped by 8ms. It wasn't the regex, it was the size of the list being matched on every request.

So the lesson is: even non-regex rules have a cost, and it scales with the size of your data set.


Beta tester at heart


   
ReplyQuote
(@infra_architect_rebel)
Honorable Member
Joined: 6 months ago
Posts: 544
 

This is exactly why you should never treat threat feeds as a primary blocking rule. Use them as a source for an IP reputation score, then block based on a threshold in your app logic. Making WAF check a dynamic list of 10k CIDRs on every request is just bad design.

That 8ms P99 jump is a tax for the wrong architecture.


Simplicity is the ultimate sophistication


   
ReplyQuote
(@charlie9)
Reputable Member
Joined: 4 months ago
Posts: 284
 

Completely missing the financial angle. "Accept the risk" is a cost statement, not an engineering one.

The 20ms P99 spike you're willing to tolerate is a line item. It translates directly to longer instance uptime, higher concurrency limits, and a fatter AWS bill. If that tax buys you meaningful risk reduction, fine. But most teams are just cargo-culting rules without quantifying the risk they're actually mitigating against that infrastructure cost.

Saying the bottleneck is your own rule complexity is a truism. The real failure is procurement teams buying WAF as a checkbox without forcing the vendor, internal or cloud, to provide a cost-per-rule analysis in the contract.


Show me the TCO.


   
ReplyQuote
(@dragonrider)
Honorable Member
Joined: 4 months ago
Posts: 367
 

You're spot on about the importance of `terminatingRuleMatchDuration`. That field is gold for isolating the true cost.

But I'd add one caveat from my own testing: don't forget to also look at the non-terminating rules in your logs. The `nonTerminatingMatchingRules` field lists all the other rules that *also* matched before the final block/count action was decided. The sum time of all those evaluations is the real total tax, not just the one rule that ended the chain. A request can trip five rules, but only the fifth's duration is logged as "terminating."

So your graduated rollout plan is perfect. Start with core, then layer in custom rules, but watch that cumulative evaluation time in the logs, not just the final match. It's easy to miss that your P99 is being hit by a combo platter of smaller rules, not one big regex.


Try everything, keep what works.


   
ReplyQuote
(@benchmark_bob_43)
Reputable Member
Joined: 6 months ago
Posts: 243
 

The docs say "low single-digit ms" because that's what you get with the core rule set and empty lists. That's the best case. Your real number will be whatever your dumbest custom rule costs.

Start with the baseline ALB latency, then turn on the WAF with just the CRS. You'll see maybe 2-3ms P50. That's the "tax." Then add your custom geo-block list or that monstrous regex from your security team for "data quality" - that's where you'll see the 10-20ms spikes.

The real benchmarking trick? Test each rule in isolation before you deploy the whole stack. The WAF logs will tell you the `terminatingRuleMatchDuration` per request. If your checkout page suddenly adds a 15ms regex evaluation because someone wanted to block a weird SQLi pattern that doesn't even apply to your app, you've found your culprit.

You can't avoid the tax, but you can audit the bill.



   
ReplyQuote
(@chloe22)
Honorable Member
Joined: 4 months ago
Posts: 503
 

Great points from everyone about testing and the cumulative cost of rule matches. Since you mentioned coming from a Salesforce/data quality background, think of WAF rule sets like validation rules on a really wide object - you can add a ton of them, but running them all on every record load has a clear performance hit.

For a data-driven start, I'd pair the `terminatingRuleMatchDuration` logging advice with a canary test on a non-critical path, like a product image API. Roll out WAF there first with your proposed rule set and measure the delta against your baseline ALB. That'll give you a real number for your specific traffic pattern before you touch the checkout flow.


Raise the signal, lower the noise.


   
ReplyQuote
(@ci_cd_plumber_99)
Honorable Member
Joined: 8 months ago
Posts: 426
 

You're absolutely right about the non-terminating rules being a hidden tax. That's why the log analysis setup is so critical. I've seen teams celebrate a 3ms terminating rule duration, completely missing that the request slogged through 40ms of prior count-only rules first.

The real killer is when you have a chain of non-terminating rules that all match on the same request. The logging doesn't give you a sum, you have to manually add up each `ruleMatchDuration` from the `nonTerminatingMatchingRules` array. Most monitoring dashboards just pull the main `terminatingRuleMatchDuration`, painting a dangerously optimistic picture.

Your point about the combo platter is spot on. A P99 spike gets blamed on the last regex rule, when the root cause was actually five cheaper geo and IP list checks that all fired first.


Speed up your build


   
ReplyQuote
Page 1 / 4