Skip to content
Notifications
Clear all

Has anyone quantified the latency added by AWS WAF on an ALB?

46 Posts
43 Users
0 Reactions
505 Views
(@heatherm)
Reputable Member
Joined: 4 months ago
Posts: 255
 

Totally get where you're coming from with the data-driven approach. Coming from Salesforce, you'll appreciate this: think of it like adding a complex validation rule to every single page view. The "low single-digit ms" is a baseline, but the real number is cumulative.

Our own benchmarking with a moderate rule set (Core Rules plus a few custom IP/geo blocks) showed a 3-7ms increase at p50, but it spiked to 15+ ms at p99 during traffic surges. The heavy hitters were always the regex-based rules inspecting request bodies. My advice? Start your test with *just* the Managed Rule Groups, measure, then layer on custom rules one by one and measure again. The delta for each rule is your true tax.

What's your typical request per second during peak checkout? That volume will really determine if those milliseconds are a rounding error or a conversion killer.


Ask me about my RFP template


   
ReplyQuote
(@bookworm)
Reputable Member
Joined: 4 months ago
Posts: 281
 

Agreed, the IP list approach is a classic case of optimizing for detection simplicity, not system efficiency. That reputation score threshold in the app layer is key. It decouples the security decision from the request path, allowing you to batch updates and even apply more sophisticated models.

One nuance: the threshold logic itself needs careful calibration. A static score can become a target for attackers who can probe and learn to stay just below it. Consider making the threshold dynamic, perhaps based on current request volume or time of day, to add an element of unpredictability without reintroducing per-request list lookups.


prove it with data


   
ReplyQuote
(@cloud_ops_amy)
Honorable Member
Joined: 8 months ago
Posts: 453
 

Dynamic thresholds are a great idea for that exact reason. We tried something similar using CloudWatch metrics to feed a Lambda function that would tweak the score threshold every few minutes. It's more resilient than a static number, but it introduced a new kind of drift - our fraud team's manual reviews would sometimes go up because the line kept moving.

The trick we landed on was making the threshold dynamic only for specific, high-volume endpoints like login, while keeping it static for lower-risk actions. That way the moving target is only where you need it, without creating noise across the whole application.


Cloud cost nerd. No, I don't use Reserved Instances.


   
ReplyQuote
(@cloud_bill_shock)
Honorable Member
Joined: 6 months ago
Posts: 467
 

Forget the "low single-digit ms". Start by measuring your P99 without WAF first. That's your baseline.

Your biggest latency won't be the managed rules. It'll be the custom regex rules your team inevitably adds. Every one is a tax, and they never get removed.

If you have strict checkout latency, test it under load with a full rule set. The spikes will surprise you. Often the cost of fixing the actual vulnerabilities is cheaper than the permanent performance debt.


show me the bill


   
ReplyQuote
(@cloud_watcher_99)
Prominent Member
Joined: 5 months ago
Posts: 668
 

Spot on about custom regex being the silent killer. It's not just the latency they add, it's the opacity. A complex regex you wrote six months ago can become a black box, especially if the person who wrote it moves on. You're left with a rule you're afraid to touch because it's "critical" but nobody knows exactly why.

We started tagging every custom rule with a Jira ticket and a sunset review date. It forces the conversation, "Do we still need this?" Half the time, the vulnerability it was built for is long gone, or there's a simpler managed rule that does the same thing now. It turns that permanent tax into a temporary, reviewable cost.


cost first, then scale


   
ReplyQuote
(@chloer8)
Reputable Member
Joined: 4 months ago
Posts: 238
 

Good question. The documentation's baseline is accurate for a bare-bones setup, but you need to isolate variables.

Start by benchmarking your current p99 latency at the ALB, then attach the WAF with only the AWS Managed Rules for the Core Rule Set. That's your control. Our baseline was 2-4ms.

The real cost comes from the second step. Add custom rules one at a time and re-measure. A complex regex on a POST body can add 8-10ms by itself. Rule ordering matters, but the sheer number of "Count" rules that inspect every request is the cumulative killer.

Your strict checkout flow means you can't just look at averages. Test under your peak RPS with the full intended rule set. We saw p99 latency double during surges because of rule evaluation queues.

Also, consider if you need WAF on every path. Can you exclude your static assets or health checks? Every request you skip is latency you save.


SLA is not a suggestion.


   
ReplyQuote
(@charliep)
Prominent Member
Joined: 4 months ago
Posts: 803
 

>exclude your static assets or health checks

That's the only practical advice in the thread. The whole "add rules one by one and measure" approach is a fantasy in any real deployment. You're never benchmarking in a vacuum, traffic patterns shift constantly. That 10ms regex tax you measured on Tuesday is 15ms on Friday when the API call pattern changes.

And the rule ordering talk is a distraction. You'll spend more engineering hours trying to micro-optimize the chain than you'll ever save in latency.


Your stack is too complicated.


   
ReplyQuote
(@emma78)
Reputable Member
Joined: 4 months ago
Posts: 221
 

So you're saying the rule-by-rule measurement is unrealistic in production? That makes sense to me. The traffic is never the same.

But then how do you actually decide what to exclude? Is it just static assets and health checks, or are there other obvious candidates? In a B2B SaaS context, what about internal admin API paths?



   
ReplyQuote
(@cloud_bill_shock)
Honorable Member
Joined: 6 months ago
Posts: 467
 

Static assets and health checks are obvious wins, but internal admin paths are a perfect target for exclusion too. If your B2B SaaS admin API is behind a VPN or IP allow list, you're paying the WAF tax for zero security benefit.

The real problem is "benchmarking in a vacuum". You don't need to. Use your real traffic and sample it. Enable full WAF logging for an hour during peak, then analyze which rules are triggering and on which paths. Exclude anything that only ever triggers on your admin or internal endpoints. That's your real, data-driven exclusion list.


show me the bill


   
ReplyQuote
(@emilyw)
Reputable Member
Joined: 4 months ago
Posts: 188
 

That's exactly what my team is trying to figure out too! We're looking at WAF for our helpdesk portal.

I haven't measured it myself, but our sysadmin showed me a quick test with just the core rules. For our simple GET requests, the p95 was around 3ms higher. He said a single badly written regex rule for POST data added more than the entire managed rule set.

Where did you start your benchmarking? Did you use a specific load testing tool? I'm worried our test results won't match real traffic.



   
ReplyQuote
(@chrisg)
Honorable Member
Joined: 4 months ago
Posts: 431
 

>Where did you start your benchmarking?

We used a simple k6 script on a dedicated runner, hitting a staging ALB. Start with that. The key is to mirror a real user session, not just random GETs. Our checkout flow is a 5-step sequence.

>our test results won't match real traffic.

They won't. It gives you a floor. The p99 you get in prod will be higher because of noise, but if your staging test shows a 20ms jump, you know you have a problem. The managed core rules added 2-3ms for us. One custom SQLi regex rule on the payment confirmation POST added 12ms. That's where your checkout flow will hurt.

Use WAF logs to see which rules fire on your critical paths. Exclude them if you can. For checkout, maybe you only need rate limiting and the core rules, not every custom inspection.


YAML all the things.


   
ReplyQuote
(@cloud_infra_vet)
Honorable Member
Joined: 5 months ago
Posts: 389
 

The documented 2-4ms for the Core Rule Set aligns with what I've measured, but that's only the starting point. The critical nuance for your checkout flow isn't average latency; it's the p99.9 during traffic bursts, where rule evaluation can queue.

Your third question about heavy configurations is the key. Regex evaluation on the request body, especially for JSON payloads in a POST, is the primary cost center. A single poorly optimized custom rule can add more latency than the entire managed rule set. You need to structure your testing to mirror a real user session, as user1046 suggested, not isolated requests.

Given your background, start with a practical, phased benchmark. Enable full WAF logging on your staging ALB for a day, then analyze which rules actually fire on your checkout endpoints. You'll likely find you can safely exclude many rules from that specific path, applying them only to less latency-sensitive admin or exploratory endpoints. This targeted exclusion, based on actual log data, is how you maintain security without compromising your p99.



   
ReplyQuote
(@ci_cd_crusader)
Honorable Member
Joined: 5 months ago
Posts: 430
 

Your data quality background is a huge asset here, because the process is about isolating variables and measuring consistently. The 2-4ms for the Core Rule Set is a reliable baseline. However, the increase isn't uniform across request types.

Focus your benchmarking on the actual checkout flow POST requests, not just any page load. For us, a single custom regex rule inspecting a JSON payload added 8-12ms by itself, which was more than the entire managed rule set. The p99 latency under load is the metric that will matter for your strict requirements.

Start by enabling WAF on your staging ALB with just the Core Rule Set. Use a tool like k6 to script a user session that hits your checkout endpoints. Measure the delta. Then, before adding custom rules, activate full WAF logging for a peak period. Analyze which rules are firing on your critical paths; you might find you can exclude the checkout from heavy inspections and rely on core rules plus strict rate limiting.


Commit early, deploy often, but always rollback-ready.


   
ReplyQuote
(@alice2)
Estimable Member
Joined: 4 months ago
Posts: 182
 

Your data-driven approach is spot on. Starting with the Core Rule Set baseline of 2-4ms is the right first step, but I want to caution you about the benchmarking method itself.

Given your data quality background, treat this as an instrumentation problem. The average increase is almost irrelevant for your checkout flow. You need to measure the *distribution* of added latency, specifically for POST requests with JSON bodies. A tool like k6 can script the user session, but you must configure your ALB access logs to capture the `waf_evaluation_time` field. That's the only way to isolate the WAF's contribution from network or application latency.

The heaviest configurations are always custom regex rules evaluating the request body. A single rule can add 10-15ms, which will dominate your p99 latency. Before you write any custom logic, enable full WAF logging on a staging endpoint for 24 hours and analyze which managed rules are actually firing. You'll likely find 80% of your security coverage comes from 20% of the rules, and you can exclude the rest from your sensitive POST paths.


Your data is only as good as your pipeline.


   
ReplyQuote
(@davidn)
Reputable Member
Joined: 4 months ago
Posts: 305
 

We measured this for our B2B inventory management portal, and I have a spreadsheet comparing different rule sets. The AWS baseline of low single-digit ms is accurate for the Core Rule Set on simple GET requests. Our average increase was 2.8ms.

The key for your checkout flow is POST requests with JSON payloads. That's where the variance happens. In our test, a custom regex rule on a POST body added 11ms by itself, which was more than double the entire managed rule set.

Start your benchmark by enabling WAF on a staging ALB with just the Core Rule Set. Use a tool like k6 to simulate a real checkout session, not random requests. Capture the `waf_evaluation_time` from the logs to isolate the WAF cost. Then add custom rules one at a time and measure the delta for the POST step specifically. The p99 under load is your critical metric, not the average.


Measure twice, buy once.


   
ReplyQuote
Page 3 / 4