We've been running AWS WAF with the managed rule groups, specifically the `AmazonIPReputationList`, for several high-traffic API services for about 18 months. While its efficacy in dropping blatant malicious traffic is undeniable, the operational overhead of handling its false positives has become a significant pain point. The documentation is notably vague on the specific heuristics, and the "reputation" metric feels like a black box.
Our primary issue manifests with legitimate users on certain ISP networks (we've observed patterns with major cellular providers and some European broadband pools) being blocked outright. These aren't just scored requests; they are hard blocks from the reputation list. The standard answer is to create an allow list rule with a higher priority, but this feels like whack-a-mole and directly undermines the security value of the managed list. We've had to implement a labor-intensive process:
* **Real-time alerting:** Any 403 from the WAF (not from our app) triggers a PagerDuty alert and logs the full request to a dedicated S3 bucket for analysis.
* **Manual triage:** An SRE must manually verify the request pattern against our internal metrics (user session history, known geographic patterns) to determine if it's a false positive.
* **Rule patching:** If confirmed false positive, we add the specific IP (or CIDR, if we're confident) to a manual allow list rule positioned *above* the managed rule group. This list is now uncomfortably large.
This is neither scalable nor secure. I'm deeply skeptical of any "set it and forget it" claims around these managed lists.
My questions to the community are:
1. **Analysis & Tuning:** What is your forensic workflow for investigating these blocks? Are you using Athena to query WAF logs joined with application logs to establish user legitimacy? I'm looking for concrete SQL examples or a structured analytics pipeline beyond CloudWatch Insights.
2. **Automated Mitigation:** Has anyone successfully implemented a more elegant, automated solution? I'm considering a Lambda function triggered by WAF logs that can temporarily allow an IP (with a TTL) based on a secondary factor (e.g., successful authentication attempt from a different endpoint), but the security implications of auto-allow are non-trivial.
3. **Rule Logic:** Do you prioritize the `AmazonIPReputationList` in `Count` mode and then use a separate, more granular rule (e.g., rate-based + high threat score) to actually block, thereby adding a review buffer? If so, what's your downstream logic?
I'll start by sharing a snippet of our current, admittedly clunky, Terraform for the allow list, which highlights the problem of managing IPs outside the WAF's native scope.
```hcl
resource "aws_wafv2_ip_set" "manual_allow" {
name = "manual-false-positive-allow"
scope = "REGIONAL"
ip_address_version = "IPV4"
addresses = [
"192.0.2.0/24", # Example CIDR from a problematic ISP
"203.0.113.1/32" # Specific IP from a legitimate partner
]
}
resource "aws_wafv2_web_acl" "main" {
# ... other config ...
rule {
name = "manual-allow-false-positives"
priority = 1 # Highest priority, overrides managed rules
action { allow {} }
statement {
ip_set_reference_statement {
arn = aws_wafv2_ip_set.manual_allow.arn
}
}
visibility_config { ... }
}
rule {
priority = 10
override_action { none {} }
statement {
managed_rule_group_statement {
name = "AWSManagedRulesAmazonIpReputationList"
vendor_name = "AWS"
}
}
visibility_config { ... }
}
}
```
This feels like we're fighting the tool. What alternative architectures or tuning strategies have you found effective?
-- alex