Check out what I made: a script to correlate WAF logs with our app errors.

For the last three major sprints, my team has been wrestling with an increasingly noisy WAF. While AWS WAF with its managed rules is a formidable shield, the sheer volume of blocked requests—particularly from the `AWSManagedRulesCommonRuleSet`—made it nearly impossible to discern actual attack patterns from legitimate traffic hitting unexpected application states. Our application logs (structured JSON in CloudWatch) were showing a rise in 5xx errors, and the question became inevitable: were these errors genuine bugs triggered by users, or were they side-effects of legitimate but oddly-formed requests being blocked by the WAF before reaching the origin, thus causing client-side retries that we weren't handling gracefully?

The standard approach of sampling WAF logs in Athena or diving into CloudWatch Insights felt too slow and detached from our operational cycle. I needed a tool that could, on-demand, correlate a spike in application errors with WAF blocks in the same timeframe and, crucially, by the same client IP. The goal was to answer one specific question: "For this batch of 5xx errors, which ones were preceded by a WAF block for that same IP, and what rule triggered it?"

I've built a Python script that leverages AWS's `boto3` library to perform this temporal correlation. It operates on a simple but effective premise:
1. Accepts a start and end time (in ISO format) and a CloudWatch Log Group for the application.
2. Queries the application logs for `5xx` HTTP status codes within that window, extracting key fields: `timestamp`, `requestId` (or `@request`), `clientIp`, and `path`.
3. Concurrently, queries the AWS WAF Logs (delivered to S3 in Parquet format via Kinesis Data Firehose) for `BLOCK` actions within a slightly preceding time window for the same set of client IPs.
4. Performs an in-memory join on `clientIp`, aligning each application error with any WAF block that occurred for that IP in the minutes before the error.
5. Outputs a consolidated report, highlighting which errors are likely collateral damage from WAF blocks and which are "true" application errors requiring developer attention.

Here is the core correlation logic of the script:

```python
def correlate_logs(app_errors, waf_blocks):
"""Correlates application errors with WAF blocks by client IP."""
correlated = []
uncorrelated = []

# Group WAF blocks by client IP for O(1) lookup
blocks_by_ip = {}
for block in waf_blocks:
ip = block['clientIp']
blocks_by_ip.setdefault(ip, []).append(block)

for error in app_errors:
error_ip = error['clientIp']
error_time = error['timestamp']

# Look for a WAF block for this IP within the last N minutes
relevant_blocks = []
if error_ip in blocks_by_ip:
for block in blocks_by_ip[error_ip]:
block_time = block['timestamp']
# Check if block occurred within a reasonable window before the error (e.g., 2 minutes)
if (error_time - block_time).total_seconds() <= 120:
relevant_blocks.append(block)

if relevant_blocks:
correlated.append({
'application_error': error,
'related_waf_blocks': relevant_blocks
})
else:
uncorrelated.append(error)

return correlated, uncorrelated
```

The script's output is a Markdown report. The most actionable section is the "Correlated Errors" table, which immediately shows patterns. In our first run, we discovered that 40% of our recent `502` errors on our API gateway were directly preceded by a WAF block from the `GenericRFI` rule against the same client IPs. This wasn't an application bug; it was clients retrying too aggressively after a block. The fix was twofold: tuning the specific rule's sensitivity and implementing a more forgiving retry policy with exponential backoff in the client SDK.

This tool has shifted our team's workflow. We now run this correlation as a first step in any incident involving elevated HTTP errors. It effectively filters the signal from the noise, allowing SREs to quickly route issues: WAF-triggered false positives go to the security/cloud-infra team for rule tuning, while pure application errors go directly to the development squad. It also provides concrete, data-driven evidence for WAF rule exclusions, moving us away from guesswork and towards a more resilient, layered defense.