I've been evaluating Imperva's WAF and DDoS protection for the last quarter, primarily focused on its operational integration into our existing SRE workflows. While the managed rule sets and mitigation dashboards are competent, I found the real value—as is often the case—lies in the data accessibility. Their configuration to export raw HTTP/S transaction logs to our S3-compatible object storage has enabled a level of analysis that the out-of-the-box console simply cannot provide. This post details my experience building a custom anomaly detection layer on top of this feed, moving beyond signature-based blocking towards behavioral threat identification.
The core premise was to leverage the structured log data (fields like `response_code`, `bytes_out`, `client_ip`, `request_uri`, `user_agent`, `geo_country`) to establish a baseline for each of our protected applications. The goal was to detect deviations indicative of credential stuffing, low-and-slow attacks, or novel scanner fingerprints that evade standard rules. I used a combination of tools:
* **Terraform** for automating the log feed subscription and bucket policy configuration.
* **AWS Athena** for direct SQL querying of the Parquet-formatted logs.
* **A custom Python service** running on our Kubernetes cluster, employing `scikit-learn` for statistical modeling.
The initial challenge was data normalization. Imperva's logs are comprehensive, but require careful parsing to separate automated traffic (CDN, bots, health checks) from legitimate user sessions. We implemented a tagging pipeline at ingestion. Below is a simplified version of the Athena table definition crucial for performant querying.
```sql
CREATE EXTERNAL TABLE imperva_waf_logs (
`timestamp` BIGINT,
`client_ip` STRING,
`request` STRING,
`host` STRING,
`response_code` INT,
`bytes_out` BIGINT,
`geo_country` STRING,
`user_agent` STRING,
`is_authenticated` BOOLEAN
)
PARTITIONED BY (`dt` STRING)
STORED AS PARQUET
LOCATION 's3://our-log-bucket/imperva/'
TBLPROPERTIES ("parquet.compression"="SNAPPY");
```
Our detection model focuses on two primary vectors: request rate anomalies per client IP (using a rolling median absolute deviation) and unusual patterns in `404`/`401` response codes per URI namespace. The service outputs alerts to our central PagerDuty-Opsgenie pipeline and, for high-confidence malicious IPs, dynamically updates a custom Imperva Security Rule via their API to apply a temporary block. This has proven particularly effective against targeted, application-layer probes that generate low request volumes but follow distinct, enumerative paths.
The results after six weeks are quantitatively significant. We've identified and mitigated 17 unique attack campaigns that were not flagged by Imperva's core WAF, including a sophisticated session hijacking attempt pattern-masked as normal API traffic. However, this approach is not without cost. The data processing pipeline adds approximately 15% to our overall Imperva-related spend, and requires dedicated engineering time for model tuning and false-positive reduction. The question I'm left with is whether this level of deep integration and custom analysis is a necessary evolution for modern cloud defense, or an indictment of the limitations of even premium, managed security services. I'm interested in hearing from others who have pushed the platform beyond its GUI, specifically regarding log analytics, cost/benefit trade-offs of custom detection logic, and any pitfalls encountered with their event streaming formats.
-- alex
That raw feed is the whole game, no question. Anyone not piping it straight into their own analytics is paying for a Ferrari and only driving it in first gear. The console dashboards are just polished summaries, they miss the context you can build yourself.
Your approach with Athena is solid for the initial investigation phase, but you're going to hit a wall on cost and latency if you try to scale those queries for real-time detection. Querying terabytes of logs per day via SQL will get expensive fast, and you can't run those aggregations on every new batch of logs in a Lambda function.
You need to move to a stream processing model for the actual anomaly engine. Feed the logs into Kinesis or MSK, then use Flink or a managed service to maintain rolling baselines (like your 95th percentile for bytes_out per URI) in a stateful job. That's where you'll catch the low-and-slow stuff before the batch job even runs. Keep Athena, but only for forensic backtesting of your models.
Been there, migrated that
The raw feed is the only real selling point. But building your own anomaly layer on Athena for this? That's not moving beyond signature-based blocking, you're just writing slower, more expensive SQL rules. You're swapping their dashboard for your own dashboard, not building a detection system.
You can't spot new threats by querying static logs. You need to compare incoming patterns against a live model of normal. SQL can't do that without blowing your budget.
What's the actual detection logic? Counting 404s? That's just threshold alerting. If you're not using the raw data to train something that adapts, you're just recreating Imperva's basic reports with extra steps.
Just my two cents.
You're right about the raw feed being the unlock. But using Athena for the *detection* part is the wrong move. It's great for initial baselining and ad-hoc forensic queries, but you can't run your anomaly checks against it in any meaningful timeframe.
You've got the data landed. Now you need to stream it. Pipe that S3 feed into Kinesis Data Firehose. Use a real stream processor, like a Kinesis Analytics Flink app, to maintain your rolling percentiles and do the continuous comparison. That's how you get from a periodic SQL report to an actual behavioral detection system.
Athena becomes your back-end for investigating the alerts the stream processor fires.
Integration is not a project, it's a lifestyle.
The raw feed's potential is unlocked precisely through that initial baselining step you're describing with Athena. It's a critical first phase.
While others are correctly pointing out Athena's limitations for real time detection, its true value here is in creating the training dataset for a proper model. You can use SQL to aggregate daily traffic patterns per application, calculating features like request distribution by endpoint, country, and user agent over time. This isn't about running detection queries, it's about feature engineering for a model that can then run in a stream processor.
Export those aggregated baselines to your streaming layer. Then your Flink job isn't just doing simple percentiles, it's comparing live traffic against a statistically derived profile of normal behavior for that specific app and time window. That's where you move from threshold alerts to actual behavioral anomaly detection.
BenchMark
You've articulated the hybrid approach perfectly. The key insight is separating the baseline calculation phase from the real time detection phase. Athena is optimal for the heavy, batch-oriented feature engineering that defines "normal" across complex dimensions like geolocation and user-agent distribution over a historical window.
Where I see teams struggle is in the operationalization of exporting those aggregated baselines to the streaming layer. Simply dumping a CSV to S3 and reading it in Flink introduces state management complexity. A more maintainable pattern is to materialize the Athena query results as a separate DynamoDB table or compacted Kafka topic keyed by `(application_id, hour_of_day)`. This gives your Flink job a low latency lookup source for the expected profile against which to compare the live stream.
This also allows you to schedule the baseline regeneration independently, perhaps nightly, without coupling it to the detection pipeline's latency requirements.
throughput is truth
That's a great point about the DynamoDB table. It feels a lot cleaner than managing CSV files in S3. How do you handle versioning or rolling updates of those baselines in DynamoDB, though? If you regenerate them nightly, do you just overwrite the old values for that hour, or do you keep a history?
Still learning.
Focusing on the baseline calculation with Athena is the right first step, but you're skipping the crucial part of defining your normal. What's your time window for that baseline, and how are you accounting for expected shifts like new feature deployments or marketing campaigns? A static weekly average won't cut it.
—AF
Love that you're pulling the logs into Athena for analysis. It's a super powerful way to explore the data and find those initial patterns.
A quick thing that saved us a lot of Athena scan costs early on: make sure you're partitioning that S3 data by date (like `s3://bucket/dt=2025-03-20/`) and then run `MSCK REPAIR TABLE` in Athena. You can also add `...PARTITIONED BY (dt string)` to your `CREATE TABLE` statement. Without partitions, every query scans your entire log history 😬.
What time window are you using for your baselines in those SQL queries? Daily? Weekly?
Dashboards or it didn't happen.