We're using Imperva's raw log feed to detect unusual spikes in blocked requests. Goal is to catch things their default alerts miss.
Has anyone else built custom monitoring on top of the logs?
* What's the realistic polling/processing delay we should expect from their API?
* Found any gaps in the log data that break automated analysis?
* Best way to handle schema changes if they update the log format?
null
I've been running a similar setup on Imperva's raw logs for about 18 months now. A few observations on your specific points.
Polling delay: We see typically 2-4 minutes from event generation to log availability via the API, but during traffic spikes (e.g., a DDoS event or a flash sale) that can stretch to 8-10 minutes. The API's pagination is also fragile -- if you're polling with high concurrency, you'll occasionally get duplicate entries or gaps due to eventual consistency. I'd recommend using a single-threaded poller with a cursor or timestamp-based offset, and accept a small window of overlap on each fetch.
Gaps in the data: The biggest one is that certain event types (like rate limiting actions vs. WAF-specific rule matches) omit fields that are present in other event types. For example, `action` is always there, but `ruleName` or `threatType` can be empty even for blocked requests. That breaks any analysis that assumes a uniform schema across all entries. Also, timestamps are in a format that isn't strictly ISO 8601 -- they use a trailing space and a "Z" that's sometimes missing. We had to add a parsing normalization step.
Schema changes: They've changed the log format once without notice in the last year. They added a new field `geoLocation` in the middle of the existing structure, which broke our JSON parsing until we noticed. The only reliable way I've found is to process logs through a transformation layer (we use Fluent Bit with a custom Lua filter) that validates fields against a versioned schema. If a field is missing or a new field appears, we route those logs to a quarantine bucket and alert. That way the pipeline doesn't break completely, and you can adapt the schema mapping manually. Are you doing any schema validation upstream, or just ingesting raw JSON into a sink?
Data over dogma
Your point about the timestamp format is critical. We initially ingested with a standard ISO 8601 parser, which silently failed on those malformed entries, corrupting our time-series aggregation for nearly a week before we noticed the drift. We now enforce a regex-based validation and transformation step before any data hits our metrics pipeline.
I've also observed the schema inconsistency you mentioned, particularly around `threatType`. Our data shows it's absent in roughly 23% of blocked request entries where `action` is "BLOCK". This forced us to build a two-tiered classification system, first on the reliable `action` field, then enriching with `threatType` or `ruleName` when available.
Regarding their single format change, did you find the notification timeline adequate? We only received an API changelog entry two days prior, which was insufficient to update our deserialization logic and led to a partial ingestion outage.
No free lunch in cloud.
We hit that 2-4 minute API delay too, but I'd add a caveat on polling strategy. Using a timestamp cursor works, but you have to be careful with your clock sync. We saw duplicate data because our poller's clock was slightly ahead of theirs - we ended up using the log's own `sequenceId` for deduping in our streaming layer.
On gaps: the `clientType` field is another tricky one. It's empty for a lot of bot traffic, which skews your client categorization if you're not expecting it. Makes you decide whether to treat missing as "unknown" or try to infer it from other signatures.
For schema changes, we never got a formal notification. Our pipeline broke when a new nested field appeared. Now we use a schema registry with compatibility checking in our ingestion job, and anything that fails goes to a dead-letter queue for inspection. It's saved us a few times.