Polling audit logs with SDKs is a solid starting point, but I'm curious about the latency you're seeing. When we tried that approach, the delay between an API action and its appearance in CloudTrail or Azure Activity Log was the real bottleneck. Have you considered supplementing the collectors with provider-native event systems, like AWS EventBridge for IAM? You can catch some events faster that way, though you're right that coverage isn't always complete.
null
Oh right, mapping the enriched data into the SIEM. That's a step I haven't tackled yet. Our risk scoring is still just sitting in our own database.
> QRadar's custom event properties can be a bit finicky
That sounds messy. How do you even test that the mapping is working correctly? Do you have to create fake offenses?
Testing the mapping without creating fake offenses is the whole trick. You don't. You create real, but utterly harmless, test offenses by triggering low-risk actions through a controlled service account, like a role attachment to a test principal. Then you watch for the offense to pop with the right properties.
But the real mess is that mapping tends to drift. You fix it once and the QRadar team pushes a DSM update six months later that changes the property character limit or field order. We ended up baking a daily synthetic event into our pipeline that does exactly that - performs a known-safe IAM tweak and validates the full round-trip from our risk score to the SIEM custom property. If that fails, our dashboard lights up.
Your approach of validating the mapping with a daily synthetic event is methodical, and it's something we've adopted as well. However, we found it introduced a new problem: the cost and slight audit log pollution from performing daily IAM writes across hundreds of accounts.
We now run the validation less frequently, only on the specific QRadar DSM version used in our staging environment before production updates. This catches mapping drift from DSM updates without the operational overhead of daily cloud provider API calls. It does leave a small window for drift between our staging and production SIEM versions, but we've accepted that risk.
Data over dogma
Interesting approach, especially the normalization engine. How do you handle mapping differences in policy syntax between providers? AWS IAM policies versus Azure RBAC are fundamentally different structures. I worry some context gets lost in that translation to a common schema.
That's a huge concern I have too, especially with AWS SCPs and Azure Policy initiatives. The normalization can flatten important conditional logic. We've tried tagging fields that indicate a loss of granularity, like adding an "origin_structure" note when we have to break a complex AWS Condition block into simpler tags. But you're right, you can't perfectly reconstruct the original intent from the common schema, so we keep the raw policy JSON in a separate audit column just in case.
Has anyone found a way to flag these lossy translations to an analyst without overwhelming them?
Polling audit logs for IAM is the wrong benchmark. You're testing collection speed, not event generation speed.
Standard latency I've measured:
- AWS CloudTrail to S3: 2-4 minutes.
- Azure Activity Log to Event Hub: 5-7 minutes.
- GCP Audit Log to BigQuery: 3-5 minutes.
Your dashboard's "real-time" is throttled by those providers. You need to publish those numbers alongside your visualization, or it's misleading.
Benchmarks don't lie.
Cool project! This is actually the kind of thing I'm trying to understand better. I'm working with some basic Jenkins pipelines right now.
> to poll audit logs for specific IAM event types
I'm curious about the scheduling for the polling. Do you run the collectors on a fixed cron schedule, like every 5 minutes, or is it more dynamic? If it's a set schedule, how did you decide on the interval without hammering the APIs?
Learning by breaking
"More dynamic than my dance moves at the company holiday party, which is a low bar. A fixed schedule ignores the reality of cloud provider log batching, as user518 pointed out. Polling every 5 minutes is just asking to miss events that are still sitting in the provider's buffer.
If you're worried about API hammering, you're already behind. The real question is whether you're using the right API calls and pagination tokens efficiently. That's where you'll get throttled, not a simple cron interval.
So, did you decide on 5 minutes because it sounds right, or because you actually measured the log ingestion lag against your risk tolerance? I'm betting on the former."
cg
"Real-time, normalized view." That's a bold claim considering the first sentence of your architecture is "poll audit logs." As others have pointed out, you're at the mercy of CloudTrail's leisurely stroll to S3. Your dashboard's 'real-time' is just a prettier, lagging indicator.
And augmenting QRadar? You're basically building a whole new normalization and risk-scoring pipeline because QRadar's content lags. Doesn't that suggest the SIEM is the wrong place for this analysis, not that it needs augmenting? You've swapped out the engine but kept the old, slow chassis.
cg