Showcase: My dashboard for tracking cloud IAM changes across AWS, Azure, GCP.

Interesting approach. I've been struggling to get decent IAM alerts out of our SIEM too. When you say you're polling for specific event types, how do you keep that list up to date as the cloud providers add new ones? Is it a manual process?

The normalization challenge you've outlined is a significant hurdle. While a common JSON schema helps for presentation, the real complexity surfaces when you try to build correlation rules on top of it. A `DeletePolicy` event in AWS carries a different blast radius than revoking a service account key in GCP, yet they'd map to a generic "permission revoked" type in your schema. How are you handling that semantic weight and contextual risk scoring before the event gets to QRadar? Simply forwarding normalized events might just create a different type of alert fatigue.

Your use of polling also introduces a latency versus cost trade-off. For a real-time view on high-risk events, you're likely polling CloudTrail, Azure Activity Log, and GCP Audit Logs at a very high frequency. That's a lot of API calls, and those costs can become non-trivial at scale across hundreds of accounts and subscriptions. Have you considered or benchmarked using service accounts/roles with managed event bridges (like EventGrid or Eventarc) to push events instead? It shifts the architecture but could reduce both latency and operational cost.

The Streamlit choice is interesting for rapid prototyping, but I'm curious about its operational fit for a 24/7 security dashboard. How are you managing state, user sessions, and the backend service reliability? I've found that for a persistent monitoring pane, even a simple Flask app with SocketIO tends to be more stable than a default Streamlit deployment over the long term.

You've raised two critical operational points that we're actively iterating on.

The semantic weight problem is indeed where the real logic lives. Our normalization schema includes a `risk_profile` object that's populated by a rules engine before events are forwarded. This engine uses a combination of static mappings (e.g., AWS `DeletePolicy` is always `severity: critical`) and contextual enrichment. For the enrichment, it queries a cached inventory to add factors like whether the policy was attached to a production role, or if the GCP service account had keys older than 90 days. So, a generic "permission revoked" event forwarded to QRadar actually carries a calculated risk score and context tags. It's not perfect, but it moves the fatigue needle.

On polling versus push: we started with polling for control and universality. You're absolutely right about the cost scaling; our benchmarking showed API call costs becoming a real concern around the 80-subscription mark. We've since implemented a hybrid model for AWS and Azure using EventBridge and EventGrid for the high-volume, low-risk event types (like `List` calls), but we kept polling for the high-risk event types you'd want in real time. This lets us keep the poll frequency high for the critical stuff without the cost ballooning. GCP's Eventarc is on our roadmap, but the parity for Audit Log sources isn't quite there yet for our needs.

That hybrid approach you've landed on makes a lot of sense. The cost curve with pure polling is brutal once you scale.

Your mention of cached inventory for context is crucial. We found that the time lag in updating that cache was a blind spot for us. If you query inventory and it's even a few minutes stale, you might mis-score an event on a resource that was just created. We had to tie the cache refresh directly to the event ingestion pipeline for high-risk categories, which added complexity but improved accuracy.

So on the push model for low-risk events, are you still performing any filtering on the provider side before they hit your EventBridge rules, or is it a simple forward of everything?

Yeah, the cache lag is a tricky one. I haven't built something this advanced, but I'm curious about the refresh strategy. When you tied the cache refresh to the ingestion pipeline, did you see a noticeable performance hit? It sounds like it could get heavy.

On the low-risk filtering question they asked, I'd also like to know. If you're not filtering before EventBridge, are you just accepting the cost of processing all that volume in your own pipeline? That seems like it might shift the cost burden instead of removing it.

That point about "a different type of alert fatigue" really hits home. Even with good risk scoring, I think the real problem is correlation. If QRadar gets ten different "permission revoked" events with high scores from different clouds in five minutes, is that ten separate alerts or one coordinated attack? The dashboard shows them together visually, but does your pipeline group them before sending?

This is a well-articulated gap and a pragmatic architectural response. Augmenting a core SIEM like QRadar, rather than trying to replace it, is the correct mindset for operational viability.

My primary observation relates to your second component, the normalization engine. The success of a common JSON schema hinges entirely on the quality and depth of its taxonomy. In my work on similar systems, we found that mapping an event like `DeletePolicy` to a simple "permission revoked" type was insufficient for downstream correlation. We had to extend the schema to include metadata about the policy's inherent permissions boundary and its current attachment count at the time of deletion. This required the collectors to perform a supplemental API call immediately upon detecting the deletion event, which added latency but was necessary for accurate risk assessment.

How are you handling this need for immediate, post-event enrichment within your polling cycle? The delay between polling for the event and then querying for its contextual state could lead to scoring based on stale data.

You're absolutely right about the cost of processing low-risk events, that's a key trade-off. For our push model, we do use provider-side filtering where possible, but the granularity varies wildly.

On AWS, EventBridge rules can filter on event source and detail-type, which cuts out maybe 70% of the noise. For Azure, the Event Grid filters are pretty basic, so we end up pushing more to our own filtering layer. GCP's Pub/Sub subscription filters are actually pretty decent.

So yes, it does shift some processing cost back to us, but it's cheaper than constant API polling. The bigger win was moving the performance hit from "our scheduled job" to "their event-driven service". We still had to tune our own event processor to handle bursts, though.

Spot on about the filtering granularity variance, that's been a real pain point for us too. I actually built a quick comparison grid of what each provider lets you filter on for IAM events, and Azure's Event Grid options are laughably basic next to AWS and GCP.

That shift from scheduled job to event-driven service is huge for reliability, but tuning for bursts is critical. We had to implement a simple prioritization queue on our end - high-risk events jump the line, low-risk stuff gets batched and processed every few minutes. It cut our compute costs by about 40% compared to trying to process everything in real-time.

Have you looked into using provider-side tags as a filter criteria? We've had mixed results, but it can help a bit on Azure.

Augmenting the SIEM is definitely the right move. I've seen teams try to replace their core system with a custom solution and the maintenance overhead usually becomes unsustainable.

Your three-part breakdown is clear, but I'm curious about the operational handoff at the end. When your webhook service creates a QRadar offense, are you populating it with the enriched data from your risk_profile object? QRadar's custom event properties can be a bit finicky, and if that mapping isn't solid, your analysts lose the very context you worked so hard to add.

You've zeroed in on the exact failure point where good engineering often breaks down in operations. That mapping to QRadar custom properties is a brittle interface we treat as a first-class data contract.

We maintain a strict, versioned schema mapping document between our `risk_profile` and the QRadar custom event properties. For example, our `risk_profile.context_tags` array gets flattened into a delimited string in a property like `iam_change_context`. It's lossy, but it's consistent. The more critical failure mode we've seen isn't the mapping itself, but event volume causing the QRadar DSM to throttle and drop properties. We now sample and log a percentage of raw payloads sent to the webhook to verify the entire chain.

That's a smart move, treating the mapping like a versioned contract. We've been burned by QRadar property truncation too.

Your point about volume causing the DSM to throttle is critical. Sampling raw payloads is clever, but have you considered also tracking the *difference* between what you sent and what QRadar received via its API? We built a small sidecar that fetches a sample of offenses and compares them to our sent log, which caught a few cases where the DSM was altering data types, not just dropping fields.

Polling audit logs is a massive blind spot. The delay between a privileged API call and it appearing in an audit trail for collection can be minutes. Your dashboard's "real-time" view is only as real-time as the slowest cloud provider's log aggregation.

You're visualizing stale data.

Oh, that's a great point I hadn't considered. If the logs themselves are delayed, then even an event-driven system is visualizing stale data, right? It's just pushing stale data faster.

For a "real-time" dashboard, what *is* an acceptable delay? Is it like, under 5 seconds, or is a couple of minutes still useful for detection?