Having observed a critical gap in traditional SIEM solutions regarding cloud-native identity and access management (IAM) visibility, I developed a custom dashboard to track IAM changes across multi-cloud environments. While IBM QRadar provides robust log aggregation and correlation, its out-of-the-box content for cloud IAM often lags behind the rapid, API-driven changes inherent to AWS IAM, Azure AD, and Google Cloud IAM. This dashboard is designed to augment QRadar by providing a real-time, normalized view of high-risk identity events, which can then be forwarded as offenses for deeper investigation.
The core architecture involves three primary components:
1. **Cloud-Specific Collectors:** Lightweight Python agents utilizing official SDKs (Boto3, MS Graph API, Google Cloud Client Library) to poll audit logs for specific IAM event types.
2. **Normalization Engine:** A central service that maps heterogeneous cloud events (e.g., AWS's `CreateUser`, Azure's `Add user`, GCP's `google.iam.admin.v1.CreateServiceAccount`) into a common JSON schema.
3. **Dashboard & Alerting Layer:** A Streamlit application for visualization, with a secondary webhook service to create QRadar offenses for critical changes.
The normalization schema is pivotal. Here is a simplified example of its structure:
```json
{
"event_id": "cloud-iam-event-001",
"timestamp": "2023-10-27T10:00:00Z",
"cloud_provider": "aws",
"normalized_action": "USER_CREATED",
"target_principal": "arn:aws:iam::123456789012:user/new_admin",
"initiating_principal": "arn:aws:iam::123456789012:user/deploy_bot",
"source_ip": "192.0.2.1",
"raw_event": { ... },
"risk_score": 85
}
```
Key metrics and visualizations tracked by the dashboard include:
- **Principal Velocity:** Number of new users, service accounts, or roles created per cloud account over a rolling 24-hour window.
- **Permission Escalation Events:** Additions of high-privilege policies (e.g., `AdministratorAccess`, `Owner`) to existing principals.
- **Credential Lifecycle:** Key creation, deletion, or inactivity for service accounts and IAM users.
- **Cross-Account Trust Modifications:** Changes to STS assume-role policies or resource-based trust relationships.
Integration with QRadar is achieved via the HTTP-based QRadar API. When the dashboard's risk score for an event exceeds a configured threshold (e.g., 75), it triggers a POST request to the QRadar console, creating an offense with a meaningful description and the normalized payload attached as a custom event property. This allows analysts to pivot from the QRadar offense directly to the contextual dashboard view.
The main pitfalls encountered during development were:
- **API Rate Limiting:** Cloud providers enforce strict API call limits, necessitating intelligent batching and backoff logic in the collectors.
- **Log Latency:** Cloud audit logs are often delivered with a delay of several minutes, making true real-time monitoring challenging.
- **Schema Drift:** Cloud providers frequently update their audit log schemas, requiring continuous maintenance of the normalization mappings.
This approach does not replace QRadar's core functions but acts as a specialized, high-fidelity feed. It addresses the specific need for proactive IAM governance in cloud environments, an area where generic log parsing can miss nuanced but critical threats like gradual privilege creep or service account compromise. The code for the collectors and normalization engine is available in a GitHub repository for those interested in implementation details.
Interesting approach. I've been struggling to get decent IAM alerts out of our SIEM too. When you say you're polling for specific event types, how do you keep that list up to date as the cloud providers add new ones? Is it a manual process?
CloudNewbie
The normalization challenge you've outlined is a significant hurdle. While a common JSON schema helps for presentation, the real complexity surfaces when you try to build correlation rules on top of it. A `DeletePolicy` event in AWS carries a different blast radius than revoking a service account key in GCP, yet they'd map to a generic "permission revoked" type in your schema. How are you handling that semantic weight and contextual risk scoring before the event gets to QRadar? Simply forwarding normalized events might just create a different type of alert fatigue.
Your use of polling also introduces a latency versus cost trade-off. For a real-time view on high-risk events, you're likely polling CloudTrail, Azure Activity Log, and GCP Audit Logs at a very high frequency. That's a lot of API calls, and those costs can become non-trivial at scale across hundreds of accounts and subscriptions. Have you considered or benchmarked using service accounts/roles with managed event bridges (like EventGrid or Eventarc) to push events instead? It shifts the architecture but could reduce both latency and operational cost.
The Streamlit choice is interesting for rapid prototyping, but I'm curious about its operational fit for a 24/7 security dashboard. How are you managing state, user sessions, and the backend service reliability? I've found that for a persistent monitoring pane, even a simple Flask app with SocketIO tends to be more stable than a default Streamlit deployment over the long term.
Plan the exit before entry.
You've raised two critical operational points that we're actively iterating on.
The semantic weight problem is indeed where the real logic lives. Our normalization schema includes a `risk_profile` object that's populated by a rules engine before events are forwarded. This engine uses a combination of static mappings (e.g., AWS `DeletePolicy` is always `severity: critical`) and contextual enrichment. For the enrichment, it queries a cached inventory to add factors like whether the policy was attached to a production role, or if the GCP service account had keys older than 90 days. So, a generic "permission revoked" event forwarded to QRadar actually carries a calculated risk score and context tags. It's not perfect, but it moves the fatigue needle.
On polling versus push: we started with polling for control and universality. You're absolutely right about the cost scaling; our benchmarking showed API call costs becoming a real concern around the 80-subscription mark. We've since implemented a hybrid model for AWS and Azure using EventBridge and EventGrid for the high-volume, low-risk event types (like `List` calls), but we kept polling for the high-risk event types you'd want in real time. This lets us keep the poll frequency high for the critical stuff without the cost ballooning. GCP's Eventarc is on our roadmap, but the parity for Audit Log sources isn't quite there yet for our needs.
Data > opinions
That hybrid approach you've landed on makes a lot of sense. The cost curve with pure polling is brutal once you scale.
Your mention of cached inventory for context is crucial. We found that the time lag in updating that cache was a blind spot for us. If you query inventory and it's even a few minutes stale, you might mis-score an event on a resource that was just created. We had to tie the cache refresh directly to the event ingestion pipeline for high-risk categories, which added complexity but improved accuracy.
So on the push model for low-risk events, are you still performing any filtering on the provider side before they hit your EventBridge rules, or is it a simple forward of everything?
Data is sacred.
Yeah, the cache lag is a tricky one. I haven't built something this advanced, but I'm curious about the refresh strategy. When you tied the cache refresh to the ingestion pipeline, did you see a noticeable performance hit? It sounds like it could get heavy.
On the low-risk filtering question they asked, I'd also like to know. If you're not filtering before EventBridge, are you just accepting the cost of processing all that volume in your own pipeline? That seems like it might shift the cost burden instead of removing it.
learning every day