Hey everyone, I've been wrestling with this for months and I think our use case is a bit of a curveball for traditional SIEMs. We run a pretty complex data pipeline built mostly in Python (Airflow DAGs, Spark jobs, FastAPI services) on Kubernetes. Logs are a wild mix of structured JSON from our apps, stdout from legacy scripts, and metric exports.
We tried Sentinel. The log ingestion is fine, and I love the tight Azure integration, but the pain point is **correlating pipeline failures with infrastructure events**. When a DAG fails because a node pool scaled down, finding that link feels manual. KQL is powerful, but writing queries that join our custom Python application logs (which have things like `dag_id`, `task_id`, `experiment_id`) with Azure Resource Health events or AKS diagnostics... it's been clunky.
I built a proof-of-concept with a different stack, and I'm curious if anyone has gone down a similar path. Here's what I pieced together:
```yaml
# Example of our custom log structure (from a Python task)
{
"log_level": "ERROR",
"timestamp": "2024-01-15T10:23:45Z",
"component": "data_pipeline",
"context": {
"dag_id": "user_behavior_aggregation",
"task_id": "validate_s3_staging",
"run_id": "manual__2024-01-15T10:20:00",
"experiment_id": "exp_987"
},
"message": "FileNotFoundError on path s3://bucket/staging/2024-01-15/events.json"
}
```
The dream is to automatically see that this error occurred 2 minutes after a specific Kubernetes node was evicted, or that this `experiment_id` has a cascade of failures linked to a specific Azure region's network latency.
**My core questions:**
* For those with Python/K8s data workloads, what's your actual, working SIEM or observability pipeline? Do you use Sentinel primarily, or something else?
* How do you handle custom, high-cardinality context fields (like `experiment_id`) in your queries and alerts? Sentinel's cost model made us hesitant to ingest all that as custom dimensions.
* Have you built connectors to pull pipeline metadata (e.g., Airflow context) into your security/operational alerts?
I'll share my current benchmark setup in a follow-up, but really hoping to learn from this community's real-world scars and victories.
— francesc
— francesc