Hey folks, I've been evaluating our monitoring and security stack lately, and I keep hitting this conceptual wall. Vendors are all saying "our SIEM is now a data lake" or "our data lake can do SIEM." It's getting muddy.
From my platform engineering view, I think of them by their primary *job*. Our SIEM (we use a couple, depending on the cloud) is for **real-time, correlated detection and response**. It's about ingesting logs, applying rules, generating alerts, and maybe running a SOAR playbook. The data might only live there for 90-180 days because hot storage is expensive.
The data lake, in our case built on S3 + Spark/Trino, is for **massive-scale, cost-effective storage and broad analytics**. We dump *everything* there—not just security logs, but app metrics, billing data, low-fidelity traces—for years. We use it for historical threat hunting, compliance audits over long periods, and even feeding ML models.
The blur comes because:
* SIEM vendors are adding cheaper, longer storage tiers (but it's often still pricier than a proper data lake).
* Data lake platforms are adding real-time query engines and detection rule frameworks.
Here’s my practical, stack-based take:
* **Use the SIEM** for what it's optimized for: the security team's daily console, active investigations, and alert triage.
* **Use the data lake** as your single source of truth archive. You can run heavy, historical queries there without impacting your SIEM's performance or budget. Some setups even forward "interesting" events from the lake back to the SIEM for action.
What's your experience? Have you managed to keep them separate, or are you leaning into a consolidated vendor platform? I'm especially curious about cost control—the ingestion fees for some SIEMs give me heartburn 😅.
K8s enthusiast