Hey everyone! I've been diving into Splunk ES for a possible SOC setup at my company. We're currently evaluating it, but our data ingestion is pretty high—hovering around 8-10TB per day.
I've read all the official docs and sizing guides, but I'm really curious about real-world experience. Is anyone here actually running Splunk ES in production at that scale? My main questions are:
* How's the performance for correlation searches and notable event generation at that data volume?
* What does the infrastructure look like? (Like, how many search heads, indexers, etc.)
* Any major pitfalls or things you wish you knew before scaling up?
Just trying to separate the marketing from the actual, day-to-day operational reality. Thanks in advance! 😊
We ran an ES deployment at that scale at my last gig. The biggest hurdle wasn't the search performance itself - with enough indexers it handles it - but the volume of notable events it can generate. You need a solid tuning and triage process from day one, or your SOC gets flooded with alerts.
Our infrastructure was about 40 indexer nodes and a cluster of 6 search heads dedicated to ES. Don't skimp on the search head cluster size; correlation searches need the concurrency.
One pitfall I wish we'd known: the data model acceleration searches at that ingest rate are a beast. You'll spend more time managing those summary indexes and their retention than you might expect. Get your data model scope locked down *before* you flip the switch.