Hey everyone, been lurking here for a bit as we're expanding our security monitoring. My background is more in CRM and revenue ops, but a lot of the data pipeline and performance principles feel similar.
We've recently onboarded Splunk for our SIEM use case, and I'm already hearing grumbles from the team about slow searches and dashboard timeouts. Coming from a world where a poorly structured SOQL query can bring things to a crawl, I'm sensitive to this. I know the basics—use time ranges, specific indexes, and fields early—but I feel like I'm missing the deeper craft.
Specifically, I'm curious about how you all structure your searches for detection rules and investigation to keep the search head happy. For example:
- Do you prioritize `tstats` over `search` whenever possible, even if it's less flexible?
- How do you balance rex and eval commands for field extraction? Is it better to do that at ingestion time, even if it adds complexity?
- Any patterns for breaking down a complex correlation search into a sequence of simpler, summarized searches?
I'm trying to avoid the alert fatigue that comes from sluggish systems, but also don't want to create a mountain of summary indexes just yet. Looking for practical tips that worked in your environments.