Skip to content
Notifications
Clear all

Results after tuning: Reduced our ES daily alert volume from 200 to under 20.

3 Posts
3 Users
0 Reactions
71 Views
(@datadog_dave_3)
Reputable Member
Joined: 7 months ago
Posts: 359
Topic starter   [#4788]

Our team recently concluded a significant tuning exercise on our Splunk Enterprise Security deployment, and the results were more dramatic than we anticipated. We successfully reduced our daily alert volume from an average of 200 to consistently under 20. This wasn't achieved by simply turning down sensitivity, but through a methodical, data-driven process.

The primary issue was alert fatigue, largely driven by redundant and low-fidelity correlation searches. We started by analyzing a 30-day history of notable events, categorizing them by source, urgency, and outcome. We found that over 70% of our alerts were from a small subset of rules that triggered on benign operational noise. For example, a single search for "unusual process execution" was generating dozens of daily alerts, almost all of which were from approved administrative tools.

Our tuning process followed these steps:
1. **Baselining:** For each noisy correlation search, we ran it over a historical period and exported the results to a CSV. We then worked with the relevant system owners to annotate each event as "true positive," "false positive," or "expected behavior."
2. **Refinement:** Using that data, we refined the search logic to exclude known-good patterns. This often involved adding more specific exclusions or requiring a stricter sequence of events.
3. **Threshold Adjustment:** For searches tracking failed logins or access attempts, we moved from static thresholds to statistically derived ones based on day-of-week and system role.

Here is a simplified example of a search before and after tuning:

**Original Search:**
```
index=windows EventCode=4688 | stats count by Process, host | where count > 5
```

**Tuned Search:**
```
index=windows EventCode=4688 Process IN ("msiexec.exe", "powershell.exe", "cmd.exe")
| search NOT [ | inputlookup admin_workstations | fields host ]
| stats count by Process, host | where count > 10
| lookup process_whitelist_lookup Process OUTPUT is_whitelisted | where is_whitelisted!=1
```

The key additions were filtering for specific processes of interest, excluding events from known administrative workstations via a lookup, and implementing a higher, more contextual threshold followed by a whitelist lookup.

The outcome is a more manageable signal-to-noise ratio. SOC analysts now engage with nearly every notable event, and mean time to response has improved. It's a clear reminder that the default content packs require substantial adaptation to your specific environment. While this level of manual tuning is intensive, the operational payoff is undeniable. In comparable Datadog workflows, this iterative tuning is often more integrated into the alert creation and lifecycle management interface, but the underlying principle of refining based on historical data remains universal.


null


   
Quote
(@martech_hoarder)
Trusted Member
Joined: 7 months ago
Posts: 47
 

Fantastic results, and I love that you led with the data. That baseline and annotation step is everything. It turns a gut-feeling tuning session into a repeatable process.

It's so similar to cleaning up a marketing automation flow. You start with a "lead scored" alert that fires 50 times a day, and 48 are from your own internal team's testing IPs. Without that annotation step, you're just guessing.

One thing I'd watch out for now is the change management side. When you cut volume that dramatically, some teams get nervous they're missing something. We found it helpful to create a simple weekly digest of the *suppressed* alerts (just a count by rule) to show stability. It keeps everyone confident the baby wasn't thrown out with the bathwater.


one stack at a time


   
ReplyQuote
(@devops_rookie_james)
Reputable Member
Joined: 5 months ago
Posts: 335
 

That's a huge improvement, and the data-driven approach really resonates. I'm just starting with alert tuning in my own CI/CD monitoring, so seeing the concrete steps helps a lot.

When you exported to CSV for baselining, how did you handle the collaboration with system owners? Was it a shared spreadsheet, or something more integrated? I'm trying to avoid creating a ton of manual work for other teams while we sort out our noisy pipeline alerts.

The shift from 200 to under 20 daily alerts is pretty inspiring. Makes me think we should run a similar analysis on our Jenkins failure notifications.


Learning by breaking


   
ReplyQuote