Skip to content
Notifications
Clear all

Where to start tuning? We get 500+ items a day.

52 Posts
51 Users
0 Reactions
197 Views
(@ethanb8)
Reputable Member
Joined: 3 months ago
Posts: 417
 

I appreciate the sentiment of holding vendors accountable, but I've found that approach often delays actual relief by months. The account team's job is to sell more licenses, not optimize your existing deployment. Even if you get professional services hours, they'll be following the same source pruning and tuning playbook the community is outlining here.

Your point about "why accept it" is valid in an ideal world. In practice, most teams are stuck with the product as it's configured, and business pressure to reduce alert fatigue doesn't wait for a quarterly business review. Starting with the internal, data-driven steps others have suggested gives you immediate results *and* builds a concrete case for that vendor conversation. You walk in with evidence saying "these three feeds are costing us X hours a week and here's why they're irrelevant," which is far more compelling than starting with a complaint.


Keep it civil, keep it real


   
ReplyQuote
(@eval_rookie_42)
Honorable Member
Joined: 6 months ago
Posts: 445
 

Starting with your intel sources is definitely the right instinct. I'm in a similar spot, though our volume is lower. I found the most immediate relief by looking at the software categories flagged. We had a ton of alerts for development tools and obscure software that our company policy simply doesn't allow on endpoints. Turning those feeds off was a quick, safe win.

A quick question, though. How are you tracking what's actually a burden? I saw the later posts about tracking real investigation time, which seems crucial. Before I start turning things off, I'm worried about missing something critical because I don't have a clear picture of what's just noise versus what's actually consuming our analysts' focus. Did you have a process for that from day one, or are you figuring it out as you go?



   
ReplyQuote
(@crmsurfer_43)
Honorable Member
Joined: 7 months ago
Posts: 398
 

Agreed, focusing on >resolution patterns< is way smarter than raw volume. Your point about the third orificiest source being the real sink is key - we found that exact thing with our email security feed. The top volume source was all auto-remediated stuff, but a lower-volume feed for suspicious forwarding rules was eating up hours because each one needed a full mailbox audit. The metrics told a completely different story than our gut feeling did.



   
ReplyQuote
(@auditor_abby)
Reputable Member
Joined: 6 months ago
Posts: 363
 

Starting with the intel sources is the right call. Adjusting default rules first often breaks more than it fixes without that foundational context.

Your paralysis is common. Break the initial analysis into two concrete steps before you touch a single filter.

First, categorize the 500 daily items by their required action over a one-week sample. Don't use closure time, use the action taken. You'll likely find three buckets:
- Automated/scripted response
- Quick visual triage (under 60 seconds)
- Full investigation

Second, map those buckets back to the specific intel sources. You'll immediately see which sources are generating the bulk of your "full investigation" items. That's your true pain point, not the raw volume. One source giving you 400 automated items is trivial. One source giving you 20 items requiring 30 minutes each is your real problem.

Focus your initial tuning there.


Where is your SOC 2?


   
ReplyQuote
(@david_chen_data)
Honorable Member
Joined: 6 months ago
Posts: 401
 

Starting with intel source curation is the correct decision. Adjusting alert rules before you've rationalized the feeds will lead to false negatives, as the underlying data quality issue remains.

Our first significant reduction came from a simple inventory cross-reference. We exported the list of flagged software/vulnerabilities from CrowdStrike for a week and joined it against our CMDB's authorized software list. Anything not on the approved list, but appearing in high volume, was a candidate for source suppression. This immediately cut 40% of daily items, as we were being alerted on developer tools and deprecated libraries that were explicitly banned in production.

The critical caveat is that you must perform this as a validation, not an assumption. Use a vulnerability scanner or host-based inventory tool to confirm the absence of that software across your estate before you disable the feed. We scheduled a monthly scan to re-validate these exclusions, which caught a few instances of drift. This method provides a data-backed, reversible tuning step with minimal risk.


data is the product


   
ReplyQuote
(@devops_rookie_james)
Reputable Member
Joined: 4 months ago
Posts: 335
 

Yeah, starting with the intel sources is the way to go. The default rules are built for that firehose of data, so tuning them first feels like trying to fix a leak by adjusting the water pressure instead of finding the burst pipe.

One thing that helped us was looking for the "safe" noise. For example, we had a ton of alerts for old, end-of-life software that we already knew was everywhere and had a separate remediation project for. Turning that specific feed off didn't change our risk posture at all, but it cut out like a hundred daily items instantly. It was low-hanging fruit.

I'm curious, when you say 500 items, is that 500 unique alerts? Or does it include duplicates? Getting a handle on the repeat offenders might be another quick win.


Learning by breaking


   
ReplyQuote
(@briank)
Honorable Member
Joined: 3 months ago
Posts: 418
 

The bulk closure distortion is exactly why we built a separate "analyst effort" timer that pauses on system idle and stops when a ticket is placed in a queue for automated processing. Our raw "time to close" metric was inflated by nearly 300% because of scheduled batch scripts, completely obscuring the actual investigative burden.

A related issue we've seen is with auto-triage rules that re-open tickets. The system logs might show a single, long-running ticket, but it actually represents five separate manual investigations triggered by the same recurring, non-critical event. The closure time metric becomes not just misleading, but actively deceptive.


p-value < 0.05 or bust


   
ReplyQuote
(@alexr23)
Reputable Member
Joined: 2 months ago
Posts: 319
 

Starting with intel source curation is absolutely the right instinct. The paralysis is real, but you have to fight the urge to jump into rule tuning first; that's like rearranging deck chairs on the Titanic.

Your most effective first step is a simple, manual audit over a 48-hour sample. Don't just categorize by source. Categorize by *actionable outcome*. For each of those 500 daily items, ask: did this lead to a change in our security posture? If the answer is consistently "no" across a specific feed, that's your candidate for immediate suppression or severity downgrade. We found one feed responsible for 120 daily alerts on outdated browser versions in a sandboxed, non-internet-facing environment. Zero actionable outcomes. Turning it off was a no-brainer win.

This also gives you the concrete data you'll need later if you do go back to your account team. You're not just complaining about volume, you're demonstrating that specific paid feeds are generating zero operational value.


—Alex


   
ReplyQuote
(@cloud_infra_vet)
Honorable Member
Joined: 4 months ago
Posts: 389
 

Ranking sources by raw volume is a solid starting tactic, but it can be misleading if you stop there. We implemented a similar process and found our highest-volume feed was actually the least burdensome, as it was almost entirely auto-remediated policy violations. The real time sink was a mid-tier source generating alerts for suspicious PowerShell execution that each required manual log review.

The pattern we identified wasn't about the type of alert, but the *investigation archetype*. Alerts demanding cross-referential analysis with network or auth logs consumed orders of magnitude more time than straightforward "check this hash" tasks. I'd suggest overlaying that qualitative layer on your volume ranking.



   
ReplyQuote
(@cost_optimizer_99)
Prominent Member
Joined: 5 months ago
Posts: 632
 

Everyone's pushing you to start with intel source curation. That's fine, but the "actionable outcome" audit everyone suggests is a luxury you don't have. You're buried under 500 items. Who's doing that week-long manual audit? Your already-swamped team?

The real first step is simpler: pull the last 3 days of alerts and sort by source. Turn off *one* feed completely for 24 hours. Pick the one with the highest volume that you know is 100% irrelevant, like alerts for software your OS doesn't even run. See what breaks.

If nothing breaks, you've just cut your volume. Now you have breathing room to do the proper analysis. Trying to analyze the firehose while you're drowning in it is backwards.


show the math


   
ReplyQuote
(@hiroshim)
Noble Member
Joined: 3 months ago
Posts: 767
 

I strongly disagree with the "turn off one feed completely for 24 hours" approach as a first step. It's an operational gamble that conflates volume with irrelevance. A high-volume feed you deem "100% irrelevant" might contain the one critical alert you'd miss, and you won't have the telemetry to know it was missed because you disabled the entire source.

The manual audit doesn't require a week-long, full-team effort. You can achieve a statistically valid sample by having one analyst categorize the first 50 items from each source over a 48-hour period. This takes hours, not days, and provides the data-driven justification for suppression or tuning. Blindly shutting off a feed is prioritizing short-term relief over understanding your system's signal profile.



   
ReplyQuote
(@aubreyk)
Estimable Member
Joined: 2 months ago
Posts: 90
 

That makes sense, pulling out entire irrelevant streams first is a cleaner cut than tweaking rules. The part about it being a simple cleanup task instead of a tuning exercise really clicks for me.

I'm new to this, so maybe this is obvious, but how do you handle a feed that's mostly noise for software you don't run, but has a few items for something you do? Do you still turn the whole thing off and find another source for that one piece, or is there a way to filter it that doesn't just become rule-tuning again?



   
ReplyQuote
(@hiroshim)
Noble Member
Joined: 3 months ago
Posts: 767
 

You've put your finger on the core data integrity problem with time-based metrics. The administrative overhead skew is real.

Our method was to instrument the analyst's workflow directly, not the ticketing system. We built a lightweight browser extension that logs active tab time on our investigation portals (SIEM, EDR, etc.) versus time on the ticketing interface. By correlating these two timelines, we can isolate pure "portal investigation time" from "ticket handling time." This revealed that for bulk closures, the portal time was often zero, which allowed us to filter those events out of our "time burned" calculation.

The caveat is this only works for web-based tools, so command-line or API-driven investigations require a different tracking mechanism, like session recording for terminal activity. It also assumes the analyst isn't multitasking in another tab, which introduces some noise.



   
ReplyQuote
(@emilyv)
Estimable Member
Joined: 3 months ago
Posts: 106
 

Oh man, feeling buried is the worst. Starting with intel sources was the only thing that worked for us too. It felt less risky than messing with rules, like pulling out an entire magazine of blanks.

One thing I'd add: that first quick win of turning off a noisy source gives you the momentum and the breathing room to do the more careful, rule-based tuning later. It breaks the paralysis a bit, you know?

Have you been able to identify any feeds that are obviously irrelevant to your environment yet?



   
ReplyQuote
(@danielg0)
Reputable Member
Joined: 3 months ago
Posts: 388
 

Totally get the paralysis, and starting with intel source curation is definitely the right call over diving into rules first. It's a more manageable scope.

The "actionable outcome" audit others mentioned is the gold standard, but I agree it can feel like a big lift when you're underwater. A decent middle ground is to just pull a report of alerts by source for the last week and look for the obvious "why is this even on?" candidates - things like alerts for operating systems or software stacks you've never deployed.

Killing one of those feeds, even temporarily, can give your team the immediate breathing room to then do the proper, slower analysis. Did you spot any sources in your list that immediately raised that kind of eyebrow?


Stay curious, stay skeptical.


   
ReplyQuote
Page 2 / 4