Skip to content
Notifications
Clear all

Guide: Using webhooks to notify Slack channels for new critical vulnerabilities.

10 Posts
10 Users
0 Reactions
1 Views
(@devops_barbarian)
Reputable Member
Joined: 3 months ago
Posts: 219
Topic starter   [#23472]

Everyone's setting up Slack alerts for critical vulns from SonarQube. It's a great way to create alert fatigue and get your notifications ignored.

The webhook integration is trivial. The problem is SonarQube's "critical" severity. It's not the same as your production system's "critical". A vulnerable dev dependency in a prototype repo shouldn't page the on-call. You'll get flooded.

You need a filter. The SonarQube webhook payload is huge. Use a small middleware script to parse it and apply logic before Slack. Here's a basic Python filter that only fires for criticals on your main branch in specific projects.

```python
#!/usr/bin/env python3
import sys, json, requests
payload = json.load(sys.stdin)

if (payload['project']['key'] not in ['PROD_MAIN', 'CORE_API']) or
(payload['branch']['name'] != 'main') or
(not any(issue['severity'] == 'CRITICAL' for issue in payload['issues'])):
sys.exit(0)

slack_msg = {"text": f"Critical vuln in {payload['project']['name']}"}
requests.post('https://hooks.slack.com/...', json=slack_msg)
```

Configure the SonarQube webhook to hit this script. Run it in a container. Now you're actually alerting on signal, not noise.


Don't panic, have a rollback plan.


   
Quote
(@datadog_dave)
Reputable Member
Joined: 2 months ago
Posts: 268
 

Totally agree - we burned ourselves with this exact flood scenario last quarter. That filter script is spot on.

One tweak we made: we also check if the issue type is a "VULNERABILITY" (vs "BUG" or "CODE_SMELL"). Sometimes critical bugs sneak in otherwise.

Also, we ended up piping filtered alerts to a dedicated #vuln-alerts channel instead of the main #infra channel. Keeps the noise contained but still visible to the right people. The Slack webhook can specify the channel in its payload, which is handy.

Your point about prototype repos is so true - we were getting paged for PoC code that hadn't touched prod in months.


Dashboards or it didn't happen.


   
ReplyQuote
(@elenar)
Estimable Member
Joined: 3 weeks ago
Posts: 150
 

The issue type filter is a good addition. We had a similar experience where high-severity bugs in legacy error handling logic triggered alerts, but they were false positives for security response.

Your dedicated channel approach works, but consider routing logic based on the component field in the payload. We direct vulnerabilities in our auth service to a channel with the security engineering team, while core API issues go to the platform channel. This requires maintaining a mapping, but it targets the notification to the group that can actually triage it.

The channel specification in the webhook payload is indeed the right method, as it prevents your middleware from needing separate Slack app tokens for each destination.


Data doesn't lie, but folks sometimes do.


   
ReplyQuote
(@amyc)
Estimable Member
Joined: 3 weeks ago
Posts: 193
 

Great point about routing by component, that's a smart escalation of the dedicated channel idea. The maintenance overhead of that mapping scares me a bit, though. Has your team automated keeping that component-to-channel list in sync as services are added or renamed? That's where I've seen these setups break down.

Also, routing to a team-specific channel implicitly adds another filter: the audience. A vuln in the auth service hitting the security team's channel gets a different kind of scrutiny than a broad #infra alert. That's a nice, subtle way to add triage context.



   
ReplyQuote
(@devops_shift_worker)
Reputable Member
Joined: 2 months ago
Posts: 174
 

Yep, that filter script is the bare minimum. Been there.

If you're gonna run it in a container, make sure it's got a timeout and a retry for the Slack post. Lost a real alert once because Slack was having a wobble and our script just 500'd and died. Now it's a five-line wrapper with ten seconds of retry logic.

Also, stash that Slack webhook URL in an env var, not the script. Makes it less painful when you inevitably rotate the token.


NightOps


   
ReplyQuote
(@hannahg)
Estimable Member
Joined: 3 weeks ago
Posts: 141
 

Yeah, the maintenance overhead is real. We tried the component mapping and it became a spreadsheet that was wrong more than it was right.

Our hack was to make the mapping a living document in the repo itself - a YAML file that's part of our service template. When a team creates a new service via the template, they have to declare the Slack channel for their component. It's not perfect, but it ties the mapping to the creation event, which is a natural point to think about it.

Your point about audience-as-filter is the real gem, though. It forces ownership. If auth-service vulns are hitting the security team's channel, they're the ones who have to decide if it's a real fire or not. It moves the triage burden to the experts, not a general on-call.



   
ReplyQuote
(@davek)
Estimable Member
Joined: 3 weeks ago
Posts: 126
 

Integrating the channel mapping into the service template is a smart approach to the maintenance problem. It codifies a governance step at the point of creation, which is far more reliable than expecting a centralized spreadsheet to be updated later.

The trade-off, of course, is that this only works for new services provisioned through that template. You still need a parallel process - likely manual - to backfill the mapping for existing components, which can be a significant lift in a large organization. This often leads to a two-tiered alerting system for a while.

> It moves the triage burden to the experts.
This is the core benefit. By routing to a team-owned channel, you're not just filtering noise, you're delegating the initial assessment to the group with the most context. It transforms the alert from a generic "something is broken" to a domain-specific "your thing has a potential issue." The ownership model is built into the notification path.


CPU cycles matter


   
ReplyQuote
(@devops_dad_joke)
Estimable Member
Joined: 5 months ago
Posts: 157
 

Yep, that backfill is the killer. We tried the "living YAML" approach and ended up with a six-month project just to catalog existing services. The two-tier alerting is real - new stuff works great, old stuff just dumps into a graveyard channel that nobody watches.

That ownership transformation you mentioned is the real win, though. Once a team knows critical vulns for *their* service hit *their* channel, they start caring about the quality of the alerts. We saw teams start tuning their own SonarQube quality gates because they were tired of the noise. It's a nice feedback loop.

The template method is the right long-term play, but you gotta accept that the migration period will be messy. Just don't try to boil the ocean on day one.



   
ReplyQuote
(@finnj)
Estimable Member
Joined: 3 weeks ago
Posts: 126
 

Trivial? Let's not get carried away. That script is a start, but it's just swapping one hard-coded list for another. What happens when PROD_MAIN gets renamed or you sunset CORE_API? You're back editing the script, which is just a different flavor of maintenance pain.

The real contrarian take? You shouldn't be filtering projects at all. You should be filtering by deployment target, or better yet, by what's actually running in an environment. A "critical" in a library that's packaged into a live container is an alert. The same "critical" in a dormant branch or a prototype is just data. SonarQube's metadata often doesn't know the difference, so your middleware needs to.

Also, sys.exit(0) on a non-match? That's a silent discard. At least log the filtered events somewhere, or you'll be wondering why your webhook "stopped working" when a legit project gets added but you forgot the list.


FOSS advocate


   
ReplyQuote
(@gregoryt)
Estimable Member
Joined: 2 weeks ago
Posts: 155
 

Routing by component makes a lot of sense. I'm just starting to set up something similar and I'm curious: how do you manage that mapping in your middleware? Is it a simple config file, or do you pull from a service registry or something? The idea of not needing separate tokens for each channel is a big win, for sure.



   
ReplyQuote