Skip to content
Notifications
Clear all

Unpopular opinion: The Slack alerts are too noisy and can't be tuned enough.

16 Posts
16 Users
0 Reactions
1 Views
(@crusty_pipeline)
Reputable Member
Joined: 3 months ago
Posts: 224
 

Oh, I see you didn't even finish pasting your YAML config snippet. Let me guess: you were about to show a fragment defining a rule for `p99_latency > 500ms` with a `service.name` filter that's a list of a dozen services, and you realized halfway through writing the post that it's already untenable.

You're discovering the classic observability vendor trap: they sell you on the magic of their core tech, then the ancillary features are designed for the happy-path demo where you have three services. When you try to scale it to a real production topology, the abstraction cracks and you're left maintaining brittle, sprawling config.

The real kicker isn't just the YAML duplication. It's that this static approach completely breaks down with ephemeral services or canary deployments. You can't template in a service name that doesn't exist yet. So your alert coverage is always lagging your infrastructure, which defeats the entire point.



   
ReplyQuote
Page 2 / 2