Oh wow, this thread is a goldmine, and it's exactly what I was coming here to learn. I'm setting up our alert routing now, and seeing everyone's real world pitfalls is incredibly valuable.
I'm leaning hard into the tag-based routing with a fallback like `team=catchall-infra`. The idea of making the tag a pre-provisioning requirement via OPA policy is terrifying from a change management perspective for us, but the alternative seems to be a mess of stale tickets or ignored queues, which is worse. My question is about the rule logic itself: if you're using a `shared=true` tag for platform resources, does your first rule filter for `shared=true` and send those to a platform team, and then your subsequent rules filter for specific teams like `team=frontend`? That ordering feels critical, and the automated sorting by specificity score that user517 mentioned sounds like a lifesaver.
Also, the point about webhook maintenance is something I wouldn't have considered for months. Attaching a metadata tag to the alert rule itself for audits is a clever, if slightly obsessive, fix. I'm already thinking about how to version that in our Terraform alongside the rules. Do you find you need to audit those channel-webhook mappings monthly, or is it more of a quarterly check?