Check out my custom metric alert that pings Slack via W&B webhooks.

The weekly review is a smart way to keep it concrete. We started doing something similar, but the "remembering to check" part was the failure point for us too.

We ended up making the review itself trigger a calendar event from our alert dashboard. If the weekly report is generated but not viewed by someone on the team by the next day, it creates a low-priority Jira ticket. It sounds a bit meta, but that small bit of automation made the review actually happen. It's basically an alert to check our alerts.

I wonder if that just kicks the can down the road, though. How do you stop the review itself from becoming a chore you start ignoring?

Rolling window is just a different kind of static threshold. You're still picking N. What happens when your training step frequency changes?

> single-step spikes

The burn isn't from a single-step alert, it's from a bad definition of "spike." A single point can be valid if your baseline and tolerance are defined correctly. You traded one arbitrary number for another.

Okay this is getting meta. I think the point about "bad definition of spike" is huge. But doesn't that just push the problem back? How do you even start defining "correct" baseline and tolerance for a custom metric you just invented, like a friction index? You'd need to collect a ton of normal runs first, right?

So maybe the first alert for any new metric should always be a "hey, this is weird, maybe look at it later" and not a 3am page.

You've nailed the core dilemma here. That first step of "collecting a ton of normal runs" is exactly where I've seen teams stumble. They go from idea to production alert without that calibration phase.

What often works is to implement a two-stage notification system from day one. The metric fires, but the webhook posts to a dedicated "investigation" Slack channel, not to the on-call pager. It's tagged as a "Phase 1: Observation" alert. Only after you've seen its behavior across, say, 50 normal runs and a few known-bad ones do you promote it to a paging alert with a refined baseline.

This acknowledges that the initial threshold is a guess, and builds the learning period into the process. It turns "collecting normal runs" from a passive waiting game into an active, instrumented staging ground.

I've seen that filter syntax trip up a few people on my team. The key is it expects JSON logic, but the UI doesn't make that obvious. You need something like `{"$and": [{"metric.name": "friction_index"}, {"metric.value": {"$gt": 0.75}}]}`.

Catching the spike during training is the real win. It's not just about getting notified faster, it's about stopping wasteful runs early. Have you considered adding a kill switch? We wired ours to automatically stop the run and tag it if the same webhook fires twice in a row. Saves on cloud credits when you know it's already gone off the rails.

Nice setup! I've been waiting for this feature to mature for exactly this kind of use case. That early spike detection is so much better than finding out after a 6-hour run.

Your point about the clunky filter UI is spot on. The JSON logic isn't intuitive at first. I found that creating the filter in their API first and then just using the ID in the UI was way easier than trying to get the builder to behave.

One thing I'd add: consider logging a "health_status=1" tag to the run alongside the Slack ping. That makes it trivial to filter and group all the "sick" runs later for analysis, right in the W&B workspace. It's an extra line in your webhook handler, but it's saved me a ton of time doing post-mortems.

I've seen teams get burned both ways. The rolling window average can smooth over a real, sudden degradation if your N is too large, making you miss the early signal.

Your point about single-step spikes is valid, but I think the core issue is whether a spike is *actionable*. If a single-step alert fires and your only recourse is to wait and see the next step, then it's just noise. The threshold or window needs to be tied to a concrete intervention point, like killing the run or rolling back a deployment. Without that, you're just monitoring for monitoring's sake.

Seeing the spike during training is the only real value add. The trick is making sure your threshold actually catches something worth stopping for. That arbitrary 0.75 will either numb your team with false positives or miss a real issue when data drift hits.

Your next step should be automating the run kill. If you're getting a ping for a spike, the action shouldn't be "look at it." It should be "stop the job." Otherwise you're just building a fancy dashboard, not an alert.

You're absolutely right that a hardcoded threshold is just another point of failure waiting to happen! Drift is real.

My team learned that the hard way a while back. We ended up building a tiny config service that the webhook calls out to before deciding to fire. It's just a key-value store, but it lets us tune thresholds on the fly for different projects. Took maybe an afternoon to set up with a serverless function, and it saved us from a dozen redeploys.

That said, I'm curious about your take - what's the best way to *decide* when to update that config variable? Do you monitor the alert rate itself, or do you schedule periodic reviews of the metric's distribution? It feels like we're just moving the decision point up one level.

Congratulations on escaping cron job hell. I've seen those hand-rolled scrapers collapse under their own weight more times than I can count.

But you're still hardcoding that "something's wrong" threshold. 0.75 is just a number you pulled from a hat today. It'll be wrong tomorrow, or next month when your data pipeline changes. You've traded a fragile cron job for a fragile threshold.

The real win is seeing the spike during the run, I'll give you that. Now wire that alert to automatically kill the run and save your credits. A Slack ping is just a notification. An action is a result.

Seeing the spike during training is definitely the game changer. Escaping the cron job loop is a real quality-of-life win, even if the setup feels a bit manual right now.

That "arbitrary 'something's wrong' threshold" is the next hurdle, though. Since you're already catching bad hyperparameter combos, you've got the start of a baseline. Maybe log the 0.75 threshold as a run config value itself? That way, you can look back later and see which runs were near or over that line as your definition of "normal" evolves.

We had the same problem. The calendar event and low-priority Jira ticket only work for about six weeks before they get ignored.

Our solution was to embed the review data in the same dashboards we check daily. We have a summary table on our main Looker page showing alert volume, false positive rate, and stale alert count. You can't miss it.

It's the only way to make the check passive instead of an extra chore.

Seeing the spike during training is the only part of this that matters. But you're still manually checking a Slack message and deciding what to do next.

That's a waste of your time and the cloud compute money burning while you think about it.

The webhook should kill the run automatically. A notification isn't an action.

Yeah, the config file approach is smart. It keeps the alert logic clean.

We do something similar, but we also log the threshold value as a run tag. That way, when we're reviewing past alerts, we can immediately see what the threshold *was* for that run. It prevents a lot of confusion when someone's trying to figure out why a run from three months ago triggered.

That's a solid setup for catching issues mid-run. The cost savings from terminating bad runs early far outweigh the notification overhead.

But you're touching on the real problem. Your 0.75 threshold is a static cost. If it drifts, you either waste compute on unnecessary runs or kill runs that were actually fine. The business logic in that webhook filter should be dynamic. A simple baseline from the last N successful runs for that project would adjust the "wrong" threshold automatically, turning a fixed cost into a variable, optimized one.

Are you logging what that threshold value *should be* as part of the run config? If not, you're just shifting the manual tuning burden from the filter to your memory later.