Hey folks, I've been living in Freeplay for the past few months, using it to orchestrate and monitor our evaluation workflows. The platform itself is super powerful for tracking prompts, versions, and scores... but oh my goodness 😅, the default Slack alerts almost drove my entire platform engineering team to mute the channel.
We were getting a *firehose* of notifications. Every single test run completion, every metric calculation—it was like the boy who cried wolf, but for AI evals. The signal-to-noise ratio was practically zero, and important regressions or failures were getting lost in the shuffle. I know alerts are a first-class citizen in Freeplay's setup, so I figured there had to be a better way.
After a deep dive into their notification policies and some trial-and-error, I've managed to tame the beast. The key is moving beyond the default "notify on everything" setup and leveraging **filters, thresholds, and routing**. Here's the config pattern that finally made our Slack alerts actionable and actually useful.
First, you'll want to navigate to your Project's **Notification Policies** in Freeplay. The magic is in creating specific policies for specific events, not using the catch-all. For example, we only want to hear about:
- **Failures** in test runs (not completions).
- **Significant regressions** in key metrics (e.g., `cost_per_output` spikes by >15%, or `quality_score` drops below a threshold).
- **Production pipeline events** only, ignoring all alerts from our staging/experimental branches.
Here’s a snippet of a YAML policy we applied via their API (you can also build this in the UI). This one only alerts on severe quality drops in our main `prod` Git branch:
```yaml
policy:
name: "Prod-Severe-Quality-Drop"
triggers:
- event_type: "test_run.completed"
filters:
- field: "branch"
operator: "equals"
value: "main"
- field: "metrics.quality_score.value"
operator: "less_than"
value: 0.7
notification_channels:
- slack:
channel_id: "C1234567" # Your specific channel
message_template: "🚨 Quality alert in production!nTest Run: {{test_run_name}}nScore: {{metrics.quality_score.value}}nLink: {{test_run_link}}"
```
Secondly, we **routed different severity levels to different channels**. Critical failures go to a dedicated `#ai-alerts-critical` channel that pings the on-call. Minor regressions or informational events go to a `#ai-evals-feed` for passive awareness. This is done by creating multiple policies with different filters and channel mappings.
A few other lifesaver tips:
- Use **aggregation** where possible. Freeplay can batch notifications over a time window to prevent a burst of 50 messages.
- Leverage the **`custom_properties`** field in your test runs to add tags like `alert_tier: high` and filter on those.
- Don't forget to **mute notifications** for your automated, scheduled regression tests during off-hours. We set up a policy that only alerts on those runs if they fail between 9 AM and 5 PM local time.
The result? Our Slack is quiet again, but when something *actually* important happens, we see it immediately and can act. It turned Freeplay from a source of noise into a legitimately valuable observability layer for our LLM apps.
Has anyone else tackled this? I'm curious if you've found other filtering strategies or maybe even built a custom webhook to further process alerts before they hit Slack. Let's compare notes!
bw
Automate all the things.