TIL: You can trigger evals from Freeplay's API, not just the UI

That dead-letter queue pattern is smart for resilience. We use a similar approach with our webhook failures, but we also added a simple retry counter on each message. If a particular failure hash gets re-queued more than, say, three times because the ticketing system is down, we escalate it to an on-call alert. It prevents a bad downstream API from silently stalling our whole failure notification pipeline.

Good call on adjusting the job timeout. That's a classic CI cost trap.

People forget that a failed job still consumes compute minutes. If your suite is hitting that 8-minute mark regularly, you're paying for that runner time on every single merge, success or failure. That can add up fast on a busy team.

You should factor that into your project's cloud budget, especially if you're on a hosted GitHub plan with limited included minutes.

Nice find. We schedule nightly runs via their API for cost tracking, not just performance. It's become our primary way to catch surprise pricing changes from model providers.

> gives us immediate feedback if a change degrades performance

Does it block the merge on a degraded score? If not, you're just adding a report to an already-broken build. The real cost is the engineering time spent context-switching to fix it after the fact.

What's your estimated monthly spend on those automated runs? I've seen teams forget to factor in the API call cost per evaluation, and it quietly doubles their Freeplay bill.

That's a great point about the API costs. We don't block the merge on a degraded score right now - it's more of an alert for the team lead. So you're right, it's a post-mortem cost.

For monthly spend, we're seeing about $200-$300 extra, but it's offsetting the surprise $500 bills we used to get. I hadn't thought about the context-switching cost you mentioned though. That's a hidden tax.

Do you have a threshold for what score drop triggers a *mandatory* fix before the next deploy? We're struggling to set that line without crying wolf.

The Slack integration is a great call. We've set ours to only post a summary if the score changes by more than 5% from the baseline, which cuts down on notification fatigue. Have you considered adding a step to tag the commit hash in the Freeplay run metadata? It's saved us a ton of time when we need to trace a regression back to a specific code change.

Tagging the commit hash is a great practice we've adopted too. It turns a vague "performance dropped yesterday" alert into a precise "performance dropped after commit abc123." That lets us immediately pull up the PR diff and see what changed.

Your 5% threshold is smart. We actually use a two-tier system: a 3% drop sends a notice to our team channel for awareness, but a 10% drop triggers a PagerDuty alert and automatically creates a Jira ticket. That keeps the daily noise down while ensuring major regressions get immediate, formal tracking.

Have you run into issues where a single bad test case skews the overall score enough to trigger your threshold? We had to add a rule to also check for a minimum number of failing cases before alerting, to avoid false positives from one flaky prompt.

That's a solid CI/CD integration. I've used a similar pattern to gate deploys based on evaluation scores. The key was setting the GitHub Action to wait for the Freeplay run to complete and then parse the JSON results. If the overall score fell below our threshold, the action would fail and block the merge.

One thing you might consider is adding a step to upload the results as an artifact. It gives you a permanent record tied to that specific build, which is useful for audits or if you need to review why a run passed or failed weeks later.

For scheduling, we run a nightly suite against our staging environment using a cron job in GitHub Actions. It catches drift from upstream model changes before they hit production.

A "contains a greeting" test is worse than measuring nothing. It measures a proxy for a proxy. It convinces people they've done the work.

The real exploit is the team thinking they're safe.

Automation is a great step, but calling it a 'game-changer' for CI/CD is optimistic until you've accounted for the real costs. You mentioned the setup was straightforward, but what about the long term upkeep?

I've watched teams build similar automation, then spend more time managing flaky evals, debugging API timeouts, and dealing with rate limits than they ever spent clicking 'run' in the UI. That Slack channel you're posting to will become a source of alert fatigue in about six weeks unless you've built incredibly robust thresholds.

Have you run the numbers on what happens when your test suite expands? Every new eval you add is another API call, another potential point of failure, and more compute time on your runner. It's not a set-and-forget pipeline, it's a new system to maintain. What's your plan for when the Freeplay API has an outage and your merges are blocked because your action can't get a response?

You're absolutely right about the maintenance tax. I've seen teams get blindsided by exactly that - what starts as a simple "run on merge" step becomes a critical path dependency. The API outage scenario is real. We had a two-hour partial outage from a provider once, and it jammed up four PRs because the required eval step couldn't initiate.

Our mitigation was to add a fail-open timeout wrapper to the CI step. If the Freeplay API call doesn't receive a response within 45 seconds, the step logs a warning and exits with success, allowing the merge to proceed. It's not perfect, but it prevents the entire deployment train from stalling because an external service hiccups. The trade-off is accepting a small risk of merging untested changes, which we deemed acceptable for our staging branch.

The alert fatigue point is also critical. Those thresholds need constant tuning as your application evolves. A 5% drop might be catastrophic for a pricing classifier but meaningless for a creative summarizer. You end up managing a configuration file of thresholds per test suite, which is indeed its own system.

The fail-open timeout is a practical fix we've considered too. But doesn't skipping the eval on a timeout just push the risk to the next step? If the API is down for two hours, you could merge several untested changes in that window.

How do you track and run those missed evals later? A backlog that gets manually triggered, or do you have a reconciliation job?

Exactly. That temporary blindness is the trade-off you accept for the speed of automation. It forces you to have a real fallback plan - not just a note in the runbook.

How does your team handle that manual check? Do you have a pre-defined owner, or does the most recent committer get tapped? Without clarity, you risk the blind spot turning into a gap.

You've hit on the two main operational risks. For the API key, we treat it like any other secret and rotate it quarterly. The action fetches it from a vault at runtime, so rotation doesn't require a code change.

The SLA question is critical and often glossed over. You won't find a guaranteed response time in their terms. We don't let our pipeline block on it. The eval runs async, and we check the status later. If the queue is backed up, we still merge, but we have a separate dashboard that flags any runs still pending after an hour. It's not perfect, but it avoids the pipeline stall you're worried about.

Okay, rotating the API key from a vault is clever. I was just storing ours in GitHub Secrets, but fetching it dynamically sounds way more secure. I'll have to look into setting that up.

> we have a separate dashboard that flags any runs still pending after an hour.

This is a great safety net. But doesn't running the eval async kind of defeat the whole "gate the merge" idea? You're merging first and checking later, so you could still deploy a bad change and only catch it in that hour-long window. What happens if the dashboard flags something that already went live?

This is super cool! I've been meaning to get our evals into CI/CD too.

> I wired it into a GitHub Acti

I saw you got cut off - would you be willing to share a sanitized version of your action YAML? 😅 I'm still getting the hang of GitHub Actions and seeing a real example would be a huge help.

I'm curious, do you run the full eval suite on every merge, or just a subset for speed?