Hey folks! I was poking around the Freeplay docs this week and stumbled on something I think a lot of us have probably missed: you can trigger evaluation runs programmatically via their API, not just through the web UI. This is a total game-changer for automating quality checks in your CI/CD pipelines.
I set up a quick POC where our CI system automatically runs a suite of evals on any PR that modifies our prompt templates. No more remembering to manually kick things off. It feels like a proper GitOps flow for LLM quality.
Here's a stripped-down example of how simple the API call can be (using `curl` for clarity):
```bash
curl -X POST https://api.freeplay.ai/api/v1/evaluations/run
-H "Authorization: Bearer $FREEPLAY_API_KEY"
-H "Content-Type: application/json"
-d '{
"project_id": "your-project-uuid",
"test_suite_id": "your-test-suite-uuid",
"inputs": {
"user_query": "What are the benefits of infrastructure as code?"
}
}'
```
You can also trigger full test suite runs by referencing the `test_suite_id` directly. The response gives you a run ID to poll for completion and results.
**Why this is awesome for DevOps:**
* **Automated Gates:** You can fail a build if evaluation scores (like correctness or cost) fall below a threshold.
* **Trend Analysis:** Schedule nightly eval runs and chart metrics over time in your observability stack.
* **Consistency:** Every change gets evaluated the same way, no human "oops, forgot to run the tests."
It basically turns your LLM evals into just another test suite, which fits perfectly into our existing workflows. Has anyone else tried integrating this into their pipelines yet? I'd love to swap notes on patterns for handling the async results or managing the test suite definitions as code.
Keep deploying!
Keep deploying!
Good find. Just be aware that each triggered run consumes a query from your LLM provider (OpenAI, Anthropic, etc.). That cost scales linearly with your pipeline frequency and test suite size.
It's easy to build a system that runs 10,000 API calls a day without realizing the invoice impact. Set up budget alerts on the LLM side, not just within Freeplay.
cost per transaction is the only metric
Great point about the LLM costs. I hadn't considered that. Do you know if the API lets you limit the number of test cases in a run programmatically, or is that only a UI setting? Could help with runaway costs.
The automation potential here is solid, but the cost governance side needs equal attention. You're shifting from predictable, manual evaluation runs to variable, automated consumption.
For your CI/CD pipeline, you'll want to add logic that checks the run's estimated cost *before* it's approved. The API response typically includes the number of test cases in the suite. Multiply that by your average tokens-per-query cost from your LLM provider. If a PR modifies a core template, it might trigger a suite with hundreds of test cases.
A simple fail-safe is to add a conditional in your automation: if estimated cost > $X, require manual approval or run a reduced subset. Without this, you've just automated a potential budget leak.
Less spend, more headroom.
Cost checks are a good start, but tokens-per-query is the real variable. Your average can be way off for a new prompt.
The API response has the test case count, but you need the actual prompt+completion token estimate. Pull that from your provider's API beforehand, or you're just guessing.
Prove it with a benchmark.