Hey everyone 👋 Been lurking here for a bit while setting up our first real CI/CD pipeline at work. I'm a DevOps newbie, so I'm trying to learn by doing. We have a customer support chatbot, and my team lead wanted a better way to test prompt changes before they go live. We decided to try Freeplay for evaluation.
I just finished building a basic eval suite. The idea was to run our new prompts against a set of ~50 historical customer questions and grade the responses for accuracy, tone, and hallucination. Freeplay's SDK made it pretty straightforward to feed in the questions and record the results. Here's a snippet of how I set up the test run in a Python script:
```python
import freeplay
# Setup the test suite run
suite_run = freeplay.TestSuite.run(
suite_id="chatbot-accuracy",
environment="staging",
metadata={"prompt_version": "v2.1", "model": "gpt-4"}
)
print(f"View results: {suite_run.dashboard_url}")
```
After the run, the dashboard gives a really clean breakdown. You can see pass/fail rates per test case, drill into individual conversations, and it even highlights which specific assertions (like "contains correct product code") failed. Super helpful for spotting patterns.
Has anyone else used Freeplay for something similar? I'm curious about a couple things:
* How do you integrate this kind of evaluation into your deployment pipeline? Do you run it on every PR, or only before production deploys?
* I'm worried about the test data getting stale. How often do you update your evaluation question bank?
* Any pitfalls with the automated grading? I'm using a mix of model-graded and code-based assertions.
The dashboard is great for showing the team why a prompt needs more work, but I'm still figuring out the operational part—making this a repeatable, automated step. Would love to hear how you've set it up with Jenkins or GitHub Actions.
Learning by breaking
50 historical questions? That's a laughably small sample size. You're basically measuring the weather on a Tuesday and calling it climate.
> grades the responses for accuracy, tone, and hallucination
How are you grading those? LLM-as-judge? Hardcoded regex? If you're using an LLM to evaluate an LLM, you're just propagating the same biases. I've seen automated eval suites give 95% pass rates while the actual human review shows 40% of responses are garbage. Trust me, I've run the same tests.
Freeplay's dashboard is pretty, but pretty doesn't mean accurate. Manual spot-check the first 100 runs. I bet you'll find at least 5-10 false positives in the "pass" bucket. What's your assertion for "contains correct product code" - a substring match? That's going to miss half the cases where the code is paraphrased or the product name changed.
Also, 50 historical questions isn't going to cover edge cases like multi-turn context, adversarial inputs, or domain-specific jargon. You'll get a false sense of confidence and ship a prompt that breaks on the first real customer who asks about a refund policy.
How many of those 50 questions actually test tone? I'd be shocked if a grading rubric can reliably distinguish professional from robotic without manual oversight.
-- bb