We were wasting hours manually testing LLM prompt changes across our RAG pipelines. Freeplay's SDK and API gave us the hooks to automate it.
Key setup:
* Freeplay project for each major workflow (e.g., `support_agent`, `doc_summarizer`)
* GitHub Actions workflow triggered on PRs with prompt changes
The workflow:
1. On PR, action extracts the changed prompt files.
2. Runs a test suite that uses Freeplay's SDK to log `test_scenario` runs against the new prompts.
3. Fetches evaluation results (latency, cost, scored outputs) via Freeplay's API.
4. Posts a comment on the PR with a summary table.
Example GitHub Actions step (simplified):
```yaml
- name: Run Freeplay Scenario Tests
env:
FREEPLAY_API_KEY: ${{ secrets.FREEPLAY_API_KEY }}
run: |
python scripts/test_prompts.py
--scenario "support_triage"
--prompt-version "${{ github.head_ref }}"
--test-file "test_data/triage_cases.json"
```
Output posted to PR:
```markdown
## Freeplay Test Summary | support_triage
| Metric | Baseline | This PR | Delta |
|--------|----------|---------|-------|
| Avg Latency | 1.2s | 0.9s | -25% |
| Avg Cost/Req | $0.003 | $0.002 | -33% |
| Accuracy Score | 0.92 | 0.94 | +0.02 |
```
Results:
* Catch performance regressions before merge.
* Quantify cost impact of prompt changes.
* Stopped guessing about "better" prompts. We have data.
null
Oh this is such a smart setup! I love the idea of posting that summary table directly to the PR. It gives immediate, actionable feedback right where the change is happening.
We did something similar but integrated it with our Pipedrive CRM for lead scoring prompts. One thing we learned the hard way: you need to make sure your baseline data is truly representative, not just a snapshot of a good day. We had a "delta" that looked amazing because our baseline test data was too easy, and it didn't catch a regression in handling edge cases.
Do you also log the raw outputs to Freeplay for a human to spot-check later, or is your process purely metric-driven from the PR comment?
Your baseline point is spot on. We've seen the same issue where a curated "golden" dataset masks real regressions until they hit production. It's a classic case of Goodhart's law.
We log the raw outputs, but that's a trap in itself. Now you're paying for storage in their system and building a review process dependent on their UI. The lock-in isn't just the API, it's your entire test history.
Did you factor in the ongoing cost of that data retention, or are you just hoping their startup pricing holds?