Skip to content
Notifications
Clear all

Showcase: Automated our prompt testing with Freeplay and GitHub Actions

3 Posts
3 Users
0 Reactions
3 Views
(@k8s_cost_ninja)
Estimable Member
Joined: 5 months ago
Posts: 70
Topic starter   [#11987]

We were wasting hours manually testing LLM prompt changes across our RAG pipelines. Freeplay's SDK and API gave us the hooks to automate it.

Key setup:
* Freeplay project for each major workflow (e.g., `support_agent`, `doc_summarizer`)
* GitHub Actions workflow triggered on PRs with prompt changes

The workflow:
1. On PR, action extracts the changed prompt files.
2. Runs a test suite that uses Freeplay's SDK to log `test_scenario` runs against the new prompts.
3. Fetches evaluation results (latency, cost, scored outputs) via Freeplay's API.
4. Posts a comment on the PR with a summary table.

Example GitHub Actions step (simplified):

```yaml
- name: Run Freeplay Scenario Tests
env:
FREEPLAY_API_KEY: ${{ secrets.FREEPLAY_API_KEY }}
run: |
python scripts/test_prompts.py
--scenario "support_triage"
--prompt-version "${{ github.head_ref }}"
--test-file "test_data/triage_cases.json"
```

Output posted to PR:

```markdown
## Freeplay Test Summary | support_triage
| Metric | Baseline | This PR | Delta |
|--------|----------|---------|-------|
| Avg Latency | 1.2s | 0.9s | -25% |
| Avg Cost/Req | $0.003 | $0.002 | -33% |
| Accuracy Score | 0.92 | 0.94 | +0.02 |
```

Results:
* Catch performance regressions before merge.
* Quantify cost impact of prompt changes.
* Stopped guessing about "better" prompts. We have data.


null


   
Quote
(@jessiew)
Eminent Member
Joined: 6 days ago
Posts: 17
 

Oh this is such a smart setup! I love the idea of posting that summary table directly to the PR. It gives immediate, actionable feedback right where the change is happening.

We did something similar but integrated it with our Pipedrive CRM for lead scoring prompts. One thing we learned the hard way: you need to make sure your baseline data is truly representative, not just a snapshot of a good day. We had a "delta" that looked amazing because our baseline test data was too easy, and it didn't catch a regression in handling edge cases.

Do you also log the raw outputs to Freeplay for a human to spot-check later, or is your process purely metric-driven from the PR comment?



   
ReplyQuote
(@kevinb)
Estimable Member
Joined: 1 week ago
Posts: 55
 

Your baseline point is spot on. We've seen the same issue where a curated "golden" dataset masks real regressions until they hit production. It's a classic case of Goodhart's law.

We log the raw outputs, but that's a trap in itself. Now you're paying for storage in their system and building a review process dependent on their UI. The lock-in isn't just the API, it's your entire test history.

Did you factor in the ongoing cost of that data retention, or are you just hoping their startup pricing holds?



   
ReplyQuote