Having recently completed a rigorous evaluation of Freeplay for our machine learning operations pipeline, I feel compelled to document the methodology and outcome. Our team operates a complex, multi-cloud inference system serving several product lines, and the need for a structured framework to evaluate, manage, and iterate on LLM prompts and chains had become acute. The proliferation of ad-hoc notebooks and slack-channel-driven "testing" was creating significant observability debt and reproducibility issues.
We established a critical constraint: any new platform must integrate seamlessly into our existing engineering workflow, which is ticket-driven via Jira and code-centric via Git. The goal was to demonstrate a closed-loop from a reported production issue to a tested, peer-reviewed solution candidate within a single business day. Here is the workflow we designed and executed:
**Phase 1: Issue Triage & Branch Creation (Hours 0-1)**
* A Jira ticket is created detailing a symptom (e.g., "Customer support agent response quality degradation for Topic X").
* A developer parses the ticket, identifies the likely prompt chain in our application, and creates a new Git branch.
* The corresponding `promptchain.yaml` (our abstraction) is located, and its identifier is noted for Freeplay.
**Phase 2: Freeplay Evaluation & Hypothesis Testing (Hours 1-4)**
* Within Freeplay, we navigate to the relevant Prompt Chain and create a new Experiment, linking the Jira ticket ID in the description.
* We import the production dataset slice relevant to the issue (in our case, a CSV of recent problematic interactions) as a Test Suite.
* The experiment involves creating variants. We modify system prompts, few-shot examples, or model parameters (e.g., switching from `gpt-4-turbo` to `claude-3-sonnet`). Crucially, we version these changes conceptually, though the actual code change is still local.
```yaml
# Example variant structure we documented in Freeplay
variant_name: "claude3_more_context"
hypothesis: "Providing a more structured context window and switching to Claude will improve adherence to guidelines."
changes:
- target_model: anthropic.claude-3-sonnet-20240229
- system_prompt_addendum: |
- Always structure your response with a 'Summary' and a 'Recommended Action' section.
- Reference the following policy document ID: #SUPPORT-45.
```
* We execute the Test Suite against all variants (including a control group using the current production prompt). Freeplay's side-by-side comparison and metric scoring (using our custom evaluator for "policy adherence") allowed for rapid, quantitative analysis.
**Phase 3: Code Integration & Peer Review (Hours 4-7)**
* The winning variant's configuration changes are formally applied to the local `promptchain.yaml` file in the Git branch.
* A Pull Request is opened, which includes a link to the Freeplay Experiment report as the primary evidence for the change.
* Peers review both the code diff *and* the experimental results in Freeplay, commenting on both before approval.
**Phase 4: Deployment & Observation (Hours 7-8)**
* Upon merge, our CI/CD pipeline deploys the updated prompt chain to a staging environment.
* We use Freeplay's monitoring features to tag all subsequent staging inferences with the new experiment ID, allowing us to observe real-world performance before a staged production rollout.
**Outcome and Architectural Critique:**
The process was successful; we validated a fix for the reported issue within the day. However, this exercise revealed several key points:
* **Strengths:** Freeplay's integration of testing, experimentation, and monitoring into a single pane is its principal advantage. The ability to attach a dataset and run batch evaluations against multiple variants is transformative compared to manual API testing.
* **Weaknesses:** The "infrastructure as code" gap is present. While Freeplay has APIs, the definition of a Prompt Chain or Test Suite is not natively expressed in a format like Terraform or YAML checked into Git. This creates a synchronization burden; our `promptchain.yaml` is the source of truth, and Freeplay becomes a derived, operational view. A true GitOps workflow would require deeper CI integration.
* **Recommendation:** For teams earlier in their LLM ops journey, Freeplay provides essential guardrails. For mature teams with heavy IaC investment, plan for a synchronization layer from your code repository to Freeplay's entities to maintain a single source of truth. The value, however, in replacing subjective prompt debates with data-driven experiments is undeniable and justifies the operational overhead.
--from the trenches
infrastructure is code