So I've been helping a few teams implement Freeplay over the last year, mostly for LLM evaluation and playground work. It's solid tech—don't get me wrong. The UI is slick, the evaluation workflows are powerful when you get them dialed in, and it genuinely speeds up iteration.
But here's the rub: I've seen it get oversold. Twice now, I've been brought into clients who were promised a "fully automated CI/CD pipeline for AI" that would "orchestrate everything from prompt management to canary releases." What they actually got was a great evaluation platform that still needs a *ton* of surrounding engineering to fit into a real production pipeline.
The biggest gap I see is around actual deployment and GitOps. Freeplay tracks your experiments and prompts, but pushing a winning prompt config to your live API? That's on you. You're stitching together webhooks, writing custom exporters, or managing a separate config repo. It's not the seamless end-to-end system some sales decks imply.
For example, to get a "prompt promotion" flow, one team ended up building this:
```yaml
# Simplified version of their GitHub Action step
- name: Export Freeplay prompt to config map
run: |
curl -s -H "Authorization: Bearer ${{ secrets.FREEPLAY_KEY }}"
https://api.freeplay.com/v1/prompts/${{ env.PROMPT_ID }}
| jq '.content' > ./prompt-template.yaml
# Then kubectl apply to update the ConfigMap in staging
```
It works, but it's custom glue code they now have to maintain.
My take: Freeplay is an excellent tool **if** you frame it correctly. It's a world-class experimentation and evaluation layer. It is *not* a deployment orchestrator, a model registry, or a cost-monitoring dashboard. If you go in expecting it to be the central nervous system of your AI stack, you'll be disappointed. Bring your own CI/CD, your own monitoring, and your own cost-tracking.
Anyone else run into this? How are you bridging the gap between Freeplay's playground and your live environments? I'm curious about Helm chart strategies or ArgoCD integrations people might be using.
-jk
Your point about deployment and GitOps is exactly where I've seen teams stumble. Even when you manage to cobble together a webhook-to-Kubernetes flow, you're left with a state synchronization problem that Freeplay doesn't solve. The live config map in your cluster becomes the source of truth, while Freeplay holds the canonical experiment history. Drift is inevitable without a reconciliation loop.
One team I advised ended up forking their Freeplay Terraform provider to write prompts directly as HCL, treating them as infrastructure-as-code artifacts. That at least gave them versioning and a promotion path via their existing pipeline tools. It was more engineering, but it acknowledged the platform's boundaries rather than fighting them.
It's a fantastic evaluation workbench, but expecting it to be an autonomous release manager is like expecting Terraform Cloud to also run your container orchestrator.