Hi everyone. I’ve been exploring Freeplay for a few weeks now, primarily to manage prompt iterations for our internal tools. I wanted to share a concrete workflow I’ve settled into, which has been really helpful for connecting user feedback directly to prompt changes. I’d love to hear how others are tackling this.
Our process starts when a user reports an issue or gives feedback via our support system (we use Linear). For example, a user might say, "The summary generated for my report is too verbose and misses key figures." Previously, this would be a ticket for a developer to manually tweak a prompt string in code.
Now, I tag that ticket with `freeplay`. I head to Freeplay and go to the specific project and `feedback` panel. I log the exact user comment there, linking it to the deployed version of the prompt chain we're using. This gives me a traceable record.
From there, I use the "Compare" feature. I'll take the exact user input that triggered the poor output and run it against our current production prompt. Then, I create a new experiment branch and iterate—maybe adjusting the instructions, adding a `##` style constraint for conciseness, or reformatting the expected output structure. Freeplay lets me run these side-by-side comparisons instantly, which is great.
Once I'm happy with a new variant, I use the CI/CD integration (a feature I particularly appreciate) to create a pull request. Our pipeline automatically updates the prompt in our staging environment. We validate it there before finally deploying the new prompt version to production, all without a full code deployment.
This loop has made what felt like a nebulous process much more tangible. It feels like applying version control and CI/CD principles directly to the LLM layer, which fits my mental model perfectly 😊. Has anyone else built a similar pipeline? I'm especially curious about how you handle validation steps or if you've integrated automated testing with tools like pytest for your prompts.
still learning
Interesting approach, but it sounds like you've just moved the manual tweaking from your code editor into another UI. How are you getting these prompt changes back into your actual deployments? If it's still a manual copy-paste step into your CI/CD pipeline, you haven't solved the automation problem, you've just added a third-party dependency.
I'd be more convinced if the feedback loop automatically created a pull request or triggered a test suite run against the new prompt variant. Otherwise, you're just trading one type of toil for another.
null
Moving feedback out of the codebase is still progress. That prompt string was a source of truth buried in a repo, which made it a dev-only domain. Shifting it to a tool like Freeplay makes it accessible to PMs or support leads who understand the nuance better than I do.
The copy-paste back into CI/CD is a real gap, though. I'd be wary of adding another SaaS that doesn't plug directly into my deployment pipeline. Have you looked at their API to see if you can automate that last-mile sync? Otherwise, you're just building a new silo.
Cloud costs are not destiny.
That's a great point about the new silo. You're right, the API is key to making this feel integrated and not just another manual step.
I did look into their API, and it's pretty straightforward for pulling prompt versions. My current experiment is using a GitHub Action that fetches the "approved" prompt from a specific Freeplay project and commits it to a JSON file in our repo. That then kicks off our usual CI/CD pipeline. It's not fully automatic from feedback to deploy, but it bridges that gap without manual copy-paste.
Have you set up anything similar, or have you found other ways to keep the source of truth accessible without sacrificing automation?
still learning
I appreciate the focus on automation, as it's the critical piece that determines whether a tool like this becomes infrastructure or just another manual layer. You're right that simply shifting the editing interface doesn't inherently solve the deployment problem.
However, the value proposition often starts with centralization and visibility before full automation is built. Getting prompt logic out of the codebase and into a shared system creates a single source of truth, which is a prerequisite for any automated pipeline. It's a stepwise improvement - you can't automate a source of truth that's fragmented across individual developer branches.
The real challenge, which your comment gets at, is whether the tool's design anticipates that eventual automation. If its API is an afterthought, then you're absolutely building a silo. But if it's built from the ground up to be machine-readable, then the copy-paste phase is just a temporary stage in the maturity of the workflow. The question for any team isn't just "are we automating now?" but "are we building on a foundation that allows automation later?"
Let's keep it constructive
That traceability is the most interesting part of your workflow. The ability to link a Linear ticket directly to a specific prompt version and then run the *exact* user input through your Compare feature is genuinely useful for diagnosis.
But I'm skeptical about the "iterate" step you gloss over. You mention adjusting instructions or adding constraints, which is the real work. How do you know your adjustments actually fix the problem for the general case and not just that one user's input? Without structured A/B testing back in your app, you're just making an educated guess and hoping the next user doesn't expose a different edge case.
It feels like you've built a great system for documenting failures, but the remediation is still based on intuition.
Question everything
You're pinpointing the exact vulnerability in any prompt management workflow that stops at iteration. The "educated guess" problem is real. My team's approach has been to treat the Compare feature not as a final validation step, but as a hypothesis generator.
We use the failing user input to create a small, targeted test suite. For the "verbose summary missing key figures" example, we'd add a few more diverse inputs that should also produce concise, figure-heavy summaries. The iteration happens in Freeplay against that entire suite, not just the original failing case. The prompt version only gets the "approved" label if it passes all of them.
But you're right, this is still synthetic. The real A/B test happens when we deploy that as a new variant in our canary and use our observability stack to compare real-world performance against the previous version across a broader set of metrics - not just correctness, but also latency and token cost. Without that production feedback loop, you're just optimizing for your own test cases.