How do I use Freeplay to track prompt performance across our user segments?

You've got the basic setup, but I bet your "manually scored" accuracy evaluation is a bottleneck. How many prompts per week are you actually scoring manually? That method doesn't scale and introduces rater bias.

Instead of manual scoring for everything, you should build proxy metrics. We use automated checks for things like code syntax validity (if that's the output), answer length versus the retrieved context chunk, or sentiment alignment. These correlate well with actual quality and you can run them on 100% of traffic.

Also, tracking cost per session is fine, but the real insight is cost per *successful* session. Segment your cost data by your accuracy or guardrail pass/fail status. You'll likely find your "enterprise" tier costs more because they get longer, more accurate responses, and that's probably the right tradeoff.

Totally agree on proxy metrics, it's a lifesaver. We actually set up a rule checking for markdown formatting in our support bot replies - it's a surprisingly good indicator of a structured, helpful response.

That cost per successful session point is gold. We found our "pro" users were getting cheap, fast answers that failed a lot of the nuance checks. Had to rebalance that.

Your workflow is a solid foundation, especially the automatic tagging via session metadata. That's crucial for clean data.

However, I must caution against relying too heavily on a **manually scored Accuracy Score** for any business-critical decision making. Even on a sample, manual scoring introduces significant subjective bias and is a major compliance risk if you're in a regulated space. For audits (think SOC 2 or GDPR data subject request handling), you need consistently applied, objective criteria. An auditor will question the validity of a manual 1-5 scale.

I'd recommend defining your "accuracy" as a composite of several automated, deterministic checks that align with your compliance requirements. For instance, you could create a rule that verifies the response does not contain unverified personal data, or that it correctly cites an internal knowledge base article ID. These are measurable, auditable, and can be run on 100% of your enterprise-tier traffic.

The compliance angle is a huge point I hadn't considered. When you say auditors will question a manual 1-5 scale, does that apply even if the manual scoring is just for an internal product team to spot-check trends, not for official compliance reports?

I'm curious about building that composite accuracy score. If you have an automated check for "cites a knowledge base article ID," how do you handle cases where a correct answer shouldn't cite anything? Do you end up with a ton of rules for different prompt types?

Excellent start on the foundational setup. I strongly endorse tagging sessions at the source - it's the only way to guarantee clean, queryable dimensions later.

You mentioned defining a **Custom Guardrail**. I'd be curious to know its logic. In our data pipelines, we found early success by implementing guardrails that check for JSON validity or SQL syntax when the expected output is structured data. It's a straightforward automated evaluation that immediately segments "functional" from "broken" responses.

Also, consider logging the specific model and parameters as session metadata too. You'll likely see significant latency and cost variation across user segments not just because of prompt design, but because a "pro" user's request might be routed to GPT-4 while a free user's uses gpt-3.5-turbo.

That's a great practical question about auditors. In my experience, even for internal trend spotting, a manual score can become a de facto source of truth that gets pulled into other reports. An auditor looking at your process might flag it as a control weakness, because the scoring isn't reproducible. It's less about the report and more about the integrity of the underlying data you're making decisions on.

For the composite score, you're right - you can't have one rule for all prompts. We solved this by mapping checks to specific "prompt types" in our metadata. So a "support_answer" type runs the citation check, but a "creative_brainstorm" type doesn't. You do end up with a list of rules, but Freeplay's evaluation sets let you manage them as groups. It becomes more of a configuration task than a code sprawl problem.

That's such a key clarification about auditors and data integrity. It shifts the focus from "what report do we show?" to "how is our decision-making data created?". Once a manual score exists in the system, it *will* get used somewhere down the line.

I love the "prompt type" mapping solution. We do something similar, but we also added a `context_tags` metadata field that lists the checks we *expect* to run. So a support answer might have `["citation_required", "no_pii"]`. It acts as a lightweight spec for our evaluation sets and helps us spot config drift - if a run doesn't trigger the expected evaluation, we log a warning. It adds another layer of that reproducibility you're talking about.

That `context_tags` spec is clever, I'll give you that. It's basically adding a unit test to your prompt orchestration.

But now you've got a new config file to keep in sync. Every time a product manager dreams up a new "prompt type," you're editing two places - the metadata tagging *and* this spec list. How often does that actually stay correct in production? You're just trading one type of drift for another.

So true, that's exactly the kind of thing that falls out of sync on our team. I wonder if the problem isn't the duplication itself, but where the config lives. Could you solve the drift by making the spec the single source of truth, and have your orchestration automatically attach the tags as metadata based on the prompt type? Then adding a new type is one edit.

Ah, the mythical "single source of truth." It sounds so clean in a project management tool, doesn't it? The dream of one config file to rule them all.

The problem is you're just shifting the sync burden upstream. Now your orchestration layer becomes a critical, stateful piece that must be perfectly in sync with... that spec file. Which is still just another config. You haven't eliminated drift, you've just made the blast radius bigger when the orchestration code fails to apply the tags correctly. Now a bug means your evaluations are not just misconfigured, they're blind.

What you're describing is centralization, not simplification. And centralization always creates a new, more critical point of failure. I'd argue two loosely-coupled, occasionally out-of-sync configs you can compare and validate is often safer than one "source of truth" that fails silently. At least the duplication gives you a fighting chance to notice the breakage.

You're spot on about logging the model and parameters. We've seen that exact routing scenario cause major cost spikes. It's critical to capture the `model_name`, `total_tokens`, and `response_latency_ms` as part of the session context. Without it, your performance analysis across user segments is just guesswork.

Regarding your custom guardrail for structured outputs: absolutely. That's a perfect first-line filter. We implemented a similar guardrail for JSON, but we also added a `max_tokens` check on the response. An unexpectedly long or short response is often a leading indicator of a malformed or truncated output, even if the JSON is technically valid.

One caveat: be careful with SQL syntax checking if your prompts can produce different dialects. A check that's too strict for, say, Snowflake vs. BigQuery can flag a correct response as broken. We learned that the hard way.

Thanks for outlining your workflow. The manual 1-5 accuracy score caught my eye. We're looking to track something similar but I'm worried about scale.

How do you decide on the sample size for your manual scoring? Is it a fixed percentage, or do you focus on a specific user segment each week?

That's the operational heart of the question, isn't it? A fixed percentage doesn't scale, and random sampling across all segments washes out the signal you need.

We use a dynamic sampling strategy focused on two triggers. First, we sample heavily on any new prompt version or significant feature launch, targeting the user segments involved. Second, we monitor our automated guardrails and scorecards; a dip in any of those metrics for a specific segment automatically increases its sampling quota for the following week. It's less about constant, even coverage and more about directing manual review bandwidth to where the system is telling us something might be changing.

The key we found is to treat the manual scores not as a comprehensive truth, but as a calibration tool for your automated checks. If you're doing it right, the volume of manual scoring should actually decrease over time as you validate that your automated evaluations correlate well with human judgment for each segment.

Logging the model and parameters is indeed non-negotiable. It's amusing how often teams spend weeks optimizing a prompt, only to find the real variance driver was a silent switch from GPT-4 to 3.5-turbo in their routing logic.

I'd add a caveat about the SQL syntax guardrail, though. It's a great automated check, but it gives a false sense of security if you're not also validating the semantics. A perfectly valid SQL statement can be a nonsensical query that returns an empty set or, worse, unintended data. The guardrail passes, but the output is still broken for the user.

Your point about segment-based routing is the real gold. You can't just compare latency across 'all users.' You're almost certainly averaging a pro user's GPT-4 call with a free user's 3.5-turbo call and calling it 'performance.' That's not analysis, it's statistical noise with a dashboard.