A common challenge when operationalizing generative AI features is moving beyond aggregate metrics. While a global "average latency" or "average cost per token" is a useful health signal, it often masks critical performance cliffs for specific user cohorts. In our case, we needed to understand if our premium-tier customers were experiencing degraded response quality compared to our free tier, or if our new EU-based deployment was meeting the same SLOs as our US one. Freeplay’s evaluation and experiment tracking features became the cornerstone of this analysis.
The core concept is to treat each user interaction not just as a call to an LLM, but as a test case within a structured experiment. Here’s the workflow we implemented:
1. **Instrumentation & Context Enrichment:** We modified our application's LLM call wrapper to send metadata to Freeplay's SDK, categorizing each request into a logical experiment (e.g., `"summarization_feature_v2"`). Crucially, we also attach key user dimensions as **tags**.
```python
# Example Python snippet for context enrichment
from freeplay import Freeplay
freeplay_client = Freeplay(api_key=os.getenv("FREEPLAY_API_KEY"))
with freeplay_client.start_experiment_run(
experiment_name="prod_summarization_v2",
tags={
"user_tier": "premium", # or "free"
"region": "eu-west-1",
"model": "gpt-4-turbo",
"customer_id_hash": "abc123" # For cohort drilling
}
) as run:
# Your existing LLM call logic here
response = llm_client.chat.completions.create(...)
# Log the inputs and outputs
run.log_prompt(
template=your_freeplay_template,
variables={"text": user_text}
)
run.log_completion(response.choices[0].message.content)
```
2. **Automated Evaluation Setup:** For performance, we care about both system SLIs (latency, cost) and quality SLIs (relevance, completeness). We configured Freeplay to compute:
* **System Metrics:** Latency and token usage/cost are automatically captured.
* **Quality Metrics:** We created a small set of custom Python evaluators (e.g., checking if a summary contains a specific key point) and used Freeplay's judge LLM evaluators for more subjective aspects like tone alignment.
3. **Segmented Analysis in the UI:** Once data flows in, Freeplay's experiment dashboard allows slicing by any tag. You can immediately compare:
* The 95th percentile latency for `user_tier:premium` vs. `user_tier:free`.
* The average "completeness" score for `region:eu` vs. `region:us`.
* Track cost drift over time for a specific `customer_id_hash` cohort.
The key insight was realizing we could define our SLOs per segment. For instance, our SLO for premium users mandates a 99th percentile latency under 2s, while for free users it's under 5s. Freeplay's dashboards let us monitor these segmented SLOs directly. We also uncovered a pitfall: a certain customer segment was consistently triggering a more expensive model fallback due to their input patterns, which was invisible in the global averages and directly impacted our capacity planning.
For teams running canary releases or A/B tests on prompt versions, this tagging and evaluation approach is indispensable. It transforms prompt performance from a monolithic concern into a granular, segment-aware observability problem, which is much closer to how engineering and product teams actually need to reason about their features.
Error budgets are for spending.
Absolutely. Tagging user dimensions as metadata is the key. We've extended that pattern to also tag the underlying infrastructure layer for correlation - for instance, tagging which specific Redis cluster or PostgreSQL replica a session's RAG data was pulled from.
That lets you answer questions like, "Did latency spike for premium EU users because of a slow vector search on replica-3, or was it actually the LLM provider?" Without that granularity, you're just chasing averages.
sub-100ms or bust