What actually works for A/B testing prompts in production

Great question. This is a common hurdle when moving from manual comparisons to a proper A/B test.

The core idea is to handle the traffic split and tagging *before* the prompt is sent to the LLM. In your FastAPI app, you'd generate a variant ID (like "v1" or "v2") at the very start of the request, perhaps using a simple random function. Attach that ID to the request context.

Then, every downstream log - the prompt sent, the raw response, latency, and any later user feedback - needs to include that same variant ID as a metadata field. This lets you slice and dice all your metrics by prompt version later. Don't try to compare responses directly; compare aggregated metrics like average latency, user satisfaction scores, or conversion events.

For a simple start, you could log all this to a separate table in your database, with the variant as a column. The key is making the assignment and tagging consistent across your entire data pipeline.

That's a really clear explanation of the tagging mechanism. I'm also working on a similar setup and have been thinking about the metric collection side. I agree that aggregating metrics is the way to go, but I'm curious about the practical step of actually comparing them.

You mention aggregating user satisfaction scores. In a dashboard tool like Power BI or Tableau, would you recommend creating a single data model that joins the variant assignment logs with the downstream feedback events on a common request ID? Or is it simpler to just have a single wide table logged from the start, even if it means some null columns for events that happen later? I'm worried about the complexity of correlating events that happen in different services.

To build on what's been said about the tagging mechanism, you need to separate the logic into three distinct components: assignment, execution, and logging. You're right to focus on the routing code.

Here's a minimal structure for your FastAPI app. The key is to generate a variant and attach it to the request state before anything else happens.

```python
import random
from contextvars import ContextVar
from fastapi import FastAPI, Request

app = FastAPI()
variant_context = ContextVar("variant", default="v1")

@app.middleware("http")
async def assign_variant(request: Request, call_next):
# Simple 50/50 split at the request entry point
assigned_variant = random.choice(["prompt_v1", "prompt_v2"])
variant_context.set(assigned_variant)
request.state.variant = assigned_variant
response = await call_next(request)
return response

@app.post("/generate")
async def generate_endpoint(request: Request, user_query: dict):
variant = request.state.variant
prompt = assemble_prompt(variant, user_query)

# Log the *start* event with the variant immediately
log_event(request_id=request.state.id, variant=variant, stage="start", prompt_text=prompt)

# Call your LLM
start_time = time.time()
llm_response = await call_llm(prompt)
latency = time.time() - start_time

# Log the response and latency, always including the variant
log_event(request_id=request.state.id, variant=variant, stage="complete", response=llm_response, latency=latency)

return {"response": llm_response, "variant": variant}
```
You then need a separate logging function that writes all these events, with the variant attached, to a database or event stream. The `request.state.id` is a unique identifier you'd need to generate per request to tie all events together. Without that correlation ID, you'll have a very hard time joining the data later for analysis.

That middleware's fine for a proof of concept, but you're baking the split logic into the application code. What happens when you need a 75/25 split tomorrow? Or a gradual ramp? You'll be redeploying your API for every traffic change.

You'll want that assignment logic in a separate service or at least a configuration layer. That way the product team can fiddle with percentages without needing a full CI/CD cycle. It's a small step from the code shown, but it's the one that prevents the next "urgent" hotfix.

The tagging approach described is the right first step. Your real challenge will likely be logging consistently across your services. If your feedback or latency metrics come from different systems, you absolutely need that variant ID to flow through to each one. A common trip-up is forgetting to include it in external API call logs or client-side event tracking.

For your FastAPI setup, I'd suggest storing that variant in a request-scoped context, like the `state` object, immediately. Then, every time you write a log entry or emit a metric from within that request's lifecycle, you pull the variant from that context. That's cleaner than trying to pass it as an argument through every function.

What does your current logging pipeline look like? Are you using something structured like JSON logs?

Love that you're starting with a concrete example and a real stack. Everyone's already correctly zeroing in on tagging, but let's talk about what you actually *do* with those tagged requests.

You'll need a decision layer before your LLM call. Think of it as a tiny router function that reads `request.state.variant` and selects the corresponding prompt template. The key is that your business logic shouldn't be littered with `if variant == 'v1'` checks. Wrap that selection up once.

Then, the logging. The temptation is to just print the variant with the response. Resist. You need to emit a structured event to somewhere you can query later, like a metrics database or a data warehouse. Every log line for that request should include the variant ID as a field. That includes latency metrics, any error logs, and especially any downstream feedback events you capture. If your feedback comes from a separate system, you *must* pass that variant ID along, or your analysis is sunk.

So your diagram is basically: Request -> Assign Variant (middleware) -> Router (picks prompt) -> LLM Call -> Log Event (with variant) -> Response. The "compare" part happens entirely in your analytics layer, days later, by grouping those logged events by the variant ID. Start simple: log to a CSV or a dedicated PostgreSQL table from your FastAPI app before you invest in a complex pipeline.

You're focusing on the right problem, but you're about to make the classic first mistake: trying to compare raw responses side-by-side. Don't.

The architecture is simple. A middleware assigns a variant ID at the request's edge. That ID gets stapled to every log, metric, and downstream event for that request's lifecycle. Your comparison happens in aggregate later, on things like average token count or user thumbs-down rate, not on individual outputs.

Your real fight won't be the routing code; it'll be ensuring that variant ID survives into your monitoring and feedback systems. If your user feedback comes from a separate front-end event, and you didn't pass that variant ID along, your test is broken.

I'd skip baking the 50/50 logic into your app code. Put it in a config file or a simple internal service. You'll want to change that split tomorrow without a redeploy.

The tagging approach everyone's describing is spot on, but there's a critical nuance for your FastAPI setup. When you use the `request.state` object in a middleware, remember it's tied to that single request object. If you spawn background tasks or use async functions that outlive the request's immediate lifecycle, you'll lose that context.

A safer pattern is to use a `ContextVar`, as user621 started to show, but also pass it explicitly to any logging or telemetry functions. For example, when you log the latency and response, you'd do something like:

```python
logger.info("llm_response", extra={"variant": current_variant, "latency": latency_ms})
```

This keeps the variant bound to the structured log entry itself, not just the request flow. It's the difference between hoping the metadata travels and guaranteeing it's embedded in the data you care about.