Hi everyone! 👋 I've been trying to set up a real A/B testing workflow for our LLM prompts in production, but I'm hitting a wall.
We can compare prompts manually, but I'm struggling to automate it. How do you actually route traffic between prompt versions, log the responses, and compare metrics like latency or user feedback? I'd love a beginner-friendly breakdown of the components needed. Our current setup is Docker containers serving a simple FastAPI app.
For example, if I have two prompts, `prompt_v1` and `prompt_v2`, how do I structure the code to send 50% of requests to each and track which response came from which version? A simple code example or a diagram of the architecture would be incredibly helpful. Thanks in advance to anyone who can point me in the right direction!
Great question. This is a common hurdle when moving from manual comparisons to a proper A/B test.
The core idea is to handle the traffic split and tagging *before* the prompt is sent to the LLM. In your FastAPI app, you'd generate a variant ID (like "v1" or "v2") at the very start of the request, perhaps using a simple random function. Attach that ID to the request context.
Then, every downstream log - the prompt sent, the raw response, latency, and any later user feedback - needs to include that same variant ID as a metadata field. This lets you slice and dice all your metrics by prompt version later. Don't try to compare responses directly; compare aggregated metrics like average latency, user satisfaction scores, or conversion events.
For a simple start, you could log all this to a separate table in your database, with the variant as a column. The key is making the assignment and tagging consistent across your entire data pipeline.
Integrate or die
That's a really clear explanation of the tagging mechanism. I'm also working on a similar setup and have been thinking about the metric collection side. I agree that aggregating metrics is the way to go, but I'm curious about the practical step of actually comparing them.
You mention aggregating user satisfaction scores. In a dashboard tool like Power BI or Tableau, would you recommend creating a single data model that joins the variant assignment logs with the downstream feedback events on a common request ID? Or is it simpler to just have a single wide table logged from the start, even if it means some null columns for events that happen later? I'm worried about the complexity of correlating events that happen in different services.