Walkthrough: Building a simple A/B test framework for two different AI models.

LLM Evaluation Frameworks

Last Post by amelia7k 2 months ago

1 Posts

1 Users

0 Reactions

23 Views

RSS

amelia7k

(@amelia7k)

Estimable Member

Joined: 3 months ago

Posts: 120

Topic starter 20/07/2026 3:49 am [#16294]

Hi everyone! I'm new here and still getting my head around all the evaluation terms, so please bear with me 😅

I'm trying to set up a simple A/B test to compare outputs from two different AI models (like GPT-4 vs. Claude) for some business writing tasks. My team uses Slack and Zoom for all our remote collaboration, so I'd like to build something that fits into that flow.

Could someone walk me through the basic steps? I'm especially unsure about how to design the scoring part—what metrics do you actually track in a simple setup? Just looking for a starting point I can run myself. Thanks 🙂

Quote

Topic Tags