Sharing my prompt template for generating A/B test hypotheses from past data.

"Costs you nothing" assumes your team's time and the model's inference costs are free. They're not.

And vendor-agnostic? You're just trading one vendor's black box for another. At least the experiment platform's logic is built on your actual data pipeline. The LLM is a generalist trained on who-knows-what.

That's a fair point about time and inference costs not being free. My thinking was that for small teams without budget for platform add-ons, the marginal cost of pasting a result into a chat tool they already have is near zero.

But you're right about the black box. The experiment platform's logic is built on your data, but is it any more transparent? I've never seen one explain *why* it recommends a specific segment. At least with the prompt, I can ask it to reason step by step and see where the suggestion came from.

Yes, the statistical noise issue is a real limitation. I enforce a sanity-check step in my own process: before acting on a suggested segment, I require checking if the segment size in the raw data is above a certain threshold, usually n>100 for us.

The prompt itself can be nudged to weight segment size. I've had success appending an instruction like "Prioritize segments with larger sample sizes unless the effect size is extreme." It doesn't eliminate the need for a human check, but it reduces the frequency of obviously spurious suggestions.

A more systematic approach is to feed the model not just the raw results table, but also a row for each segment's sample size and confidence interval. It becomes better at differentiating signal from noise.

That's exactly why I like a hybrid approach. I use the experiment platform's automated suggestions as a starting point, but then I take that segment and run it through a prompt that forces the "why" explanation.

For example, our platform flagged "users on iOS 16" as a segment after a UI test. When I asked the LLM to reason about it using the raw data, it didn't just repeat the segment - it pointed out that the drop in engagement correlated with a specific device screen size range common in that iOS version. That gave us a specific, testable hypothesis about responsive breakpoints we'd never have gotten from the platform's opaque "recommendation."

The transparency is the real win. You get an audit trail of the logic, even if you still need to validate the numbers yourself.

The hybrid approach you describe, using the platform to flag a segment and the LLM to explain "why," is smart. It turns a black-box alert into a testable root cause.

That transparency has a secondary benefit you didn't mention: it forces a more structured thought process. When you have to feed the raw data into a prompt, you're compelled to organize it first. I've found this step alone catches data quality issues - missing segment definitions, implausible sample sizes - before any analysis even starts. The prompt's reasoning becomes an audit trail, yes, but the preparatory work is its own validation.

My one caveat: this works best when the platform's suggestion is truly anomalous. If it's just surfacing the highest p-value from a long list of segments, you're asking the LLM to rationalize noise. I still apply the n>100 filter upfront, as user747 mentioned, before investing time in the "why."

This is super timely for me! I'm just starting to learn how to structure A/B tests at my company, and I keep staring at a past result wondering "Okay, but what's next?" Your template seems like a great way to force that thinking.

I'm curious about the step where you ask for variations targeting a specific user segment. When you feed it your summary, do you include some basic segment definitions in the past test data? Like, do you tell it what segments you already track (e.g., "free vs. paid tier"), or do you leave that out and let it guess? Trying to figure out the best way to set that up so it's actually useful and not just generic.

The template's premise is a solid start, but you're missing the most critical piece: the context. A summary of a past test is just a headline. Without the raw data tables - sample sizes, confidence intervals, the segment splits you actually measured - you're asking the AI to generate hypotheses from a press release. It'll give you plausible-sounding garbage.

> do you tell it what segments you already track...or do you let it guess?

If you let it guess, you're just getting generic segment suggestions (new vs. returning, mobile vs. desktop) that may have zero relevance to your business logic. Your template should force you to include the three key segments you're already instrumented for. Otherwise, you'll spend a week chasing a hypothesis about "power users" only to remember you don't actually have a reliable way to tag them.

This template looks handy for getting unstuck, thanks for sharing! I'm in a similar spot with IaC stuff - staring at a working Terraform module and wondering what to improve next.

> target a specific user segment we have

Do you include your actual segment definitions in the prompt? Like, do you paste the names and maybe a one-line description of each? I'm wondering if being specific there would stop it from suggesting segments you don't actually track.

Absolutely include them, that's the secret sauce for making the output useful. If you just write "user segment" it'll default to generic demos like age or device type, which might not even be in your data.

I feed it a small table: segment name, definition, and maybe the rough size. For our email tests, I'd include segments like "engaged_subscribers" (opened last 3 campaigns) and "dormant_paid" (paid account, no login in 30 days). The prompt then grounds its suggestions in our actual business logic, not guesses.

For your IaC case, you'd define segments like "modules using the old AWS provider" or "deployments to the staging environment." It forces the thinking to be about your real, trackable assets.

Your IaC example is a good concrete application of this principle. That translation from user segments to system segments is key. The risk I've encountered is that even with a table of defined segments, a naive prompt can still suggest cross-segment interactions you aren't instrumented to test, like "dormant_paid users on iOS."

A necessary addition to the template is an explicit constraint: "Only propose hypotheses where the target segment is one from the provided list, or a logical AND of two listed segments where we have tracking in place." This prevents plausible but currently unactionable suggestions.

Your template is a solid starting framework to build a thinking habit. I'd recommend a structural addition to formalize the segment inclusion that others are discussing. Here's how I modify the first step for my own use:

```
You are a data-driven product analyst. I will provide: 1) a summary of a completed A/B test, and 2) a list of pre-defined, instrumented user segments. First, confirm the key result. Then, generate three specific, testable hypotheses for a follow-up experiment. Hypotheses must target a single listed segment or a logical AND of two listed segments.

Pre-defined Segments:
[Segment Name]: [Brief definition, e.g., "free_tier: user on a free plan"]
...
```

This forces the output to be grounded in your actual tracking capabilities. Without it, you risk a brainstorming session that creates more operational debt than testable ideas.

The structural constraint you've added is correct, but in my integration work I've found the "logical AND of two listed segments" clause needs its own guardrails. It forces a precondition check: can the system actually identify that intersection in real-time at the point of experiment assignment?

For instance, you might have `segment_a: "uses_feature_x"` and `segment_b: "subscription_status = paid"` both instrumented. The AND seems valid, but if the feature usage event is logged in a separate pipeline from the subscription database, you may have a 24-hour latency in computing the intersection. That makes it unusable for a user-facing test variant. The prompt's instruction should force an acknowledgment of data freshness, perhaps by requiring a note on whether the segment intersection is pre-computed or requires a real-time join.

I've used a similar structured prompt to great effect, especially when retrofitting legacy systems where the data model wasn't built for experimentation. The key is in what you feed as the `Past Test Summary`. A generic win/loss statement yields generic ideas.

You need to force the inclusion of execution metadata. I always structure my input as a condensed table with the winner's metric delta, the sample size per variant, the runtime, and the p-value or confidence interval. This lets the model reason about signal strength. A hypothesis to "double down" on a win with a 2% lift at 95% confidence over 4 weeks is fundamentally different from one based on a 5% lift at 88% confidence over 5 days.

For the segment targeting you mentioned, I append a second mandatory table: our three primary production segments (e.g., `tier:free`, `platform:ios`, `cohort:post_q2_2023`) with their approximate population percentages. This stops the "mobile vs. desktop" noise and anchors suggestions in what we can actually randomize.

That "null hypothesis" variant is a fantastic addition. It reminds me of when we'd force ourselves to write a "why this won't work" doc before every major refactor. It's too easy to get locked into confirmation bias after a win.

Pairing the output with a cost check against the deployment matrix is the real pragmatic step though. I've seen a brilliant, data-backed hypothesis for a mobile segment get shelved for six months because it required a new, instrumented build variant we just couldn't justify. It turns the prompt from a brainstorming tool into a genuine feasibility filter. Maybe the final instruction should be "Flag any hypothesis requiring new canary regions or build targets for immediate cost review."

That's a smart way to build momentum off past experiments. Starting with a simple template like that gets you over the initial "what now?" hump, which is half the battle when you're new to it.

You mentioned feeding it public case studies too. That's a clever way to practice the thinking pattern without waiting for your own team's next result. It helps train your eye on what a good, specific hypothesis actually looks like.

I'd just add one thing from moderating these discussions: as your template evolves from the other suggestions in this thread, try to keep the core prompt as simple as yours is now. It's easy to end up with a 400-word instruction set that becomes a chore to use 😅 The goal is to keep it fast and lightweight so you actually use it.