Exactly, saving that first successful output as a template is the workflow shift that actually makes this manageable for real work. We've adopted a similar system, but we version those templates in a shared drive because they inevitably need tweaks as requirements change.
One caveat: referencing a past successful example works brilliantly for code structure, but I've found it can backfire for API call specs. If my template JSON response includes a `notes` field from that one time it added something extra, the model sometimes interprets that as an instruction to *always* include a `notes` field, even if I delete it from the reference block. It's like it pattern-matches on the historical output, not just the structure I want. Now we have to scrub any superfluous keys from the template before we use it as a reference.
api first
"Prompt drift" is a great term for that. We haven't automated checks yet, but we started tagging outputs in a spreadsheet for basic script work. We track columns like "matches_format" and "extra_keys."
It's manual, but seeing the variance as data helps us pinpoint where the drift happens. For key term diffs, would you check against your original prompt text or just the last known good output?
You definitely didn't do anything wrong. I've run into the same issue when trying to follow onboarding tutorials for a new collaboration platform. The documented example would use a specific dashboard widget setup, but when I'd ask the AI assistant for the same thing, it would give me instructions for a different widget suite entirely, using terminology that wasn't even in our version.
What you're calling "the invisible hourly rate" is so real for small teams trying to adopt these tools. You spend more time verifying and reconciling outputs than you would have just building the script from a known template. It makes the training process for new team members incredibly shaky.
Have you considered treating that case study's published code as your canonical template from the start, and then using it as a reference point in your prompts? I'm curious if explicitly naming pandas and pasting their logic structure into a new request would anchor the output closer to what you expect.
Yep, that's the reality with most of these assistants - the output is just inherently random. You didn't do anything wrong. The model you used probably has a huge variety of customer segmentation examples in its training data, from simple pandas logic to some obscure research library. It just picked a different one.
What you're running into is the biggest hidden cost for small teams. I wasted a week trying to get consistent CloudWatch dashboard code, getting a different AWS SDK version pattern every other try. The only reliable fix is what others mentioned - save a good output as a strict template for next time. Use the case study's function itself as that starting template.
It makes evaluating the tools frustrating, because you're not just testing capability, you're testing output stability. Have you tried setting the temperature to zero? It helps a bit, but not completely.
cost first, then scale
You didn't do anything wrong, the output is just that random. It pulled a different example from its training data.
I get this all the time when asking for basic scripts. The variance is the main thing you're actually testing when you evaluate these tools. For a small business workflow, you can't trust the first output. You have to run the prompt a few times and pick the cleanest one to save as your template, like user425 said.
That case study's code *is* your template now. Use it directly.
"Vendor reliability issue" is spot on. People act like prompt engineering is the only variable, but you're right, the baseline prompt-to-output mapping itself is a moving target. It's like building on a foundation you didn't pour.
>whatever pattern the provider decided was the "best" completion for your prompt this month
This is the part that gets me. Temperature=0 doesn't mean deterministic output, it just means the provider's current "most likely" token chain, which they can and do recalculate on their end. Your consistent business process now depends on their undisclosed ranking metrics. That's a support contract you never signed.
The real advice for a small business isn't "use the API," it's "don't build anything that can't survive the next model refresh." Which, ironically, means not using the tool for the very things it's advertised for.
Anecdotes aren't data.
The "recurring verification costs" point is exactly what makes these tools a trap for any budget-conscious team. You're not just paying the monthly subscription, you're signing up for an indefinite QA role.
I saw a vendor demo where they bragged about 99% accuracy on their canned examples. When someone asked about the cost of validating that last 1% across ten thousand daily operations, they just called it "standard DevOps." That's the hidden line item nobody budgets for.
—DW