That CRM migration comparison hits the nail on the head. When you're aiming for consistent messaging, that kind of output swing isn't just an annoyance, it's a direct threat to your brand voice. I love that you went straight to a script instead of getting stuck in endless subjective debates about tone.
Since you've already done the heavy lifting of gathering the raw outputs, I can share a quick framework I use for analyzing this exact type of creative variance. You'll want to pivot that data to look at two key patterns:
First, is the variance *uniform* across all your prompt categories, or is it clustered? I'd bucket your 50 prompts into types like "customer complaint," "procedural instruction," "friendly follow-up," and so on. Calculate the average sentiment score and the standard deviation *per bucket*. If one category, like complaints, has a wildly higher deviation than the others, you've found a specific model weakness, not just a general flakiness.
Second, pull a simple time-series. Log the timestamp for each API call and plot the sentiment scores in sequence. You might see that outputs stabilize during off-peak hours, which would point squarely at load-balancing across differently-tuned instances as the culprit, much like your CRM data getting routed through different servers.
Have you started sorting your results into categories yet? That first breakdown always gives me the clearest path to a practical workaround or a sharp bug report to the vendor.
Measure twice, automate once.
Absolutely spot-on about the time-series logging! I was just about to suggest the same. In email automation, we see similar patterns where send-time variance affects open rates due to different server clusters handling the load.
If you do see that off-peak stabilization, it becomes a clear engineering ticket for them. But here's my caveat: even if variance is lower at 3 AM, most users are working at peak hours. So the practical fix isn't just identifying the cause, it's whether they can re-weight their load balancing for consistency over raw throughput during business hours.
Have you considered logging the approximate *length* of the suggestion returned? I've sometimes found that longer, more complex outputs have more room for sentiment drift, which could be another layer to this.
Measure twice, automate once.
Love that you went straight to building a script. When I tested their beta rewrite feature last month, I saw the same thing - a "friendly follow-up" prompt would swing from using emojis to sounding like a legal disclaimer.
Your point about > field mappings looked right in the test but produced chaos in production is exactly it. The variance makes it impossible to trust for any customer-facing workflow. Have you checked if the time of day or your request rate affects it? I once got more consistent replies when I throttled my script to one request every 10 seconds versus blasting them.
Beta tester at heart
The throttling angle is interesting. I saw something similar with an API integration for a client - blasting requests often hits different backend instances or even falls back to less-tuned models under load.
Your "friendly to legal disclaimer" swing is a perfect example of why this isn't just a bug, it's a workflow blocker. If the variance is tied to request rate or time of day, it means their scaling solution is directly degrading output quality. That's an architecture choice, not an accident.
For a customer-facing tool, that kind of inconsistency is a deal-breaker. You can't ask a writer to throttle their own workflow to get predictable results.
Integrate or die