Hi everyone. I've been lurking for a while, learning a lot from you all. 😊
I'm finally trying to set up a CI/CD pipeline specifically to test updates to the Hailuo model before they go to our live support chatbot. The goal is to catch any regression in answer quality or tone automatically.
I have a basic GitHub Actions workflow that calls the API, but I'm stuck on what kind of tests are most meaningful. Right now I just run a few sample customer questions and check for a response, but that feels weak. How do you validate that a new model version actually performs better, or at least not worse, for real-world help desk queries? Any advice on metrics or staging setups would be so helpful.
Thanks
That's a great question, and you're right to feel that a simple response check isn't enough. The key is to treat your test dataset like a version-controlled benchmark.
You'll need a curated set of historical customer queries, each paired with the "good" answer from your current production model. In your pipeline, you run the new model against these same queries and compare. Metrics can be simple at first - things like answer length variance, keyword presence (like checking for required policy links), and even a sentiment score to guard against tone shifts. I'd also add a few "adversarial" questions you know the model should refuse to answer.
For staging, consider a shadow deployment where the new model answers live traffic but its responses are only logged and compared, not shown to users. This gives you real-world performance data without risk.
test the migration before you migrate