Migrated from OpenAI Evals to Giskard - 6 month report on accuracy gains

elliotk · 2026-08-05T10:26:11Z

Hey folks, I’ve been absolutely buried in evaluation land for the last half-year, and I wanted to share a pretty significant shift in my workflow that’s paid off big time. For context, I was a pretty heavy user of OpenAI’s Evals framework for a long time—it was my go-to for benchmarking chatbot responses and RAG pipeline outputs. But around six months ago, I decided to port my primary evaluation suite over to Giskard, and the difference in measurable accuracy and, frankly, my own debugging sanity has been night and day. So, why the move? I hit a wall with scaling and reproducibility. With OpenAI Evals, I was stitching together a lot of custom Python scripts to handle things like: * Dynamic test case generation based on my knowledge base * Visualizing drift between model versions * Automatically generating “adversarial” test cases to find edge cases Giskard essentially bundles a lot of that philosophy into a single, opinionated framework. The core concept of wrapping your model (LLM or otherwise) and dataset into their objects felt clunky at first, but it creates a unified surface area for applying tests. Here’s a concrete example of the accuracy gain. I have a RAG pipeline for internal documentation. With my old Evals setup, I was measuring accuracy mostly on a set of ~50 static Q&A pairs. My “accuracy” was a score derived from LLM-as-judge grading on faithfulness and relevance. After migrating to Giskard, I used their `scan` function on my model. It automatically generated over 200 new test cases by perturbing my original questions—things like introducing typos, synonyms, or negations. Running my pipeline against this *expanded* test suite revealed a whole class of failures where the retrieval was brittle to minor rephrasing. Fixing those (mostly through better query transformation and chunking) boosted my *real-world* accuracy on user queries by an estimated 15-20%. The old static suite simply wasn’t finding those holes. The other massive win is in monitoring. Setting up a “testing as code” regimen where my evaluation suite runs automatically on PRs has been a game-changer. Giskard’s integration with the CI/CD pipeline means I can now get a report showing: * Performance regression on any core functionality * New vulnerabilities introduced (e.g., sensitivity to personal identifiable information, even if not in my training data) * A clear, visual diff of what changed in the outputs It’s not all sunshine—the learning curve is steeper, and you have to buy into their way of structuring things. But for a tinkerer who loves to compare tools side-by-side, the data doesn’t lie. My evaluation coverage is broader, the tests are more robust, and I’m catching issues much earlier. For anyone else feeling like their eval process is becoming a pile of scripts held together with hope, I’d really recommend giving Giskard a look. It’s turned evaluation from a periodic report card into a live, interactive debugging dashboard.

Page 3 / 3 Prev

LLM Evaluation Frameworks

Last Post by Emily_R 32 minutes ago

32 Posts

32 Users

0 Reactions

4 Views

RSS

EthanM

(@ethanm)

Estimable Member

Joined: 3 weeks ago

Posts: 79

06/08/2026 9:32 pm

Yeah, the "stale test infrastructure" point hits home. It's like you're building a new pipeline just to run tests, and then you're stuck maintaining that too.

So, what's the solution? Just accepting that lag as a tax, or did you find a way to auto-generate those datasets from the actual pipeline output?

ReplyQuote

Emily_R

(@emilyr22)

Estimable Member

Joined: 3 weeks ago

Posts: 110

06/08/2026 10:00 pm

That's a great question. We saw a similar pattern. The initial accuracy jump was all from fixing those hidden bugs during the port, like you said. It felt like a one-time correction.

But on the sustained improvement, I'm not sure. Our benchmark scores plateaued after that initial fix. The ongoing scans haven't pushed them higher, but they've been crucial for catching the kind of regressions user1235 mentioned, especially after we update our product documentation. It's more about protecting the baseline.

So the ROI for us is in stability, not continuous gains. Has that been your experience, or did your scores keep climbing?

ReplyQuote

Page 3 / 3 Prev