Migrated from OpenAI Evals to Giskard - 6 month report on accuracy gains

elliotk · 2026-08-05T10:26:11Z

Hey folks, I’ve been absolutely buried in evaluation land for the last half-year, and I wanted to share a pretty significant shift in my workflow that’s paid off big time. For context, I was a pretty heavy user of OpenAI’s Evals framework for a long time—it was my go-to for benchmarking chatbot responses and RAG pipeline outputs. But around six months ago, I decided to port my primary evaluation suite over to Giskard, and the difference in measurable accuracy and, frankly, my own debugging sanity has been night and day. So, why the move? I hit a wall with scaling and reproducibility. With OpenAI Evals, I was stitching together a lot of custom Python scripts to handle things like: * Dynamic test case generation based on my knowledge base * Visualizing drift between model versions * Automatically generating “adversarial” test cases to find edge cases Giskard essentially bundles a lot of that philosophy into a single, opinionated framework. The core concept of wrapping your model (LLM or otherwise) and dataset into their objects felt clunky at first, but it creates a unified surface area for applying tests. Here’s a concrete example of the accuracy gain. I have a RAG pipeline for internal documentation. With my old Evals setup, I was measuring accuracy mostly on a set of ~50 static Q&A pairs. My “accuracy” was a score derived from LLM-as-judge grading on faithfulness and relevance. After migrating to Giskard, I used their `scan` function on my model. It automatically generated over 200 new test cases by perturbing my original questions—things like introducing typos, synonyms, or negations. Running my pipeline against this *expanded* test suite revealed a whole class of failures where the retrieval was brittle to minor rephrasing. Fixing those (mostly through better query transformation and chunking) boosted my *real-world* accuracy on user queries by an estimated 15-20%. The old static suite simply wasn’t finding those holes. The other massive win is in monitoring. Setting up a “testing as code” regimen where my evaluation suite runs automatically on PRs has been a game-changer. Giskard’s integration with the CI/CD pipeline means I can now get a report showing: * Performance regression on any core functionality * New vulnerabilities introduced (e.g., sensitivity to personal identifiable information, even if not in my training data) * A clear, visual diff of what changed in the outputs It’s not all sunshine—the learning curve is steeper, and you have to buy into their way of structuring things. But for a tinkerer who loves to compare tools side-by-side, the data doesn’t lie. My evaluation coverage is broader, the tests are more robust, and I’m catching issues much earlier. For anyone else feeling like their eval process is becoming a pile of scripts held together with hope, I’d really recommend giving Giskard a look. It’s turned evaluation from a periodic report card into a live, interactive debugging dashboard.

Page 2 / 2 Prev

LLM Evaluation Frameworks

Last Post by HarryJ 1 hour ago

16 Posts

16 Users

0 Reactions

3 Views

RSS

HarryJ

(@harryj)

Reputable Member

Joined: 3 weeks ago

Posts: 191

06/08/2026 5:49 am

That forced model wrapping step is exactly where I saw the biggest immediate win too. It flagged a bunch of inconsistent data cleaning steps across my different scripts that were skewing old results.

Your point about visualizing drift is key. Having that built in meant we finally started tracking it properly, which caught a nasty performance drop after a knowledge base update last quarter. We would've missed it for weeks before.

I'm curious about your adversarial test generation based on your knowledge base though. Did you find the auto-generated ones from Giskard's scanners actually useful out of the box, or did you have to heavily curate them like we did? We ended up writing most of our own based on actual user queries.

Automate the boring stuff.

ReplyQuote

Page 2 / 2 Prev