Hey everyone, just read the new Freeplay blog post about '10x faster evals' and I've got some questions. As someone still getting my head around CI/CD for ML, that kind of claim really grabs your attention, but I'm trying to figure out what it actually means in practice.
They talk about parallelizing evaluations and using their CLI to run things concurrently. From a DevOps angle, that makes sense—if you can split your test suite across multiple runners, you'll get a speed boost. But 10x feels like a best-case scenario, right? It probably depends heavily on how your evals are structured. If they're a bunch of small, independent tasks, then sure. But if there's a lot of shared state or sequential steps, the gains might be smaller.
I'd love to see a more concrete example. Like, a before-and-after config snippet. How does their system manage the orchestration and aggregation of results? Do they spin up containers for each parallel run? Coming from a GitHub Actions background, I'm imagining something like a matrix strategy, but for eval scenarios.
Also, what are the common pitfalls with parallelizing evals? I'm thinking about stuff like:
- Throttling or rate limits on model APIs when you fire off 50 requests at once.
- How to handle shared, costly resources (like a GPU pool).
- Making sure the results are collected reliably if one of the parallel runs fails.
Has anyone tried implementing this yet? I'm curious if the speed increase came with a big jump in complexity or cost. Sometimes "faster" in the blog post just means "spend more compute credits," which isn't the same thing 😅
Learning by breaking