Everyone pushes these heavyweight evaluation suites as the solution. They're not. They're anchors.
I need to run automated checks on LLM outputs in CI/CD. Current framework takes minutes per check. That's useless. Looking for something that runs in seconds. Must handle basic criteria: answer relevance, format compliance, no hallucinations. Don't need a 50-metric PhD thesis. What are you actually using that doesn't break the pipeline?
Trust but verify.
Exactly. I built my own tiny Python script for this. It just calls GPT-4o's API with a scoring rubric in the system prompt, asks for a simple 1-5 score and a yes/no on format. Costs a fraction of a cent and finishes in a few seconds.
Have you considered rolling your own like that, or are you looking for an off-the-shelf tool?
That's a really interesting approach. I hadn't considered using the model to evaluate itself that way. My worry with rolling my own would be consistency, especially with a simple 1-5 score. How do you handle the scoring staying reliable across different runs or API versions? Do you find you need to constantly tweak your rubric prompt?
I feel your pain. Those big suites are overkill for a pipeline check. I've been using Deepchecks' LLM Evaluation for exactly this - it runs a focused suite on relevance and factuality in under 30 seconds for most of our outputs. It's basically their open-source library with a subset of checks tuned for speed.
The key for us was disabling the heavy statistical metrics and just keeping the direct LLM-as-judge ones, which sounds similar to what user537 described but pre-packaged. It plugs right into our GitHub Actions. Have you looked at stripping down an existing library instead of a full rebuild?