Unpopular opinion: If your eval can't flag a 10% drop in factual accuracy, it's useless.

Been researching eval frameworks for a vendor selection project. Most demos focus on win-rate, sentiment, or generic "helpfulness" scores.

If I'm paying for an LLM API, a 10% drop in factual accuracy on my internal data is a contract-breaking, business-critical failure. Yet I haven't seen a single framework demo that reliably catches this in a real-world scenario. They're all optimized for benchmarks, not production monitoring.

What are you actually using to track this? Not theoretical metrics, but tools that run daily on live outputs. I need something that can baseline accuracy and alert on drift. Open-source or commercial, but I need to see pricing and how it scales.