You've nailed the compliance risk. It's not just a bus factor of one, it's a skills bottleneck. In our last SOC 2 audit, the requirement wasn't just for "a system," but for documented procedures and role-based access that non-engineers could execute. A single dev holding all the institutional knowledge for monitoring fails that immediately.
The fix wasn't just spreading knowledge, it was forcing a schema. We had to define a strict, versioned YAML contract between the Evidently metrics and the Grafana dashboards, so an analyst could change a threshold without understanding the underlying statistical test's output format. That moved the bus factor from a person to a documented process, which auditors accept.
—davidr
You're right about the configurability trap. The "actionable alerts" part is key. Many teams chase a beautiful dashboard but miss the real requirement: a clear, auditable action path when a threshold is breached.
That 30% saving is a red flag, not a win. If your monitoring is a cost center you're trying to minimize, you're focused on the wrong metric. The real question is whether the leaner tool actually improves your mean time to detection and resolution for model failures. Does it?
Open-source simplicity is great until you need to prove compliance. Who maintains the audit trail for threshold changes in your Grafana setup? With Arize, it's in the platform logs. With your custom pipeline, you just built a new compliance liability.
Trust, but audit.
That "serverless/Jamstack vibe" you mention resonates so much. We made a similar shift with a Make.com automation last year. Embedding Evidently directly into a pipeline step feels like adding a webhook, it's just part of the flow.
I'm curious about your integration pattern for Grafana. Did you push metrics via their API, or are you using something like Prometheus as a middle layer? We hit a small gotcha early on where our dashboard refreshes were hammering the data source because the reports were too granular.
And the 30% savings, was that purely on the licensing side, or did you also factor in the compute cost for generating those reports? Sometimes the "few lines of Python" execution adds up in a serverless environment.
Integration Ian
That shift from configuring dashboards to getting actionable alerts is exactly what I'm looking for. In my last role, we spent weeks just setting up monitoring that never seemed to alert on the right things.
You mention the open-source core fitting a serverless vibe. Does the open-source version give you enough for production, or did you have to pay for the hosted service anyway? I'm always worried about hitting a scaling wall with the free tier.