Right, so your fancy new LLM monitoring dashboard lights up like a Christmas tree. The "Answer Relevance" score for your customer support bot has plummeted 40 points overnight. Panic starts to set in. Before you start yelling at the data science team or ripping out your prompt templates, let's walk through how you actually *debug* this with a tool like Arize AI. The goal here isn't just to see the dip—any graph can do that—it's to find the *why* buried in the inferential rubble.
First, you rule out the obvious. This is where most teams waste a week.
* **Did a model version get silently swapped in production?** Check your inference logs. A new GPT-4-turbo version or a tweak to your Claude API default can do this. Arize's model performance comparison should be your first stop.
* **Was there a deployment or a prompt change?** Talk to engineering. Someone might have "optimized" a system prompt and forgotten to tell the folks monitoring business metrics. Track prompt versions like code versions.
* **Is it a data skew issue?** A sudden influx of a new type of query (e.g., "fix my billing" after a system outage) that your model was never trained on will tank your aggregate score. Segment your traffic.
Assuming none of that panned out, you dig into the tool. The value isn't in the alert; it's in the lineage. You need to trace a low-scoring prediction back through its lifecycle.
1. **Pull up the problematic cohort.** Filter for predictions from the time the score dropped, with low relevance scores. Don't look at aggregates—look at individual examples.
2. **Examine the ground truth vs. prediction.** How is "relevance" even measured? Is it human-labeled? An LLM-as-a-judge? If it's the latter, your "relevance score" drop could actually be a failure of your judge model, not your primary model. I've seen it happen.
3. **Follow the embedding projections.** This is often the killer feature. Arize should show you a UMAP/t-SNE plot of your query embeddings. That sudden drop likely corresponds to a new, distinct cluster of queries that have appeared far from your training data. Visually, you'll see a blob of points that have drifted into uncharted territory. *That's* your root cause: a new, unrecognized intent.
4. **Correlate with other metrics.** Did latency spike? Did cost per query jump? Maybe a fallback model or a different retrieval pathway was triggered for this new query cluster, one that gives worse answers.
The post-mortem usually reveals something wonderfully mundane: a new marketing campaign drove users to ask about a product feature your support bot never learned, or a third-party knowledge base used for retrieval had a document schema change that broke your RAG pipeline. The tool showed you *where* to look, but you still needed to know *how* to think about the problem.
The lesson, as always, is that monitoring is not about dashboards. It's about having a forensic workflow to connect a statistical anomaly in a metric to a concrete, deployable fix. Otherwise, you're just watching your numbers get worse in high definition.
-- Carl
Test the migration.