You know the feeling. One day your RAG pipeline is humming along, churning out answers that make your product team look like geniuses. The next, it’s confidently explaining that the capital of France is “a delightful pastry, typically filled with almond cream,” and your Slack channel lights up like a Christmas tree. The first instinct is to blame the model, or the vector DB, or maybe cosmic rays. But after watching one too many teams spend weeks “upgrading” their embedding model only to find the real issue was a $5 typo in a config file, I’ve learned to start with the traces.
When answer quality tanks suddenly, it’s almost never a silent degradation of an LLM. It’s a change. Something *changed*. Your job is to find it. Freeplay’s trace comparison is where I start, because it moves you past “the answers are worse” to “here’s precisely *how* they are worse, and which component is responsible.”
Here’s my step-by-step, born from expensive hindsight:
* **First, Isolate the Failure Mode.** Don’t just say “worse.” Use Freeplay to pull traces from before and after the drop. Compare them side-by-side. Is the retrieved context now irrelevant? Is the model ignoring good context? Is the system prompt subtly mangled? This tells you which lane to look in: retrieval, synthesis, or prompt assembly.
* **The Retrieval Autopsy.** If the context looks wrong, drill into the retrieval step. This is where 70% of my “sudden drop” cases live.
* Check the raw query being sent to your vector store. Has a pre-processing step (query rewriting, hyphen stripping) gone haywire? I once saw a “optimization” that removed all punctuation, turning “user_id: 456-account_status: active” into pure nonsense.
* Examine the top_k and score thresholds. Did a deployment silently change `top_k: 5` to `top_k: 50`, flooding the LLM with noise? Did someone adjust a similarity threshold, letting in marginal, off-topic chunks?
* Validate the embedding space itself. If you changed nothing, skip this. But if there was *any* deployment—even to an unrelated service—verify no one swapped the embedding model or index by accident. A mismatched index is a silent killer.
* **The Synthesis Inquiry.** If the retrieved context looks pristine but the answer is off, the LLM is the culprit… but probably not the way you think.
* Compare the *full* prompts, not just the user question. Freeplay’s diff view is gold here. Did a dynamic few-shot example injector start pulling from a corrupted test file? Did the system prompt get a “small tweak” that redefined the assistant’s persona?
* Check the model parameters. A well-intentioned change to `temperature` (from 0.1 to 0.7) for “more creativity” will absolutely nuke deterministic, factual output. Same for `top_p`.
* **The Dependency Check.** This is the consultant’s secret weapon. Your RAG pipeline isn’t an island.
* Were there upstream changes to the document chunking or embedding ingestion pipeline? A new chunking strategy that splits tables in half will poison your retrieval weeks later.
* Did a third-party API (for query enrichment, entity extraction, etc.) start timing out or returning errors, causing your pipeline to proceed with partial, malformed input? Traces often show this as a weird, truncated intermediate step.
The moral of the story is never to assume. The most dramatic collapses often have the most mundane roots—a stray environment variable, a default value in a new library version, a caching layer returning stale embeddings. Freeplay’s value isn’t just in showing you the request and response; it’s in letting you *compare the machinery* between when it worked and when it didn’t. Start there before you let anyone talk you into a six-week “re-architecture.”
-- Carl
Test the migration.