Hey everyone 👋 I've been integrating LangSmith into our CI/CD pipelines for monitoring our RAG applications, and I've hit the same wall a few of you probably have: the official cookbook examples for custom evaluators are a great start, but they don't quite bridge the gap to real-world, production-grade checks.
For instance, the example shows a basic correctness check against a static answer, but our needs are more nuanced. We need to validate responses against dynamic data, assess the quality of retrieved context for hallucinations even when the final answer is okay, and track drift in chain behavior over deployments.
I'm trying to build a set of evaluators that can:
1. Compare an LLM's output against a snippet of our internal knowledge base (not just a static string).
2. Score the "completeness" of an answer when dealing with multi-part questions.
3. Measure the relevance of *all* retrieved documents in a RAG flow, not just the final synthesis.
My initial attempt involved subclassing `StringEvaluator`, but I quickly got tangled in the run trace structure. Where's the best place to hook in for complex logic?
Here's a snippet of where I'm currently stuck. I'm trying to access the retrieved documents from the trace, but the structure seems to vary:
```python
from langsmith.evaluation import evaluate
from langsmith.schemas import Example, Run
class MyContextRelevanceEvaluator(Evaluator):
def __init__(self):
super().__init__()
def _evaluate(self, run: Run, example: Example = None) -> dict:
# How do I reliably get the retrieved contexts from a RAG run?
# The run outputs might be just the answer string.
# Are the documents buried in the run events or metadata?
retrieved_docs = ???
# My logic to score each doc's relevance to the query...
```
My main questions for the community are:
* What's the most robust pattern for extracting intermediate steps (like retrieval output) from a run for evaluation?
* For those running evaluations in CI, how are you managing the evaluation dataset? Are you versioning it alongside code?
* Any examples of evaluators that make external API calls (e.g., to a model-as-a-judge endpoint) and handle failures gracefully?
I'd love to see some more complex, battle-tested patterns. If you've built custom evaluators that go beyond simple string comparison, sharing your structure would be a huge help to many of us trying to move from prototyping to production monitoring.
ā francesc
ā francesc
You're right, the jump from those basic examples to production monitoring is significant. The key is to bypass `StringEvaluator` for these cases and work directly with the `RunEvaluator` class. It gives you the full trace object, not just inputs and outputs.
For your three points: comparing to a knowledge base snippet means you need evaluator state or a retrieval step within the eval itself. You'd pass a document ID or query to the evaluator constructor, then fetch the ground truth during the `evaluate_run` call. For completeness and multi-document relevance, you'll need to parse the trace to find the retriever step and inspect the `outputs['documents']` list. That's where the run structure gets messy.
Post your snippet, even if it's broken. The most common trip-up is assuming the run's `outputs` is the LLM's final answer, when you often need to walk back through the parent run tree to find the retrieval step's output.
Plan the exit before entry.
Your snippet cutting off is the real issue. We can't see where you're stuck.
Forget subclassing anything at first. Write a plain Python function that takes the entire LangChain run dict and prints what you need. That's your debug step. The evaluator class just wraps it later.
To answer your hook question: your complex logic goes in the evaluate_run method. That's where you have access to the run's parent trace, inputs, outputs, and the full nested structure. You can walk the child runs to find the specific retriever step and pull its document list.
Start with that function. Post the actual trace path you're trying to follow and the error.
Absolutely agree with the plain function approach - that's how I always prototype these things. It's too easy to get tangled in the class structure before you even know what data you need.
One practical snag I've hit: that run dict can be enormous. When you start walking child runs in a complex trace, printing the whole thing is a mess. I'd suggest isolating the specific key path you need first, like grabbing `run.outputs['documents']` or `run.child_runs[3].inputs`. Maybe even write a tiny helper to search for run names by type.
For user1290's original question, that debug function is also the perfect place to test your knowledge base lookup before you bake it into an evaluator class. You can mock the retrieval call and see if your scoring logic makes sense with real trace data.
don't spam bro
You've identified the core limitation. The issue isn't just with your code, it's a conceptual mismatch. Subclassing `StringEvaluator` forces you into a pattern of comparing two strings. Your requirements - dynamic knowledge base lookups, multi-document relevance, completeness scoring - aren't string-to-string operations. They are audit functions on the entire trace.
That's why the suggestion to start with a plain function is correct. But take it a step further: write that function as if LangSmith didn't exist. Its sole job is to ingest a complete run trace dictionary and return a score with a reasoning key. Once that works reliably, wrapping it in a `RunEvaluator` is a trivial, mechanical step.
The real complexity is in designing a stable way to navigate your specific trace schema to find the retriever step and its document list every time. That's what you should be printing and validating first.
Trust but verify ā especially the fine print.
Good grief, you're already down the rabbit hole of trying to hook into the trace before you've even decided what your scoring criteria are. You're trying to build the scaffolding before you know the shape of the building.
Forget about subclassing anything for now. Your three requirements are basically three separate auditing functions. Write each as a standalone script that ingests a *saved* LangSmith run JSON dump. If you can't produce a score from that static file, you'll never make it work in a live evaluator. The run trace structure is a distraction at this phase.
You'll also hit a more practical wall: how are you sourcing that "snippet of our internal knowledge base" dynamically during the eval? If that requires a database call or API fetch, you've just introduced a network dependency and latency into your evaluation suite. That's a cost-benefit discussion nobody wants to have at 2 a.m. when the pipeline is failing.
Test the migration.
User330 is right about the static JSON dump, that's the only sane way to start. But the suggestion that the trace structure is a "distraction" misses the point - figuring out how to navigate that mess is half the battle. You can't write a reliable evaluator unless you know exactly where in the sprawling run your retriever step lives, and that varies by framework and version.
Their last point about the network dependency is the real gotcha, though. It's not just latency. You're now signing up to maintain and pay for a separate evaluator service that has its own failure modes, separate from your main pipeline. Every vendor loves to sell you on "observability," but they never mention the extra infrastructure you'll need just to judge the first one.
ā skeptical but fair
Totally feel your pain with subclassing `StringEvaluator`. It's like trying to fit a square peg in a round hole for what you're describing.
> Where's the best place to hook in for complex logic?
The `RunEvaluator` class is the hook you need. Its `evaluate_run` method gives you the entire run object, so you can traverse the trace to find the retriever step and its documents. That's how you'd tackle your point #3 on measuring all retrieved docs.
For your knowledge base lookup, your evaluator can accept a client or loader in its `__init__`, then use that inside `evaluate_run` to fetch the dynamic ground truth on the fly. Just be mindful it adds latency and a new point of failure.
Have you managed to pull a full trace JSON from LangSmith yet? That's the best first step to prototype your scoring logic without any framework code getting in the way.
Beta tester at heart
>Have you managed to pull a full trace JSON from LangSmith yet?
That's a good practical step. I used the API client's `client.read_run(run_id)` and exported it to a file. The challenge was finding the right keys in the nested structure without getting lost.
The suggestion to pass a client in `__init__` for a knowledge base lookup seems necessary, but it makes testing the evaluator in isolation much harder. You now need a mock for that external service.
Oh good, you've hit the network dependency wall. Everyone glosses over that part. user1364 is right about the testing headache, but the real cost is operational.
You're now running a separate service (your evaluator) that makes outbound calls (to your knowledge base) to judge your primary service. That's a new billing line and a new failure mode. If your KB has an outage, your monitoring goes dark. The vendor docs never show you that line item.
Start with a stub. Make your evaluator's __init__ accept a fetch function, but default it to a lambda that returns a hardcoded string. Get your trace navigation working first. You can swap in the real fetch later, after you've priced out the extra latency and API calls.
Cloud costs are not destiny.
>Start with a stub.
That's the core principle, but a stub only solves prototyping. The architectural problem is that your evaluator's dependency on a live knowledge base makes it a stateful, side-effectful service, not a pure scoring function.
You can design around this by making the evaluator's input a pre-fetched "ground truth context" object, not a client. Your orchestration layer (the thing calling the evaluator) becomes responsible for fetching the KB snippet and injecting it alongside the run trace. This separates the pure scoring logic from the impure data fetching, making the evaluator stateless, cacheable, and trivial to unit test. The trade-off is you push the network dependency and its cost one layer up, but at least it's a conscious design choice, not a hidden side effect.
infrastructure is code