I've been conducting a systematic evaluation of LangSmith's automated evaluation features for orchestrating complex, multi-step workflows that involve custom function calls within a ReAct-style agent. My primary objective is to establish a reproducible, event-driven evaluation pipeline that can validate not only the final output but also the integrity of the intermediate reasoning steps and the correct invocation of tools with precise parameters. However, I am encountering persistent and somewhat opaque failures when the evaluator processes traces containing these custom function calls.
The core issue manifests when I configure an evaluation run against a dataset of input queries that are designed to trigger specific tool usage. The evaluator, which operates correctly on simpler LLM-only chains, consistently returns a failure state. The LangSmith UI indicates the evaluation itself crashed, but the diagnostic details are insufficient for root-cause analysis. My hypothesis centers on a serialization or context management problem when the evaluation runtime attempts to execute or simulate the custom function within its isolated assessment environment.
To illustrate my configuration, I have a custom function registered as a tool within my agent's runtime:
```python
from langchain.tools import tool
@tool
def fetch_order_status(order_id: str) -> str:
"""Fetches the current status of a given order from the ERP system."""
# Simulated integration logic
erp_client = get_erp_client()
status = erp_client.orders.retrieve(order_id).status
return f"Order {order_id} status: {status}"
```
My evaluation setup is as follows:
* **Dataset:** Contains queries like "What is the status of order #12345?".
* **Evaluator:** A custom `CriteriaEvalChain` configured to check for "correct_tool_use" among other criteria.
* **Run Settings:** The evaluation is configured to run on the full trace of the agent, which includes the tool call node.
The failure does not seem to be in the agent's execution—it runs perfectly in production and returns the correct data from the mock ERP. The breakage is isolated to the evaluation phase. I have attempted several mitigation strategies:
* Wrapping the function body in extensive try-catch blocks and logging, but no internal exceptions are thrown before the eval crash.
* Verifying that all necessary context (API clients, environment variables) is available in the evaluation runtime environment.
* Testing with a simpler, pure-Python tool that performs no I/O, which surprisingly also fails.
This points to a potential mismatch in how LangSmith's evaluation executor hydrates the tool's callable context versus how the agent runtime does it. My critical questions for the community are:
* Has anyone successfully implemented auto-evals on agents with custom function calls, and if so, what was the required configuration pattern?
* Is there a specific method for registering tools or defining evaluators that must be adhered to for evaluation compatibility?
* Are there known limitations regarding the complexity of the callable object or its dependencies when run within the evaluator?
The inability to automatically evaluate these workflows creates a significant gap in our CI/CD pipeline for conversational agents, as we cannot regression-test tool-integration logic without manual review. Any insights into the architecture of the evaluation runner or shared experiences would be invaluable.
-- Ivan
Single source of truth is a myth.