I'm evaluating LangSmith for a sales qualification bot that's about to go into production. The vendor's own "evaluation" docs are full of fluff metrics like "relevance score" and "helpfulness" on a 1-5 scale. Not useful for a system that needs to handle high volume and directly impact pipeline.
I need to instrument my RAG pipeline properly from the start. What raw, quantifiable metrics should I be logging for each chain/agent step in a sales context? I'm thinking beyond token counts and latency.
My non-negotiables so far:
* **Cost per conversation:** Model + retrieval API calls broken down.
* **Intent classification accuracy:** Did the bot correctly identify a "lead" vs. "customer support" vs. "spam" query? Need a hard label comparison.
* **Data retrieval precision:** For RAG, what percentage of returned chunks were actually relevant to answering? This requires manual scoring a sample.
* **Fallback rate:** How often did the chain hit a "I don't know" or off-topic deflection?
What am I missing that's specific to sales/business development?
* Should I track the percentage of conversations where a qualifying *question* (e.g., "What's your budget?") was correctly asked based on the playbook?
* How are you measuring the quality of extracted contact info or meeting request parameters?
Show me your trace tagging or custom feedback setup. I want to see code, not theory.
```python
# Rough sketch of what I'm considering for custom scoring
def evaluate_sales_trace(run):
metrics = {
"asked_budget_question": False,
"extracted_company": None,
"call_scheduled": False
}
# ... parse messages for patterns
return metrics
```
--- pw
pipelines are code
You've got a solid start on the operational metrics. The critical gap I see is tying those metrics directly to financial outcomes, which is the entire point of a sales bot. You need to measure its influence on the pipeline stages it's designed to handle.
Your missing metrics should be conversion rates between stages the bot owns. For example:
* Lead capture to qualified meeting booked: What percentage of conversations where the bot identified a lead actually resulted in a calendar event?
* Qualification question to answer captured: When the bot asks "What's your budget?", log if a specific answer was extracted and stored, not just if the question was asked.
* Cost per qualified lead: This is your "cost per conversation" divided by the lead capture rate. It's the real unit economics.
Also, instrument for downstream contamination. Track how often a conversation the bot tagged as a "qualified lead" was later rejected by a human SDR for being off-target. That false positive rate directly translates to wasted sales labor costs.
Read the fine print