After spending the last quarter evaluating various agent-building platforms for a high-touch B2B sales use case, I’ve concluded that most solutions are either too brittle for complex logic or become prohibitively expensive when scaled to meaningful conversation volumes. My hypothesis was that a well-architected agent, built on our own enriched lead data, could significantly outperform our current manual pre-qualification process, particularly in scoring intent and identifying champion signals.
This post details my implementation of a lead qualification agent using Relevance AI, which I selected after a structured evaluation against several orchestration frameworks. The primary goal was to automate the initial 10-15 minute discovery call qualification, pulling from our internal PostgreSQL database containing lead interaction data (website sessions, content downloads, support ticket history) and enriching it with real-time LLM analysis.
The core agent workflow is structured as a sequential chain with parallel data retrieval:
* **Step 1: Data Hydration.** The agent receives a lead's email, then executes parallel tool calls to our internal API and a Relevance AI vector dataset of past sales call transcripts.
* **Step 2: Profile Synthesis.** A dedicated LLM task synthesizes the retrieved raw data into a structured profile, focusing on firmographics, observed behavioral signals, and potential pain points.
* **Step 3: Scoring & Qualification.** A final LLM task uses the synthesized profile against our BANT-based criteria, outputting a JSON object with scores, a confidence level, and specific reasoning citations.
The configuration for the final scoring task, defined in Relevance's studio, is as follows:
```yaml
task: "qualification_scoring"
instruction: >
Using the synthesized lead profile, score against the BANT framework.
Budget (0-10): Evidence of allocated budget or capacity to purchase.
Authority (0-10): Contact's role and influence in decision chain.
Need (0-10): Specificity and urgency of pain points.
Timeline (0-10): Explicit or inferred project timeline.
Output a JSON object with scores, overall confidence (High/Medium/Low),
and a bulleted list of key supporting signals.
input_variables:
- lead_profile
model: gpt-4-turbo
output_type: json_object
```
Initial results over a 50-lead pilot, compared against our SDR team's manual baseline, are promising but highlight specific dependencies:
* **Accuracy:** The agent achieved a 94% alignment with senior SDR qualification decisions on high-confidence scores (where confidence was 'High'). Discrepancies occurred primarily with leads having sparse internal data.
* **Speed:** Qualification time reduced from an average of 12 minutes manual review to under 90 seconds per lead.
* **Critical Finding:** The agent's reliability is directly contingent on the quality and latency of the data retrieval step. Without our enriched internal data, it falls back to generic questioning, undermining its value proposition.
The most substantial pitfall encountered was not in the agent logic itself, but in data pipeline readiness. Relevance's tool calling is effective, but it assumes accessible and well-structured data sources. Teams considering this approach must audit their data accessibility first. The platform's strength lies in its granular control over the reasoning chain and the ability to embed deterministic data retrieval before LLM analysis, which reduces hallucination compared to a pure chat-based qualifier.
Attached screenshots illustrate the agent workflow in the Relevance studio, a sample of the synthesized profile output, and a comparison dashboard of agent vs. manual qualification scores. I welcome critiques of the methodology and am particularly interested in how others are validating the statistical significance of agent-derived scores against eventual conversion rates.
p-value < 0.05 or bust
That initial cost analysis for scaling conversation volumes is a critical filter. Many teams get the pilot working, only to get a seven-figure invoice when they try to roll it out to the full sales team. You mentioned a structured evaluation. Did you model the total cost of ownership at 10x your current qualified lead volume? That's where the pricing models of these orchestration platforms tend to break, especially around token consumption for those real-time enrichments.
Trust but verify — especially the fine print.
Your point about identifying champion signals is a key one that often gets overlooked. Many agents get good at scoring general intent but struggle to separate the interested prospect from the person who can actually drive a purchase decision internally.
How are you prompting the LLM to distinguish a signal of budget ownership versus a signal of technical influence? That's a nuance our own team has grappled with.
Stay curious, stay critical.
The problem is trying to prompt for this distinction in a vacuum. Budget ownership and technical influence are data points, not just linguistic patterns an LLM can reliably sniff out.
You need to enrich the conversation with internal meta-data *before* the LLM call. Is this person's role in the finance department? Have they previously been tagged as an 'influencer' in our CRM? That context has to be injected as system-level facts. Relying on the agent to infer it from "I need to run this by my director" versus "I'll need to check the API specs" is a path to noisy, useless output.
We tried this. Without structured enrichment, the false positive rate on 'champion' signals was over 40%. The agent kept flagging eager engineers as budget holders. The prompt is the last mile, not the foundation.
- Nina