Everyone's obsessed with tracking "cost per token" these days, but I'm more interested in the cost per *customer*. Specifically, how many of those expensive LLM-generated Marketing Qualified Leads ever turn into something real. I built an OpenPipe dashboard to track exactly that, and the initial numbers are... sobering.
Here's the core of my setup. I pipe our chatbot conversations (where MQLs are captured) to OpenPipe, and then join that data with our internal Postgres `sales` table on a common identifier.
```sql
-- OpenPipe log query, simplified
SELECT
o.response->'parsed'->>'lead_id' as lead_id,
COUNT(o.id) as prompt_calls,
SUM(o.cost) as acquisition_cost
FROM openpipe_logs o
WHERE o.response->'parsed'->>'mql' = 'true'
GROUP BY lead_id;
```
I then join this against the SQL data in a dashboard widget. The key metric:
`(SUM(sales.amount) / SUM(openpipe_logs.cost))` for converted leads only.
The early results challenge a few sacred cows:
* **"More MQLs are always better."** Not if your cost per MQL is high and conversion is abysmal. We're seeing a 22% conversion from MQL to SQL, which sounds okay until you see the aggregate LLM cost involved.
* **"Optimizing for cheaper tokens is the priority."** The real leverage isn't in shaving millicents off a 1k completion. It's in tightening the prompt to disqualify bad leads *earlier*, even if it adds a few tokens.
* **"Our pipeline velocity is healthy."** Velocity matters, but not if the feedstock is gold-plated. I'm now tracking the time from LLM interaction to SQL, not just from MQL to SQL.
The dashboard makes it painfully clear: we're spending too much to generate leads that our sales team immediately discards. The next step is A/B testing different prompt strategies and measuring the *downstream* conversion impact, not just the immediate cost. Anyone else moved beyond tracking mere token economics to measuring actual business outcomes? I'd be curious to see your approaches, especially if they contradict the common wisdom.
profile before you optimize