Just had a huge "aha!" moment and wanted to share. We've been running a custom chatbot for sales support for months, but had zero visibility into what prompts our team was actually using or how much each conversation was costing us. We were flying blind on both performance and budget.
I finally wired up Langfuse to log every single interaction. The setup was surprisingly smooth. Now, every time our bot is called, we capture:
* The exact user prompt and the full response
* Token usage and cost per call (broken down by model)
* Trace timelines to see where any slowdowns happen
* User session IDs to track a lead's entire journey
The immediate win? We spotted a few overly verbose—and expensive—system prompts that were adding no value. Trimming those alone should save us hundreds a month. But the real excitement is for our sales managers. Now we can see which prompts consistently lead to qualified lead info being captured, and which ones fall flat. It’s like turning on the lights for our sales enablement AI.
Next step is using this data to refine our prompt chains and maybe even build a simple lead scoring flag based on the quality of the bot's interactions. Anyone else using Langfuse for cost control and pipeline insights? Would love to compare notes on how you're structuring your traces.
— Aiden
Let the machines do the grunt work
Fantastic move. That transition from "flying blind" to having a complete audit trail is transformative, and you've hit on the exact reason why. It shifts the conversation from speculation about AI performance to actual analysis of user behavior and cost drivers.
You mentioned spotting verbose system prompts for immediate savings, which is the low-hanging fruit. The next tier, which you're already moving toward, is using that data for optimization. One thing we've found valuable is to cross-reference the token cost per call with the outcome of that call. A high-cost interaction that doesn't move the needle on lead qualification is a prime candidate for refactoring, perhaps by simplifying the model choice or breaking the prompt into smaller, cheaper steps.
I'm curious about your lead scoring flag idea. Are you planning to derive that score purely from the structure of the captured data (e.g., presence of certain entities in the response) or will you incorporate the trace timelines to score the *process* of how the bot arrived at the answer?
That's a really smart approach. Capturing the *user prompt* alongside the cost is the key part most people miss initially. You can have an expensive system prompt that's justified if it reliably converts tricky customer questions into solid leads.
One thing I'd watch out for as you build lead scoring off this data is the cold start problem. You'll need a decent volume of logged interactions where a human later marked the lead as "qualified" or not to train even a simple scoring flag. Otherwise, you might just be amplifying noise. Have you thought about how you'll gather that initial ground truth data - maybe a simple thumbs up/down from the sales rep after a bot-handled call?
Prod is the only environment that matters.
Absolutely agree on the principle of cost versus outcome analysis. In practice, I've found the most effective first step is to build a simple dimensional model where a "conversation fact" table is linked to separate dimensions for prompt cost, user session, and, crucially, a downstream sales event. This lets you run the exact cross-reference you mentioned.
> score the *process* of how the bot arrived at the answer
This is a critical nuance. We built a process score using the trace timelines, specifically the number of sequential LLM calls and their cumulative latency. We found that conversations requiring multiple expensive reasoning steps rarely correlated with better lead qualification in our case - the initial prompt and response structure were more predictive. The timeline data was better for engineering optimization (finding bottlenecks) than for scoring lead quality.
Garbage in, garbage out.
That's a really interesting split you found between using trace data for engineering versus for scoring quality. I hadn't thought about it that way.
So the timeline is best for spotting bottlenecks to make the bot faster/cheaper, but not necessarily a signal for *what* it's saying being better. That makes sense, now that I think about it. A slow, expensive chain of thought might just mean the initial prompt was poorly defined.
> a downstream sales event. This lets you run the exact cross-reference you mentioned.
This is the part I'm still trying to figure out how to build. How do you reliably link a chatbot session weeks earlier to a final deal in the CRM? Is it just based on the user session ID being attached to a contact record, or do you have something more complex?
That linking problem is exactly what I'm trying to solve, too. From what I've read, just using a session ID often fails if someone comes back later as a new lead in the CRM.
One idea I've seen is to also log and hash a stable identifier, like a user's email from a previous form fill, and link on that instead. But that feels like it needs extra data capture early on.
Nice you caught the verbose system prompt waste. That's the easy part. The harder sell will be when your sales managers want you to "optimize" prompts based on their gut feeling about what makes a "good lead," with zero correlation to actual closed deals.
Langfuse gives you data, not answers. Building a scoring flag off bot interaction quality is a fast track to overengineering a metric that's just guessing. How are you defining "quality" without a closed-loop back to the CRM?
Keep it simple
That's awesome, turning on the lights is the perfect way to put it. I set up something similar last month, but I'm stuck on the dashboard side.
I can see all the cost data in Langfuse, but I'm trying to pull it into Grafana to get alerts when token usage spikes for a certain model. Did you run into any issues setting up the data source or queries for that? My PromQL is still pretty rough 😅