Alright folks, buckle up, because I just spent the last weekend deep in a rabbit hole and I've emerged with something I'm genuinely excited to share. We all know Langfuse is fantastic for diving deep into LLM traces, scores, and user sessions directly in their UI. But what if you want that data living right next to all your other operational metrics? What if your team lives in Grafana for everything from server health to business KPIs?
I decided to bridge that gap. I set up Langfuse's webhooks to pipe trace and score data directly into a time-series database (I used Postgres with the TimescaleDB extension, but any will do), and then built a custom Grafana dashboard on top of it. The goal? A single pane of glass for my product's "AI health."
Here’s a peek at what the dashboard tracks now:
* **Real-time Latency & Cost Tracking:** I get a live view of average trace duration and cost per trace, broken down by `trace_name` (which I use for major features like "chat_completion" or "document_summarizer"). Seeing a sudden latency spike in Grafana at the same time as a cloud provider alert is *chef's kiss* for correlation.
* **Feature Adoption via Trace Volumes:** I've created graphs that plot the daily count of traces for each of our core LLM features. This is product-led growth gold—you can literally see which new AI feature is catching on immediately after a release.
* **Score Trends Over Time:** This is the juicy part. I'm visualizing the average for each of my custom scores (like `correctness` or `helpfulness`) week-over-week. I can filter by a user cohort (stored as a trace tag) to see if our recent model swap improved scores for our "power_user" cohort versus new users. It turns abstract "quality" into a moving line chart.
* **Error & Exception Rate:** A simple but vital panel that counts traces where `level` is set to "ERROR." It's now part of our general system health check.
The setup wasn't trivial, but it was so worth it. The key was configuring the Langfuse webhook to send to a small ingestion service (I used a simple Python FastAPI app) that parses the JSON, flattens the key bits (trace ID, timestamp, scores, total cost, etc.), and writes it to the time-series table. The most fiddly part was deciding on a schema that could handle the dynamic nature of scores.
I'm now experimenting with setting up Grafana alerts on score degradation for specific features. Imagine getting a PagerDuty alert because your `retrieval_accuracy` score for the Q&A bot dropped below a threshold for 30 minutes straight. That's proactive observability!
Has anyone else tried something similar? I'm curious about alternative data pipeline setups or what other metrics you'd consider essential to include. I'm thinking of adding a panel that correlates trace cost with user segment (free vs. enterprise) to understand ROI better.
🔥
Try everything, keep what works.
Nice, the idea of a single pane for AI health is solid. I'm curious about the ingestion though - when you're piping webhook data directly into a timescale table, what's your schema look like for the trace data? Are you flattening the JSON payload or storing it as a column to parse later?
Also, have you run into any issues with webhook delivery during high load? I've seen similar setups where the queue can back up if the database write isn't fast enough, causing gaps in that real-time view.
grep is my friend.
Flattened it. Storing the full JSON blob felt lazy and would make queries slow. I extracted the key metrics we actually graph - latency, token counts, scores - into separate columns. The raw payload goes to cold storage if we ever need it.
You're right about the queue. I hit that early on. The webhook endpoint just dumps into a queue, and a separate worker processes inserts. If the DB gets slow, the queue builds up but the endpoint stays responsive. It's not real-time, but it's close enough.