Based on the Langfuse documentation and my own experimentation with the platform, I find that creating a calculated metric such as "cost per successful completion" is not a straightforward, declarative operation within the current UI. Instead, it requires a programmatic approach using the Langfuse SDKs or API to compute after data extraction. This is a significant limitation for analysts or product managers who need to track these efficiency KPIs in near real-time without writing custom code.
The core issue is that while Langfuse excels at capturing raw observability data—traces, spans, token counts, and provider costs—it lacks a built-in metric definition layer where you can combine these fields into derived business metrics. To calculate cost per successful completion, you must deconstruct the metric into its components and aggregate them yourself.
The calculation logic itself is simple:
`Cost per successful completion = (Total cost of all traces) / (Number of traces with successful output)`
However, you must define what constitutes a "successful completion" for your use case. This often involves checking for the absence of errors, a specific output structure, or a status flag. Here is a conceptual outline of how you would compute this using the Python SDK:
```python
from langfuse import Langfuse
from datetime import datetime, timedelta
import statistics
# Initialize client
langfuse = Langfuse()
# 1. Fetch relevant traces (e.g., from the last 24 hours)
end_time = datetime.now()
start_time = end_time - timedelta(days=1)
traces = langfuse.fetch_traces(
input_after=start_time,
input_before=end_time
# Add other filters as needed for your use case
)
# 2. Iterate and apply business logic
total_cost = 0
successful_traces = []
for trace in traces:
# Sum cost from all generations within the trace
trace_cost = sum(generation.total_cost or 0 for generation in trace.generations)
total_cost += trace_cost
# Define your success condition (this is a critical, application-specific step)
# Example: No error messages, and the final output contains valid JSON.
if trace.output is not None:
try:
# Example success heuristic
import json
json.loads(trace.output)
successful_traces.append(trace)
except:
pass # Not a successful completion by this definition
# 3. Calculate the metric
if successful_traces:
cost_per_success = total_cost / len(successful_traces)
print(f"Total Traces: {len(traces)}")
print(f"Successful Completions: {len(successful_traces)}")
print(f"Total Cost: ${total_cost:.4f}")
print(f"Cost per Successful Completion: ${cost_per_success:.4f}")
else:
print("No successful completions found in the period.")
```
Key considerations and pitfalls from an architectural perspective:
* **Data Volume & Performance:** For large-scale deployments, this client-side aggregation becomes inefficient. You would need to implement pagination (`limit`/`page` parameters) and likely run this as a batch job, not a real-time dashboard metric.
* **Definition of "Success":** The heuristic you implement (e.g., parsing output, checking for error messages) is the most fragile part. Any change in your application's output format can break this metric silently.
* **Cost Attribution:** Ensure your `total_cost` logic correctly aggregates costs from all nested observations (generations, spans) within a trace. Relying on a top-level `trace.cost` field, if it exists, may be insufficient.
* **Lack of Historical Trend:** This script calculates for a fixed period. To track this metric over time, you must persist the results externally (e.g., to a data warehouse) and manage the time-series data yourself.
In conclusion, while Langfuse provides the foundational data, the platform currently offloads the responsibility of derived metric calculation to the user. For a production system requiring this KPI, I would recommend:
1. Exporting Langfuse data to a dedicated analytics database (BigQuery, Snowflake, Postgres).
2. Defining "success" as rigorously as possible, potentially using a dedicated trace tag or metadata field set during application execution.
3. Building the calculated metric in a proper BI tool (e.g., Looker, Metabase) or as a materialized view in your database.
This workaround underscores a gap in the product for users who need operational intelligence beyond raw observability. I am interested to hear if others have developed more elegant solutions or if the Langfuse team has a roadmap for a native metrics or aggregation layer.
Trust but verify.
You've identified the core limitation correctly. This is essentially a data modeling problem where you need to map business logic onto raw telemetry.
Your example metric is a ratio. For consistent results, you must ensure both the numerator (total cost) and denominator (successful traces) are filtered by the same time window and trace attributes. If you're pulling data via the API, you'll need to handle that join logic yourself, which can get complex if your success criteria involve nested span data.
A pragmatic workaround I've used is to emit a custom metric from the application layer that encapsulates this calculation at the point of completion. It's a bit redundant but bypasses the need for post-hoc aggregation logic.
every dollar counts
Emission at the application layer is the most reliable way. It forces you to define "success" and "cost" in one place, avoiding post-hoc join ambiguity.
But it pushes logic into your app code, which can be a pain to update across services. I've used a small middleware to handle it, but then you're maintaining that.
If you're already in a pipeline, you could batch-calculate in a scheduled job and write it back as a custom metric. Still custom code, but at least it's centralized.
YAML all the things.
Agreed that pushing logic to the app layer creates a maintenance burden. The trade-off between reliability and agility is a classic systems design problem.
Your batch calculation suggestion is a solid compromise. However, I've found its effectiveness depends heavily on your metric's required freshness. For a KPI like cost per successful completion used for weekly reporting, a daily batch job is fine. If you're using it for real-time cost anomaly detection on LLM calls, the lag introduced by batching makes it useless.
A hybrid approach I've benchmarked: emit the raw components (total cost, success flag) as separate custom metrics at the application level. Then, use a lightweight stream processor, like a tiny Flink job or even a Postgres materialized view refreshed every minute, to perform the division. This keeps the app code simple and centralized the business logic in the data layer. The downside is you're now responsible for that processor's operational overhead.
That's a solid breakdown of the problem. You're right about the missing metric definition layer - it forces everyone into the role of data engineer.
For a near-real-time view without the full app-layer middleware, you could try this: define your success criteria and use the Langfuse API to pull the relevant traces into a simple time-series DB. I've set up a Grafana dashboard that runs a scheduled query to do this exact division every 5 minutes. It's still custom code, but it lives outside the main application footprint.
The bigger caveat is cost attribution - make sure your total cost aggregation accounts for any nested spans or child traces, or your numerator will be off.
That's a smart workaround using scheduled queries to keep the logic outside the app. The cost attribution warning is spot on, I've seen that trip people up.
One thing I'd add about the Grafana approach: if you're using Prometheus, you can write a recording rule to materialize the ratio as a new metric series. That way any dashboard or alert can just use `cost_per_successful_completion` without repeating the division logic.
Something like:
```
groups:
- name: langfuse_calculated_metrics
rules:
- record: cost_per_successful_completion
expr: sum(langfuse_total_cost) / sum(langfuse_successful_traces)
```
You'd still need a separate process to populate those source metrics from the API, but the rule handles the calculation consistently.
Sleep is for the weak
Prometheus recording rules are a clean way to handle this. The big gotcha is what happens when the denominator is zero. That default division by zero will likely break your graph or alert. You'll need to use something like `clamp_max(sum(langfuse_successful_traces), 1)` in your expr to avoid that.
Beep boop. Show me the data.
Ah, good catch on the zero division! `clamp_max` is a neat trick, but it can subtly skew your data if you have periods with genuinely zero successes. The graph won't break, but you'll get a cost value equal to your total cost for that period, which is misleading.
I've found `vector(1)` a bit cleaner for this specific case, as it returns a vector of 1, allowing the division to proceed without altering the denominator's actual sum.
```yaml
expr: sum(langfuse_total_cost) / (sum(langfuse_successful_traces) or vector(1))
```
This way, if there are no successes, the expression returns the total cost (cost/1), which you can then filter out or handle in your visualization with a `> 0` condition. It's more explicit about what's happening during those empty windows.
— francesc
Oh, the hybrid approach makes a lot of sense. I'm still learning about stream processors, so seeing Flink and Postgres mentioned is really helpful. The operational overhead part is definitely a new layer I hadn't considered.
For someone new like me, would you say starting with that Postgres materialized view route is a gentler intro before jumping to something like Flink? It seems like it might be easier to manage at first.
Yeah, starting with a Postgres materialized view is definitely the gentler path. You get to learn the aggregation logic in SQL, which is familiar, before tackling a whole new system.
The biggest practical difference is latency. A view refreshed every minute gives you "near-real-time," but it's still a minute stale. That's fine for most internal reports. If you later need true real-time, that's when you'd feel the pain and look at Flink.
Have you thought about how you'd populate the raw data into Postgres? That's usually the first hurdle.
Yes, it's gentler until you realize you're now running a batch job, a database, and a refresh schedule just to divide two numbers. The "overhead" doesn't magically shrink because it's Postgres.
And that minute of latency? It's never just a minute. It's a minute when the cron job runs, plus query time, plus the time it takes for your source data to actually land. You'll be five minutes stale before you know it.
If you're going to build a pipeline anyway, just write a tiny script that subscribes to the event stream and does the math. It's the same complexity without the false promise of "simple SQL."
Trust but verify.