I've been running Langfuse in production for about six months now to trace our RAG pipeline and agent workflows. The core tracing functionality is, in my opinion, excellent. The ability to see a full trace, with timings, token usage, and the exact flow of events through a complex chain, has been invaluable for debugging and performance tuning. The SDKs integrate cleanly, and the data model makes intuitive sense.
However, I've hit consistent friction when trying to adopt their evaluation features for any kind of systematic quality checks. The toolset feels like it's in a proof-of-concept stage, lacking the rigor needed for a continuous evaluation loop. My primary issues:
* **Evaluation "Templates" Lack Flexibility:** The pre-built templates (e.g., for "factuality" or "coherence") are useful for a quick demo, but they're essentially black boxes with hard-coded prompts. There's no clear path to modify the underlying prompt for our specific domain without completely abandoning the framework and writing a custom scorer from scratch. In a real pipeline, evaluation criteria are rarely one-size-fits-all.
* **Weak Integration for Automated Runs:** The API and webhook structure seems designed for manual, one-off evaluations from the UI, not for automated batch evaluation as part of a CI/CD or post-deployment monitoring workflow. We wanted to automatically score a sample of production traces daily against our SLAs, but orchestrating that through the Langfuse API was more cumbersome than just building our own lightweight service that writes results back as scores.
* **Missing Statistical Aggregation:** The dashboard shows scores for individual traces, but I found the aggregation and trend analysis to be very basic. For release engineering, we need to track metrics like "95th percentile of correctness score for version 2.1 over the last week" and see if a new model deployment caused a regression. This requires computing distributions over scores, which currently means exporting all data and analyzing it elsewhere.
For example, here's a simplified version of the custom scorer we ended up writing because the template system couldn't accommodate our need to check citations against a specific knowledge base version:
```python
# Our custom evaluator, run outside of Langfuse's eval system
def evaluate_citation_fidelity(trace_output, knowledge_base_version):
"""
Checks if cited document IDs are valid for the given KB version.
Returns a score and a failure reason.
"""
# ... our logic ...
valid_citations = [...]
score = len(valid_citations) / total_citations if total_citations > 0 else 1.0
# We then log this manually to Langfuse as a score
langfuse_client.score(
trace_id=trace_id,
name="citation_fidelity",
value=score,
comment=f"KB Version: {knowledge_base_version}"
)
return score
```
This works, but it defeats the purpose of an integrated eval framework. I'm left wondering if the team's roadmap prioritizes expanding the tracing feature set over hardening the evaluation offering.
Has anyone else pushed Langfuse evaluations beyond simple demos? Did you find a way to make the templates truly adaptable, or did you also revert to custom code? I'm particularly interested in patterns for running batch evals on a schedule.
-jf