I've been using OpenAI's evals framework for a while to benchmark my LLM outputs, but I always felt like I was missing the "so what?" after running them. The scores were isolated snapshots. I wanted to see trends, correlate them with other data (like latency or cost), and track them per user or session in my app.
Enter Langfuse. I realized I could pipe my eval results into it to get that long-term, analytical view. Here's a quick walkthrough of my setup for tracking custom eval metrics.
First, I define my custom evaluator. This one checks if a response contains a specific keyword (you can imagine more complex logic).
```python
from langfuse.openai import OpenAI
import openai
# Initialize the Langfuse-wrapped client
client = OpenAI()
def keyword_evaluator(response, expected_keyword="integration"):
score = 1 if expected_keyword.lower() in response.lower() else 0
return {
"score": score,
"name": "keyword_match",
"expected_keyword": expected_keyword
}
```
Then, in my evaluation loop, I log both the trace (the generation) and the score as a feedback event.
```python
# Inside my eval batch loop
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
)
response_text = completion.choices[0].message.content
# Run my custom evaluator
eval_result = keyword_evaluator(response_text)
# Log score as feedback, linked to the trace
client.feedback(
trace_id=completion.langfuse.trace_id, # Automatically linked
name=eval_result["name"],
score=eval_result["score"],
comment=f"Expected keyword: {eval_result['expected_keyword']}"
)
```
The beauty is that now, in the Langfuse dashboard, I can:
* See the distribution of my `keyword_match` scores over time.
* Filter traces by low scores to debug bad responses.
* Correlate scores with other attributes like model parameters or prompt version.
Has anyone else tried something similar? I'm curious about:
- Other custom metrics you're tracking (factuality, tone, etc.)
- If you've integrated more complex, model-graded evals from the OpenAI evals repo directly.
- How you're using the Langfuse Python SDK versus the OpenAI SDK wrapper for this.
The combo feels powerful for moving from one-off evaluations to continuous monitoring.