Showcase: using the API to build a custom leaderboard for prompt templates

Hey everyone, I've been diving into Langfuse for a few weeks to track some LLM experiments, and I wanted to share a small project I just finished. I was manually comparing different prompt templates in the UI, but it got messy with more than a few versions. I thought: why not build a custom leaderboard outside of Langfuse to see which template performs best across key metrics?

I used the Langfuse API to fetch the trace data. My main goal was to rank templates by average latency and total cost, but also factor in a custom score I calculate (like correctness from human feedback). Here's the core Python snippet I used to pull and structure the data:

```python
import requests
import pandas as pd

API_KEY = "your-key-here"
BASE_URL = "https://cloud.langfuse.com/api"

headers = {"Authorization": f"Bearer {API_KEY}"}

def get_traces(project_id, limit=500):
url = f"{BASE_URL}/traces"
params = {"projectId": project_id, "limit": limit}
response = requests.get(url, headers=headers, params=params)
return response.json()['data']

traces = get_traces("your-project-id")
df = pd.DataFrame(traces)

# Filter for traces with a specific tag, e.g., 'prompt_eval'
df = df[df['tags'].apply(lambda x: 'prompt_eval' in x if x else False)]

# Extract the prompt template name from metadata
df['template_name'] = df['metadata'].apply(lambda x: x.get('template_version', 'unknown'))

# Group by template and calculate metrics
leaderboard = df.groupby('template_name').agg(
avg_latency=('duration', 'mean'),
total_cost=('total_cost', 'sum'),
count=('id', 'count')
).reset_index()

print(leaderboard.sort_values('avg_latency'))
```

This gave me a nice table. I then pushed it into a simple Streamlit app to make it interactive, adding filters for date ranges and model names. The cool part was being able to combine Langfuse's built-in metrics with my own derived score from the trace's output or tags.

Has anyone else built something similar? I'm curious about a couple of things:
- Are there better ways to fetch larger datasets? I hit some limits with the default pagination.
- How do you handle calculating composite scores when the data comes from different traces? I'm currently doing a separate post-processing step, but maybe there's a smarter way.

This was a really practical way for me to learn the API, and it's super useful for our team's weekly reviews. The documentation was good, but I had to piece together the filtering parts.