Ran a 4-week comparison. Switched our production tracing from Langfuse to Arize AI.
**Setup:**
* Same FastAPI app with 3 LLM calls per chain.
* Instrumented both SDKs.
* Primary metrics: trace latency overhead, dashboard load time, filter speed for 10k+ traces.
**Raw Findings:**
**Langfuse (v2.6.1):**
* SDK latency overhead: ~120-150ms per trace.
* UI is fast for simple views.
* Complex filtering (e.g., `input_tokens > 500 AND tags contains 'retry'`) slowed significantly at scale.
* Pricing model got complex with high-volume users.
**Arize AI (v8.1.0):**
* SDK latency overhead: ~80-110ms per trace.
* UI dashboard loads ~1.5s faster on average.
* Filtering performance consistent at scale.
* Pricing is per-observation, simpler for our volume.
**Code snippet for trace comparison:**
```python
# Langfuse trace creation
langfuse.trace(
name="llm_chain",
input=user_input,
metadata={"model": "gpt-4"}
)
# Arize trace creation
arize.trace(
name="llm_chain",
prediction_id=str(uuid.uuid4()),
input=user_input,
model_version="gpt-4"
)
```
**Conclusion for my stack:** Arize wins on performance at scale. Langfuse UI is cleaner for small teams.
Has anyone else benchmarked both? Looking for data on cost at >1M traces/month.
- bench_beast
Benchmarks don't lie.