Looking to instrument our internal apps that call various LLMs (mostly OpenAI, some Anthropic, and a self-hosted Mistral). Team of five backend engineers, mandate is open source only—no commercial SaaS. Need to trace calls, see latency breakdowns per provider/model, and ideally track costs.
Primary concern is the overhead of the tracing system itself. We can't afford to add 500ms of latency per LLM call. Also, we need something that can handle multi-step, nested LangChain/LlamaIndex workflows without drowning us in noise.
So far, I've been benchmarking a few options locally:
- **OpenLLMetry** (OpenTelemetry-based): Seems promising, but the automatic instrumentation for some libraries felt a bit heavy. Need to test with a realistic load.
- **Langfuse**: Open source core, but I'm unclear on the self-hosted resource requirements. Their tracing looks detailed.
- **Phoenix** (by Arize): Purely OSS. More focused on eval and tracing. The tracing spans looked clean in my quick test.
Has anyone run a similar stack and measured the actual observability tax? I'm particularly interested in:
- Latency added per LLM span (p99 values would be golden)
- Ease of integrating with existing Grafana/Prometheus for dashboards
- Whether the tool can break down latency by the network hop to the LLM provider vs. time in first token vs. time in our own post-processing chain
A dream setup would output trace data to Tempo or Jaeger, and let us build custom latency histograms per model. Any open source tools that get close to that?
ms matters