We're scaling up. Need to give ~100 engineers and data scientists visibility into their LLM apps, from prototypes to production. Main requirements:
* Track cost, latency, token usage per user/feature.
* Trace complex chains (LangChain, custom agents) without a rewrite.
* Alert on drift in response quality or sudden error spikes.
Tried a few. The big split is between open-source/in-house and managed platforms.
**Managed (easier, but watch the lock-in):**
* **Arize Phoenix** - Good for evals and tracing on notebooks. Their hosted dashboard is solid for midsize teams. Cost attribution is clear.
* **Langfuse** - Open-source but also cloud. Strong on tracing and sessions. Their prompt management is getting better.
* **Helicone** - Proxy-based. Simple to bolt on if you're using OpenAI SDK. Good for basic metrics and caching.
**DIY/OSS (more control, more work):**
* OpenTelemetry with LlamaIndex/LangChain auto-instrumentation. You'll need to pipe traces to something like Tempo/Grafana or SigNoz.
* Build your own wrapper and log to ClickHouse. Gets messy fast at 100 users.
Biggest gotcha: tracing is useless if it kills your app's performance. Sample aggressively in prod.
What's everyone else using at this scale? Specifically:
- How are you handling multi-tenant data isolation?
- Any platform that does well with non-OpenAI models (Claude, open-source)?
// chris
metrics not myths
That point about sampling aggressively is crucial, and something I've seen bite teams in my own area. A team I work with built a custom wrapper and logged everything to start, thinking they'd mine the data later. The volume from just a few dozen active devs building prototypes absolutely choked their logging pipeline and added noticeable latency to the actual apps. They ended up having to retroactively implement sampling by user session and by error type, which was messy.
Your managed list seems focused on the newer, dedicated LLM observability tools. For a group of 100, I'm curious if anyone has experience with more general-purpose APM platforms, like DataDog or New Relic, that have added LLM-specific features. They might already be in your stack for other services, and the integration story could be simpler for a large, mixed team. But I've heard their tracing for complex LangChain agents can be superficial compared to something like Langfuse.