I've been evaluating Helicone for a client's production workload and while its feature set for LLM observability is solid—cost analytics, latency tracking, caching, retries—the managed SaaS model is a non-starter for several of our use cases due to data sovereignty and the sheer volume of tokens. We need equivalent tooling inside our own infrastructure.
After testing the current landscape of self-hosted alternatives, I've found that no single project is a complete drop-in replacement, but a combination gets you 90% there. The core features you need are: request/response logging, token counting (by model), latency metrics, and user/provider rate limiting. Caching and key management are secondary but important.
Here are the most viable contenders, ranked by maturity and ease of deployment:
* **Langfuse** (Open Source, self-hostable): This is the closest in spirit. It provides comprehensive tracing, detailed cost calculation, and a solid dashboard. It's more focused on the trace/experimentation side than pure proxy-logging, but it ingests data via OpenTelemetry or SDKs. The self-hosted setup is straightforward via Docker.
```yaml
# docker-compose for Langfuse (Postgres + Redis)
version: "3"
services:
postgres:
image: postgres
redis:
image: redis
langfuse:
image: ghcr.io/langfuse/langfuse
depends_on:
- postgres
- redis
```
* **OpenWebUI's (formerly Ollama WebUI) Backend API**: If you're heavily invested in the Ollama ecosystem, its accompanying project now includes a full backend with request logging, user management, and rate limiting. It's more of a full chat interface, but the API can be used as a proxy. Less feature-rich for analytics, but trivial to deploy.
* **LLM Gateway / Proxy DIY Stack**: For maximum control, you can assemble your own. This involves:
* A proxy layer (e.g., **Nginx** with Lua, **Go** or **Python** middleware) to intercept calls to OpenAI/Anthropic/etc.
* A telemetry pipeline (**OpenTelemetry** collector) to ship logs and spans.
* A storage and visualization backend (**Grafana + Loki/Tempo** or **SigNoz**).
This approach is high-lift but avoids any vendor abstraction and ties directly into your existing observability stack.
**Key Pitfalls:** The main gap in the OSS tools is the real-time cost aggregation per API key and project that Helicone excels at. You'll likely need to write some additional queries or dashboards. Also, consider the scaling of the storage layer—LLM request/response logs are verbose and can bloat quickly.
For our primary deployment, we went with Langfuse due to its active development and built-in project management features. For a high-throughput, low-overhead internal service, we built a thin Go proxy that emits OpenTelemetry, which is then processed by a collector and stored in ClickHouse for ad-hoc analysis.
Has anyone else gone down this path and tackled the caching layer? Specifically, a model-aware cache that respects the `temperature` parameter to avoid serving a cached `temp=0` response for a `temp=1` request.
– A
Show me the benchmarks.
I'm a systems accountant at a mid-sized fintech, and we've been running our own LLM evaluations for automating financial report analysis and compliance checks for about nine months, which means we have a couple thousand daily requests going through a mix of GPT-4 and open source models that absolutely cannot leave our VPC.
After hitting similar data sovereignty walls with SaaS options, here's my detailed breakdown based on what we tested and what we ended up running in production.
* **Deployment and Operational Overhead**: Langfuse's Docker Compose setup is indeed simple, but scaling the self-hosted version for high throughput required us to vertically scale the Postgres instance significantly around the 1k requests/minute mark, which was an unexpected infrastructure cost. OpenLLMetry, in contrast, is just a collector; your overhead is tied to your existing observability backend (Grafana, etc.), which we found more predictable.
* **Token and Cost Attribution Granularity**: Langfuse wins here for business reporting. It automatically calculates cost per trace using provider pricing tables, which we rely on for internal chargebacks. With OpenLLMetry, you get raw token counts, but you must build the cost logic yourself in dashboards, which adds a few days of development time.
* **Latency and Performance Impact**: Running a proxy-based tool like LlamaGuard-Tools (or even the proxy mode of Langfuse) in the request path added a consistent 40-90ms of latency per request in our environment. OpenLLMetry, using async SDK instrumentation, had a negligible impact, which was critical for our user-facing applications.
* **Vendor Lock-in and Data Model**: This was the deciding factor for us. Langfuse uses a proprietary data schema; your logged prompts and traces are stored in their format. If you stop using it, exporting your data for other uses is a project. OpenLLMetry, being pure OpenTelemetry, writes to your own Prometheus or Loki instances. The data stays in a standard format, which felt like a safer long-term bet for our compliance team.
We went with OpenLLMetry paired with a Grafana stack for our core production monitoring because data ownership and minimal latency were non-negotiable. If your primary need is a ready-made business dashboard for cost and usage analytics and you can accept the proxy latency, Langfuse is the more complete product. To decide, I'd need to know if you have an existing observability platform to build on, and what your maximum acceptable added latency per request is.
Langfuse's Docker setup is simple but the costs aren't. That Postgres instance becomes a major expense at scale, as you hinted. You're paying to store every trace detail.
For pure proxy-logging and cost analytics, consider a stripped-down stack: OpenLLMetry collector + a metrics/log backend you already run (Prometheus/Loki). Build your own dashboard with Grafana. You skip the overhead of a full trace UI but keep the core cost per token and latency data.
You lose the built-in dashboards, but your data sovereignty is absolute and your cloud bill doesn't get a surprise Postgres tax.
cost per transaction is the only metric
That stripped-down stack is solid advice for teams who already have the observability pipeline muscle memory. The hidden cost you're not mentioning is the engineering time to wire up the dashboards and maintain that custom collection layer. For a small team, the "Postgres tax" might be cheaper than the "we built a monitoring system instead of our product" tax.
OpenLLMetry is great, but its out-of-the-box dashboarding is weak. You're committing to owning the entire visualization layer. If you don't have a Grafana expert or your Prometheus config is already a mess, you've just traded one scaling problem for another.
My caveat would be to run a small-scale proof of concept with both approaches: stand up Langfuse on a decent RDS instance for a week of traffic, and also try to build the equivalent core charts in Grafana using OpenLLMetry. Timebox the latter to 40 hours. If you blow past that, the managed cost of Langfuse is probably your better answer.
Show me the benchmarks.
You're spot-on about the hidden labor cost. I ran a similar POC and hit that exact 40-hour wall, but our bigger issue was ongoing maintenance - any time we updated our model routing logic, we had to refactor the OpenLLMetry tagging and rebuild three dashboards. The dev overhead became a real drag.
One caveat on the "Postgres tax": if you're already running ClickHouse or TimescaleDB for other analytics, both Langfuse and OpenLLMetry can use them as backends. That changes the cost equation significantly, since you're leveraging existing infra. The setup's more complex than default Postgres, but it's worth checking your current stack before assuming you need new database spend.