We’ve been running a production RAG pipeline on Kubernetes for about eight months, and we’ve been using Freeplay for prompt management, versioning, and some basic observability. It’s served its purpose, but as we scale the number of models and experiments, its limitations are becoming apparent—specifically around custom metrics, granular cost tracking per model route, and the need for a more programmatic, GitOps-friendly approach. LangSmith is the obvious next step everyone shouts about, but I have philosophical and practical issues with locking into another closed, vendor-specific ecosystem.
I’m looking for alternatives that are designed to run within our own infrastructure (or are at least highly Kubernetes-native), prioritizing the following:
* **Open Source or Self-Hostable:** Must be something we can run inside our k8s cluster. Agent-based telemetry collectors are fine, but the control plane needs to be under our control.
* **Prometheus/OpenTelemetry Native:** Our entire stack is monitored via Prometheus metrics and Grafana dashboards. Any solution should emit standard OTel metrics or custom metrics we can scrape directly, not just live inside a proprietary UI.
* **Trace-Centric Debugging:** Need to see the full chain of retrievals, LLM calls, and tool usage as a single trace, similar to distributed tracing in microservices.
* **Programmatic Configuration:** Prompt templates, evaluation criteria, and test datasets should be definable as code (YAML, Python SDK) and deployable via our existing CI/CD.
* **Cost Tracking Per Model/Route:** We route to a mix of OpenAI, Anthropic, and self-hosted open-source models (via vLLM/TGI). We need to attribute token usage and cost to specific pipeline branches and feature flags.
What we don’t need: a full MLOps platform for training, or a low-code GUI for non-technical users.
I’ve done a preliminary scan and have a few candidates, but I’m keen to hear from teams who have actually deployed them in a similar environment:
* **Phoenix by Arize AI** (Apache 2.0): Appears to be trace collection and evals, with a Jupyter-centric UI. Can it be run as a service with a persistent store in k8s? How robust is its programmatic test suite management?
* **Langfuse** (Self-hostable MIT license): Often mentioned. Its architecture looks k8s-friendly, but how does its tracing and metric emission compare to a more OTel-native approach? Is the prompt management truly GitOps compatible?
* **Helicone** (Self-hostable AGPL): Primarily a proxy with logging/analytics. Does its k8s operator support advanced trace linking and custom evaluations, or is it mostly a cost and latency dashboard?
* **Custom OTel Instrumentation + DAG Studio:** The nuclear option: instrumenting everything with OpenTelemetry, storing traces in Tempo/Jaeger, and building a thin UI for trace comparison. Has anyone gone this route and regretted it?
My biggest concern is investing in a platform that becomes a bottleneck itself. I’d rather have a set of composable tools than another monolithic "AI DevOps" suite.
If you’re running something in this space, I’d appreciate specifics on:
- Your deployment method (Helm, raw manifests, operator)
- How you handle prompt versioning and A/B testing rollouts
- How you integrate evaluation scores (e.g., answer relevance) back into your monitoring alerts
- Any non-obvious resource overhead (e.g., the vector DB for trace storage becoming a scale issue)
Here’s a snippet of our current clunky integration for context, where we manually emit metrics after a chain execution:
```python
# What we do now - want to replace this manual wiring
from prometheus_client import Counter, Histogram
PROMPT_TOKENS = Counter('llm_prompt_tokens_total', 'Total prompt tokens', ['model', 'route'])
EVAL_SCORE = Histogram('rag_answer_relevance', 'Relevance score 0-1', ['experiment_id'])
def run_chain(question, model_name, experiment_id):
# ... chain execution ...
# Manual instrumentation
PROMPT_TOKENS.labels(model=model_name, route='main').inc(prompt_tokens)
EVAL_SCORE.labels(experiment_id=experiment_id).observe(relevance_score)
```
Show me the benchmarks