Hey folks! 👋 I've been deep in the LangSmith trial for a few weeks now, and it's fantastic for tracing and debugging our LLM calls. The visualization and dataset management are top-notch. However, our security team is pushing back hard on any third-party SaaS that would have visibility into our internal prompts and data flows. The obvious answer seems to be "self-host the open-source alternatives," but that's a non-starter for our small platform team—we don't have the bandwidth to manage another complex infrastructure stack.
So, I'm hunting for something in the middle: a managed, closed-source (or source-available) service that gives us similar observability for our LangChain/LlamaIndex pipelines, but where the vendor handles the hosting, scaling, and upkeep. Essentially, we need a "LangSmith Cloud" alternative that isn't self-hosted. Our core needs are:
* **Detailed tracing** of LLM calls, tools, and agents.
* **Prompt management** and versioning.
* **Dataset creation** for evaluation.
* **API-based integration**, not just a UI.
I've done some initial digging and found a few names, but I'd love real-world feedback. Has anyone moved from LangSmith to a different *managed* service? How did the switch impact your workflow?
A couple I'm looking at (but haven't tested yet):
* **Arize Phoenix** - I know it's OSS, but they have a managed offering now. Anyone tried it?
* **Portkey** - Seems more gateway-focused, but promises observability.
* **Weights & Biases** - Their Prompts product looks comparable.
Bonus points if the service has a generous free tier for evaluation! Our current pipeline is built with GitHub Actions, and I'd love to plug evaluations into our CI. Something like:
```yaml
- name: Run LLM Evaluation Suite
run: |
python eval_pipeline.py --dataset staging --target prod
```
Would appreciate any experiences, especially around pricing pitfalls or integration ease. Let's compare notes!
-pipelinepilot
Pipeline Pilot
I've been researching this exact gap for our internal evaluation pipeline, and the managed service options do feel limited compared to the self-hosted open-source landscape. One name that came up in my search was Aporia, which is a fully managed ML observability platform. They have specific features for LLM monitoring, including trace visualization and prompt management, and their sales engineering team stressed the data residency and encryption options that might help with your security team's concerns.
However, from the demos I've seen, their dataset creation and versioning capabilities for LLM evaluation aren't as mature as LangSmith's. It felt more geared towards monitoring production drift than iterative prompt development. Have you looked at any of the newer players like Helicone? I recall they offer a managed cloud version in addition to their open-source option, but I'm not sure how their feature set compares on the prompt management side.