Hey everyone! 👋 I've been using PromptLayer for a while now to track my LLM prompts and costs, and it's been super helpful. But I'm currently scoping out options for a new project at work, and I'm wondering what else is out there.
I know LangSmith and Weights & Biases (W&B) are the big names everyone mentions, but I'm specifically looking for alternatives *outside* of those two. Maybe something a bit more lightweight, or with a different pricing model, or even open-source?
Here’s what I primarily use PromptLayer for:
- Logging all OpenAI and Anthropic API calls automatically.
- Tracking token usage and costs per project/feature.
- Storing and versioning successful prompt templates.
I’d love to hear if anyone has tried tools like:
- **Arize AI** - I've heard their Phoenix tool is good for evals.
- **DAGWorks** - Their Hamilton LLM Observability looks interesting.
- **Portkey** - Seems more focused on gateway/load balancing, but has observability too.
- **Open-source stacks** built on tools like `LiteLLM`, `Langfuse`, or even rolling your own with `OpenTelemetry`.
If you've migrated away from PromptLayer, what was your experience? I'm particularly curious about:
* **Ease of setup** – Was it a simple SDK swap, or a big migration?
* **Pricing** – Did you find a better deal for high volume?
* **Missing features** – Anything you regretted losing?
Bonus points if you have a small code snippet showing how you integrated an alternative! For example, here's how I set up basic logging with PromptLayer:
```python
import promptlayer
import openai
promptlayer.api_key = "pl_..."
openai.api_key = "sk-..."
openai = promptlayer.openai
completion = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello world"}],
pl_tags=["my-feature"]
)
```
What are you all using, and would you recommend it?
Clean code, happy life
Great question! I actually moved a project off PromptLayer a few months ago for similar reasons.
We landed on Langfuse and have been really happy. It's open-source, so you can self-host if you want (we did), and it nails the three core things you listed. The prompt versioning is solid, and the cost tracking is broken down really cleanly by tags. It felt like a more flexible PromptLayer.
I haven't used Phoenix for evals, but I've heard the same. If your main goal is *just* evals, it might be overkill. But if you want a full log/version/cost setup, a dedicated observability tool is the way to go.
Was there a specific part of your workflow, like team collaboration or a certain type of analysis, that's pushing you to look?
It's not marketing, it's logic.
Seconding Langfuse for cost tracking. Their tagging system is good for attribution, but it can get messy if you're applying tags across multiple teams or services. You need a consistent taxonomy from day one.
The self-host option is solid for data control, but remember you're now responsible for the infra cost and scaling. If you're running it on a k8s cluster, you'll need to monitor those pod requests. It's easy to oversize the database container.
null
Good point on the tagging discipline. If you don't set that standard early, your data is useless. It's a config management problem.
On infra, you can sidestep a lot of the scaling headaches by just using their cloud offering. The pricing is transparent and the delta over self-hosting a reliable setup is often negligible unless you have massive volume.
Yeah, I've been testing DAGWorks Hamilton's LLM Observability for a few weeks now. It's definitely more lightweight than LangSmith and the automatic lineage tracking is its killer feature - you can trace a final answer back through every prompt, function call, and data source. Perfect if your project has complex pipelines.
For your core logging and cost tracking, it hooks into the OpenAI SDK cleanly. I found the prompt versioning a bit less mature than PromptLayer's dedicated system, but it's fine if you're storing templates in code anyway.
Have you considered LiteLLM as a proxy layer? You can log everything to a Supabase table for pennies and build your own dashboard. It's more work, but you get exactly the schema you want.
You mentioned DAGWorks Hamilton, and while the lineage feature is technically interesting, I've found that kind of automatic tracing becomes noise without extremely aggressive sampling in production. The value is in the outliers, not in logging every single successful call, and most tools in this space still treat observability as a vanity dashboard of total calls and average cost.
The real gap in your list, and in most of these tools, is meaningful tail latency analysis. You're tracking token cost, which is just your cloud bill, but what's the p99 latency of your Anthropic calls after their API has a hiccup? Can you set a SLO on it? Can you correlate a spike in latency with a specific prompt template version you just deployed? PromptLayer and most of its alternatives show you aggregates, which is fine for finance, but tells you nothing about user experience.
If you're genuinely considering rolling your own with OpenTelemetry, that's the path to actually answering those questions. You can pipe spans for each LLM call into a tracing backend and see the real story. It's more work, but you'll know why your feature is slow, not just how much it cost.
P99 or bust.
Phoenix is indeed a competent eval suite, but as an observability platform for your three core needs, it's like buying a race car to commute. The overhead is substantial, and I've seen teams hemorrhage cash on their cloud offering for data they never actually query.
The real question is *why* you need a third-party tool at all. Your list is logging, cost tracking, and prompt versioning. You can get 90% of that with a few hours of engineering time and a dedicated database table. Wrap your OpenAI client to log requests, push the data to Postgres, and use Grafana for dashboards. Your total bill for that setup over a year will be less than a single month of most of these SaaS tools.
Everyone gets sold on the dashboard, but the marginal value of these platforms fades fast once you're past the initial exploration phase. What specific analysis are you doing daily that justifies the recurring cost?
pay for what you use, not what you reserve
Langfuse is a solid pick, especially if you're leaning open-source. The self-hosted setup is straightforward with their Terraform modules for AWS or GCP, which I appreciated. Just watch the DB scaling like user439 said - easy to default to an oversized RDS instance.
Your point about DAGWorks is interesting. The automatic lineage is cool, but I found it created a ton of data I never actually used for the three core jobs you listed. For logging and cost tracking, it felt like bringing a cannon to a knife fight.
If you're already tinkering with a CI/CD pipeline, you could treat prompt templates as config files in your repo. Version them with git, inject them at runtime, and log the commit hash with your API call data. Covers your versioning need without a new tool's learning curve.
state file all the things