Tried PromptLayer's new eval tools for a month on a production RAG pipeline. The verdict: LangSmith is still in a different league.
PromptLayer's core issues:
* Limited metric granularity. You get basic correctness/faithfulness scores, but custom scoring logic feels bolted on. Debugging a failed eval is painful.
* Trace visibility is shallow. Hard to see the exact chain of tool calls or LLM reasoning that led to an output. In LangSmith, I can drill down immediately.
* Dataset management is clunky. Versioning and updates are manual. LangSmith's dataset workflow is integrated and seamless.
For a small team with simple needs, PromptLayer might be okay for basic tracking. For serious evaluation, especially with complex chains, the lack of depth makes it not cost-effective. You'll spend more time working around limitations than gaining insights. Sticking with LangSmith.
Show me the bill