LangSmith's new cost tracking is a step in the right direction. But is it accurate? I'm skeptical.
My early tests show discrepancies. My own cloud provider billing for the same period shows higher costs. The breakdown is too opaque.
* Are they counting all token usage correctly across nested chains?
* How are embeddings or vector DB calls priced?
* Does it include the overhead of the LangSmith infrastructure itself? Probably not.
Without transparent, granular logging of the raw LLM provider API calls, it's just an estimate. A useful estimate, but not a source of truth for cost optimization.
If you're serious about cost, you still need:
* Direct API logging from Azure/OpenAI/AWS.
* Your own aggregation and reporting.
* To treat LangSmith's numbers as a relative guide, not an absolute metric.
Simplicity is the ultimate sophistication
You're absolutely right to be skeptical. I'm seeing the same gaps in my pipelines.
> treat LangSmith's numbers as a relative guide
This is the key. I use it for spotting sudden cost *spikes* across deployments - that trend data is super useful. But for actual budgeting? Nah.
I built a simple sidecar script that scrapes our Azure OpenAI logs and dumps the totals into a monitoring dashboard. The numbers are always 15-LOL, ok, 20% higher than LangSmith's. It's the hidden orchestration hops that add up.
So yeah, great for alerts, bad for invoices.
Pipeline Pilot