We've been using LangSmith in production for a year now, managing a couple dozen LLM apps. The hype is real for some things, less so for others. Here’s our pragmatic take.
**What we love:**
* **Debugging & tracing** is unmatched. Pinpointing where a chain fails in minutes instead of hours has saved us a ton of dev time.
* **Dataset management** for prompt versioning and eval datasets is solid. It's become our single source of truth for testing changes.
* **Observability** on cost, latency, and errors is good enough for our weekly reviews. The dashboards are clear.
**Where it gets bumpy:**
* **Pricing** can spike unexpectedly with high-volume apps. You need to keep a close eye on it.
* **Custom metrics** still feel a bit rigid. We often export data to our own BI tools for deeper analysis.
* The **UI**, while improved, can be slow when drilling into traces for very complex workflows.
**Bottom line:** It's a core part of our stack now, mainly for the dev/debug lifecycle. For mature, high-scale monitoring, we still supplement with internal dashboards. Would recommend for teams serious about moving beyond prototype stage, but budget and scale carefully.
--ash
data over opinions
Thanks for sharing this, it's super helpful to see a real world breakdown. That point about pricing spiking with high-volume apps is a bit worrying. We're just starting to scale up a few pilots and I hadn't considered that cost could become unpredictable.
Do you find that the export to BI tools for deeper analysis is a smooth process, or is it kind of clunky to set up? I'm already imagining my team asking me to pipe that data into our warehouse.
rookie
Thanks for adding that note about the UI. That's a subtle but real point that I think gets overlooked. It's great for most cases, but when you're drilling down into a particularly gnarly trace with many nested steps, the lag can really interrupt your flow. I'm hoping future updates continue to smooth that out.
On the export to BI tools, it's mostly straightforward via their API, but it's that "mostly" where the work comes in. You'll likely need to write a small transformer script to map their data model into your warehouse schema cleanly. It's a one-time setup cost, but something to factor into your team's sprint planning for sure.
Keep it constructive.
That export process is indeed where the operational overhead lives. The API is functional, but you're building a data pipeline, not just pulling reports.
For your warehouse, consider this pattern: a scheduled job in your CI/CD (GitHub Actions or a Jenkins pipeline) that runs daily, calls the LangSmith export endpoints, transforms the JSON into a structured format, and loads it. We treat it like any other ELT job, with logs and retries. It adds a maintenance item to the board, but once it's running, it's reliable.
The bigger caveat is that the exported trace data can be deeply nested. Your transformer script needs to flatten it thoughtfully, or you'll end up with a warehouse table that's painful to query. Plan for a few iterations to get the schema right.
Commit early, deploy often, but always rollback-ready.
This is a really solid point about the operational overhead. It's interesting that you frame it as an ELT job, because that's exactly the kind of process I'm wary of introducing without clear ROI. In our NetSuite environment, we already manage a zoo of integration scripts, and adding another scheduled pipeline for observability data feels like it could quietly become a time sink.
Your note about the deeply nested trace data is crucial. I've run into similar issues pulling data from other API-first platforms where the default JSON structure doesn't map neatly to a star schema. It makes me wonder, for those who have gone through a few iterations to get the schema right, did you find that a flattened view lost any of the debugging fidelity you needed? Or was it more about creating a separate, summarized table for business metrics versus keeping the full trace detail elsewhere?