We recently completed a migration of our evaluation and observability stack from MLflow to LangSmith for a production system of approximately 50 LLM-based agents. The primary drivers were LangSmith's native support for complex chains and agent traces, which promised better operational visibility. While the core tracing and evaluation functionalities are now superior, the migration surfaced several non-obvious breakages in our FinOps and operational cost reporting.
The immediate, expected issues were around recreating experiment tracking paradigms and adapting our existing prompt versioning. However, the more significant disruptions were:
* **Loss of historical cost data correlation:** Our MLflow instance was integrated with our internal billing data, allowing us to tie model run costs directly to specific project codes and departments. LangSmith's native pricing model focuses on token counts per provider, but our existing granular cost allocation (which factored in GPU-hour overhead, network egress, and support tiers) did not map cleanly. We are now building a separate reconciliation layer.
* **Vendor contract compliance became opaque:** Several of our model provider contracts have commit tiers and custom pricing based on usage patterns. Our previous dashboard highlighted proximity to these thresholds. LangSmith's provider-level spend tracking is good, but it does not natively alert or project against our specific contractual commitments, creating a renewal risk.
* **Benchmarking overhead increased:** We maintain a suite of performance/cost benchmarks for agent variants. In MLflow, this was a dedicated artifact. Migrating these to LangSmith's dataset and evaluation workflows required re-architecting the benchmark runs themselves to fit the "evaluation" paradigm, adding unexpected project time.
Has anyone else managed a similar-scale migration from a general-purpose MLOps platform to a specialized LLMops one? I'm particularly interested in how you bridged the total cost of ownership and vendor management gaps that such a move can create. What middleware or integration patterns proved necessary to maintain financial oversight?
Buy once, cry once.
I manage a community platform where we run about 20 support and content-moderation agents in production, and we evaluated both MLflow and LangSmith last year before settling on a hybrid approach.
**Target Audience Fit:** MLflow feels built for in-house platform teams who own their own infra and need to stitch together custom reporting. LangSmith targets product teams building with OpenAI/Anthropic who want the tracing out of the box. Our 20-agent system is a comfortable mid-market fit for LangSmith; at 50 agents, you're hitting its scaling edge where cost control gets fuzzy.
**Real Operational Cost:** LangSmith's per-token tracking is transparent but narrow. Our hidden cost was the engineering time to rebuild the cost-allocation layer you mentioned - roughly 3 person-weeks for us. MLflow's "free" software has a high internal tax for building and maintaining the integrations you described.
**Deployment & Integration Effort:** Migrating *traces* was straightforward. Migrating our *governance* (cost tags, compliance flags, project mappings) was a 6-week project. The breakage point is that LangSmith's API expects its own ontology; retrofitting our existing departmental billing codes required a middleware proxy to avoid altering all our agent calls.
**Vendor & Support Experience:** At our scale, LangSmith support was responsive on technical bugs (1-2 day turnaround). They were unable to help with our custom billing integration, which was expected. MLflow support is your own team or paid enterprise support, which changes the dynamic entirely.
I'd recommend LangSmith if your primary need is developer velocity on complex, multi-step agent logic and you can tolerate a quarter-long project to rebuild FinOps. If granular, pre-existing cost allocation and vendor compliance are non-negotiable, you should tell us your team size for rebuilding pipelines and whether you can tolerate a multi-tool stack.