LangSmith? For a startup? Lol.
You're pre-series A. Your "observability" is logging to stdout and praying. You don't need another SaaS bill, especially not at LangChain prices. You need something that costs zero dollars and runs where your code runs.
My stack for this:
* **Tracing:** OpenTelemetry. Instrument your LLM calls, pipe it to a collector.
* **Visualization:** Grafana (free tier) on a cheap VM, or even better, just use the console output for now. If you must, Jaeger for traces.
* **Evals/Testing:** Write your own damn pytest scripts. It's not that hard. Store prompts/completions in your own database, run nightly diffs.
LangSmith's real value is the tight integration if you're *all-in* on LangChain. Are you? Probably not. You're using raw OpenAI or maybe LiteLLM.
Spending engineering cycles on "best practice" LLM ops before you have a working product is a fantastic way to burn runway. Get the traces, log the costs, iterate fast. Fancy UI can come later.
fight me