I’ve been tracking my LangSmith usage for a few months now, primarily to correlate the platform’s reported token counts and latency with my own infrastructure’s performance and cost data. The LangSmith UI is fine for a quick glance, but for serious analysis—especially when you’re running multiple projects, comparing model versions over time, or trying to pinpoint regressions—it falls short. I needed something that could pull historical metrics, overlay them with my own monitoring, and allow for custom dashboards.
So I built a data source plugin for Grafana. It queries the LangSmith HTTP API directly, transforms the data, and makes it available for visualization alongside everything else. This lets me create a single pane of glass for inference performance. Here’s what I can now track in real-time on a dashboard:
* **Average latency per model/chain** across all traced requests, with percentiles (P90, P95) to catch tail latency issues.
* **Token consumption per project**, broken down by I/O, so I can validate pricing and catch unexpected token bloat.
* **Error rate per deployment**, correlated with my own application logs.
* **Cost per trace** (estimated), using my actual negotiated rates with the model providers, which often differ from list price.
The setup requires a Grafana instance and a LangSmith API key. The plugin is written in Go. Here’s a minimal example of the plugin’s configuration in your `grafana.ini`:
```ini
[plugin.langsmith-langsmith]
api_key = ${LANGSMITH_API_KEY}
api_url = https://api.smith.langchain.com
default_org = your-org-id-here
```
Once configured, you can write queries in the Grafana query editor. For example, to get the average latency for traces tagged with `production` and model `gpt-4-turbo` over the last 24 hours, you’d use a query like this in the plugin:
```
project="my-llm-app"; tags="production"; model_name="gpt-4-turbo"; metric="latency"
```
The real power comes from combining this with other data sources. I have a dashboard that plots LangSmith-reported token counts against the actual GPU utilization on my inference servers (for open-source models), and another that compares the cost-per-trace of `claude-3-opus` versus `gpt-4o` for the same chain over the past week. The discrepancies between what you *think* is happening and what the data shows can be significant.
I’m considering adding support for pulling full trace details for offline analysis and maybe integrating with the dataset run comparisons to visualize performance regressions. For now, it’s already uncovered a few inefficiencies in my prompt chains that weren’t obvious from the standard LangSmith table view. If you’re running LangSmith at any scale and care about measurable performance, you should be feeding its data into a proper observability stack.
Show me the benchmarks
That single pane of glass idea is exactly why Grafana shines. Pulling LangSmith metrics into the same dashboard as your own infrastructure data is the critical move - you can actually see if a latency spike was in the model call or your downstream services.
> cost per trace (estimated)
This is smart. Have you thought about tagging traces with a feature flag identifier or a release version? It could let you chart cost changes directly against a deployment, which is gold for a postmortem. "Rollout of v2.3 increased average token consumption by 18%" is the kind of hard fact that saves weeks of speculation.
How are you handling authentication with the LangSmith API? Did you have to write a custom backend for the plugin, or is it a straightforward data source config?
ship early, test often