How do I audit LLM usage per client for billing using LangSmith data?

A common challenge when moving from prototype to production with LLM applications is implementing a granular, client-aware cost attribution model. While LangSmith excels at tracing and debugging, its raw telemetry data requires a deliberate pipeline to transform into a per-client billing feed. I've recently architected such a system and can share a reproducible pattern.

The core concept is to treat LangSmith's dataset of execution traces as a source stream. Each trace contains the necessary metadata—model, tokens, prompts—but must be enriched with a client identifier. The most reliable method is to instrument your application to inject a client ID into the trace metadata at runtime. This can be done via the `metadata` field when calling your LangChain or LLM application.

```python
# Example: Setting client context in a LangChain chain invocation
from langsmith import Client

# Assuming you have a chain already defined
result = chain.invoke(
{"input": "user query"},
config={
"metadata": {"client_id": "acme_corp_2024", "tenant": "subscription_tier_1"},
"tags": ["production", "billing"],
},
)
```

Once your traces are tagged, you can extract the data for transformation. I recommend using the LangSmith SDK to periodically query completed traces, then aggregate token counts and model calls. A robust pipeline involves the following stages:

1. **Extraction**: Schedule a script or use a framework like Airflow to fetch traces for a given time window via `client.list_runs`. Filter on project, tags, or status. Pagination is essential.
2. **Transformation & Enrichment**: Flatten the nested trace structure. Key fields for billing are:
* `run_id` & `trace_id` (for deduplication)
* `start_time` (for time-window aggregation)
* `metadata.client_id` (your injected identifier)
* `model_name` (e.g., `gpt-4-turbo-preview`)
* `usage` (total tokens, prompt tokens, completion tokens)
* `total_cost` (if you have enabled LangSmith's cost calculation)
3. **Aggregation**: Group the flattened records by `client_id`, `model_name`, and your billing period (e.g., day). Sum the token usage and cost. If LangSmith's `total_cost` is not used, you will need to apply your own cost calculation based on the official model pricing and the aggregated token counts.
4. **Load**: Write the aggregated records to your billing database or data warehouse (e.g., BigQuery). This becomes the source of truth for invoice generation.

A critical pitfall is the handling of incomplete or nested traces. Ensure your aggregation logic correctly attributes child runs (like tool calls or retrievers) to the parent's client metadata. You may need to traverse the trace tree to propagate the `client_id` from the root.

For teams already using dbt, this is an ideal transformation layer. The raw trace data can be ingested into a staging table, then the grouping and cost calculations can be defined as modular, testable SQL models. This also allows for easy integration with existing business intelligence tools for client-facing dashboards.

Has anyone else implemented a similar pipeline? I'm particularly interested in strategies for real-time streaming of this data, perhaps using LangSmith's webhook capabilities, to reduce the aggregation latency for high-volume applications.