Hi everyone. I'm in the middle of setting up monitoring for a pipeline that uses both OpenAI GPT-4 and Anthropic's Claude (via API). The goal is cost tracking and spotting latency spikes. It's my first time with LLM-specific tools, and I'm a bit overwhelmed.
I've been looking at tools like LangSmith, Helicone, and OpenTelemetry instrumentation. My initial setup just logged basic stuff to BigQuery, but I'm missing key details.
Here's my naive first attempt for a single call:
```python
# This feels insufficient
import time
start = time.time()
response = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}]
)
duration = time.time() - start
# Log to BQ
log_row = {
"model": "gpt-4",
"duration": duration,
"input_tokens": ???, # How do I get this easily?
"output_tokens": ???,
"cost": ??? # Want to calculate this
}
```
My main questions:
* For a simple pipeline, is it better to use a dedicated proxy/tool or roll your own with OTEL?
* How are you attributing costs per project or user when models have different pricing?
* Any pitfalls in tracing chains that call multiple models? I'm worried about trace sprawl.
I'd love to see concrete examples of what you log for each LLM call. The docs often show the dashboard, but not the raw data structure going into it.
Rolling your own with OTEL sounds like the right move if you're already comfortable with it. The dedicated tools are great for out-of-the-box dashboards, but you'll fight their abstractions when you need custom cost attribution per project or user.
For cost tracking, you need to intercept the raw response. Both OpenAI and Anthropic return usage data in the response object. For OpenAI, it's `response.usage.prompt_tokens` and `response.usage.completion_tokens`. Calculate cost using the published per-1K-token prices for the specific model. You'll need a lookup table in your code mapping model names to their input/output rates. Do this calculation right after the API call so you can tag the metric with the project ID or user ID from your request context.
The big pitfall with tracing chains isn't sprawl, it's consistency. Make sure you're generating and propagating a single trace ID across all model calls within a single request, whether it's OTEL or just a UUID you log. That way you can still sum total cost and latency for the entire operation, not just per-call. Log everything - the prompt, the model, tokens, cost, latency - to your BQ table, keyed by that trace ID. Then you can slice it any way you need later.
Your naive approach is close. Just fill in the blanks from the response object and add that shared trace ID. Start simple.
I agree with the trace ID propagation point. Without a single correlation key across model calls, you lose the ability to attribute total cost and latency to a specific user request. I've seen teams log each API call as a separate row with no shared identifier, then try to stitch things together by timestamp. That breaks under any concurrency.
One nuance worth adding: if you're using streaming responses from either provider, the usage object is usually only present in the final chunk or the final response object. For OpenAI's streaming, `response.usage` is only populated on the last chunk when `stream_options: {"include_usage": true}` is set. For Claude, the streaming API returns usage in the `message_delta` event. If you're not careful, you'll log zero tokens for mid-stream metrics and only get the full count at the end. That throws off any real-time latency dashboards that try to compute cost per token as data arrives.
Also, regarding the cost lookup table: keep in mind that both providers occasionally update prices or introduce new model versions. A static map in your code is fine, but version it alongside your deployments. I've seen a team deploy a new model alias (e.g., `gpt-4o` replacing `gpt-4`) and forget to update the lookup, resulting in zero-cost logs for a week.
One more thing on BQ schema: if you log the full prompt text and response, watch out for the 100 MB row limit. You might want to truncate or hash long contexts, or store the text in a separate blob store and only keep a reference.
brianh
I'll stick with rolling your own because you mentioned cost attribution per project, which becomes a nightmare with most pre-built tools when you have more than three internal teams.
You can't rely on logging after the response object without intercepting the call. You need a wrapper function or a decorator that captures both the request parameters and the response. Here's a minimal pattern I've used:
```python
def instrumented_create(client, *args, **kwargs):
start = time.perf_counter()
project_id = get_current_project() # from thread-local or request context
response = client.chat.completions.create(*args, **kwargs)
duration = time.perf_counter() - start
cost_calculator = ModelCostLookup()
cost = cost_calculator.calculate(
model=kwargs.get('model'),
input_tokens=response.usage.prompt_tokens,
output_tokens=response.usage.completion_tokens
)
log_row = { ... } # include project_id and cost
```
The ModelCostLookup class is just a hardcoded dict mapping model names to their per-1K input/output rates. Update it when providers change pricing; you'll miss it otherwise.
For tracing chains, generate a single correlation ID at the entry point of your user request and pass it through every wrapper call. Don't let each model call generate its own; that's what causes sprawl. Attach it as a custom HTTP header or thread-local variable.
Show me the benchmarks.