How do you trace calls that go through a proxy?

A common architectural pattern I've implemented for managing LLM applications involves routing all model calls—whether to Anthropic, OpenAI, or open-source endpoints—through a custom proxy layer. This provides a single point for authentication, rate limiting, and standardized logging. However, it introduces a significant observability challenge: the proxy becomes a black box, obscuring the latency, cost, and error attribution of the individual downstream LLM provider calls.

My current approach involves instrumenting the proxy itself to generate and propagate trace context. The goal is to maintain a continuous trace from the initial user request, through the proxy's routing logic, out to the external LLM API, and back. This allows for breakdowns of where time is spent and which provider is responsible for a specific error or cost increment. I'll outline a representative setup using OpenTelemetry.

First, the proxy application must be instrumented to create spans for its internal processing and to inject trace context into the outgoing HTTP request to the LLM provider. Here's a simplified Python FastAPI example using the `opentelemetry` SDK:

```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
import httpx

tracer = trace.get_tracer(__name__)

@app.post("/v1/chat/completions")
async def proxy_to_llm(request: Request):
with tracer.start_as_current_span("proxy_llm_request") as span:
# Extract context from incoming request (if any)
carrier = dict(request.headers)
ctx = TraceContextTextMapPropagator().extract(carrier=carrier)
# Add proxy-specific attributes
span.set_attribute("llm.proxy.route", request.url.path)
span.set_attribute("llm.proxy.model", request.json().get("model"))

# Prepare call to actual LLM provider
headers = {'Authorization': f'Bearer {API_KEY}'}
# INJECT trace context into outgoing headers
outbound_carrier = {}
TraceContextTextMapPropagator().inject(carrier=outbound_carrier, context=ctx)
headers.update(outbound_carrier) # Adds 'traceparent' header

async with httpx.AsyncClient() as client:
with tracer.start_as_current_span("llm_api_call", context=ctx):
response = await client.post(
"https://api.openai.com/v1/chat/completions",
json=await request.json(),
headers=headers,
timeout=30.0
)
span.set_attribute("llm.response.status_code", response.status_code)
return response.json()
```

The critical part is the injection and extraction of the `traceparent` header. For this to be truly effective, the downstream LLM provider would need to be instrumented to consume that header and continue the trace, which is not possible with a third-party service. Therefore, the `llm_api_call` span is generated locally within the proxy, representing the external call's duration and outcome.

To gain deeper insights, we must enrich these spans with LLM-specific semantic conventions. I recommend using or extending the proposed OpenTelemetry semantic conventions for LLMs. This transforms generic HTTP spans into actionable LLM observability data.

Key attributes to set on the `llm_api_call` span:
* `gen_ai.system`: e.g., "openai", "anthropic", "cohere"
* `gen_ai.request.model`: e.g., "gpt-4-turbo-preview"
* `gen_ai.request.max_tokens`
* `gen_ai.response.finish_reasons`
* `gen_ai.usage.prompt_tokens`
* `gen_ai.usage.completion_tokens`

This instrumentation allows you to answer critical questions in your tracing backend (e.g., Jaeger, Tempo, Honeycomb):
* What is the 95th percentile latency for `gpt-4` calls routed through the proxy vs. direct calls?
* Which downstream provider is causing cascading failures due to rate limits?
* What is the cost per traced user session, broken down by model?

The remaining complexity lies in aggregating token usage and cost from these traces for business metrics. I typically have the proxy emit span events containing the raw prompt/completion token counts and a calculated cost based on the provider's pricing, which can then be summed in a metrics pipeline.

I'm interested in how others are solving the final mile: correlating these proxy-internal traces with the actual, provider-side telemetry some services like Azure OpenAI or Anthropic offer. Have you managed to link a `trace_id` from your proxy to a vendor's dashboard, or is the vendor telemetry treated as a separate, correlated stream?