Hi everyone. I’ve been exploring Traceloop for monitoring and debugging our AI pipelines, and I’m quite impressed with its tracing capabilities so far. Our team currently uses a self‑hosted instance of Anthropic’s Claude API, and I’d like to integrate it with Traceloop to get better visibility into our model usage and costs.
I’ve successfully connected Traceloop to cloud‑based APIs using their standard SDK, but the documentation seems focused on those endpoints. Could someone provide a clear, step‑by‑step guide on how to configure the Traceloop SDK (or perhaps the OpenTelemetry setup) to point to a self‑hosted Claude API? I’m particularly interested in any necessary changes to the base URL or authentication parameters that differ from the default setup.
Also, if anyone has done this, I’d be curious to hear about any specific differences you noticed in the traces or metrics compared to using the standard Anthropic endpoint. For instance, does the cost tracking feature still work accurately, or does it require custom configuration?
Thanks!
Great question, and you're right that the docs don't cover this specific setup clearly. The key is to configure the OpenTelemetry instrumentation for the Anthropic client, not Traceloop directly.
When you instantiate your Anthropic client for the self-hosted endpoint, you just need to pass your custom base URL there. The Traceloop SDK, which wraps OpenTelemetry, will automatically pick up the calls from that configured client. So your code change is isolated to the client initialization, something like `client = anthropic.Anthropic(base_url="http://your.internal.endpoint/v1", api_key="your-key")`. The tracer doesn't need the endpoint separately.
On your second point about cost tracking, that's where you'll likely hit a snag. The cost tracking features typically rely on known, cloud-based model names from Anthropic's roster. If your self-hosted instance uses a different model name string, or if you've stripped out pricing details, the automated cost estimation will probably fail or default to zero. You might need to set up custom pricing metrics in your Traceloop dashboard to map your internal model names to a cost-per-token.
Has anyone else found a workaround for the pricing piece with internal models?
buyer beware, but buy smart
Agreed, the cost tracking will break with a custom endpoint. The instrumentation uses the model name from the Anthropic SDK's response. If your self-hosted Claude returns a model identifier like `claude-3-5-sonnet-20241022`, it might still work if it matches Traceloop's internal pricing table. If it returns something like `my-company-claude`, it won't.
You can work around it by setting a custom attribute on the span. For example, after initializing Traceloop, you could monkey-patch the client's completion method to add `llm.model` as a known model string. It's a bit hacky, but it forces the cost attribution.
```python
# Rough example
original_create = client.completions.create
def patched_create(*args, **kwargs):
from traceloop.tracing import get_current_tracer_span
span = get_current_tracer_span()
if span:
span.set_attribute("llm.model", "claude-3-5-sonnet-20241022")
return original_create(*args, **kwargs)
client.completions.create = patched_create
```
Without something like this, your dashboards will show zero cost for those calls.
BenchMark