I've been seeing a lot of buzz about Traceloop for LLM observability, especially in OpenAI-heavy stacks. Their docs and examples are pretty OpenAI-centric, which got me thinking.
My team is starting to integrate Anthropic's Claude (via their API) for some specific classification tasks. I'm curious if anyone has practical experience wiring Traceloop into a pipeline that uses a non-OpenAI provider. Specifically:
* Does the SDK's `traceloop.trace()` decorator or manual instrumentation work seamlessly, or did you need to write custom wrappers?
* How's the support for Claude's specific message structure (e.g., the `system` vs `human` vs `assistant` roles) in the traced spans?
* Were you able to capture key metrics like token usage and latency for Anthropic calls as easily as for OpenAI?
I threw together a quick test using the Python SDK, but I'm not sure if I'm configuring it optimally. My attempt looked something like this:
```python
import traceloop
from anthropic import Anthropic
traceloop.init(app_name="claude_poc")
client = Anthropic()
@traceloop.trace()
def query_claude(prompt):
message = client.messages.create(
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
model="claude-3-opus-20240229",
)
return message.content
```
It seems to create a trace, but I'm not seeing the rich details I'd expect. Is the out-of-the-box support just for OpenAI, and everything else requires manual span creation?
Would love to hear about your setup, any config snippets, or if you concluded it wasn't the right tool for a multi-model environment. Also, how does it compare to more provider-agnostic tools like LangSmith or Helicone for this use case?
--diver
Data is the new oil - but it's usually crude.