Tried SigNoz for our RAG pipeline. OpenTelemetry integration is a plus, but the LLM-specific insights felt thin compared to dedicated tools.
Main issues:
* Traces show generic spans, but lack LLM-specific attributes (e.g., token counts, model name) without heavy manual instrumentation.
* Dashboards require a lot of manual setup to show cost per call or prompt/response token tracking.
* The OpenAI integration is basically just HTTP-level tracing. You don't see the LLM semantics.
If you just need basic latency and error rates for your LLM gateway, it's okay. For anything deeper—like dissecting retrieval step costs or tracking embedding dimensions—you're building it yourself.
Has anyone pushed it further? Specifically:
* Custom attributes for OpenAI/Anthropic calls?
* Useful pre-built dashboards for token usage/costs?
* Any gotchas with high-volume tracing?
metrics not myths
Exactly. Their OTel collector hooks are just auto-instrumentation wrappers. You get HTTP metrics, not LLM semantics.
We added the `opentelemetry-instrumentation-openai` package directly. That populates spans with model, tokens, and finish reason. You still have to pipe those attributes into your dashboards manually. No cost tracking out of the box.
For high volume, we sampled trace data aggressively and pushed summaries to CloudWatch for alerting. Raw traces in SigNoz for ad-hoc drilling.
Show me the bill