Everyone's rushing to slap "LLM Observability" on their dashboards. Most of it is just repackaged APM with a fancy tag.
I'm evaluating for a real multi-cloud mess: GCP Vertex AI, Azure OpenAI, and a self-hosted pile of OSS models. Need to trace a single user request across these providers, see latency per model/call, and track costs.
Grafana's stack (Tempo, Loki, Mimir) is heavy but known. SigNoz pitches as an all-in-one OSS alternative. In practice, for LLMs:
* Can either actually instrument LLM SDK calls properly, or do you end up writing manual spans for everything?
* Is the cost attribution anything more than a manually tagged metric?
* Multi-cloud collection overhead – agent bloat versus missing data.
Looking for war stories, not marketing slides. Who's actually using either for this beyond basic demo setups?
More dashboards != better ops