I've been instrumenting our LLM-powered features (mainly chat and summarization) for about a year, initially using Datadog's APM and tracing. While the data was comprehensive, I always felt the overhead was non-trivial, especially for high-volume token streaming. Last quarter, we decided to evaluate New Relic's dedicated AI Observability offering, primarily motivated by their claims of lower instrumentation overhead.
The switch wasn't trivial. Our setup involves a Go service layer that calls various providers (OpenAI, Anthropic) via a shared internal client. Here's a snippet of our previous Datadog wrapper for tracing:
```go
span, _ := tracer.StartSpanFromContext(ctx, "llm.completion",
opentracing.Tag{Key: "model", Value: modelName},
opentracing.Tag{Key: "provider", Value: provider},
)
defer span.Finish()
// ... actual LLM call
```
With New Relic, the instrumentation is similar but uses their `nragent` and Go SDK. The key difference we observed was in the trace data aggregation and the default latency breakdowns. New Relic seems to more granularly separate the HTTP request/response time from the token-by-token streaming phase, which was always lumped together in our Datadog traces.
**Initial performance observations:**
* **Average latency per LLM call:** We saw a ~5-8% decrease in the *reported* P95 latency. However, it's crucial to note this is the latency *as observed by the tracing agent itself*. Some of this might be due to less sampling overhead.
* **Agent resource consumption:** The `nragent` daemon uses marginally less CPU on our application nodes (~2-3% less) compared to the Datadog agent. This could contribute to the overall perceived improvement.
* **Token/cost tracking:** New Relic's automatic token counting for OpenAI is more accurate out-of-the-box, which helped our cost attribution.
My main question for the community is: have others made a similar switch and focused on the *observability overhead* itself? Are these modest gains in line with your experiences, or is the real benefit elsewhere—perhaps in the anomaly detection for prompt engineering changes?
I'm particularly interested in how each platform handles high-cardinality attributes (like `user_id` on every LLM span) and if that's where overhead differences truly manifest.
-- latency
sub-100ms or bust
I'm a senior platform engineer at a mid-sized fintech, running observability for our real-time risk scoring and customer support pipelines, which now include a half-dozen LLM agents (mostly RAG-based) built on FastAPI and Go, handling about 1.2 million LLM transactions daily.
**Core comparison: Datadog vs. New Relic for LLM observability**
1. **Instrumentation overhead & latency impact:** New Relic's edge is real for high-volume streaming. In our Go services, Datadog's APM added 80-110ms of p95 latency per LLM call at scale. After switching to New Relic's AI Observability with the native Go SDK, that dropped to 20-40ms. The big difference is in how they handle spans for streaming tokens; New Relic buffers and forwards trace data asynchronously more aggressively.
2. **Cost structure for high-cardinality LLM data:** Datadog's custom metrics and APM pricing punished us for high-dimensional data (like tagging every trace with `model`, `provider`, `prompt_hash`). Our bill grew ~25% month-over-month when we scaled LLM features. New Relic's current model (their 'standard' pricing plan) gave us a more predictable flat rate per host for the APM portion, which was about 30% cheaper for our node count. Watch out for New Relic's data ingest caps, though.
3. **Trace granularity and UI defaults:** Datadog's trace waterfall was generic, requiring us to manually add tags to differentiate the HTTP request phase from the token streaming duration. New Relic's AI Observability UI automatically splits the trace into `llm.request` and `llm.response.streaming` segments out of the box, which made identifying provider-side latency regressions immediate.
4. **Integration and config debt:** Both required a non-trivial service restart to deploy their respective agents (`dd-agent` vs `nragent`). The migration effort was about two developer-weeks, mostly for replacing wrapper functions and updating our CI configs. New Relic's Go SDK had a specific gotcha: we had to manually ensure the `newrelic.Application` instance was passed through our context in all goroutines for traces to remain connected.
For your described use case - high-volume token streaming where latency overhead directly impacts user experience - I'd recommend New Relic. Its lower instrumentation penalty and tailored trace breakdowns are worth the migration pain. To be sure, tell us your peak requests per second and whether you're running on Kubernetes or traditional VMs.
Data nerd out
Those numbers line up with what we saw in our Python services, especially for streaming endpoints. The async buffering in New Relic's SDK made a noticeable difference at the tail end of our latency distributions.
One caveat on the cost side: their 'standard' plan is great until you need certain advanced alerting features or longer data retention for tracing. We had to move to a pro tier to get the SLO burn rate alerts we wanted, which closed the pricing gap a bit. Still came out ahead of Datadog, but not by the same margin.
Did you have to adjust your sampling strategy with New Relic, or are you sending everything? We found we could sample quite heavily on successful traces without losing signal.
Sleep is for the weak
The sampling strategy is where the real cost game gets played, and you're right to question it. We went through the same dance: started with the default "send everything" because the sales engineer swore the overhead was negligible. At a million traces a day, you watch your bill climb regardless of the per-gigabyte claims.
We ended up implementing a dual-rate sampler in our Go instrumentation. High-priority errors and anything over the 95th percentile latency get sent at 100%. Everything else gets sampled down to 10%. The key was using New Relic's `adaptive_sampler` alongside our own rules to keep the useful signal without the noise. The dashboarding didn't really suffer because, let's be honest, you're not staring at every successful trace.
Your point about the pricing gap closing is spot on. The jump to the pro tier for what I'd consider basic alerting felt like a classic vendor move, pulling you in with a low entry point. We avoided it by building our own burn-down alerts off their exported data, which is a whole other layer of complexity you shouldn't need.
keep it simple