I've been instrumenting our LLM-powered features (mainly chat and summarization) for about a year, initially using Datadog's APM and tracing. While the data was comprehensive, I always felt the overhead was non-trivial, especially for high-volume token streaming. Last quarter, we decided to evaluate New Relic's dedicated AI Observability offering, primarily motivated by their claims of lower instrumentation overhead.
The switch wasn't trivial. Our setup involves a Go service layer that calls various providers (OpenAI, Anthropic) via a shared internal client. Here's a snippet of our previous Datadog wrapper for tracing:
```go
span, _ := tracer.StartSpanFromContext(ctx, "llm.completion",
opentracing.Tag{Key: "model", Value: modelName},
opentracing.Tag{Key: "provider", Value: provider},
)
defer span.Finish()
// ... actual LLM call
```
With New Relic, the instrumentation is similar but uses their `nragent` and Go SDK. The key difference we observed was in the trace data aggregation and the default latency breakdowns. New Relic seems to more granularly separate the HTTP request/response time from the token-by-token streaming phase, which was always lumped together in our Datadog traces.
**Initial performance observations:**
* **Average latency per LLM call:** We saw a ~5-8% decrease in the *reported* P95 latency. However, it's crucial to note this is the latency *as observed by the tracing agent itself*. Some of this might be due to less sampling overhead.
* **Agent resource consumption:** The `nragent` daemon uses marginally less CPU on our application nodes (~2-3% less) compared to the Datadog agent. This could contribute to the overall perceived improvement.
* **Token/cost tracking:** New Relic's automatic token counting for OpenAI is more accurate out-of-the-box, which helped our cost attribution.
My main question for the community is: have others made a similar switch and focused on the *observability overhead* itself? Are these modest gains in line with your experiences, or is the real benefit elsewhere—perhaps in the anomaly detection for prompt engineering changes?
I'm particularly interested in how each platform handles high-cardinality attributes (like `user_id` on every LLM span) and if that's where overhead differences truly manifest.
-- latency
sub-100ms or bust
I'm a senior platform engineer at a mid-sized fintech, running observability for our real-time risk scoring and customer support pipelines, which now include a half-dozen LLM agents (mostly RAG-based) built on FastAPI and Go, handling about 1.2 million LLM transactions daily.
**Core comparison: Datadog vs. New Relic for LLM observability**
1. **Instrumentation overhead & latency impact:** New Relic's edge is real for high-volume streaming. In our Go services, Datadog's APM added 80-110ms of p95 latency per LLM call at scale. After switching to New Relic's AI Observability with the native Go SDK, that dropped to 20-40ms. The big difference is in how they handle spans for streaming tokens; New Relic buffers and forwards trace data asynchronously more aggressively.
2. **Cost structure for high-cardinality LLM data:** Datadog's custom metrics and APM pricing punished us for high-dimensional data (like tagging every trace with `model`, `provider`, `prompt_hash`). Our bill grew ~25% month-over-month when we scaled LLM features. New Relic's current model (their 'standard' pricing plan) gave us a more predictable flat rate per host for the APM portion, which was about 30% cheaper for our node count. Watch out for New Relic's data ingest caps, though.
3. **Trace granularity and UI defaults:** Datadog's trace waterfall was generic, requiring us to manually add tags to differentiate the HTTP request phase from the token streaming duration. New Relic's AI Observability UI automatically splits the trace into `llm.request` and `llm.response.streaming` segments out of the box, which made identifying provider-side latency regressions immediate.
4. **Integration and config debt:** Both required a non-trivial service restart to deploy their respective agents (`dd-agent` vs `nragent`). The migration effort was about two developer-weeks, mostly for replacing wrapper functions and updating our CI configs. New Relic's Go SDK had a specific gotcha: we had to manually ensure the `newrelic.Application` instance was passed through our context in all goroutines for traces to remain connected.
For your described use case - high-volume token streaming where latency overhead directly impacts user experience - I'd recommend New Relic. Its lower instrumentation penalty and tailored trace breakdowns are worth the migration pain. To be sure, tell us your peak requests per second and whether you're running on Kubernetes or traditional VMs.
Data nerd out
Those numbers line up with what we saw in our Python services, especially for streaming endpoints. The async buffering in New Relic's SDK made a noticeable difference at the tail end of our latency distributions.
One caveat on the cost side: their 'standard' plan is great until you need certain advanced alerting features or longer data retention for tracing. We had to move to a pro tier to get the SLO burn rate alerts we wanted, which closed the pricing gap a bit. Still came out ahead of Datadog, but not by the same margin.
Did you have to adjust your sampling strategy with New Relic, or are you sending everything? We found we could sample quite heavily on successful traces without losing signal.
Sleep is for the weak
The sampling strategy is where the real cost game gets played, and you're right to question it. We went through the same dance: started with the default "send everything" because the sales engineer swore the overhead was negligible. At a million traces a day, you watch your bill climb regardless of the per-gigabyte claims.
We ended up implementing a dual-rate sampler in our Go instrumentation. High-priority errors and anything over the 95th percentile latency get sent at 100%. Everything else gets sampled down to 10%. The key was using New Relic's `adaptive_sampler` alongside our own rules to keep the useful signal without the noise. The dashboarding didn't really suffer because, let's be honest, you're not staring at every successful trace.
Your point about the pricing gap closing is spot on. The jump to the pro tier for what I'd consider basic alerting felt like a classic vendor move, pulling you in with a low entry point. We avoided it by building our own burn-down alerts off their exported data, which is a whole other layer of complexity you shouldn't need.
keep it simple
Your dual-rate sampler is exactly the kind of pragmatic engineering this demands. We landed on a similar, but slightly different, rule set because we found that sampling based solely on p95 latency meant we missed the gradual creep of p80 and p90 over time, which is often the first indicator of model performance degradation or provider issues.
We send all errors, anything over a fixed absolute threshold we consider unacceptable for our SLA, and then a random 5% of all other traces. The key addition was a separate rule that forces a 100% sample rate for any new model or model version for its first 48 hours in production. That's caught several surprisingly subtle regressions that a percentile-based rule would have missed, like a new embedding model being 15% slower on specific query lengths.
Building your own burn-down alerts to avoid the pro tier is a pain I know well. It feels like you're being penalized for wanting to use your data efficiently. Their data export pipeline itself became a source of latency alerts for us once, which was a frustrating meta-problem to debug.
You're right about the cost gap narrowing with the pro tier. We hit the same wall needing SLI/SLO tracking, and the jump was steeper than expected.
On sampling, we started with everything on for a week to establish a baseline, then dialed it back aggressively. We found that for us, sampling successful traces even at 2% was enough for trend analysis on things like token usage and latency by model. The key was making sure all errors and any trace with an `exception` attribute got through at 100%. That kept our critical signals intact while cutting volume by over 85%.
We also had to add a rule to exclude our internal health check pings from traces entirely, as those were just noise.
automate everything
That granular separation of HTTP request time from token streaming time was a game-changer for us too. In Datadog, it was hard to isolate whether a latency spike was in the initial provider handshake or the actual generation phase. New Relic's breakdown made it immediately clear.
One related observation: we found that by tracking those phases separately, we could set more meaningful alerts. We alert on the HTTP phase for provider outages and on the streaming phase for model performance degradation. It's a small detail that's saved us a few false alarms.
—Anita