Just got our latest Datadog invoice and my jaw hit the floor. 😳 We turned on APM for a few key Kubernetes services last month and the bill *tripled*.
Our setup seemed reasonable:
* Sampled at 10%
* Only 3 main services
* Retention at the default 15 days
But with the volume of traces from our API, the ingest costs went crazy. We're now looking at **$4,200/month** just for APM data. For a mid-sized SaaS app!
Has anyone else been blindsided by this? What are you doing to keep APM costs under control? I'm looking at:
* Aggressive dynamic sampling rules
* Filtering out health checks & noisy endpoints
* Considering a switch to Grafana Tempo for storage
Here's the sampling rule we just added to the Datadog tracing library as a starting point:
```java
@datadog.trace.api.config.TracerConfig
DD_TRACE_SAMPLE_RATE=0.1
DD_TRACE_SAMPLING_RULES=[{"service": "api-gateway", "sample_rate": 0.05}, {"name": "healthcheck", "sample_rate": 0}]
```
Would love to hear your war stories and what actually worked.
#savings
Oh, the classic "triple" surprise. It's a rite of passage. Your plan is sound, but that 10% fixed sample rate is still a sledgehammer. The real trick is getting dynamic sampling *into your actual application code*, not just at the agent level. Your rule for healthchecks is good, but you need to get ruthless with the business value of each endpoint.
For us, the real killer was ignoring trace priority. We set up rules to only keep 100% of traces for high-priority errors or specific customer journeys we were debugging. Everything else gets a hair-cut. We also pipe all our traces through a sidecar that filters out entire spans for noisy, internal calls before they ever reach Datadog's intake API.
Have you calculated your actual ingest volume versus what you're paying for? I've found their pricing calculator is... optimistic. Switching storage helps, but then you're paying in engineering time to rebuild the dashboards they lock to their own storage. It's a nasty trap.
Data over dogma.
Yeah, the engineering time trade-off is a big one. We looked at moving storage too, but rebuilding all the dashboards and alerts felt like a second full-time job. It locks you in.
You mentioned a sidecar for filtering. Are you using something open source for that, or is it a custom service? I'm wondering if the maintenance of that filter ends up costing as much as the ingest savings.