Heard a team bragging about their "cost-free" distributed tracing setup because they're using open-source. Laughed out loud. Their storage and compute bill must be astronomical. They're probably ingesting 100% of traces, 99% of which are just "200 OK".
Built a dead-simple proxy to sit in front of our tracing backend. Its only job: drop spans based on rules before they incur cost. The savings are real, but I want to see others' approaches.
Our logic right now:
* Keep 100% of traces for any error (status >= 500)
* Keep 1% random sample for all successful requests
* Drop all health check pings entirely
It's a Go service, but the concept is trivial. Here's the core filtering logic:
```go
func shouldSample(trace *pb.Trace) bool {
// Rule 1: Drop all health checks
if strings.Contains(trace.HttpPath, "/health") {
return false
}
// Rule 2: Keep all errors
for _, span := range trace.Spans {
if span.StatusCode >= 500 {
return true
}
}
// Rule 3: Sample 1% of the rest
return rand.Intn(100) == 0
}
```
Key point: This runs **before** the vendor's ingestion endpoint. We own the data drop, not them. This is the only way to guarantee the cost line moves.
Questions for the room:
* What's your sampling rate for prod traffic?
* Are you filtering out specific noisy endpoints (looking at you, ELB pings)?
* Anyone doing dynamic sampling based on load or time of day?
Show me the bill.
show me the bill