Just built a simple proxy to downsample high-volume, low-error traces.

finops_auditor_ray · 2026-07-27T15:52:09Z

Heard a team bragging about their "cost-free" distributed tracing setup because they're using open-source. Laughed out loud. Their storage and compute bill must be astronomical. They're probably ingesting 100% of traces, 99% of which are just "200 OK". Built a dead-simple proxy to sit in front of our tracing backend. Its only job: drop spans based on rules before they incur cost. The savings are real, but I want to see others' approaches. Our logic right now: * Keep 100% of traces for any error (status >= 500) * Keep 1% random sample for all successful requests * Drop all health check pings entirely It's a Go service, but the concept is trivial. Here's the core filtering logic: ```go func shouldSample(trace *pb.Trace) bool { // Rule 1: Drop all health checks if strings.Contains(trace.HttpPath, "/health") { return false } // Rule 2: Keep all errors for _, span := range trace.Spans { if span.StatusCode >= 500 { return true } } // Rule 3: Sample 1% of the rest return rand.Intn(100) == 0 } ``` Key point: This runs **before** the vendor's ingestion endpoint. We own the data drop, not them. This is the only way to guarantee the cost line moves. Questions for the room: * What's your sampling rate for prod traffic? * Are you filtering out specific noisy endpoints (looking at you, ELB pings)? * Anyone doing dynamic sampling based on load or time of day? Show me the bill.

Page 3 / 3 Prev

Observability Spend & Cost Control

Last Post by fionac 2 months ago

31 Posts

30 Users

0 Reactions

169 Views

RSS

fionac

(@fionac)

Reputable Member

Joined: 3 months ago

Posts: 186

29/07/2026 5:16 am

I hadn't considered that tightening the sample rate later would amplify the skew problem. That's a good point about capacity planning.

For someone like me who's just learning this, could you explain the practical difference between a shared seed and using a deterministic hash? I get that they'd both aim for consistency, but is one generally simpler to implement and manage than the other?

ReplyQuote

Page 3 / 3 Prev

Forum Icons: Forum contains no unread posts Forum contains unread posts

Topic Icons: Not Replied Replied Active Hot Sticky Unapproved Solved Private Closed