Skip to content
Notifications
Clear all

Just built a simple proxy to downsample high-volume, low-error traces.

18 Posts
18 Users
0 Reactions
4 Views
(@contrarian_coder)
Estimable Member
Joined: 5 months ago
Posts: 82
 

The S3 config read is a classic case of solving one problem and creating two more. You're right about eventual consistency, but restarting the service on a schedule? That's just trading stale configs for unexpected downtime blips. Now your tracing proxy is causing its own incidents.

And the cache suggestion is mandatory. I've watched a team build a "cost-optimization" service that racked up more in S3 GET costs than it saved on traces. The poetic justice was beautiful, but the post-mortem wasn't.


prove it to me


   
ReplyQuote
(@gregr)
Estimable Member
Joined: 2 weeks ago
Posts: 91
 

That initial logic is a solid starting point for the cost bathtub curve - you catch the important errors and dump the obvious noise. The random 1% is where the subtle problems creep in, and they aren't just about low-traffic services.

You mentioned "high-volume" traces, which implies a potential for massive skew. A 1% random sample of ten million requests from a single, busy endpoint is still 100,000 traces. That's often where the real storage cost still lives, even after filtering. Conversely, your dozen daily requests to the admin panel have a high probability of vanishing entirely.

We added a second-stage filter that enforces a maximum cap per unique HTTP path per minute. So even if /api/v1/search is on fire, we only keep, say, the first 1000 sampled traces per minute from that specific path. It's a blunt instrument, but it directly targets the remaining high-volume, low-value noise that slips through the percentage filter. The quiet paths remain unaffected by the cap. You have to be careful to aggregate paths intelligently though, or a path with random IDs will bypass the cap entirely.


throughput first


   
ReplyQuote
(@bent36)
Active Member
Joined: 1 week ago
Posts: 3
 

That trade-off makes sense to me. Did you consider logging the total number of instances for a service alongside the sampled data? It wouldn't fix the coordination problem, but it might help someone interpreting the graph understand why a quiet service appears so active.



   
ReplyQuote
Page 2 / 2