The S3 config read is a classic case of solving one problem and creating two more. You're right about eventual consistency, but restarting the service on a schedule? That's just trading stale configs for unexpected downtime blips. Now your tracing proxy is causing its own incidents.
And the cache suggestion is mandatory. I've watched a team build a "cost-optimization" service that racked up more in S3 GET costs than it saved on traces. The poetic justice was beautiful, but the post-mortem wasn't.
prove it to me
That initial logic is a solid starting point for the cost bathtub curve - you catch the important errors and dump the obvious noise. The random 1% is where the subtle problems creep in, and they aren't just about low-traffic services.
You mentioned "high-volume" traces, which implies a potential for massive skew. A 1% random sample of ten million requests from a single, busy endpoint is still 100,000 traces. That's often where the real storage cost still lives, even after filtering. Conversely, your dozen daily requests to the admin panel have a high probability of vanishing entirely.
We added a second-stage filter that enforces a maximum cap per unique HTTP path per minute. So even if /api/v1/search is on fire, we only keep, say, the first 1000 sampled traces per minute from that specific path. It's a blunt instrument, but it directly targets the remaining high-volume, low-value noise that slips through the percentage filter. The quiet paths remain unaffected by the cap. You have to be careful to aggregate paths intelligently though, or a path with random IDs will bypass the cap entirely.
throughput first
That trade-off makes sense to me. Did you consider logging the total number of instances for a service alongside the sampled data? It wouldn't fix the coordination problem, but it might help someone interpreting the graph understand why a quiet service appears so active.