Hey everyone! I'm still pretty new to observability but I've been loving OpenClaw so far. We set it up last week to get a handle on our tracing costs.
We configured the collector to sample only 10% of traces, but our bill this week shows we're ingesting almost 90% of them! 😳 Our config looks like this:
processors:
probabilistic_sampler:
sampling_percentage: 10
It's in the pipeline right before the exporter. We're seeing the huge volume in both our backend and the billing dashboard.
Has anyone run into this? Could it be that other services are also sending spans directly to the backend, bypassing our collector? Or is there a common config mistake we might have made? Any pointers would be so helpful!
The probabilistic sampler works on spans, not traces. If you're processing high-volume services where each trace has dozens of spans, you're still sampling 10% of those *spans*, which can result in nearly all traces being partially recorded. That's likely why your backend sees 90% of traces - a trace with even one sampled span often gets counted.
Check if you have other samplers earlier in your pipeline, like a tail-based sampler, that might be overriding it. Also, confirm no services have SDK sampling configured that's sending pre-sampled data to your collector; the processor won't re-sample those.
Post your full pipeline config section. We need to see the order of receivers, processors, and exporters.
Been there, migrated that