I see a lot of teams get overwhelmed by their observability bill right after they get their first real traffic spike. They instrument everything, send all the data, and then get a nasty surprise. The knee-jerk reaction is to start cutting back instrumentation, which hurts your ability to debug.
A more sustainable approach is to start with **deterministic sampling**. The title says 10%, and that's a solid, battle-tested starting point. The goal isn't to lose data, but to be intentional about what you capture.
Here’s a simple pattern we've used for tracing with OpenTelemetry. You configure this in your collector or SDK.
```yaml
processors:
probabilistic_sampler:
sampling_percentage: 10
hash_seed: 22
```
The key is **deterministic** sampling. The same request, traced again, will have the same sampling decision. This lets you debug by recreating the request ID. You're not losing *types* of errors, you're just seeing a representative sample of them.
Start at 10%. Then ask yourself these questions over the next sprint:
* Are we missing critical, low-volume errors (e.g., 5xx on a checkout endpoint)?
* Can we still identify the root cause of performance regressions?
* Do our dashboards and alerts still function?
If the answer is "yes, we're missing things," don't abandon sampling. Instead, layer on **rules-based sampling**. Sample all errors (`span.status_code == ERROR`) at 100%, and keep the 10% for everything else. Most vendors support this, and you can do it in the OTel collector.
This approach keeps costs predictable and scales with your traffic, while protecting the data you need for debugging. The next step is looking at cardinality in your metrics, but that's a different thread.