Everyone's jumping on the high-cardinality bandwagon because it's the new hotness. Then they get a six-figure bill and blame the vendor. The problem is usually in the collector, not the backend.
If you're serious about this, start here: aggressive filtering at the receiver, drop high-churn attributes you don't use for SLOs, and use the batch processor with size/time limits tuned to your downstream's limits. And for the love of debugging, don't just copy the demo config from a blog post. Test with realistic load before it hits production.
What's your actual pipeline? Pushing to a SaaS or self-managed? That changes where you put the throttling.
Trust but verify.
I'm David, a staff engineer at a fintech that processes 80B events/day. Our data pipeline runs Kafka and Flink, pushing metrics from 1500 pods to a self-managed Cortex cluster via an OpenTelemetry Collector fleet.
**What matters for high-cardinality configs:**
1. **Receiver Filtering** - Drop attributes at the receiver, not in processors. A regex-based `include`/`exclude` matcher on the OTLP receiver stopped 60% of useless high-churn labels (like per-request internal IDs) from entering the pipeline in my last setup. This is your biggest cost lever.
2. **Batch Processor Tuning** - Defaults will blow up your backend. Set `send_batch_size` to match your downstream's max payload (e.g., 1000 for Cortex ingesters) and `timeout` to 30s. Use `send_batch_max_size` as a hard ceiling. I batch ~10k metrics/time series per batch.
3. **Memory Ballast Sizing** - If you don't set `mem_ballast_size_mib` to ~1/3 of collector memory, you'll get frequent GC pauses under load, causing queue overflows. On a 4GB container, I run a 1.5GiB ballast.
4. **Downstream Throttling Location** - For SaaS (Datadog, New Relic), you must enforce their per-host/s per-minute limits in the `batch` processor or with a dedicated throttling processor. For self-managed (Cortex, Prometheus), push throttling to the `loadbalancing` exporter's `sending_queue` size and retry logic.
I'd set up a dedicated collector tier for high-cardinality metrics with this config skeleton, isolated from traces/logs:
```yaml
receivers:
otlp:
protocols:
grpc:
include:
match_type: regexp
services: [".*service.*"]
exclude:
attributes:
- key: request.internal_id
match_type: regexp
value: ".*"
processors:
batch:
send_batch_size: 1000
send_batch_max_size: 2000
timeout: 30s
memory_limiter:
check_interval: 1s
limit_mib: 1500
```
My pick is always a self-managed collector pushing to Cortex if you have the platform team to run it, because you control the limits and cost is predictable. If you're on SaaS, tell us your vendor's exact rate limits and your average cardinality per metric so I can suggest where to clamp.
Benchmarks or bust
Missing your point #4. For self-managed Cortex you still need throttling, but push it to the `loadbalancing` exporter if you're running multiple collectors. It'll handle backoff on 429s better than a batch processor can.
Also, 1.5GiB ballast on 4GB is aggressive. It works until a memory leak eats your headroom. I run 1GiB on 4GB and tune GC percent instead.
What's your batch processor timeout? 30s is fine for high volume, but if Cortex ingesters are remote you'll hit timeouts on the first slow response.
Ship fast, review slower
Totally agree about testing with realistic load first. We learned that the hard way when our dev config couldn't handle prod traffic and the collector fell over.
We're pushing to a SaaS vendor. Does that mean we should rely on their throttling at the exporter, or is it still better to limit on our side first?