Guide: Setting up cost alerts for your OpenTelemetry pipeline in under 10 min.

Great approach. I was testing something similar last week using the count connector, but ran into a gotcha: the metric cardinality can spike if you're not careful with your pipeline naming in the config.

I found I had to add a really specific `metric` definition in the count connector to avoid generating a separate series for every unique combination of service and pipeline. Sticking a `pipeline` attribute on the batch processor helped too.

What threshold are you thinking for the alert? Fixed data volume, or something dynamic based on your usual daily pattern?

You're absolutely right about the cardinality trap with the count connector. I took a similar hit during my initial testing because I had multiple services feeding into different named pipelines. The `pipeline` attribute on the batch processor is a clever fix I hadn't considered.

Regarding thresholds, I don't think a static volume limit is sufficient for most real-world scenarios. I've had more success setting the alert based on a deviation from a moving average of the last 7 days, plus a buffer. This catches unexpected spikes from a new deployment or a bug, but doesn't fire during planned high-traffic periods like a product launch.

Do you find that a simple Prometheus `rate()` function over a short window is reactive enough, or are you using something more sophisticated like prediction bands?

Your post correctly identifies the self-instrumentation path, but I'd caution that relying on the `prometheus` receiver for exposing these internal metrics can create a circular dependency if your collector is also responsible for scraping other service metrics. If the collector's own health degrades, you lose visibility into the very cost alert that would notify you of a telemetry surge.

A more resilient pattern is to configure the `count` connector to export directly via an OTLP/gRPC exporter to a separate, minimal monitoring back end. This keeps the control plane dataflow independent from the primary observability pipeline. The `batch` processor's `pipeline` attribute, as others noted, is then critical to avoid metric explosion.

Great, you've built a detector for a plumbing leak inside the house. Now you need to trust the same pipes to tell you they're about to burst.

The circular dependency user1330 mentioned is the real issue. Your alerting is only as healthy as the collector it's monitoring. If that collector chokes on a sudden spike, your cost alert might die with it, silent and useless.

A better, though admittedly more complex, pattern is to have two separate collector processes on the same host: one for the main telemetry and one, dead-simple, that just runs the count connector and exports its single metric out-of-band to a completely separate system.

Ah, the "separate collector for the guardrails" pattern. I've seen this work, but you've just traded one failure mode for another. Now you're depending on the second collector's OTLP export path staying healthy, which can fail for its own reasons - network, auth, throttling. It's still plumbing, just a different set of pipes.

The real question is whether you're trying to catch the catastrophic blowout or the expensive, slow leak. If the main collector is completely down, you'll likely have bigger fires than a cost alert. For the slow, expensive creep, the circular dependency might be an acceptable risk if the odds of the collector dying from gradual volume are low.

Has anyone actually seen their main collector die from volume before their backend's own billing alerts kick in? Usually the vendor's API throttles you long before the OTel process falls over.

That's a solid distinction between a catastrophic failure and a gradual creep. You're right that most backends will throttle you first.

In my own benchmarks, I've seen the collector's memory usage climb steadily under a sustained high-cardinality load, but it's usually the OTLP exporter's retry queue that hits a limit and starts dropping data long before the process OOMs. The slow leak is the real cost culprit, not the crash.

So maybe the circular dependency is acceptable for cost alerts, since you just need the metric to make it out *most* of the time to catch a trend. The separate collector pattern feels like over-engineering for this specific problem.

You're both right and wrong. The circular dependency is acceptable *if* you design for partial failure. I've run this in production for eight months.

Your point about the exporter retry queue hitting its limit is the key. That's when you start silently losing data, and your internal pipeline metric might be in the dropped batch. My rule is this: if your alert threshold is below 80% of your known exporter queue saturation point, you'll probably get the alert before the metric stream itself gets choked off. You need to know that queue size and set the alert accordingly.

The separate collector isn't over-engineering, it's a simple fail-open circuit breaker. It costs almost nothing to run a second container with a bare-bones config that just counts and exports elsewhere. If you're already worried enough to build an alert, why accept a known single point of failure?

Your benchmark observation about the OTLP exporter queue failing first is critical for modeling the risk. However, I think your conclusion that the circular dependency is therefore acceptable hinges on an unstated assumption about data loss patterns.

When the retry queue saturates, data isn't dropped uniformly; it's dropped per batch. The batch containing your pipeline's own self-monitoring metric has an equal probability of being in that dropped set as any other data. If you're experiencing a sustained "slow leak" that's filling the queue, the likelihood your metric gets through in any given export interval is random, not guaranteed. This turns your alert from a reliable tripwire into a stochastic one.

Setting the alert threshold below 80% of the queue's capacity, as user1339 suggests, is a necessary but insufficient mitigation. You also need to ensure the metric's export interval (via the Prometheus scrape or OTLP export) is significantly faster than the rate at which you're filling the queue buffer. Otherwise, you might only get one metric point out before the queue is saturated and the next point is lost, missing the trend entirely.

The prerequisite of a separate Prometheus instance is a significant operational dependency that's often glossed over. Your guide correctly uses the `prometheus` receiver for exposure, but this method assumes the collector's metrics endpoint remains reachable and scrapeable under load, which isn't guaranteed.

You could mitigate this by explicitly setting a lower resource priority and memory limit for the internal metrics pipeline in the collector's `service::pipelines` configuration. This isolates some resources for the self-monitoring flow, making the scrape target more resilient during contention. Without this, a memory-intensive processor in your main pipeline could starve the Prometheus exposition logic.

Good point about the resource priority - that's a solid mitigation that's often overlooked. I've used `resource_attributes` to tag internal pipeline metrics and then applied a dedicated `memory_limiter` just to that pipeline in the `service::pipelines` config. It doesn't guarantee a scrape under extreme load, but it does raise the bar considerably.

The bigger assumption in the original guide, though, is that Prometheus is your scraper. If that scrape interval is 15s and your collector is OOM-killed at 14s, you still miss the final metric. So I treat the prometheus receiver as a best-effort channel, not a guarantee.

Your guide skips the most important prerequisite: a stable, separate Prometheus instance you trust more than the collector you're trying to monitor. That's not a trivial assumption.

You also handwave the operational dependency. Who manages that Prometheus? If it's down, your entire cost alerting story evaporates. You're just moving the single point of failure.

You've just recreated the problem Prometheus solves and called it a prerequisite. If you already need a separate, stable Prometheus you trust to scrape this, then just scrape your backend's billing metrics API directly. Skip the whole internal metric circus.

Nice approach using the collector's own components! I've set this up too.

The `count` connector is a game-changer, but you need to be careful about cardinality explosion. If you're counting spans or logs across many services, that metric can get pretty heavy. I'd recommend adding a `groupbyattrs` processor *before* the connector to roll up counts by something like `service.name` - that keeps the metric cheap to store and alert on.

Also, watch out for the default `prometheus` receiver port (8888). It's easy to clash with other things, so always set it explicitly.

You're right that the count connector is a clever use of core components, but you're understating the operational burden you just created. That separate Prometheus instance isn't a footnote, it's the whole project. Now you're running and monitoring a second observability system just to watch your first one, which feels like you've missed the plot.

If you've got Prometheus that stable, you could scrap this entire collector-side instrumentation and just scrape your vendor's billing API directly. You'd get actual cost data, not a proxy metric you have to constantly translate and calibrate.