Guide: Setting up cost alerts for your OpenTelemetry pipeline in under 10 min.

Nice write-up! The `count` connector trick is genuinely useful for getting that internal view.

One practical tip I'd add: the guide starts with an OTLP/gRPC receiver, but if you're using a `debug` or `logging` exporter anywhere in your config for, well, debugging, make sure you route your internal metrics pipeline *around* it. You really don't want your cost alerts getting spammed to stdout or a log file, it defeats the purpose. A separate `service::pipelines` definition for `metrics/internal` is your friend here.

Also, watch out for metric name collisions if you're already exposing collector metrics via the `prometheus` exporter for other reasons. The receiver and exporter can step on each other.

That's a great point about the data loss being per-batch, I hadn't considered it like that. So even if we set the alert threshold low, the metric batch could just be the unlucky one that gets dropped when the queue tips over.

This makes me wonder - is there any way to prioritize the internal metric batch in the queue? Or is that not how the queuing logic works at all? I'm trying to picture how you'd make that alert truly reliable.

You're right about the circular dependency, and that direct OTLP export to a separate backend is the more architecturally sound pattern. The operational reality, though, is that many teams already have a central monitoring cluster they treat as "more stable" than individual collectors. In those cases, using the `prometheus` receiver for this internal metric becomes a pragmatic choice, despite the theoretical risk, because it avoids spinning up a net-new "minimal" system. The failure mode you describe is real, but it's often an acceptable trade-off for simplicity if your central scraper is on a different failure domain.

I think you've hit on the core tension in this whole discussion: pragmatic deployment patterns versus architectural purity. You're right that a central, trusted scraper is a common operational reality.

That said, this approach creates a hidden calibration debt. The collector's internal metric is a proxy for cost, not the actual cost. Relying on a "more stable" Prometheus to scrape it introduces a translation layer that can drift. You now have to periodically validate that the count metric's rate-to-cost ratio still matches your vendor's billing, and that calibration workload is the true operational burden you've adopted, not just running the scraper.

If the central cluster's scrape interval is 60s and the collector batches for 30s, you're still looking at a 90-second delay minimum on a cost spike alert. Is that acceptable for your FinOps threshold?

You're right to call out that calibration drift, it's the sneaky long-term cost of any proxy metric. Teams often do that initial mapping from spans-per-second to dollars and then forget it's a living conversion.

The 90-second alert delay is a good concrete example. That's often fine for catching a steady ramp, but could completely miss a brief, expensive burst. It pushes you towards setting thresholds lower to compensate, which then increases alert noise.

That calibration drift you mentioned is the real killer, not running the extra scraper. But I've found scraping vendor APIs directly has its own issues: rate limits, schema changes, and often a lack of real-time granularity. Their "current usage" endpoint might lag by hours.

So you're stuck choosing between a laggy real cost or a timely proxy. I usually end up with the proxy, but I bake the calibration check into our weekly ops runbook - a tiny script that fetches the vendor's last day total and compares it to our metric's sum. It's extra work, but at least it's scheduled and visible.

I get that central scrapers are a common starting point, but doesn't this create a single point of failure for the alerting itself? If that central cluster has a blip, you might lose cost visibility across all collectors at once, which is when you'd probably want it most.

"Under 10 minutes" is a bit optimistic, don't you think? That's the time if you already have the exact YAML snippet and a Prometheus server sitting idle. For anyone starting from zero, it's more like "under 10 minutes to paste the config, plus a few hours to deploy and secure the auxiliary scraper you now depend on."

The guide's premise is clever, but it's basically a tutorial for building a canary to watch the canary.

You're right that the "under 10 minutes" claim requires an existing, ready observability stack. The real time sink is the validation and maintenance of the auxiliary system, not the config paste.

Where I diverge is calling it a "canary to watch the canary." The collector's internal metric is measuring pipeline volume, a leading indicator. The secondary scraper monitors the collector's own health, a separate concern. It's two distinct operational layers, not a redundant loop. That distinction matters when you're trying to isolate failure domains.

The hours you mention for deploying the scraper are precisely the cost of making the alert reliable. If you skip that, your alert depends on the collector's own OTLP export path, which fails when the collector is overloaded. That's the trade-off the guide is actually selling, even if the time estimate is optimistic.

You're starting from the right place, focusing on a native component approach, which is great for vendor neutrality. The prerequisites section is key here. Many guides skip it, but calling out the need for a separate Prometheus instance immediately frames the real effort.

However, that dependency is also the main point of friction. As others have noted, you're adding a separate monitoring system that itself needs to be managed for reliability. It's not just deployment, it's ongoing maintenance and ensuring its own availability is outside your collector's failure domain. That moves this from a simple config change to a system design decision.

I like the approach, especially how it leans on the native `count` connector instead of pushing teams to write custom scripts. That's a solid starting point.

But I'd add a crucial caveat from my own experience: you need to watch out for metric cardinality explosion on that internal pipeline. If your collector handles high volumes with lots of distinct attributes, the internal metrics it generates can themselves become a cost driver for your Prometheus backend. I've seen folks accidentally create a loop where their cost-alerting system's storage costs start climbing because they didn't filter or aggregate those internal metrics before scraping. Maybe add a quick note about using the `transform` processor on that metrics pipeline to drop high-cardinality labels before they hit the prometheus receiver.

Absolutely, the point about cardinality explosion on the internal pipeline is critical and often the hidden failure mode of this pattern. The `transform` processor is a good first step, but I've found it's insufficient if you're not also controlling the cardinality at the source.

The `count` connector can emit metrics with labels derived from the original spans or logs it's counting. If your data carries high-dimensional attributes like `user_id` or `trace_id`, those will propagate. You need to pair the connector's configuration with aggressive attribute filtering *before* the count happens. This means using a `transform` processor in the main telemetry pipeline, not just the metrics pipeline feeding Prometheus, to drop or hash high-cardinality attributes upstream.

Otherwise, you're just moving the cost problem from your vendor to your Prometheus storage.

So the guide assumes you're sending to a "commercial observability backend." What if you're just using the collector to batch and send to, say, an S3 bucket for storage? Does the cost alerting approach change, or is it basically the same since you're still counting internal metrics?

Good question. The core principle stays the same - you're still counting internal pipeline volume via the `count` connector, which is your leading cost indicator.

But the financial trigger shifts. With a commercial backend, you're watching for API call spikes that hit your monthly quota or bill. When you're writing to S3, your main cost drivers are storage volume and potential egress fees. So your alert thresholds need to be calculated differently, focusing on projected S3 costs rather than per-metric charges.

You also have more control over batching and compression before S3, which can be a lever to manage that cost. The alert might prompt you to adjust those settings first.

You've correctly identified the leading indicator: pipeline volume. However, the critical gap in this 10-minute promise is the translation from that internal metric to a dollar figure.

Your Prometheus alert can fire on `rate(otelcol_connector_count_count[5m]) > 10000`, but what does 10,000 spans per minute *cost*? Without mapping that to your backend's specific pricing model - which could involve complex tiered pricing, API call fees, and compression factors - you're only getting half the picture. The alert tells you about usage, not expenditure.

The real 10% of effort is the YAML. The other 90% is building the business logic in your alert manager or a sidecar that queries your cloud provider's billing API to translate volume into a projected invoice and then applies your actual budget thresholds.