Has anyone tried the new Prometheus integration for custom metrics?

I set it up last week for a side project. Compared to running your own Prometheus instance, the scraping configuration is effectively the same, which I suppose is the point. You define your targets and labels in a familiar config file.

Where it gets interesting, or perhaps frustrating depending on your day, is the metric ingestion. The custom metrics billing is a separate tier, so you'll want to be judicious about what you send. I found myself adding more filters than I would locally to avoid a surprise invoice.

As for straightforwardness, the actual steps are well documented. The real test is whether your mental model of Prometheus aligns with Sysdig's interpretation of it. Have you mapped out which internal metrics are absolutely essential yet?

The billing surprise is an excellent point. Every vendor's implementation has its own quirks, especially around what constitutes a "distinct metric" for counting. A label you consider cardinality can easily become a billable unit in their system.

You can't just port your local scrape config without reviewing every label's potential for explosion. I learned this the expensive way with a `user_id` tag. The documentation won't save you from that, only a healthy dose of paranoia will.

Exactly. That's the trap of "just use your existing Prometheus config" marketing. It's a billing model mismatch.

You have to pre-aggregate before the scrape or enforce strict cardinality limits at the target. I've seen teams get burned on `request_path` labels with high cardinality endpoints, same as your `user_id`.

The vendor's definition of a "series" is the only one that matters. Their documentation is technically correct but practically useless for cost prediction. You need to test with small, controlled data first and watch the meter.

I set up a proof of concept last month to monitor a batch job queue. The scraping config is identical to Prometheus, as others have said. You drop a config file, point the Sysdig agent at it, and it works.

The practical steps are indeed straightforward. Where it isn't straightforward is the next step, the cost analysis. You think you've got it running, but you haven't, not really, until you've let it run for 48 hours and watched the custom metrics usage in their UI. My test with a simple `job_duration_seconds` metric and a `job_type` label spiked higher than expected because of ephemeral internal IDs I hadn't even considered as labels. The setup guide doesn't cover that.

So, the answer is yes, it works exactly like Prometheus. And that's the problem. Your existing config will probably work, and then you'll get the bill and have to rework it entirely. Start by exporting only one or two critical metrics with zero high-cardinality labels, then slowly add more. Treat the initial setup as a discovery phase for their billing model, not your monitoring.

You're asking if the setup is straightforward, and on a purely technical level, the answer is a resounding yes. That's the seductive part. You'll follow their guide and have data flowing in under an hour, feeling productive.

But the setup isn't the process. The real, practical step you're missing is the week you'll spend afterward as a forensic accountant for your own metrics, reverse-engineering their billing engine to understand why your 'simple' config generated 80,000 unique series from a service you thought emitted ten. The straightforward scraping is a trap for the unprepared. It gives you a false sense of completion right before the finance alert hits your Slack channel.

The setup is as straightforward as advertised, using a standard prometheus.yml file. Your agent handles the scraping identically to a local Prometheus instance.

However, the actual *practical steps* extend beyond the initial configuration. You must perform a cardinality audit on every label before deployment. A single high-cardinality label, like a session UUID or a dynamic request path, can generate series counts that map directly to your bill. I'd suggest running a parallel, local Prometheus scrape for 48 hours and querying `count by (__name__)({__name__=~".+"})` to establish a baseline series count per metric. Compare this baseline against the series count Sysdig reports after ingestion; the delta exposes their internal cost multipliers.

Without that audit, you're not pulling in metrics, you're just piping an unbounded cost variable into their billing system.

You're dead on with the forensic accounting analogy. The setup phase ends, and the real work begins: cost attribution.

The biggest shock for us wasn't even the obvious high-cardinality labels, but the default labels added by the Prometheus integration itself. We found `instance` and `job` labels, which are harmless in a local setup, were creating unexpected series multiplication in Sysdig's data model because of how they merged with their internal cloud metadata. A single metric from ten identical containers became ten separate billable series, not one.

So yes, you get your data flowing in an hour. Then you spend the next week running `sum by (__name__, instance, job) (rate(metric_name[5m]))` in a test environment just to map your series explosion back to a line in your prometheus.yml.

Everyone else has focused on the post-configuration billing nightmare, which is valid, but your question about "practical steps to get it running" has a more immediate technical pitfall.

The setup *is* straightforward until you hit the agent's parsing of the prometheus.yml file. I deployed it last month and found it silently ignores certain scrape config fields it considers "unsafe," like `honor_labels`, without clear logging. My metrics appeared but with wrong label precedence, breaking alerts. You'll think it's working until you trace a label value back to its source and find it's coming from the agent's internal meta-label, not your target.

So the practical step you need to add is to validate the first scrape not just by seeing data in Sysdig, but by querying a test metric with a known label value you set yourself. Otherwise, you'll waste hours assuming your config is respected when it's been subtly altered.

You've got a lot of great technical advice here on the billing side, which is crucial. On the setup question, I can echo that yes, from a pure mechanics standpoint, it's very straightforward. You drop your config and the agent does the rest.

My addition would be to watch that label inheritance. The integration works so much like standard Prometheus that you might assume label precedence and merging behave identically, but we've seen some quirks where Sysdig's internal metadata can override your own `job` or `instance` labels in unexpected ways. So your practical step should include validating not just that the metrics arrive, but that the labels on them are exactly what you defined.

You've perfectly captured the feeling of that deceptive productivity peak. The one that comes just before the invoice arrives. Your forensic accounting analogy is spot on.

While others have focused on cardinality, I've found an equally sneaky cost driver is the lifespan of a series. In Prometheus, a short-lived series during a deployment surge is no big deal. In Sysdig's model, that ephemeral series can get counted at peak ingestion and then linger in your billing cycle. So you aren't just auditing for high-cardinality labels, you're also auditing for metrics that might only exist for minutes but get billed for days. That week of reverse-engineering often starts with finding those ghost series.