I am in the preliminary stages of architecting a monitoring and security stack for a large-scale Kubernetes environment that will scale to approximately 1000 worker nodes, hosting a mix of stateless microservices and stateful data pipelines. The shortlist for runtime security and forensics has naturally led to Sysdig, given its prominence in the CNAPP and container runtime security space. However, most case studies and vendor-provided benchmarks focus on deployments in the 50-200 node range.
My primary inquiry is for operational data from engineers managing Sysdig (specifically Sysdig Secure and Monitor) at a true 1000-node scale. The academic literature and vendor whitepapers are insufficient for a real cost/performance analysis. I require concrete, reproducible metrics to model our own deployment.
Key areas where I seek empirical data:
* **Agent Overhead:** The resource footprint of the Sysdig agent per node under sustained load. Not just "low impact," but quantifiable measurements.
* CPU millicores reserved and typical usage under normal and peak scrape/event rates.
* Memory RSS (not just allocated) for the agent pod, including any sidecars (e.g., Falco).
* Network I/O per node, particularly for policy enforcement and image scanning callbacks.
* **Backend Ingestion & Query Performance:** At 1000 nodes, the volume of metrics, events, and audit logs is substantial.
* What is your average ingest volume in GB/day or TB/day?
* Latency for complex PromQL queries over 7-day windows on high-cardinality metrics (e.g., container-level timeseries).
* Experience with the performance of the managed SaaS backend (us-east-1, eu-central-1) versus the on-premise (self-hosted) deployment during incident investigation.
* **Cost Structure at Scale:** The pricing model is opaque. Based on your production usage:
* What is your effective cost per node per month, factoring in all tiers (Monitor, Secure, cloud scanning)?
* Does the cost curve remain linear, or are there inflection points or volume discounts post ~500 nodes?
* How significant is the cost component for data retention (e.g., 90 days vs. 365 days of forensic data)?
* **Configuration Specifics:** The default out-of-the-box configuration is rarely optimal for large clusters.
* Which data streams did you find necessary to disable or sample to control costs and volume (e.g., system call events for all containers vs. only sensitive namespaces)?
* Tuning parameters for the agent `dragent.yaml` and Falco rules to reduce noise without compromising security posture.
* Any required adjustments to Kubernetes resource limits for the agent DaemonSet.
A representative snippet of our current baseline agent resource limits for comparison:
```yaml
resources:
limits:
memory: "1024Mi"
cpu: "500m"
requests:
memory: "256Mi"
cpu: "150m"
```
I intend to compile responses into a comparative benchmark against alternatives like Datadog, Elastic Agent/Fleet with the CSPM module, and a custom-built stack using OpenTelemetry collectors with Falco and Thanos. Without verifiable data from production deployments at this cardinality, any vendor selection is fundamentally speculative.
numbers don't lie.
numbers don't lie
You're right to demand real numbers. Those "low impact" claims are meaningless.
I ran it on ~800 nodes for about six months. Agent overhead was the primary reason we decommissioned it. You'll see a baseline of 50-80 millicores, but spike analysis or a security event flood pushes it past 250m consistently. Memory was worse, with the agent pod holding 120-150Mi RSS per node in steady state.
For 1000 nodes, that's a dedicated 75-100 cores and over 120Gi of RAM just for observation. That's before you pay for their platform ingestion. The cost model collapses at that scale.
If it's not a retention curve, I don't care.
Your focus on quantifiable data is exactly where this analysis needs to start, but I'd push you to expand the criteria. You're asking for CPU/RSS, which is correct, but the real cost multiplier at 1000 nodes will be the data pipeline and storage.
Even if the agent overhead is manageable, you must model the ingest and retention costs. For Monitor, you're looking at a continuous stream of metrics, events, and potentially logs; for Secure, every system call event if you enable that depth. The platform cost based on that data volume will dwarf your infrastructure overhead. Have you calculated your expected events-per-second baseline and requested a formal quote from their sales engineering with those figures? Without that, any agent resource numbers are only a fraction of the total cost picture.
CostCutter