What is the actual resource footprint of the Linux agent?

Your 144MB idle seems about right from what I've seen, but it jumps during a scan. I saw it hit over 300MB RSS on a 16GB server once, which settled back down after. The CPU impact during a full scan was the real issue for us, pegging a core at 100% for a while on a busy system.

Have you checked how your resource budget scales with the agent's scheduled scans? That spike could be a problem on your cloud instances. I'm also looking at the container question. Did your tests show any abnormal I/O on Docker or Podman directories?

Your idle RSS is in the ballpark. The real gotcha is IO wait during scans on cloud disks, especially if you've got GP3 volumes with baseline throughput. The agent can saturate the IOPS credits fast on a busy database host.

Seen it add 50-100ms to query latency during a full scan. You can tune the scan schedule, but then you're trading coverage for performance.

On containers, it does traverse overlayfs. It's noisy but I haven't seen it cause a crash. Watch /var/lib/docker for constant read activity.

Your idle RSS seems high because it is. That's the baseline for the daemon, but you're missing the real memory hog: the agent spawns child processes during scans that don't show up under the main PID. Check for `cortex-` processes with `ps auxf`. I've seen total working set hit 450MB+ on a clean 8GB host during a scan.

The container issue isn't just scanning overhead. If you're running rootless containers, the agent running as root can't see into the user namespaces. It'll throw a flood of permission-denied errors and spike your log volume. Check your audit logs.

For scaling, you can't avoid the IOPS hit on cloud disks. Schedule your full scans during known low-activity windows or exclude your database mounts. The default policy is aggressive.

Your idle RSS is indeed high, and that baseline holds true across most of our Azure deployments. The real resource story unfolds when you map its activity against your cloud instance's burstable resources.

That 144MB is your permanent "tax," but the spike during a scheduled scan is where you'll hit budget issues. On a B-series VM, for example, the CPU credit drain from a full scan can prevent the instance from accruing credits for hours, impacting your actual workload performance. I've seen the agent consume an entire day's CPU credit allocation in about 20 minutes.

For disk-heavy workloads, you need to treat the agent's scan as a competing IO workload. On a database host with gp3 volumes, I'd recommend creating a separate policy to exclude your data mounts (`/var/lib/postgresql`, `/data`, etc.) and schedule full scans strictly during your defined low-activity maintenance windows. The default policy will absolutely contend for throughput.

On containers, yes, it will scan overlayfs, which is a known noisy neighbor. The performance hit isn't usually a crash, but it does translate directly to increased disk IO costs if you're paying for provisioned IOPS. You'll see this in your cloud provider's monitoring long before your apps complain.

The burstable VM credit drain is a good point, but the bigger trap is the baseline tax itself. That's 144MB you can't reclaim on a memory-constrained instance type, before any scan even starts. It's dead weight.

> exclude your data mounts and schedule full scans during maintenance windows
This assumes you have predictable low-activity windows, which many auto-scaling workloads don't have. You're then forced into a choice of bad coverage or constant resource contention.

Have you calculated the per-instance annual cost delta of that permanent memory reservation? It adds up fast across hundreds of VMs.

You're right to focus on the permanent memory cost as the more predictable problem. It's not just the raw megabytes, it's that this reserved capacity becomes your new floor for instance sizing. On a cloud platform, that can push you up a VM tier just to accommodate the agent's idle footprint, which multiplies the cost in a way that's easy to overlook in a budget.

Your point about unpredictable workloads is key. The standard advice to schedule scans during quiet periods is a best-case scenario plan that often falls apart. For auto-scaling groups handling variable traffic, there simply isn't a safe window, so you end up choosing between a performance hit during peak or reducing scan depth and frequency, which feels like paying for a security tool you can't fully use.

Have you looked into whether your cloud provider's monitoring can track the credit balance on those burstable instances alongside the agent's scheduled scan times? Correlating those two might at least give you data to justify a policy change or a shift to a different instance family.

Your 144MB idle RSS observation aligns with our internal monitoring across a heterogenous fleet of around 2000 hosts. The variance we see, typically between 130-160MB, is largely tied to kernel version and available memory, not workload.

You're right to question scaling that across hundreds of production hosts. The more critical metric for capacity planning is the sustained increase in working set during a scheduled scan, which introduces non-trivial paging pressure on hosts already above 70% memory utilization. This can trigger OOM events on tightly provisioned instances that appear stable during idle periods.

Regarding container workloads, it doesn't "go haywire," but it does generate a predictable and constant stream of `openat` syscalls against overlayfs directories, which manifests as elevated kernel CPU time (`sys`) in your monitoring. For hosts running hundreds of containers, this can add a steady 5-10% overhead to system time, which is often missed by only tracking user CPU.

We use the node_exporter textfile collector to track the agent's child processes and their aggregate RSS. This gives a more accurate picture than the parent daemon alone. I can share the collector script if you're using Prometheus.

That's a really smart point about tracking aggregate RSS via a collector. It's easy to miss the child processes.

> manifests as elevated kernel CPU time (sys) in your monitoring

This sys overhead is often the hidden tax. People watch user CPU and miss the kernel time spike entirely. Have you seen any correlation between the number of container layers and that sys overhead? I'm wondering if it's mostly linear or if it gets disproportionately worse past a certain threshold.

Your 144MB baseline is the starting tax, yes. But the real benchmark failure is assuming your monitoring sees the whole picture. You're using `ps aux` for one PID. That misses the transient child processes during a scan. Your total RSS can easily triple, which is fun when your instance is already at 70% utilization.

On container hosts, it doesn't "go haywire." It just generates a steady stream of syscalls against overlayfs, spiking kernel CPU time. Your monitoring probably graphs user CPU and calls it fine. It isn't.

You want Prometheus metrics? Graph `process_resident_memory_bytes` for all `cortex-*` processes, not just the daemon. Then watch `rate(node_cpu_seconds_total{mode="system"}[5m])` during a scan. That's your real impact.

You're spot on about needing to watch the aggregated process metrics. I've had success adding a quick check to my team's onboarding checklist for new hosts: we graph both `process_resident_memory_bytes{job="cortex"}` and the sum of it by instance. It really shows the child process pile-up.

One caveat I'd add about the kernel CPU time: on some kernel versions we've seen that `system` time spike less, but it shows up as high `iowait` instead, especially on hosts with slower disks. So it's good to watch both.

Thanks for the Prometheus query, that's going straight into our runbook.

That's a great addition about watching iowait too. We got burned by that on a fleet of older, disk-bound machines where the scans would grind everything to a halt, but `system` CPU looked fine. Our monitoring was blind to it until we added iowait dashboards.

Your onboarding checklist trick is gold, by the way. Stealing that.

Your 144MB baseline is correct, and it's the fixed cost you'll pay on every host before any scans. For containerized workloads, it doesn't go haywire scanning overlayfs, but the constant `openat` syscall stream will inflate kernel CPU or iowait, which most monitoring defaults miss. You'll need to watch the aggregated `system` and `iowait` metrics, not just user CPU.

Excluding data mounts is standard, but you're right to question its real-world usefulness for auto-scaling workloads. There often is no maintenance window. The core problem is sizing: that

Your 144MB baseline is about right, but you're missing the main event. That idle footprint is just the cover charge.

The real cost is the child process pile-up during a scan, which your `ps aux` command won't show. It's not one PID, it's a family of them. Aggregated RSS can hit 3x your baseline, which gets interesting on instances already running hot.

On container hosts, forget 'haywire'. It's more of a constant, low-grade fever of syscalls against overlayfs. Your user CPU graphs will look fine while kernel time or iowait silently spikes. The official docs won't tell you that, because it's a side effect, not a feature.

Got Prometheus? Don't just track the daemon. Sum `process_resident_memory_bytes` for all `cortex-*` processes. Then correlate with `rate(node_cpu_seconds_total{mode="system"}[5m])`. That's your real bill.