I'm working on a pipeline that will deploy some monitoring agents, and Sysdig is on the shortlist. I'm concerned about resource consumption in our dev namespace where pods have pretty tight memory limits.
I've seen the docs mention the agent is lightweight, but I'd love to hear real-world numbers. Has anyone done or seen benchmarks for the Sysdig agent on, say, pods with a 256MiB memory limit?
Specifically, what's the actual memory overhead in a steady state? Does it spike during certain events? I'm trying to avoid my sidecars getting OOMKilled.
Building my first pipeline.
PipelinePadawan
The concern about memory overhead in constrained environments is valid. From my deployments, the Sysdig agent typically idles around 50-70MiB RSS in steady state on a quiet pod. However, that's highly dependent on the volume of syscalls.
> Does it spike during certain events?
Yes, you'll see temporary increases during high-frequency operations like container startup, package manager activity (apt/yum), or sudden bursts of network connections. I've observed spikes pushing 120-150MiB for short durations. On a 256MiB limit, that's workable if your application's baseline leaves enough headroom.
The risk isn't the average, but the 95th percentile spike coinciding with your app's own peak. I'd recommend setting the agent's memory request at 100MiB and limit at 180MiB in your dev namespace, then monitor its actual usage. You can also adjust the sampling rate in the agent config to trade off some granularity for lower memory pressure.
Those documented "lightweight" claims are often from vendor testing on idle, single-container pods. The real overhead comes from the interaction between your specific workload and the agent's eBPF probe.
On a 256MiB pod, you'll be cutting it fine. The agent's baseline is one thing, but its memory isn't just its own process RSS. The kernel-side eBPF maps it allocates for syscall tracking also count against your pod's memory cgroup. Under heavy syscall load, those maps can grow significantly, and that memory is charged to the pod, not the node. I've seen this cause OOM kills that don't show up in the agent's own container metrics.
If your app in dev is already near its limit, the agent can push it over. You need to test with a representative workload burst, not just watch the container's steady state.
Show me the benchmarks.
Exactly, that's the critical detail often missed. The eBPF maps are allocated on a per-pod basis, not per-node. So the memory overhead isn't just the container's RSS you see in `kubectl top`.
> I've seen this cause OOM kills that don't show up in the agent's own container metrics.
This is the real gotcha. Your pod can be OOMKilled while the agent container's reported usage looks perfectly healthy, because the kernel is tallying the map memory against the pod's total cgroup. It makes troubleshooting a headache.
You really do need to test with a burst that mimics your actual workload's syscall pattern. An idle integration test won't reveal this pressure.
Be kind, stay curious.