Their marketing pushes the agentless angle, but they offer an optional agent for "deeper visibility." I'm skeptical.
Has anyone done a proper, isolated benchmark on the performance impact of that agent? I'm looking for concrete numbers on:
* CPU overhead (idle and under load)
* Memory footprint
* I/O latency (especially for disk-intensive operations)
* Network throughput delta
Generic "it's lightweight" claims are useless. I need data from a controlled test, preferably on a known workload (e.g., a web server or database). If you've tested, share your methodology.
My main concern: if we run it on production database nodes, what's the real cost? A 2% hit is acceptable; 10% is not.
I ran a controlled test on a web server workload (nginx serving static files under wrk load). Here's the delta with the agent running.
* CPU overhead: ~3.5% under sustained load, <1% idle.
* Memory footprint: Added ~45 MB resident.
* I/O latency (fio random read): Negligible impact, within noise threshold (<0.5%).
* Network throughput: No measurable delta on 10Gbe.
Methodology: Isolated on a bare-metal node, agent on/off, 5 runs each, median reported. Your 2% threshold seems safe for this scenario, but database ops could be different. It's mostly sleeping until it needs to collect. The memory overhead is the main cost.
Benchmarks don't lie.
Solid methodology, and I appreciate you sharing actual numbers. Your point about database workloads is key. The agent's overhead on a web server is one thing, but its behavior during a database's I/O or locking patterns could be different.
I ran a similar test on a Postgres node doing TPC-C-like transactions. The CPU overhead was similar to yours, but I saw a 5-8% increase in query latency during peak write contention. The agent's periodic collection synced up with buffer flushes and added just enough scheduler noise to matter.
For a database, you need to benchmark under your specific load pattern, not just generic I/O. The memory overhead is fixed, but the CPU impact can become multiplicative during contention.
shift left or go home
Spot on about the scheduler noise. That's the hidden tax you don't see in average CPU charts.
We saw something similar with MySQL on high-concurrency inserts. The agent's timer going off during a group commit phase was like throwing a pebble in the gears. It wasn't about raw throughput, but it added jitter that messed with our latency SLOs.
Your 5-8% latency increase under contention is the exact kind of data point that matters for production DBs. Makes me wonder if tuning the collection interval away from flush intervals could help.