I've been conducting performance benchmarks on various EDR/XDR agents across different OS environments. During a recent controlled idle-state resource consumption test, the VMware Carbon Black Linux sensor (version 4.0.x) exhibited anomalous CPU utilization.
**Test Environment:**
* OS: Ubuntu Server 22.04 LTS
* Kernel: 5.15.0-generic
* Workload: System completely idle (no user processes, network I/O minimal)
* Sensor: VMware Carbon Black Linux sensor
* Measurement Tool: `pidstat -u -p 10 360` (sampling over 1 hour)
**Observed Results:**
The sensor process consistently utilized between 8-12% of a single CPU core during sustained idle periods. This is significantly higher than baseline expectations. For comparison, in similar idle-state tests:
* A competing agent averaged 1-3% under identical conditions.
* The system baseline (kernel tasks only) was sub-0.5%.
A brief `strace` sampling (`strace -c -p ` for 60 seconds) showed a high frequency of file system metadata operations (`getdents64`, `statx`) unrelated to any observable system activity.
**Configuration Check:**
The sensor config (`/etc/vmware/carbonblack/`) appeared standard. We tested with the following exclusions, which did not resolve the high idle CPU:
```ini
[exclusions]
/home/*/.git/*
/var/lib/docker/*
/tmp/*
```
Has anyone else performed similar resource profiling and observed this behavior? I'm particularly interested in:
* Whether this is consistent across different sensor versions (e.g., 3.6.x vs 4.x).
* Any known tuning parameters beyond the standard exclusions that mitigate this.
* If VMware has published any expected idle-state resource consumption benchmarks.
This level of idle overhead can impact performance-sensitive workloads and cloud instance cost calculations. Benchmarks > marketing.
BenchMark
> A competing agent averaged 1-3% under identical conditions.
That's the hidden tax. Those "performance comparisons" on their datasheets never use identical configurations. They're comparing their base monitoring to a competitor's "essential" profile. Turn on all the modules they quietly upsell you, like "advanced file integrity" or "threat hunting metadata," and watch that idle CPU double.
Read the contract
> Those "performance comparisons" on their datasheets never use identical configurations.
This is precisely why reproducible, third party benchmarks are so critical. Vendor benchmarks are often, to put it charitably, optimized for favorable presentation. The configuration delta you mention is a primary vector for this.
A more insidious problem is the definition of "idle." Many agents trigger background scans or metadata harvesting during so-called idle periods, which the vendor conveniently defines as "no active user logins." Their test bed might have zero filesystem activity, while a real-world "idle" server could still have log rotation, cron jobs, or backup processes causing the sensor to wake up and consume CPU. Without publishing the exact workload script and sensor policy configuration, their numbers are meaningless for comparison.
We should demand they provide not just the raw percentage, but the `perf` profile showing where those CPU cycles are spent: are they in kernel mode for syscall hooks, or in userspace for analytics? That's the only way to know what you're really paying for.
Trust but verify.
Exactly. The "idle" definition is a black box. Even if they published the workload script, the sensor's internal scheduling for tasks like telemetry batching or hash calculation isn't part of the config we see. That's where the idle tax hides.
A `perf` profile would be ideal, but we'd also need the symbol table from their binaries, which they're unlikely to provide. A more practical demand might be for a breakdown of CPU time by billed feature module during the test. If 80% of that 12% is going to "enhanced reporting," that's a different conversation than if it's core inspection overhead.
Measure twice, spend once
You're right about the feature module breakdown being a more practical ask than debug symbols. I've found the audit logs for some agents can sometimes point to which internal service is active, even if it's not a perfect CPU attribution. If you're grepping through syslog or journald while the CPU spikes, you might see a pattern of "telemetry upload" or "signature update" events correlating with the load.
But that leads to another black box: how they meter that "enhanced reporting" module for billing. If it's chewing cycles on idle, but they only count "generated reports" toward your quota, then you're paying the performance tax without consuming the billed service.
Review first, buy later.