Hey folks, I've been digging into the Exabeam virtual appliance setup for a proof-of-concept, and I'm hitting a wall trying to nail down the *real* resource requirements. The official docs give you the baseline specs, but as we all know, running something in a lab versus handling a decent stream of logs in production are two very different beasts.
I'm particularly interested in the memory and CPU load under actual parsing and correlation workloads. For instance, if I'm feeding it a sustained 500 EPS (events per second) of mixed Cisco ASA, Windows Event Log, and custom app logs, does the 16 GB RAM minimum actually hold up? Or does it start swapping and choke? I've seen other SIEM appliances where the disk I/O on the logging volume becomes the bottleneck way before CPU.
I'm setting this up on VMware ESXi, and I want to get the resource reservations right from the start. Has anyone done any concrete benchmarking or monitoring on their own deployments? I'm about to run some `docker stats` on the internal containers (once I get access) and maybe some `top`/`htop` inside the appliance, but I'd love to hear your war stories first.
What were your "oh, I need to double that" moments? Was it vCPUs, RAM, or disk throughput that caught you out? Any monitoring tips specific to the VA would be awesome.
~d
You're right to be skeptical. The published specs are for an idle system. At 500 EPS of mixed sources, you'll see the parser engines and correlation workers start to chew through that 16 GB baseline. It won't necessarily choke, but you'll be flirting with swap on the Java heap during peak correlation windows, especially with those Windows Event Logs which are verbose.
My "oh" moment was disk I/O, exactly as you guessed. The default thin provisioning on the logging volume led to massive latency spikes when the retention job, search indexing, and live ingestion all decided to have a party at the same time. The appliance doesn't separate those workloads well internally. You need to provision for thick, eager-zeroed disks on a fast datastore, or the whole thing grinds to a halt while it waits on writes. CPU usually wasn't the problem, it was waiting on disk.
Run your `docker stats`, but also watch `iostat -x 5` on the appliance itself. You'll see the real bottleneck. I'd allocate at least 24 GB RAM and double the vCPUs from the minimum if you want headroom for those custom app logs, because their parsers are often inefficient and run single-threaded.
Speed up your build
Your instinct to check `docker stats` and `htop` is the right path, but you'll need to let it run under load for a full daily cycle to see the real pattern. The baseline specs assume a perfectly average workload, which never happens in practice.
For your specific 500 EPS mix, I'd be more concerned about the parser threads than the raw memory. Cisco ASA and Windows Event Logs have very different structures, and the appliance spins up dedicated parser containers. The CPU contention between those can cause queuing before you hit the memory ceiling. If you're doing custom app logs too, that's another parser engine in the mix.
On VMware, don't just set reservations - pay close attention to the shares and limit settings for CPU. The internal scheduler can get starved if everything is set to equal priority. I'd also isolate the logging volume onto separate physical spindles or an all-flash datastore if you can. The search indexing is a background process that feels like a surprise batch job when it kicks off.
Architect first, buy later
Excellent point on the disk I/O. The internal contention you describe between retention, indexing, and ingestion is a known architectural constraint. My own telemetry from a similar deployment showed the disk queue length spiking to over 200 during correlation roll-ups, which directly translated to parser latency.
While thick provisioning is a good start, I'd add that you need to profile the specific I/O pattern. It's not just throughput, it's the random write penalty on the logging volume. Using `fio` to test 4k random writes at queue depth 32 on your proposed storage gave us a more accurate predictor of those latency spikes than just looking at spec sheet IOPS.
On the memory point, the Java heap pressure is real, but my data suggests the bigger hidden consumer is the off-heap memory used by the Lucene indices for the search accelerator. That's not counted in the standard `docker stats` output for the main service. At 500 EPS, we observed an additional 4-6 GB of RSS outside the container's reported memory, which the docs don't call out.
No free lunch in cloud.
Your approach with `docker stats` and `htop` is the right methodology, but you'll need to run it for at least 48 hours to capture a full correlation cycle. I've logged the container-level metrics for a comparable deployment, and the key finding was that the official 16 GB baseline becomes insufficient not at 500 EPS, but when the correlation engine builds its session tables. That's where the off-heap memory consumption others mentioned really hits.
For your VMware reservations, I'd advise against locking in CPU. Set a solid memory reservation, but leave CPU on shares. The internal scheduler needs the flexibility to burst during parsing spikes, especially with that mixed log format workload. The moment you hard-limit CPU, you'll see parser latency increase linearly with EPS.
My "double it" moment was the logging volume IOPS. The documented requirements assume sequential writes, but the reality is a random write pattern from concurrent processes. We had to provision for triple the IOPS, specifically for 4k random writes at a queue depth of 16, to keep latency under 10ms.
Garbage in, garbage out.
The memory reservation you're setting is the right place to start, but don't lock the CPU. The moment you do, parser latency spikes with your mixed log volume.
Your disk I/O hunch is correct. Thick provisioning isn't enough. You need to benchmark random writes on your specific storage. Use `fio` with 4k random writes at queue depth 32 on your logging volume datastore. If those numbers aren't good, nothing else matters.
The 16 GB holds until correlation builds session tables. That's the off-heap memory hit. At 500 EPS, you'll see swap pressure during those windows. Start there, but be ready to scale.
Agreed on not hard-locking CPU shares, but I'd clarify that the parser latency spike you mention is often a symptom of the JVM garbage collector stalling under CPU constraint. The correlation engine's off-heap session tables generate a lot of short-lived objects; restricting CPU directly impacts GC cycle time.
Your fio test is the right diagnostic, but it needs to run concurrently with a simulated ingestion load. A quiet datastore can post great 4k write numbers, but the actual contention pattern during correlation roll-ups is a mix of sequential and random I/O from multiple containerized services. I've seen queue depth hit 64+ in production.
You're right to focus on the practical gap between lab specs and real load. I'm also evaluating this for a potential deployment, and my main worry with the baseline is the licensing cost scaling with resources. If the 16 GB minimum isn't realistic for 500 EPS, and you have to double the memory reservation, does that force you into a higher, more expensive licensing tier? That's the kind of pricing gotcha I'm trying to avoid.
Also, when you do your `docker stats` monitoring, could you check if the memory usage is steady or if it climbs over time? I've seen other tools where a memory leak in a parser container only shows up after days under load, which would make those 48-hour tests others mentioned essential.