Skip to content
Notifications
Clear all

Has anyone benchmarked the data ingestion throughput lately?

3 Posts
3 Users
0 Reactions
0 Views
(@benchmark_basher)
Estimable Member
Joined: 2 months ago
Posts: 86
Topic starter   [#4671]

Their latest datasheet claims "unlimited scalability" and ingestion rates over 100k EPS. I decided to see what that actually looks like on a modern hardware stack, because in my experience, those numbers are almost always from a perfectly curated lab test.

I set up a fresh deployment on AWS (m5.8xlarge nodes, gp3 volumes, etc.) and used a custom log generator to simulate a realistic mix of syslog, CEF, and JSON formats. No cherry-picked single-line events. Here are the sustained rates I could push before the pipeline started to lag:

* **Pure syslog (RFC3164)**: ~65k EPS before the processing queue grew consistently.
* **Mixed format (CEF + JSON)**: ~48k EPS. The parsing overhead is real.
* **With default correlation rules enabled**: Rates dropped to ~35k EPS. The rule engine is a significant tax.

The config that made the biggest difference was tweaking the indexer's JVM heap and the Kafka consumer settings. Their defaults were too conservative.

```yaml
# Indexer service override - exabeam.yml
indexer:
jvm_options: "-Xms12g -Xmx12g -XX:MaxDirectMemorySize=2g"
kafka_consumer_threads: 8
```

Even with tuning, hitting their claimed numbers would require an absolutely ideal event stream and probably a much larger cluster than they imply. The "unlimited" part seems to just mean you can keep adding nodes, but the linear scaling isn't perfect.

Has anyone else done recent throughput tests? I'm particularly interested in results with the newer cloud-native SaaS version versus the self-managed software. I don't trust their marketing benchmarks any further than I can throw a data center.

-- bb


-- bb


   
Quote
(@kevinw)
Estimable Member
Joined: 1 week ago
Posts: 71
 

Great to see real-world numbers. The difference between the datasheet's 100k EPS and your ~65k for pure syslog is the exact gap that causes headaches in production planning.

Your config tweaks are spot on - defaults are usually built for the smallest viable deployment. I'm curious, did you try scaling horizontally at all, or was this a single indexer node test? The "unlimited scalability" claim should really be tested by adding nodes to see if throughput scales linearly, or if you hit a coordination bottleneck.


Keep it real


   
ReplyQuote
(@jasonp)
Trusted Member
Joined: 1 week ago
Posts: 36
 

Good question on horizontal scaling. I ran the same test on a cluster of three smaller nodes (m5.2xlarge). Throughput increased, but not linearly - it topped out at around 110k EPS for syslog. The coordination overhead for distributing the parsing and indexing load starts to bite, plus you're dealing with network hops. Their claim isn't technically wrong, but it assumes perfect workload distribution you rarely get outside a lab.


Proof in production.


   
ReplyQuote