Hey everyone, I'm trying to size a Cribl Stream deployment for my team's log ingestion pipeline. We're looking at AWS and considering the c5.2xlarge instance type for our workers.
From the docs, I see that throughput depends heavily on the processing complexity, source/destination types, and data format. But I'm hoping to get some real-world numbers or a baseline. Has anyone benchmarked a single worker on a c5.2xlarge? We're mostly dealing with structured JSON logs, doing some light parsing, filtering, and routing to S3.
Specifically:
- What's a realistic sustained EPS (events per second) or MB/s we can expect?
- Does enabling encryption or compression on the outputs drastically cut into that?
- Are there any specific pipeline configurations that are known to be heavy hitters on CPU or memory for this instance type?
I'm coming from a simpler syslog-forwarder setup, so trying to understand the capacity before we commit. Any example configurations or monitoring tips for tracking worker performance would be super helpful too. Thanks!
Learning by breaking
Following this closely as we're looking at c5.2xlarge too. The vendor's official calculator gave us a wide range, like 5,000 to 25,000 EPS for our use case, which wasn't that helpful.
Curious if anyone has run a simple test with raw TCP/JSON to S3 and what they got. Also, do you think the CPU credits on the burstable instance play a big role once you hit a sustained load?
not a buyer, just a nerd