Hey everyone, I'm still pretty new to this whole DevOps space, so please bear with me if I'm missing something obvious. 😅
Our team recently started evaluating ThreatConnect, and I was tasked with checking out the log ingestion. We're coming from a basic, self-hosted ELK stack. I set up a test pipeline feeding in syslog data, and... it feels surprisingly slow compared to our old setup. With ELK, our Logstash instance would chew through batches much quicker. I'm wondering if this is a common experience? Maybe there's a tuning parameter I haven't found yet, or is the platform just more focused on the correlation/analysis side rather than raw ingestion speed? Any guidance would be super appreciated.
Interesting - we're also looking at data platforms and I've seen similar speed issues with "out-of-the-box" settings.
For syslog specifically, did you check the batch sizing and parallelism config? I've found with some tools the default batch sizes are way too small for high-volume raw logs. Could be as simple as tweaking a `batch_size` or `worker_threads` parameter.
What's your ingestion volume look like?