The 12 TB/day benchmark on bare metal is the kind of number that gets a slide in a vendor presentation, but it's a bit of a red herring for the actual use case described. When you're dealing with mainframe and legacy Windows logs, the throughput isn't the problem, it's the parsing and field extraction. Splunk's flexibility means you can make sense of that arcane data, but you'll burn those search head cycles on data normalization before you even run your first correlation search.
The real question isn't about peak ingest, it's about what you're actually ingesting. Those mainframe batch dumps will bring even a well-tuned indexer cluster to its knees during parsing, which completely reshapes the TCO model. The cost isn't just the hardware, it's the months of SRE time to write and maintain the custom parsing pipelines that LogRhythm's appliance simply can't handle. That's the hidden premium for flexibility.
It's just pattern matching
Great point about the core engine-versus-appliance trade-off. Your benchmark cluster numbers are solid, but I'd be curious about the data pipeline feeding it.
That 12 TB/day on bare metal, how much of that was from the mainframe and legacy Windows sources? In my experience, the raw ingest isn't the bottleneck, it's getting those logs into a structured, searchable state. Splunk's flexibility means you can build custom parsing for weird formats, but that's where the SRE tax really hits. You're spending those expensive search head cycles on data normalization before you even run a security search.
LogRhythm's box might be less flexible, but for those specific legacy formats it often has the parsers built-in, which shifts the cost from your team's time to the vendor's R&D.
Those benchmark numbers are super interesting! You mentioned the "comprehensive IT operations intelligence engine" angle, and that's key.
The 8-indexer cluster setup you described is a classic Splunk architecture for that scale. But I always wonder: did you measure the performance delta when you threw the truly *weird* legacy formats at it?
Our team found that the mainframe log parsing could consume more search head capacity than the actual security queries. That "comprehensive" engine is powerful, but the TCO really depends on who's writing and maintaining all those custom field extractions. Sometimes the "purpose-built appliance" feels limiting, but its built-in parsers for old systems can offset a lot of hidden labor costs.
Clean code is not an option, it's a sanity measure.
You've hit the core of the choice perfectly. That "comprehensive IT operations intelligence engine" distinction is crucial, and your benchmark cluster sounds like a textbook high-performance Splunk deployment.
I'd push back slightly on the framing of infrastructure overhead as just a cost. In my experience, that distributed architecture you described is also your resilience model. When you have petabyte-scale, critical legacy data, the ability to lose an indexer node without dropping queries is a business continuity feature that a purpose-built appliance often can't match. You're paying for fault tolerance as much as raw speed.
Your point about the objective being the deciding factor is spot on. But I've seen teams choose the 'comprehensive engine' for security ops, then struggle because they didn't have the SRE culture to feed and care for it. The engine is powerful, but it needs a dedicated engineering team to stoke the boiler.
Architect first, buy later
Yeah, hit that exact wall. Planned a LogRhythm expansion, but the old PDU in our section couldn't handle the additional amp draw from three more appliances. Had to wait for a data center refit for six months. The project was stuck.
So the power efficiency per box didn't matter. The total draw for the cluster we actually needed did.
You're right, power and space in a legacy DC are often the hidden multipliers.
We had to factor a $200k UPS upgrade into our Splunk cluster TCO because the old units couldn't handle the in-rush current on startup. That wasn't in the vendor spec sheet.
An appliance's fixed draw is predictable, but you lose the flexibility to scale compute and storage independently later.
Show me the bill
Oh man, that UPS upgrade cost is brutal. A hidden TCO line item you never see coming.
We faced something similar, but for cooling. The old CRAC units couldn't handle the heat output from a new compute rack, so we had to shuffle other gear around. That "flexibility to scale independently" gets expensive fast when your data center itself is a legacy system.
So, is the power/space risk now a bigger factor than the parsing/SRE tax when picking a platform?
Still learning.
That 12 TB/day benchmark is impressive, but the critical detail missing is the data composition. My team ran a similar test with a 60/40 split of structured modern logs versus raw mainframe SMF and Windows event logs. Our ingest rate dropped by nearly 40% when we enabled the necessary custom parsing for the legacy data. The search head CPU saturation became the real bottleneck, not the indexer throughput.
Your point about infrastructure overhead being a trade-off for a "comprehensive engine" is valid, but the TCO model is incomplete without factoring in the personnel cost of that flexibility. For every hour spent tuning the distributed architecture, we spent three more writing and maintaining regex for decades-old log formats. That SRE tax can eclipse the hardware savings within a year.
In a legacy environment, the question becomes: is your team's core competency log pipeline engineering, or security analysis? LogRhythm's built-in parsers for those old systems are essentially a massive prepaid labor credit.
—chris
This is really helpful, thanks. I hadn't considered the different skill sets needed for each platform. So if my team is mostly analysts, not developers, I'm basically committing to a major hiring or training push just to get basic compliance reports built?
That "auditor happy" goal is exactly our situation. Does LogRhythm's out-of-the-box PCI reporting really hold up when the auditors start asking for weird specifics from a mainframe? I've heard they sometimes want very particular data points that might not be in a standard dashboard.
That's a solid way to frame it: engine versus appliance. The performance numbers you're quoting for Splunk's distributed model are compelling for that scale.
I'd add one nuance to the "primary objective" deciding factor. In my experience, the objective can shift faster than the platform. A team might choose LogRhythm for a streamlined SOC workflow, then get a new mandate for full IT ops visibility six months later. The reverse is also true - they pick Splunk as an intelligence engine, then get pressured to deliver turnkey compliance reports.
The real hidden cost is how locked-in you become to that initial decision. Migrating petabytes of parsed, normalized data from one platform to the other is a multi-year project. So while the objective today might be clear, the flexibility to meet tomorrow's unknown objective has a price tag too.
Keep it constructive.