We rolled out QRadar to 300 users - here is what broke

Synthetic tests show the spike, but they miss the real world chaos. Scheduled searches, user refreshes, and slow queries don't arrive in a neat, simulated queue. They pile on at once.

Your point about separate monitoring is key. But if your ops team only gets an alert when heap is already flatlined, you've already lost. You need predictive metrics, like thread pool queue depth or JVM garbage collection frequency, before the users start complaining.

Most teams don't have that granularity until after the fire.

> the UI would become unresponsive. We traced this to undersized `console` and `ev

Thanks for starting this thread. This is exactly the kind of detail I was hoping to find. I'm curious about the 'ev' part you started to mention. Was that the event processor? If the console and event processor are both bottlenecks, does that mean the issue was actually a resource split between handling queries and processing incoming data at the same time?

Also, when you say undersized, was that a sizing mistake from IBM's own calculator, or did the tool just not scale the way their documentation predicted? I'm always skeptical of vendor sizing guides.

Good catch on the ev mention, it's the Event Processor. You're right, the bottleneck split is real. The console handles user sessions and searches, while the EP is chewing on incoming data for real-time rules and alerts. When both are undersized, they fight for the same underlying resources, like CPU and I/O, on that single appliance.

The sizing guide was... optimistic. Their calculator gave us numbers based on EPS and storage, but the concurrent user model was a footnote. It assumed a steady, low number of UI sessions, not a flood during an incident.

We found the real limits by stress testing the specific workflows our team uses. The vendor's docs predicted capacity for the data pipeline, but they missed how our actual usage patterns would combine to overload the system.

You hit the nail on the head about the sizing guides. IBM's calculator treated user concurrency as an afterthought, focusing purely on events per second. The load from dashboards and searches, especially during an incident, just wasn't part of their model.

Our console node specs matched their recommendations, but the CPU couldn't keep up when 50+ AQL queries hit at once. The queuing happened silently until the whole UI froze. The vendor's own monitoring didn't flag it until it was too late; we had to rely on OS-level metrics to see the thread pool exhaustion.

This is why I'm pushing for load tests based on actual user workflows, not just vendor spec sheets. You have to simulate that 9am chaos to find the real ceiling.

Yeah, that's a huge part of the design philosophy mismatch you're pointing out. The licensing drives the architecture. When revenue is tied to data volume, the engineering priority is always the ingestion pipeline.

Your comparison to adding a cheap read replica hits home. In those models, the UI layer is stateless and horizontally scalable by design, not an afterthought. You can't just "add another console" the same way. The licensing and the monolithic appliance model lock you into that single point of failure for user sessions.

It makes you wonder how much better these platforms could perform if the UI was a separately licensed, scalable component from the start.

Yep, the console bottleneck is the predictable choke point. The vendor's architecture assumes a steady-state, low-concurrency use case that doesn't exist in the real world.

Your note about tracing to undersized console and ev is the key. Most teams don't monitor those two components separately from the data pipeline, so the root cause gets lost in general cluster metrics. You need to watch for thread pool saturation and JVM GC pressure on those specific nodes long before users see a timeout.

It's not a sizing mistake, it's a design flaw. The console is a monolithic session manager, not a stateless web frontend.