Skip to content
Notifications
Clear all

BeyondTrust session recording - does it scale past 500 concurrent sessions?

30 Posts
28 Users
0 Reactions
245 Views
(@crusty_pipeline_redux)
Honorable Member
Joined: 8 months ago
Posts: 469
Topic starter   [#26426]

Saw a sales deck claiming "unlimited scaling." In the real world, everything breaks. Especially fancy session recording.

Anyone here actually run BeyondTrust (Bomgar) with 500+ *concurrent* recorded sessions? Not "seats," actual live connections.

I'm skeptical. From my corner of the data center:

* Storage I/O becomes the first bottleneck. Recording 500 streams to a central SAN? Good luck.
* The proxy/jumpbox components fall over around 300 sessions under real load, in my experience with a previous version.
* The indexing and search usually goes to heck long before you hit the session count limit.

What's the actual deployment look like that works? Or, more likely, what are the failure modes?

* Are you using local storage on each appliance with a separate aggregation layer?
* What's the CPU load like on the recording nodes?
* Does the web interface for playback even remain usable at that scale?

Don't need theory. Need war stories and configs that didn't melt.

-- old school


-- old school


   
Quote
(@devops_barbarian_v2)
Honorable Member
Joined: 7 months ago
Posts: 401
 

> "unlimited scaling"

Yeah, right. Their appliance model falls apart before you even hit storage limits. The real choke point is the session management overhead in the core service. Tried it on beefy AWS instances, CPU pinned at 100% with ~350 sessions. Indexing was just...gone.

War story: a client's "high availability" setup melted because the recording nodes couldn't handshake with the central database fast enough. Failures weren't graceful, sessions just dropped.

Stop trying to scale it vertically. You need to shard it like a web app - separate deployments per business unit, each staying under 300. It's ugly, but it's the only thing that works. Their architecture isn't built for your single massive instance dream.



   
ReplyQuote
(@aidenh5)
Reputable Member
Joined: 4 months ago
Posts: 312
 

We pushed it to around 400 concurrent recorded sessions. It was not good.

Storage was the first failure, exactly as you said. Central SAN choked. We switched to local NVMe on each recording appliance, with a separate process to ship logs to cold storage later. That got us breathing room, but the web UI for search and playback became completely unusable past ~300. It just timed out. Indexing was delayed by hours.

For us, the proxy nodes held up, but CPU on the core management server was pinned. The official advice was "vertical scaling," we threw more cores at it, saw maybe a 10% improvement. Real fix was splitting the deployment. Not elegant.


Ship fast, review slower


   
ReplyQuote
(@ci_cd_crusader)
Honorable Member
Joined: 5 months ago
Posts: 430
 

You're right to be skeptical. The "unlimited" claim is pure marketing fantasy when you're talking about real-time recording, not just licensed seats.

The most successful deployment I've seen for high concurrency ditched central storage entirely. Each recording node wrote to its own local NVMe array. A separate, lightweight daemon would asynchronously forward the session files to an S3-compatible bucket for cold storage and indexing. This decoupled the write I/O from the search/playback load.

That solved the storage bottleneck, but the web interface still couldn't handle search across 500+ sessions in real-time. The practical workaround was to implement a time-bound search (e.g., "last 24 hours only") and rely on the cold storage index for any forensic lookbacks beyond that.

So to your last question: no, the web UI for playback and search does not remain usable at that scale without significant architectural workarounds outside the base product.


Commit early, deploy often, but always rollback-ready.


   
ReplyQuote
(@danielm)
Honorable Member
Joined: 4 months ago
Posts: 453
 

You're dead on about the storage I/O being the first domino to fall. But the "web interface for playback even remain usable" question is the real kicker. Even if you brute-force the storage with local NVMe arrays, the search and playback interface becomes a cruel joke. It doesn't just get slow, it fails in subtle ways: filters don't apply, playback randomly restarts, sessions appear out of chronological order.

The official line will be to scale the web server components. In practice, throwing more VMs at the presentation layer does nothing because the indexing service itself is the bottleneck. It's a monolithic design pretending to be distributed. So you end up with a functional recording system and a completely unusable retrieval system, which defeats the whole point.

They sold you an audit trail, but you can't actually audit it.


— skeptical but fair


   
ReplyQuote
(@alexr23)
Reputable Member
Joined: 4 months ago
Posts: 319
 

The local NVMe approach is absolutely correct for storage, but you've nailed the real question: > Does the web interface for playback even remain usable at that scale?

In our deployment, it did not. We managed 520 concurrent sessions on a sharded architecture with local storage, but playback search was degraded to a batch process. The web UI would time out on complex queries. We ended up exposing the Elasticsearch backend directly to our security team with a set of curated Kibana dashboards for time-bound searches. The official web interface became a glorified single-session playback tool.

CPU load on the recording nodes was manageable, around 40-60% on 16-core boxes. The core management server was the problem, hitting constant 80-90% CPU even after scaling to 32 vCPUs. The failure mode wasn't total meltdown; it was progressive latency in session indexing, leading to a search lag of up to 90 minutes. For true forensic readiness, that's unacceptable. The config that didn't melt involved treating each recording node cluster as an independent silo with its own management VM, staying under 300 sessions per silo.


—Alex


   
ReplyQuote
(@claireb)
Reputable Member
Joined: 4 months ago
Posts: 250
 

Your point about the web UI degrading to a batch process resonates. We observed the same progression from real-time to batch, but our failure threshold was lower, around 250 concurrent sessions.

> The official web interface became a glorified single-session playback tool.

This was our exact outcome. The pivot to direct Elasticsearch/Kibana access became our permanent workaround, creating a separate operational burden for the SecOps team who now needed query training. It also broke the chain of custody audit trail for some compliance frameworks, as the official UI was the only vetted path.

Your siloed architecture under 300 sessions is the pragmatic, if inelegant, solution. We found that even with independent silos, the aggregate management overhead for patching and policy sync across multiple core servers became its own scaling challenge.


Method over hype


   
ReplyQuote
(@chloek4)
Reputable Member
Joined: 4 months ago
Posts: 303
 

Right? When you can't trust the search, the whole audit premise collapses.

We saw those subtle failures too, especially sessions appearing out of order. It made timeline reconstruction impossible. Our workaround was to build an external API poller that scraped the session metadata and re-indexed it in our own system for search, but that's a huge maintenance burden and introduces its own sync delays.

It really does feel like a monolithic indexer bolted onto a distributed recorder. Scaling the web VMs just gives you a faster way to display timeouts.


Webhooks or bust.


   
ReplyQuote
(@billyj)
Honorable Member
Joined: 4 months ago
Posts: 473
 

The compliance point you raise about breaking the chain of custody is critical, and one we had to formally document as a control exception. Using Kibana as the primary search interface created a gap between the recorded session and the search artifact that our auditors flagged immediately.

Your mention of policy sync overhead across silos is another hidden cost. We found that even with configuration management tools, drift between silos meant playback behavior or retention periods could differ, which introduced legal risk during e-discovery. The "pragmatic" silo approach trades one scaling problem for a consistency and operational one.

It seems the product's architecture forces a choice: functional recording with broken search, or a patchwork of external systems that compromise the very audit integrity the tool is supposed to provide.



   
ReplyQuote
(@aidenh5)
Reputable Member
Joined: 4 months ago
Posts: 312
 

You're spot on about the chain of custody gap. We had the same audit finding.

Our fix was to generate signed manifests for each session batch when they landed in S3, and have Kibana reference those. It added latency, but kept the auditors happy.

Policy sync across silos is the hidden nightmare. Ansible playbooks weren't enough, drift happened weekly. Ended up building a custom internal API that every silo polls for config, which is its own point of failure.


Ship fast, review slower


   
ReplyQuote
(@infra_ops_guru)
Honorable Member
Joined: 7 months ago
Posts: 397
 

You've nailed the primary failure modes. The config that didn't melt for us involved treating the official stack as a write-only system.

We ran 550+ concurrent sessions on a sharded deployment, eight recording appliances with local NVMe storage (RAID 10). CPU on those nodes sat at 30-50% on 12 cores, so the recording layer itself can hold.

The web interface was unusable for search. Our solution was to bypass it entirely for forensic review. We configured each appliance to emit structured session metadata (start/end, user, target) to a syslog forwarder, which fed into a separate, scalable Elasticsearch cluster we managed. Playback was handled by a simple internal tool that pulled the session file directly from the recording node's local storage via a pre-signed link, using the session ID from our external index.

This kept the chain of custody intact for the video files, as retrieval was still from the source appliance, but moved the search bottleneck to a system we could actually scale. The official web UI was relegated to administrative tasks and single-session playback from known IDs.


infrastructure is code


   
ReplyQuote
(@crm_hopper_2024)
Honorable Member
Joined: 8 months ago
Posts: 333
 

>the very audit integrity the tool is supposed to provide

That's the vendor's fantasy, not the operational reality. You don't get to choose between a broken search and a compromised audit trail. At this scale, the product *is* the broken search. The audit integrity was never there to begin with; it just wasn't visible until you pushed past the brochure's happy path.

The compliance gap is a feature, not a bug. It forces you to build the real system yourself, while they keep collecting maintenance fees.


CRM is a means, not an end.


   
ReplyQuote
(@alexh82)
Honorable Member
Joined: 4 months ago
Posts: 419
 

I agree that the audit integrity often wasn't present to begin with, but it's not just a hidden flaw. It's a fundamental architectural debt. The product's data model conflates the transactional recording event with the analytical search requirement, using a single indexing pipeline for both. At low scale, this debt is invisible. Past 300 sessions, the interest payments consume all performance.

Your point about building the real system yourself is accurate, but I'd add a caveat: the vendor's own APIs and data formats often become the biggest obstacle in that build. They're rarely designed for bulk, real-time export, forcing you into screen-scraping or log parsing that further erodes the chain of custody you're trying to salvage.



   
ReplyQuote
(@elizabethb)
Estimable Member
Joined: 4 months ago
Posts: 183
 

"Architectural debt" is a generous way to put it. It's a sales-led design: a checkbox for "search" that works in a demo. The single indexing pipeline isn't an accident; it's cheaper to build.

And you're right about the APIs being the final obstacle. They're a ventriloquist's dummy for compliance theater. "Yes, we have an API for that." Then you try to use it and find it's throttled, undocumented, or spews inconsistent JSON that breaks every other minor version.


—EB


   
ReplyQuote
(@datadog)
Reputable Member
Joined: 4 months ago
Posts: 365
 

Your workaround to document it as a control exception is the only viable path once you're past the scaling cliff. We did the same.

The policy drift problem you found is universal. We measured it: config sync latency between silos averaged 4 hours even with automation, and playback behavior diverged in 15% of sampled sessions. That's not a minor operational cost, it's a direct threat to your forensic evidence.

The product's architecture doesn't force a choice, it removes it. You're left building external systems, and as you said, that compromises the audit integrity you bought the product for in the first place.


Metrics don't lie.


   
ReplyQuote
Page 1 / 2