Alright, I need to vent and get some eyes on this. We've been rolling out BeyondTrust's Privilege Management for Unix & Linux (PMUL) and their session recording for compliance. The privilege escalation part works fine, but the session recording is a dumpster fire. It fails silently about 40% of the time on our RHEL 8 and Ubuntu 20.04 boxes, leaving us with gaps in the audit trail. Security team is rightfully pissed.
We followed the deployment guide, but the documentation feels like it's written for a perfect lab environment. Our real-world estate has a mix of on-prem and AWS EC2 instances. The failures seem random, but there might be a pattern under heavy load or with specific types of interactive sessions (like `su -` or direct SSH to a shared service account).
Here's our typical `beyondtrust.conf` on a Linux host, anonymized. We're using the central policy server model.
```ini
[SessionRecording]
Enabled = true
PolicyServerHost = beyondtrust-policy.internal.domain.com
PolicyServerPort = 8555
LogLevel = INFO
StoragePath = /var/lib/beyondtrust/sessionrecordings
```
The syslog entries (`/var/log/messages` or `journalctl`) aren't super helpful. We see a lot of these:
```
beyondtrust-sr: [12345] Failed to allocate session buffer.
beyondtrust-sr: [12345] Policy server connection timeout.
```
Things we've tried, with zero success:
* Increasing the `ulimit` values (`nofile`, `memlock`) for the beyondtrust user.
* Doubling the local buffer size in the advanced config snippet they provided.
* Whitelisting the policy server FQDN and IP in the host firewall (even though other components talk fine).
* Deploying a newer agent version (from 6.0.1 to 6.0.3).
My main questions for anyone else in the trenches:
1. Is the session recording fundamentally fragile, or did we misconfigure something obvious?
2. Are there specific kernel parameters (`vm.max_map_count`, `kernel.pty.max`) that are critical but not in the main install guide?
3. Has anyone built a monitoring solution around this? We're thinking of scraping the agent's local status port with Prometheus and alerting on "session recording disabled" states, but that's a band-aid.
4. The storage path fills up with corrupted `.sr` files sometimes. Any scripts you use to clean and alert on that?
I'll share our Terraform snippet for deploying the agent config if it helps, but right now I'm more interested in the operational stability. This feels like a product that passed QA on three clean VMs but falls over at scale.
Automate everything. Twice.
Ugh, silent failures are the worst for compliance. The logs cutting off at "beyondtru" are a big red flag. I'd bet it's a buffer or I/O issue on the client side when shipping data to that central PolicyServerHost.
Have you checked the disk space and inode usage on your `/var/lib/beyondtrust/sessionrecordings` path across the fleet? Also, on a busy EC2 instance, the process might be getting starved for resources. You could try increasing the LogLevel to DEBUG temporarily on a test box to see if it's timing out on the network call.
Good points, especially about the EC2 resource starvation. The session recorder process gets a really low OOM score by default, so it's often first to get killed under memory pressure. Check your kernel logs for OOM kills around the failure times.
We also had intermittent issues that traced back to the NFS mount holding /var/lib/beyondtrust. The client would lose the connection during a session write and just die. Switched to local SSD with a cron job to sync out, and the failure rate dropped.
Proof in production.
The central policy server model is your first mistake for anything time-sensitive. That single point of failure and network hop is murder on consistency, especially in a hybrid setup.
> heavy load or with specific types of interactive sessions
Your failure rate tracks with our early tests. It's the burst I/O from recording a fast `su -` or `sudo vim` that the local buffer can't flush to the network in time. The process hits an internal timeout and bails. Silent failure by design.
You're running this on general-purpose EC2 instances, aren't you? The baseline I/O performance on a `t3.medium` is a joke for constant streaming. You need either bigger instances or, better yet, local storage with async uploads. Our silent failure rate went from ~35% to near-zero when we stopped trying to stream live over the network for every keystroke.
show the math
Oh man, that 40% silent failure rate would give our security team a meltdown too. Your config looks identical to how we started, and we hit the same wall.
The key for us wasn't the storage path, but the fact that the default buffer for shipping data is way too small for real-world terminal activity. When a user pastes a huge config or runs something interactive, that buffer fills instantly and the client just... gives up.
We added a local buffer file directive to the config as a stopgap, but honestly, user400 is onto something about the central server being a bottleneck. We only got reliable after moving to a regional collector model.
good UX is non-negotiable
Checking the OOM score is a really good call. We had a similar issue with a different agent on t3 instances, and the OOM killer logs weren't super obvious until we knew what to grep for.
Your NFS point makes me wonder about latency in general. Even on EBS volumes, the I/O wait could cause a timeout if the central server is busy. Maybe the cron sync is the real fix, not just the SSD itself.