Hello everyone. I've been digging into a persistent issue with our BeyondTrust Password Safe deployment and wanted to share my findings, as well as see if others have encountered similar patterns. We've been struggling with scheduled password changes for service accounts (Windows and SQL primarily) failing intermittently, but with increasing frequency. The failures aren't consistent—they work for weeks, then a batch will fail, often with cryptic or unhelpful error messages from the BeyondTrust platform itself.
Our initial triage pointed to the obvious culprits: network connectivity, permissions, or password complexity policies. We've methodically ruled those out. The managed hosts are reachable, the rotating account has the required administrative privileges (we even tried domain admin as a test), and the new passwords generated meet the domain's historical complexity requirements.
What's more interesting is the pattern we observed in the logs. The failures often correlate with:
* **Concurrency issues:** When multiple rotations are scheduled for the same managed host in a tight window, the later ones seem to fail more often.
* **Session or handle exhaustion:** We see occasional `Win32 Error 1053` or timeouts that suggest the remote procedure call doesn't complete.
* **Silent failures on verification:** The rotation job reports success, but the subsequent verification step (where the vault account attempts to log in with the new credential) fails, leaving the password in an unknown state. This is the most dangerous scenario.
We've developed a runbook for our on-call to handle these failures, but it's a band-aid. The core issue seems to be in the interaction between the Password Safe rotation engine and the target systems under load. Here's a snippet of the enhanced logging we enabled in our rotation scripts to capture more context, which has been invaluable:
```powershell
# Wrapper snippet for Windows account rotation
Try {
$changeResult = Invoke-BCPasswordRotation -Target $hostname -Account $svcAccount
# Log full result object, not just status
Write-Log -Level INFO -Message "Rotation attempt metadata: $($changeResult | ConvertTo-Json -Depth 3)"
} Catch [Exception] {
Write-Log -Level ERROR -Message "Rotation exception: $($_.Exception.Message)"
Write-Log -Level DEBUG -Message "Full exception: $($_.Exception | Format-List * -Force | Out-String)"
}
```
Has anyone else performed a deep-dive postmortem on similar flaky rotations? Specifically:
1. Did you find tuning parameters within Password Safe (like timeouts, retry intervals, or concurrent job limits) that improved stability?
2. Have you moved to a different model for service account rotation (like using Group Managed Service Accounts where possible) and only using Password Safe as a credential store?
3. Are there known issues with specific versions of the Password Safe remote agent that we should be aware of?
The error budgets for our core services are taking a hit because of these silent credential failures, and we're considering whether this is an architectural limitation or a configuration gap. Any shared experiences or runbooks would be greatly appreciated.
pagerduty certified lifer