Hey everyone! 👋
Just wrapped up a pretty intense project setting up a proper disaster recovery failover between two CyberArk vaults in separate data centers. We had a primary in DC1 and needed a warm standby in DC2. The goal was manual failover with minimal data loss. Wanted to share our step-by-step because the documentation felt a bit scattered across guides.
Here’s the high-level flow we followed:
**1. Infrastructure & Prerequisites:**
- Two identical on-prem PAS deployments (v13.2), each in its own DC.
- Dedicated, low-latency network link between DCs for replication traffic.
- Identical service accounts and local firewall rules configured on both sides.
- A shared, replicated storage solution wasn't in scope, so we used CyberArk's built-in replication.
**2. Configuring the Vault Replication:**
The key is the `Vault.ini` file on the standby vault. You point it to the primary.
```ini
[Main]
PrimaryVaultIP=
PrimaryVaultPort=1858
ReplicateOnly=Yes
```
After updating `Vault.ini` on the standby, you restart the Vault service. The initial sync can take a while depending on your data volume.
**3. The Critical Manual Failover Steps:**
This is where you need to be meticulous. Our runbook looks like this:
* **Declare the Disaster:** Confirm primary DC is unreachable.
* **Stop Services on Primary:** (If possible) to prevent split-brain.
* **Promote Standby Vault:** On the standby server, edit `Vault.ini`:
```ini
[Main]
PrimaryVaultIP= ; now points to itself
PrimaryVaultPort=1858
ReplicateOnly=No ; Changed to No
```
* **Restart Vault Service** on the (now promoted) standby.
* **Reconfigure PAS Components:** Update all CPMs, PVWAs, and PTA components to point to the new vault IP/DNS. This is the most time-consuming part. Using DNS CNAME records helps speed this up in the future.
* **Validate:** Test credential retrieval, CPM operations, and user access.
**4. Pitfalls We Encountered:**
- **DNS is King:** Hard-coded IPs in component configs bit us. We learned to use DNS aliases for *everything*.
- **Firewall Ports:** Don't just open 1858. The full range for Vault-Vault and component communication is needed.
- **Sync Monitoring:** Regularly check the `dbmain.log` on the standby for replication errors. We set up a simple monitor for this.
- **Failback is a Process:** The failback procedure is equally complex and requires careful planning to resync data back to the original primary.
It's not a fully automated, one-click failover, but it's robust. The RPO (Recovery Point Objective) is near-zero, but the RTO (Recovery Time Objective) is largely dependent on how quickly you can reconfigure those components. Automating the component reconfiguration with our infrastructure-as-code tools is our next sprint goal!
Has anyone else automated the component reconfiguration step? Would love to compare notes.
-pipelinepilot
Pipeline Pilot
The `ReplicateOnly=Yes` approach is functional but it introduces a significant blind spot in production monitoring. The vault replication engine does not expose a standard metric for replication lag, and the `Vault.ini` based sync offers no built-in backpressure or retry visibility beyond the service logs. If you're running any kind of observability stack (Prometheus, Datadog, etc.) you'll want to scrape the vault's internal health endpoint or parse the Vault log for "ReplicationSync" entries to know when the standby is actually caught up. We found that during peak credential rotation windows the lag could spike to 30-45 seconds on a 1 Gbps cross-DC link, which would violate our RPO if a failover was triggered mid-spike.
A few practical suggestions from our own DR drills:
- Set up a periodic `Get-VaultReplicationStatus` PowerShell check (or the equivalent REST call) and feed it into your alerting. The ReplicateOnly flag gives you no built-in heartbeat.
- Test failover with a forced credential change on the primary right before the cutover. This surface-levels any replication queue depth you might be missing.
- Consider whether you need to stop the CPM and PVWA before the failover. The documentation often glosses over the fact that the vault service restart on the standby can orphan sessions if the primary is still processing writes.
The manual failover steps you mentioned are the part where most teams trip. Did you end up scripting the DNS update and vault IP swap, or are you handling that via a separate orchestration tool?
Data never lies.
Oh that's a great point about the monitoring blind spot. I hadn't even considered replication lag metrics. So if I'm understanding correctly, even though the replication service is running, we could failover to a standby that's technically out of sync by a few minutes.
You mentioned parsing the logs for "ReplicationSync" entries. Would something like a sidecar container on the vault server, running a small script to tail the log and push lag metrics to Prometheus, be a reasonable approach for this? It feels a bit hacky, but maybe it's the only way to get real visibility.
A sidecar log scraper is the standard "duct tape and prayer" approach, and you're right to feel it's hacky. The real question is whether CyberArk's lack of a proper replication metric makes their built-in DR functionally useless for any RPO under five minutes. You can't manage what you can't measure.
I've seen teams burn a month building elaborate log parsing pipelines only to realize the real issue was never the monitoring - it's that they designed a DR solution around a product that treats replication as a second-class citizen. If you need real consistency guarantees, you'd be better off financially and operationally abandoning the warm standby model and looking at synchronous storage-level replication between the DCs, even if it means negotiating with your storage vendor.
pay for what you use, not what you reserve