Seeing this a lot lately. CyberArk health dashboard is all green, but automated backups are failing silently. Classic case of monitoring not aligned with actual service health.
What I've checked:
* PVWA/CPM services running
* Disk space and permissions on backup target
* Scheduled task configuration
Common root causes I've found:
* Backup service account password expired (health check uses different creds)
* Underlying Vault/DR replication issues not reflected in component status
* Firewall rule changes blocking outbound backup traffic to secondary site
Post your component versions and any errors from the `BackupProcess.log`. I'll help you trace it.
—cp
> "health check uses different creds"
You've hit the nail on the head there. That credential mismatch is probably the single most common trap I've seen in production environments over the past couple of years. The health dashboard is often polling a lightweight service endpoint that doesn't really exercise the full backup pipeline, so it stays green while the actual backup job trips over a stale password.
One thing I'd add to your list: check whether the backup schedule is using the same CyberArk component version as the rest of the environment. I've run into a case where the backup task was still pointing at an older CPM instance that had been decommissioned in the dashboard but not cleaned up in the scheduler. The green light was for the new CPM, the failing process was for the old one.
What version of the vault are you sitting on? And are you seeing any correlation between the backup failures and recent credential rotation cycles?
Keep it constructive.
That log won't tell you the whole story. The health dashboard and BackupProcess.log often exist in different universes.
The real failure is usually in the scheduler or the job wrapper. I've seen the scheduled task succeed with exit code 0 while the actual backup script it called failed. Health checks the task, not the outcome.
Check the Windows Event Log for the backup task, or the cron output if it's Linux. The .log file can be clean if the process never actually launches.
Don't panic, have a rollback plan.
Yeah, spot on about the exit code deception. I had a Windows Scheduled Task that was calling a PowerShell wrapper script. The script would log a failure, but because the task's "Stop if the running task exceeds" duration was longer than the script's runtime, it would return success.
Found the actual error by adding `$host.SetShouldExit(1)` in the catch block of the PS script. The health check just saw "task completed", not the internal exit code.
Great checklist! I'd add one more spot to check: the time zone on the server running the scheduled task versus the backup target. I've seen backups fail because the task ran on local time, but the retention policy on the storage side was based on UTC, causing a conflict that didn't log an obvious error.
null