That optimistic description of the built-in tool is painfully familiar.
We had a similar failure. Our script does the raw database dump first, then immediately copies the config directories. But after reading the thread, I realize we've never checked the disk load between those steps.
The point about config directories containing API tokens that the dump misses has me worried now. If the configs are partially written, those tokens would be corrupt too, right? Does anyone have a method to verify the integrity of those config files after the tarball is made, without a full restore?
Ugh, that "optimistic" description hits home. We were in that same boat a year ago.
The method that finally worked for us is a layered script that grabs the raw `pg_dump -Fc` first, then the config directories, but we added a crucial step *before* the tarball: we run a simple `find /etc/anomali -type f -exec sha256sum {} ;` and save that manifest alongside the backup. It doesn't prevent corruption during the copy, but it at least lets you verify later that the files you *meant* to back up are bit-for-bit identical in the archive. It saved us once when we discovered a cron job was overwriting a config file mid-backup.
The real answer to your question is yes, you have to test the restore, and not just once. We schedule a restore to a sandbox every quarter, and it's the only way we caught a missing environment variable that wasn't in any config file.
Backup first.
>Forget the config directories at first.
That's a solid first test to isolate the core problem. It forces you to prove the database itself is sound.
I'd just add that this test also reveals whether your configs are *truly* decoupled. If the app fails to start with only the restored database, it often means there's hidden state or logic in those config files that the application expects on boot. That's a different, but equally important, finding than a broken backup method.
Passing this test is a great milestone, but it's only the first checkpoint in a full recovery.