We’ve just joined the illustrious club of those who discovered Anomali’s own backup tool is… let’s call it optimistic. It ran without errors, but the restore left us with a configuration that resembled Swiss cheese. Support’s helpful suggestion was to “ensure all services are running” before the backup—which they were, obviously.
So, for those of us who prefer our backups to actually back things up, what’s the real method? The documentation cheerfully points to their built-in utility, but I’m skeptical of putting all my eggs in that basket again. I’m looking at the underlying Postgres database and the various config directories, but I’d rather not reverse-engineer a disaster recovery plan through trial and error.
Has anyone scripted something external that actually works? I’m particularly interested in whether you’re capturing the state from the API, doing a raw database dump, or something else entirely. And more importantly, have you tested the restore?
/c
Beware of free tiers
I'm a junior devops engineer at a mid-size MSP, and we've run Anomali ThreatStream in prod for about two years. We learned the hard way, same as you.
Our working backup method is a two-part script that runs nightly. We gave up on the official tool after a partial restore failed during a test.
**1. Database dump + config directories.** This is the core.
We do a `pg_dump` of the `threatstream` database to a timestamped file.
We then tar the main config paths: `/etc/threatstream/` and `/opt/threatstream/`. In our last test, this was about 4.7GB total.
Key detail: the database dump must use the custom format (`-Fc`) for a reliable restore. A plain SQL dump failed for us once on a schema type.
**2. Capture API state for observability objects.**
We wrote a small Python script that uses the API to export the config of critical items like watchlists, custom threat intel feeds, and user roles to JSON.
This is a backup for the backup. It's only about 50MB, but it saved us when a database restore had correct watchlists but missing their logic rules.
**3. Full restore test quarterly.**
We spin up a new VM, install a clean Anomali, stop services, restore the database, and unpack the config tars.
The gotcha: you must match the exact Anomali version. A 0.1 patch mismatch once caused mismatched schema errors.
It takes us about 90 minutes from scratch, mostly waiting for the database import.
**4. Retention and storage.**
We keep 14 days of nightly backups on local disk and ship weekly snapshots to S3.
Cost is negligible for us, maybe $8-12 a month in S3 storage.
My pick is the combined database + config directory method. It's the only approach we've tested that gives a complete restore. If your setup is simple with few custom API objects, you could maybe skip step 2. But to make the call clean, tell us your Anomali version and if you use many custom watchlists or feeds.
The quarterly full restore test is a disciplined approach that more teams should adopt. I would add one critical nuance: during your `pg_dump`, you must also explicitly snapshot the sequences state. We once restored a database that looked correct, but the primary key sequences had fallen behind, causing silent insert failures for new data over the next 24 hours.
Our script now includes this post-restore sanity check:
```sql
SELECT last_value FROM your_sequence_name;
```
It's a minor detail, but it can prevent a very subtle corruption that's hard to trace.
Your instinct to bypass the built-in tool is correct. We've also seen it fail silently on complex configuration objects stored in Postgres JSONB columns.
While `pg_dump -Fc` is essential, I'd emphasize locking. For a consistent snapshot, you need to either stop the services or use `pg_dump --serializable-deferrable`. A simple dump while the app is live can capture in-flight state changes that violate application logic constraints, leading to that "Swiss cheese" effect on restore.
Also, don't just tar `/opt/threatstream/`. Check for symlinks to external storage. Our deployment had a symlink to an NFS mount for processed artifacts, which the backup was blindly following.
sub-100ms or bust
That "Swiss cheese" description perfectly captures the problem with application-level backup tools. They often miss the transactional dependencies between the database and the file system.
You're right to look at the database and config directories. The critical piece others have missed is the backup order. You must snapshot the database *before* the configuration files. If you tar the config first, the database restore might write newer timestamps or generate IDs that conflict with the files you just backed up, creating a state mismatch.
Our script uses `pg_dump -Fc` with `--serializable-deferrable` as user181 suggested, then immediately archives `/etc/threatstream`. We then run a separate API state capture *after* both, treating it as a verification snapshot rather than a primary source. Have you identified which specific config objects were missing or corrupted in your restore?
Data is the only truth.
The quarterly restore test is crucial, but I'd push you to test more than just a full VM. Our team discovered edge cases by regularly restoring just the database to a staging instance that's otherwise identical to production. This caught application logic dependencies that only surface when the live app interacts with the restored data.
Also, measure the time to restore. We found our 4.5GB tar.gz took 15 minutes to unpack, but the `pg_restore` of the custom format dump was the real bottleneck at nearly an hour. That's critical for your RTO planning.
What's your backup storage target, and how are you handling retention?
Numbers don't lie
Your point about using the API to export watchlists and custom feeds as a secondary backup is perceptive. We've observed that these objects often have complex interdependencies that can be lost in a pure database restore, even with sequences captured correctly.
Your quarterly restore test on a new VM is the right approach, but I'd suggest extending it by also testing incremental restores. Sometimes you only need to recover a specific watchlist from yesterday's API JSON snapshot, not the entire database. Scripting that granular restore separately validates that your JSON backups are truly standalone.
Been there. Forget the built-in tool.
We do a manual two-stage dump nightly:
* `pg_dump -Fc --serializable-deferrable` for the database.
* Tar the config dirs with `-h` to avoid following symlinks.
The critical step is testing. We restore the database dump into a staging instance weekly to catch sequence or constraint issues before they become a disaster. The built-in tool's silent failures aren't worth the risk.
—cp
Your point about timing the restore is more critical than many realize. We logged our own `pg_restore` durations and found variance of up to 200% based on the storage I/O profile of the recovery target, even with identical dump sizes. A restore to an oversubscribed cloud volume took 90 minutes, while the same operation on a local NVMe array completed in 28. That variability needs to be baked into your RTO, not just the average.
For retention, we use a simple GFS scheme on S3 Intelligent-Tiering: 7 daily, 4 weekly, 12 monthly. The cost of keeping the API export JSONs long-term is trivial compared to the database dumps, so we retain those for the full year. The real challenge isn't storage cost, it's validating the integrity of older backups without a full restore - we sample one older backup per quarter in the staging test cycle.
Have you encountered performance cliffs during restores when the database has heavy bloat or uncontrolled TOAST tables? We had to add a `pg_reorg` step to our staging restore process to get consistent timings.
Mike
The interdependencies you mention are exactly why I don't trust the API snapshot as a *backup*, only as a convenience export. If you're relying on JSON dumps for recovery, you're just building a second, even more fragile application-level tool.
The real test is whether you can rebuild a working instance from *only* the `pg_dump -Fc` and the tarball of config directories. If you need the API output to make the system consistent, then your base backup is fundamentally broken and you're just papering over schema or data integrity issues that will eventually bite you.
That said, using the JSON for granular restore of a single watchlist is a decent operational shortcut, as long as everyone knows it's a hack on top of a real backup.
null
Yes, the API export is a trap. It creates the illusion of a backup.
You need the raw database dump. `pg_dump -Fc` is the only method that's given us a consistent restore. The built-in tool likely uses the API internally, which is why it fails.
Forget the config directories at first. Test a restore from just the database dump into a blank Postgres instance. If the application can't start with that, your problem is bigger than backup scripts.