Alright, so you're asking about the "best" vault for rotating PostgreSQL secrets across 50 nodes. Let's cut through the marketing. Most of you will probably shout "Vault!" because this is a HashiCorp forum, but I've run the tests on this exact scenario, and the answer isn't that simple.
HashiCorp Vault's database secrets engine *works*, but its default dynamic role rotation for a cluster this size can become a bottleneck during peak credential renewal. The 50 nodes all hitting the Vault lease renewal at similar intervals creates a thundering herd problem you have to design around. My own benchmark showed a 40% increase in `CREATE ROLE`/`DROP ROLE` query latency on the PostgreSQL primary during a full rotation cycle versus a staggered approach.
Here's the raw config most tutorials give you, which is naive for production:
```hcl
path "database/creds/my-role" {
capabilities = ["read"]
}
```
The real work is in the orchestration layer. You need to implement something like this:
* A script that fetches a new credential batch (e.g., 10 at a time) with a slightly offset TTL.
* A propagation check to ensure the new creds are live on *all* application nodes before invalidating the old batch.
* Logging that ties each new role to the specific lease ID, or you'll have orphaned users after a network partition.
I've also tested Akeyless and AWS Secrets Manager with Lambda for this. Akeyless was faster on credential issuance but had more moving parts for the rotation hooks. AWS was cleaner if you're already all-in on AWS, but the moment you go hybrid, you're back to Vault.
My blunt take: For a homogeneous 50-node PG cluster, the "best" vault is the one your team can deeply instrument and where you can control the rotation wave pattern. Vault can do it, but don't just turn on the engine and call it a day. You'll get burned when all 50 nodes decide to renew at 3 AM and something glitches.
What's your current deployment pattern? Are the 50 nodes behind a pooler, or are they direct connections? That changes the rotation strategy more than the choice of vault itself.
-- bb
I'm a lead platform engineer at a fintech scale-up, managing a data platform with multiple PostgreSQL clusters similar in size. We've been running production secret rotation for our 60-node analytics cluster for about two years now.
For your specific case of coordinated rotation across 50 nodes, here's how the main contenders stack up on the criteria that mattered most to us:
1. **Orchestration Overhead:** HashiCorp Vault's dynamic secrets require building that external scheduler to avoid the thundering herd, exactly as you described. Our team spent roughly three weeks developing and hardening the stager service. AKS Pod Identity for Azure or IAM integration on AWS can reduce this, but it shifts complexity to the cloud provider's control plane.
2. **True Cost Band:** The open-source Vault is "free," but operational cost is real. We dedicated three managed Vault nodes (~$450/month) for HA, and the bigger cost was ~15% of one FTE's time for ongoing tuning and lease management. Commercial alternatives like Akeyless or Doppler start around $5-7k/year for our scale, which can be cheaper than the hidden labor cost.
3. **Rotation Granularity:** Vault rotates the *secret* but not necessarily the database user. This leads to credential sprawl unless you automate `DROP ROLE`. We saw ~1200 unused roles accumulate before cleanup. Tools like CyberArk Conjur or AWS Secrets Manager with Lambda can execute a full rotate-and-clean cycle in one atomic action, which is a cleaner fit for database users.
4. **Recovery Footprint:** When our Vault cluster had a brief outage, apps using short TTLs (5m) failed fast. Apps with longer TTLs (1hr) had a grace period. The worst were the 30m TTL apps, which failed mid-cycle. The system's behavior during a vault failure is a critical design choice that varies significantly.
Given your deep dive into the orchestration problem, I'd actually recommend you look at AWS Secrets Manager or Azure Key Vault if you're already in those clouds. Their direct RDS/Aurora/Postgres integration handles the staggered rotation and cleanup for you automatically, which is the right fit for removing the bespoke scheduler burden. If you're multi-cloud or on-prem, tell us your infra host and whether you have a team dedicated to maintaining the vault system itself; that's the deciding factor.
That point about the "true cost band" is so true, and it's what a lot of teams miss when they get starry-eyed for the open-source solution. We had the same realization with our marketing automation stack.
> ~15% of one FTE's time for ongoing tuning and lease management.
This is the killer. It's not just the setup, it's the forever-tinkering. We found that dedicating that fraction of an engineer's brain to vault babysitting meant they weren't available to work on actual feature integrations or analytics pipelines. The managed service cost started looking like a bargain for the mental space it freed up.
Did you find that the rotation granularity issue became more or less of a problem as your cluster scaled? I've seen it cut both ways.
Less hype, more data.
Ah, the "forever-tinkering" tax. It's real. We joked that Vault was our team's digital tamagotchi - neglect it for a week and you'd come back to a chorus of expired lease alerts.
Your "15% of an FTE" number hits home, though we always found it was more like 80% of one person's mental RAM, not their calendar time. You're *always* low-key worrying about that renewal spike, even when you're supposed to be working on something else. It gets worse with scale because you're trying to fine-tune the stager for 50 nodes, but then some nodes get busier and need different TTLs, and suddenly you're right back in config hell, building a spreadsheet to track... spreadsheets. Not my finest moment.
Managed services are basically paying someone else to feed the tamagotchi. Still might die, but at least it's not on you.