Hey folks, had a bit of a debate in our team this week and wanted to get the community's take.
We're reviewing our break-glass procedures for critical systems (think core cloud infra, our main identity provider). The classic dilemma came up: should the emergency admin account passwords be **stored** (in a secure vault, sealed envelope, etc.) or **generated on-demand** via some automated system?
I've seen both approaches:
* **Stored:** Password is created once, put in a physical safe or a dedicated PAM vault that requires multiple approvers to access. Simple, works offline, but there's always that tiny risk of the secret becoming stale or the envelope walking away.
* **Generated on-demand:** System creates a one-time password or temporary credential when a break-glass workflow is initiated (with approvals/alerting). More dynamic, can be integrated with JIT principles, but adds complexity. What if the system that generates the password is part of the outage?
My gut leans towards a hybrid: a super-secure, rarely-touched vault holds the primary break-glass credentials, but we also have a secondary, automated JIT method as a backup. Overkill?
What's working (or not working) in your environments? Anyone using a cool PAM feature for this specifically?
Automate the boring stuff.
Your hybrid instinct is correct, but the implementation is what separates a procedure from a theatre prop. The core failure mode you must engineer against is the "meta-outage," where the very systems needed for break-glass are impaired.
Storing a static credential in a vault works until the IAM policy or authentication endpoint for that account changes without a corresponding vault update. That's your staleness vector. A purely on-demand system fails if your control plane is down.
The pattern I've enforced is: the **primary** method is a just-in-time generation system (like HashiCorp Vault with a static role for a TTL'd password), because it ensures credential freshness and auditability. The **secondary** is a stored, physical envelope, but with a critical twist: that envelope doesn't contain the "admin" password. It contains credentials to a dedicated, highly restricted service account whose *only* permission is to *invoke* the JIT system to generate the actual break-glass credential. This breaks the chain of trust issues and keeps the stored secret from directly accessing production.
Boring is beautiful
That's a clever twist on the hybrid approach, using the envelope as a key to the JIT system rather than holding the direct credential. It addresses the staleness problem elegantly.
My one practical caveat from experience is around testing that workflow. You need to regularly validate that the service account hasn't had its permissions drift and, more importantly, that the stored envelope credentials still work to log into the restricted subsystem to initiate the process. It's an extra step in the fire drill that teams sometimes forget to schedule.
Your gut on the hybrid model is right, it's where most mature setups end up. The key is making sure those two paths are truly independent.
For the automated JIT backup, consider making it a completely separate system, maybe even a low-tech script running on a standalone Raspberry Pi locked in a cabinet. That way it's not dependent on your primary orchestration or cloud control plane.
The real trick is the drill. You have to test both paths under simulated duress - like actually cutting network access to your main vault - to prove the fallback works. It's the only way to trust the envelope isn't just security theater.
Testing is the make-or-break step, and it's where most plans turn into fiction. You're right about teams forgetting to schedule it.
The permission drift you mentioned is an absolute killer. I've seen teams run flawless fire drills for the envelope-to-JIT process, only to find the underlying service account lost its "generate temporary credential" permission six months prior during a "security cleanup." Your test cadence needs to validate the entire chain, not just the first link.
Consider scheduling your test to coincide with another mandatory, but unrelated, procedural review. That's the only way it consistently happens.
—JW
Your gut's right, hybrid is the only sane way. But your vault shouldn't hold the main password, that's how you get stale.
Store the *seed* credential in the vault. That gets you into the isolated, low-tech system that *then* spits out the actual break-glass password. Two layers, different failure modes. Anything else is just playing house.
And for the love of all that's holy, don't make it a cloud vault. It's a metal box or it's theater. Seen too many "break-glass" procedures fail because the SSO provider was the thing down.
CRM is a means, not an end.