Alright, let me start by saying I know this is going to be a contentious take in some circles, but after spending the last six months deep in a vendor evaluation and TCO project for our PAM solution, I’ve become convinced of this: we need to stop treating SSH keys as a valid, long-term authentication method and mandate certificate-based authentication for all our servers.
I come from an IT procurement background, so my brain immediately goes to the operational and financial overhead. The sheer scale of the SSH key problem feels invisible until you have to manage it. From what I’ve seen:
* **Key Inventory is a Nightmare:** We have no true, centralized inventory. Keys are scattered across individual `.ssh/authorized_keys` files. Who has access to what prod server right now? Honestly, we can’t answer that with confidence without a massive, manual audit.
* **The Lifecycle Management Gap:** There’s no built-in expiry or automated rotation for standard keys. An engineer leaves the company, or moves teams—how do we guarantee all their keys are revoked? We rely on a manual process ticket, which is a huge risk.
* **Auditing is Painful:** Tracing a specific SSH session back to an individual is indirect at best. You’re correlating IPs and usernames, not a strong, cryptographically-backed identity.
Certificate-based auth (like using a short-lived certificate issued by a CA, say HashiCorp Vault or a dedicated SSH CA) seems to solve these pain points directly. The user authenticates to a central authority (integrating with our IdP), gets a certificate valid for, say, 8 hours, and uses that to connect. The server trusts the CA’s public key.
My questions for the community, because I want to be thorough before I even propose this internally:
1. **Implementation Timeline & Real-World Cost:** For those who’ve done a full migration from key-based to certificate-based SSH, what was the actual timeline? Not the vendor’s best-case scenario, but the real-world crawl-walk-run. Were there hidden costs in developer training or unexpected compatibility issues with legacy tooling?
2. **User Experience & Adoption Friction:** How did your developer teams react? Was the change transparent (e.g., a wrapper script or agent) or did it require a significant change in their daily workflow? User reviews from actual engineers would be invaluable.
3. **Exit Strategy & Vendor Lock-in:** This is a big one for me. If we build our process around a specific vendor’s CA solution (commercial or open-source), how portable is the *process*? Is the standard (SSH certificates) robust enough that we could swap the backend CA without rebuilding everything, or are we tying ourselves to a specific ecosystem?
4. **Break-Glass / Emergency Access:** How do you handle true break-glass scenarios when the CA or the issuing service is unavailable? Do you keep a tiny, tightly-controlled island of key-based access, or is there a certificate-based model for that too?
I’m convinced the TCO argument wins in the long run when you factor in reduced risk and less manual overhead, but the devil is always in the details of the migration. I’d rather ask too many questions now than face a failed rollout later.
You've nailed the real-world management pain, no question. But I think there's a huge hidden step between "scattered authorized_keys files" and a CA rollout that often gets skipped.
For teams already drowning in key sprawl, trying to mandate a whole new auth system is like telling someone to rebuild their house while it's on fire. The initial credential discovery and cleanup is its own multi-month project, and if you don't do it first, you just end up with certificate sprawl on top of key sprawl. I've seen it happen.
Have you found any good tools or scripts for that initial key inventory phase that actually work at scale? Everyone talks about the shiny CA, but I'm more interested in how you clean up the mess to get there.
api first