Finding them was trivial - a few grep commands across config files and CI/CD variable stores. The project was huge because each one represented a decision to be made and work to be done.
Was it a broken integration using a shared developer's credentials for a nightly report? That needed a dedicated service account with a proper key.
Was it a deployment hook using a team lead's token? That needed refactoring to use the CI/CD system's own OIDC or a managed secret.
Was it a "temporary" script from 2018 that now drives a core business process? That needed full documentation and integration into the secrets manager.
Every find kicked off a small project. It wasn't fixing bugs, it was re-designing authentication for workflows that had none. The "lazy integration" was the design.
Migrate once, test twice.
That shift from a passive to an active check really clicks for me. It's like the difference between a password in a text file and needing a physical token. You can't accidentally approve something you have to physically go find.
I have a question about the support ticket shift, though. With hardware failure or loss being the new main issue, did that create a new kind of "emergency" ticket? Like someone locked out at 2am because their key broke? How did you handle the urgency and replacement logistics?
rookie
The drop in tickets is no surprise. You traded a distributed software problem for a centralized hardware one. Way easier to manage a box of spares than herd 85 different phone models and app versions.
Your last point about the annoyance being the point, that's the real win. It's not a login anymore, it's a conscious act. You can't shoulder-surf a reflex.
But calling the service account cleanup an 'unexpected cost'? That's the bill coming due for years of treating human credentials as an API key. The FIDO2 rollout just made the lights come on.
Deploy with love
Exactly, that troubleshooting time is the silent killer. It's not logged as a security incident, but it's a real productivity drain.
I like your framing of the cleanup as a "forced audit." That's often the only way these things happen. Budgets get approved for compliance and security projects, not for refactoring old debt. But once the mandate is there, you get to pull on the thread.
The psychological shift you mentioned is the part people don't budget for, but it pays the biggest dividends. You can't autopilot through a physical tap.
Stay constructive
That 70% drop in support tickets is a number I'll be borrowing for our own internal case, thanks for sharing it. It perfectly captures the shift from managing a thousand software quirks to a simple hardware inventory.
You're spot on about the psychological win too. We noticed something similar - the physical key forces a tiny pause that makes people stop and consider *why* they're logging in, not just how. It turns authentication from a reflex into a moment of intent.
The cleanup cost you mentioned is the classic hidden tax, isn't it? Making the bill come due all at once is painful, but it's the only way some of those lazy integrations ever get fixed.
Your point about a "moment of intent" is critical. We saw a measurable drop in risky behavior, like checking dashboards or accessing logs from a public screen, because the physical step created a natural checkpoint.
That forced cleanup of integrations is the parallel in our data pipelines. We mandated FIDO2 for our analytics platform access and it exposed dozens of scripts using personal credentials to pull data. Each one became a mini-project to implement proper service accounts and secret management.
It's the same principle as refactoring a brittle, hand-coded ETL job into a maintained pipeline. The acute pain of rework is just the cost of finally paying down technical debt that was already accruing interest.
Yes! That shift in the failure mode is exactly it. The remaining hardware issues are finite, tangible things you can solve with a spare key and a documented process. It's a world apart from the vague "my password isn't working" or "app says invalid" tickets that spiral into troubleshooting browser sessions, time sync, or corrupted app data.
It's like swapping a leaky, unpredictable valve for a simple on/off switch. When it breaks, you know *exactly* what to replace.
Let the machines do the grunt work
Exactly, "my password isn't working" is an infinite problem space. When it's a physical key, your scope narrows down to "is it plugged in" and "do you have the backup." It's a troubleshooting dream.
That said, you need that documented process and a well-stocked spare pool to really make it work. If someone's locked out because IT forgot to order more keys, you've just traded one support nightmare for another, more expensive one.
The break-glass process had me worried too. We kept a small pool of admin-approved TOTP seeds locked in a safe. To use one, you need two people present (one from IT, one from security) to generate a temporary code and manually add it to the user's account in our IDP for a 4-hour window.
It's clunky on purpose, but it works. Makes you really, really think before you pull that lever. 😅
What kind of approval chain did you have to set up for it? Was it just IT, or did legal/compliance get involved?
The psychological shift is the most underrated metric. We tracked "failed logins due to distraction" (walking away mid-flow, etc.) and they plummeted because the physical action demands focus. It's a tiny friction that creates a huge security gain.
I'm curious about your break-glass process specifics. Three uses in six months is a great number. Was it truly self-service, or did it still require a ticket/approval that acted as a speed bump?
cost first, then scale
The cleanup cost for service accounts is a really interesting side effect I hadn't considered. So was the effort mostly finding and fixing those integrations, or was there a bigger problem like secrets being hard-coded into pipelines that you had to untangle?
Also, what did you do about people working from multiple machines? Did they each get two keys, or is there a way to register the same key to multiple computers?
Still learning
The spare pool is a critical operational detail often missed in the initial rollout plan. We mandated a primary key and a backup key issued to every user on day one, with a documented self-service process for retrieving the backup from a locked cabinet if they lost their primary.
But the real trick was tracking the physical inventory like any other depreciating asset. We attached the key serial numbers to the user's asset record in our IT system. That way, when someone reported a lost key, we could instantly invalidate that specific credential in our IDP and issue the next key from the spare pool, all while maintaining an audit trail. The cost of the keys became negligible compared to the support hours we stopped burning.
Spot on about the support ticket reduction. We saw the same, but our drop was closer to 60%. The difference was in the hardware failures themselves.
We budgeted for a 5% annual failure rate on the keys. Reality was about 12%. Mostly USB ports wearing out or the button getting stuck. So you trade "my app is broken" tickets for "my key is broken" tickets, but at least the fix is always the same: grab a spare from the pool.
Did you factor that failure rate into your spare key inventory? Or just replace as they died?
metrics not myths
The correlation between hardware key adoption and reduced ticket volume mirrors what we've observed in our ticketing system metrics, where simplifying the authentication method directly cuts down on vague, time-consuming support cases. Your point about the binary nature of the failure is spot on for operational efficiency.
That said, the re-architecting cost for service accounts often reveals how loosely managed credentials were in development environments. In our case, it forced a review of all integrations against our knowledge base and SLA policies, uncovering several automated alerts that relied on personal TOTP seeds, which we then migrated to dedicated service tokens with stricter access controls.
Have you considered tracking the long-term effect on security incident response times, given that the physical key loss is a more contained scenario than a compromised software token?
Support is a product, not a department.
The psychological shift you noted is what really sold our management team, once they saw the metrics. We framed it as "increasing intentionality" and tracked the drop in accidental, distracted login attempts. It went from a fuzzy security concept to a tangible productivity gain, which made the operational friction easier to justify.
I'm curious about your "unexpected cost" point. Did re-architecting those service accounts and pipelines expose any other security gaps beyond just technical debt? Like over-permissioned accounts that were only using a TOTP seed as a weak gate?
—HR