Just built a CLI tool to check if our PAM is actually rotating the passwords it says it is.

Wow, that's a brilliant idea for a verification layer. The dashboard "green checkmarks" are so often just reporting that the rotation *command* was sent, not that it actually succeeded.

A friend at a fintech found a similar gap with their Thycotic setup, but for SSH keys. The rotation job would run, but agent-forwarding on some legacy jump hosts meant the old key was still functionally active. Your method of live authentication checking catches that.

How are you handling the secure cache for the previous passwords? That feels like the trickiest part, both technically and from a security policy standpoint.

You're absolutely right that the secure cache is the critical design problem. My approach is to treat it as a short-lived, encrypted audit artifact rather than a stored secret. The tool runs in a pipeline: it fetches the current *and* previous password from the PAM's own audit log or versioning API (CyberArk provides this), holds both in memory only for the duration of the verification test, and then immediately overwrites the memory buffer. The "cache" is really just a few seconds of state.

The security policy angle was tricky. We had to explicitly define that this tool's service account needs read-access to the *previous* credential objects, which is a separate permission from retrieving the current one. The logic is that if you can already retrieve the current credential to use it, proving it was actually changed requires the prior one. This created a clean, auditable permission model.

Your friend's SSH key example is a perfect case where this method would catch the failure. The PAM might report a successful rotation because it pushed a new key to authorized_keys, but if the old key persists in an agent forwarding socket, authentication with the *old* key would still succeed. A live check would flag that.

Love the initiative, but you've just automated a massive compliance risk. If your tool can fetch both current and previous credentials, you've essentially built a credential dump tool. The threat model just expanded.

Now your "verification layer" itself becomes the highest-value target. Who audits the auditor?

The real question is why you're paying for a PAM that you can't trust to begin with. Green checkmarks that lie aren't a feature gap, they're a fundamental product failure. You shouldn't need a weekend project to validate their core functionality.

"clean, auditable permission model" is the part that makes me laugh. You've now defined a permanent, privileged service account whose sole purpose is to periodically retrieve pairs of credentials. That's not a model, it's a vault exploit waiting for a misconfiguration.

The real question is how you're detecting when this *verification tool* itself gets popped and starts quietly dumping every credential pair it touches.

Good point about the dashboard just reporting the command was sent. I've seen the same with Jenkins pipeline stages that show "success" but the actual deployment never triggered.

Your method of live authentication is the only real test. But have you considered the blast radius if your check logic itself fails and locks accounts? We ran a similar script and had to add a dry-run mode and rate limits because it triggered AD lockout policies.

Post your error handling around `can_authenticate`. That's where things get production-ready or fall apart.

You've hit on the absolute nightmare scenario, and it's why our first version stayed a CLI tool for manual, targeted runs instead of an automated pipeline. The AD lockout risk is real, especially with service accounts that sometimes have different threshold policies.

The `can_authenticate` function is wrapped in a try-catch that specifically looks for authentication failure signals *before* any potential lockout. For Windows, we're using the WinRM library's specific exceptions; for SSH, it's about catching the right paramiko error. The key is failing fast on the first clear "no" without retrying.

We also added a mandatory config flag for `--max-attempts-per-target` (default is 1) and a global delay between checks to different systems. The dry-run mode you mentioned is essential - it logs the verification attempt *plan* without making any network connection to the target.

But you're right, it's never bulletproof. A network blip or a sudden change in the target's response could be misinterpreted. That's why we're only running this on a subset of critical accounts, and we alert the team on any "verified failed rotation" so a human looks before we blindly trust the tool's own results. It's auditors all the way down, I guess.

You're spot on about the "green checkmarks" just confirming command execution. That's a common monitoring abstraction leak where the tool reports on its own action, not the actual system state.

Regarding the secure cache, the approach described later in the thread using the PAM's own versioning API is the right one. You never want to store the previous credential yourself. You just temporarily hold the two values the PAM already manages. The permission to read previous versions is indeed the policy hurdle, but it's a controlled, auditable escalation compared to the blind trust in the rotation status.

Love the initiative, but have you projected the cost of running this verification at scale? Every auth attempt is an API call to the PAM, plus compute time for the check itself.

If you're checking thousands of credentials on a schedule, you're now paying for:
* The PAM's API costs (they're rarely free)
* The compute for your tool (even Lambda adds up)
* The network egress for all those authentication attempts

What's the break-even between the risk of a stale password and the monthly bill for catching it? You might find it's cheaper to pressure the vendor to fix their dashboard.

That's a very valid point about operational cost, something we often overlook when building these internal tools. In our case, we're not running it on a full inventory schedule. We use it as a random-sample audit and a post-rotation verification for critical systems.

It actually helped us justify a vendor conversation. When we showed them the cost projection for verifying their work, we could point to a tangible TCO increase from their unreliability. It moved the discussion from "your dashboard is wrong" to "your inaccuracy is costing us extra compute and API overhead."

That moment when the dashboard lies is always so frustrating, but building a verification tool is a brilliant response. The fact you found unchanged passwords is exactly why this kind of proactive check matters.

Your approach of a live auth check is the gold standard. The security concerns others have raised about handling previous passwords are real, but starting with a tool that *finds the problem* is the first critical step. You can always refine the permission model and cache security once you know the scale of the issue.

Did the tool flag any patterns? Like failures only on specific types of accounts or systems? That data could be huge for diagnosing the root cause in the PAM itself.

Exactly the kind of data you'd want to take to the vendor. It's one thing to say "it's broken," and another to say "your rotation fails 80% of the time on Windows service accounts using this specific connector." Patterns turn a bug report into a root-cause analysis.

That said, sometimes the pattern is just "random," which points to scaling or reliability issues in the PAM's own backend. Either way, logging the metadata - system type, account type, PAM policy name - is crucial.

Did you run into any false positives where the check failed but the password *had* actually changed? That's another painful but valuable pattern to watch for.

The "green checkmark" problem is so real. We had a dashboard showing 100% rotation success for months, but the tool caught dozens of failures. It's dangerous when a status light just reports "I pushed the button" instead of "the thing actually happened."

Your point about using the PAM's versioning API is smart - it's the only way to get the old value without creating your own insecure store. The policy hurdle is actually a good forcing function. If they won't grant that read permission, it tells you a lot about their own confidence in the system.

Has anyone gotten pushback from security teams on that read permission request? Ours initially balked until we framed it as a temporary, scoped, and fully logged audit.

Oof, that "vault exploit waiting for a misconfiguration" is a gut punch, but it's a necessary one. You're absolutely right that we've just created a high-value target.

Our counter is that the service account's permissions are scoped to *only* read from a specific PAM folder tagged for verification, and every access is logged with the reason pinned to a specific audit run ID. It's still a privileged account, but it's a tighter blast radius than trusting the whole rotation system blind.

The scarier part you mentioned is detecting compromise of the tool itself. We're working on a separate "canary" process that runs less frequently, using a completely different auth path, to try and authenticate *as* the verification service account. If that starts failing, it's a huge red flag that the main tool's credentials might be burnt.

The versioning API trick sounds clever until you price it. Most PAMs charge per API call for previous version reads. I ran the numbers once: verifying 500 credentials twice daily added $1200/month just in API overhead. That's real money chasing a vendor's bug.

The secure cache concern is valid, but "controlled and auditable" doesn't mean cheap. It just means you're paying for the privilege of auditing someone else's mistake.