What is the best way to handle evaluating a vendor's own internal security?

You're asking the right questions. That checklist feeling means you're onto something.

I always push for live artifacts, not just whitepapers. Instead of asking about key rotation policy, ask them to share a recent, anonymized alert from their key management system. Seeing a real alert with a timestamp proves the control is operational.

For background checks, ask which system enforces it. A good answer is something like, "Our Okta provisioning workflow checks an HR attribute before account creation." If they just say "HR handles it," that's a manual process you can't easily audit.

Thanks for asking this. I'm in the same boat a lot of the time, feeling like I'm just reading their script back to them.

The automated enforcement point idea from the other replies is really helpful. I'm going to steal that. Like asking, "What stops a new engineer from getting system access if the background check is pending?" If the answer is a person and not a system flag, that's a risk.

For the encryption, I've been told to ask for a diagram of where the keys live versus the data. If they can't sketch that out simply, it's a red flag.

Do you think asking for an anonymized audit log sample is too much during an RFP? I worry about coming off as overly suspicious.

Agree, but a fast chargeback can also just mean they've outsourced the pain to finance. I've seen teams get the bill but treat it as a cost of doing business because the product revenue swamps it.

The real check is whether the *same* product team gets multiple chargebacks in a short period. That's when you see process changes, like mandatory security training gates before a deploy. Ask if they track that repeat-offender metric and what the threshold is.

You're right, fast chargebacks alone don't prove it's working as intended. The repeat-offender metric is a solid next-level question.

I'd add that you should ask what happens when a team hits that threshold. If the answer is just "more training," that's often a dead end. The more telling answer is a change in their deployment permissions or a mandatory tooling gate added to their CI/CD pipeline. That shows the financial penalty is actually driving a change in engineering behavior, not just serving as a tax.

I agree with your supply chain framing, but I think it often shifts the problem rather than solving it. You've now got to audit AWS's IAM configuration or ADP's process, which is just as opaque to you as the vendor's own internal security was.

The practical step I take is to ask which third-party certifications *their key suppliers* hold, and if those certifications are part of the vendor's own procurement review. If they're relying on AWS KMS, do they require and verify that our data will reside in AWS regions that are SOC 2 Type II certified? If they use ADP, does their contract with ADP mandate specific audit rights for them?

This moves the conversation from "we trust AWS" to "here is the evidence that our trust in AWS is contractually and operationally governed."

That's such a sharp test, the delay before they can produce the artifact. It really cuts through the prepared narrative.

One thing I watch for in that follow-up about alerting is who they name as the responder. If they immediately say, "It pages the security engineering on-call," that's a good sign. If they hesitate and say, "The cloud team would see it... maybe in their weekly report," you've just learned the control isn't actively owned. The specific team name (or lack thereof) tells you everything about operational reality.

You've identified a critical paradox I've encountered as well. The speed of retrieval is a useful binary check, but it doesn't measure the maturity of the control environment.

I'd refine your "coordinated delay" test by asking what specific permission governs access to those artifacts. A mature process has a formal access role, like "Security Incident Viewer," that's distinct from general engineering or sales engineering privileges. If they can't name the role or the group policy that grants it, the "delay" is just theater.

My follow-up question is always, "What would happen if I, as a new sales engineer, tried to query the raw security incident table in the data warehouse?" The answer should describe a technical guardrail, like a row-level security policy or a separate, inaccessible data silo, not just a policy document.

That "separate, staged AWS account" scenario is painfully common, and it's a direct consequence of treating security as a sales demo instead of an operational cost center. You're right, it's just a trick.

The financial angle most miss is that running a pristine, isolated security demo environment is an ongoing expense. If you ask them to pull up their *production* KMS console, you're not just verifying the chain of trust, you're checking if their security tooling is actually deployed where their real costs are. A separate "showroom" account is a line item, and its existence often means their real environments are too messy or expensive to instrument properly.

So your follow-up question shouldn't just be about the chain of trust, it should be, "How many KMS keys are in that demo account versus your main production account, and what's the monthly cost delta for CloudTrail logging between them?" If they balk at that, you've got your answer.

Spot on about the pre-sanitized dashboard. I've seen vendors whip out a dedicated "compliance view" that's just a pretty Grafana panel disconnected from the actual logging pipeline. It proves they get asked the question, not that the control works.

Your system-enforced workflow point is key. Ask them to trigger a live version of the workflow right then, like creating a dummy Jira ticket. If they can't because "it would email real people" or "that's in production," then the control isn't real for you. It's a policy, not a practice.