We get a lot of vendor pitches and blog posts with big security claims. You don't need to be a security architect to poke holes in them. Here's my quick checklist.
**First, ask for the evidence.**
* "That's a great claim. Can you share the CVE you're referring to?"
* "Do you have a public penetration test report or third-party audit?"
* "Show me the specific test results or benchmarks."
**Then, check for concrete details.**
A real claim will have specifics. Vague language is a red flag.
* Bad: "Hardened against attacks."
* Good: "Prevents SSRF via a validated allow-list, as shown in this code snippet."
```yaml
# Example of a specific, verifiable control
steps:
- name: Scan for secrets
uses: gitleaks/gitleaks-action@v2
with:
config-path: .gitleaks.toml
# This shows *how* it's done, not just that it's "secure"
```
Finally, see if they can explain it simply. If they can't break it down for a non-expert, they might not understand it themselves.
cg
YAML all the things.
This is a solid starting framework, especially the insistence on moving from marketing language to evidence. Your point about requesting the CVE is crucial; it forces the vendor to anchor their claim to a known, documented vulnerability. I'd add a step between asking for evidence and checking details.
You must analyze the provenance of the evidence they provide. A third-party audit from a no-name consultancy carries a different weight than one from a firm like Cure53 or NCC Group. Similarly, a "test result" from an internal QA team is not equivalent to a report from a contracted red team. Always ask *who* produced the evidence and under what scope of work.
The other piece missing here is the follow-up on recency. Security isn't a snapshot. A penetration test from 2022 is practically obsolete for a product under active development. Your final question about explaining it simply is a good proxy for internal understanding, but don't mistake a polished sales engineer's simplified explanation for actual engineering depth. The real test is whether they can also explain the *limitations* of their control in simple terms.
Good list. The part about explaining it simply is often overlooked. If their sales engineer can't walk me through the mitigation in plain language during a demo, that's a huge red flag. They're either hiding complexity or don't understand their own product.
The code snippet example is key. Asking "can you show me the config or the policy rule that enforces this?" separates features from reality. I've seen tools claim "drift detection" where the actual output is just a JSON blob of raw cloud API calls with no analysis. Concrete output beats a marketing slide every time.
Build once, deploy everywhere
Agree on the code snippet. My addition: ask for the *negative* test case too.
"Show me the config that enforces this" is good. Then ask "Now show me a malicious input your tool is designed to block, and the log/alert it generates." If they can't produce the failure output, their detection is probably theoretical.
Also, that gitleaks example is a CI step. That's prevention. Many tools claim "detection" but just scan for known signatures. The real test is catching a novel, obfuscated secret. Ask for their false positive/negative rate on that.
Trust but verify, then don't trust.