Welcome to the vendor evaluation ritual where the compliance checkbox meets reality. That gap you're feeling is the standard mismatch between their audit posture and your actual usage.
Everyone gets dazzled by the SOC 2 seal. It proves they have a process, not that your specific data pipeline is secure. Their pen test scope being limited to the core app is a classic move. It keeps their costs down and their report clean, while offloading the risk of extended features onto you, the customer.
The real question isn't how to weigh them against each other, it's why you'd accept the risk at all. If their API and webhooks are critical to you, they're part of your threat model and should have been in theirs. Asking for a one-off test now is just paying them to do the work they should have done already.
prove it to me
Exactly right, and it's not just a feeling - it's a quantifiable risk delta. The SOC 2 validates their control environment, but the pen test scope defines the actual tested boundary. When your data flow crosses that boundary, you're operating in an effectively unvalidated zone.
I map this by creating a simple table for procurement: one column lists our critical data flows (API calls for X, webhook ingestion for Y), and the adjacent column shows whether each flow is inside or outside the pen test scope. The result is a clear visual showing the percentage of our intended usage that lacks independent adversarial validation. I've found presenting that gap as a percentage, often 40-60% for integration-heavy use cases, gets immediate attention.
Have you asked for their asset inventory and compared it to the systems listed in the pen test's "Targets In Scope" section? The disconnect is usually right there in black and white.
—Alex
That percentage gap is a neat trick. I'd ask how they define "critical data flow" for the table. If it's based on volume, a low-volume auth endpoint could be excluded but still be the crown jewels.
—EB
Good catch. A volume-based criticality filter misses the point entirely.
When I build that table, "critical" means any data flow that touches sensitive customer data or a privileged function, regardless of call count. The single daily cron job that syncs all user permissions? That's a crown jewel. A high-volume public marketing content API? Probably not.
You have to define "critical" by potential blast radius, not traffic logs. Otherwise, you're just measuring noise.