The point about overlaying real-time latency metrics is the key. Asking for them to annotate the failover demo with the specific performance data from the PoP you'd actually be routed through changes the request from hypothetical to contractual.
If they can't or won't provide that, you've revealed a gap in their operational transparency. It also subtly tests if their sales engineers have a conduit to the NOC for live data, which tells you about their internal collaboration. A delay or refusal here often means the demo environment is a completely isolated sandbox, and those failover timers are idealized.
RTFM — then ask for the audit
Agree completely on the pillars, especially the need for your own data. Your point about mapping their constructs to your infrastructure is where I've seen deals get stuck for months after a demo.
One addition to the "critical applications" list: you must know the exact protocol and port behavior, not just the name. Saying "SAP" isn't enough. Tell them you need to see a policy built live that isolates your specific SAP GUI traffic on port 32xx from the background RFC traffic, and then watch a simulated failover to confirm the policy state is preserved. If they can't demo policy statefulness during a path switch for your actual app signatures, their "zero trust" claims are just talk.
The TCO model falls apart if you don't also list your current incident counts and mean-time-to-resolution per site. Their operational transformation promise hinges on them reducing those. If you can't quantify your current pain, you can't measure their improvement.
Been there, migrated that
Exactly. The "policy statefulness during a failover" ask is a perfect filter. I saw a vendor demo where the firewall rule stuck, but the SD-WAN QoS markings for that specific app got wiped, causing packet loss on the new path. They considered it a successful failover because the tunnel was up.
You're spot on about quantifying current incidents for TCO. I'd add to ask them *how* their platform reduces MTTR. If they just say "better visibility," ask them to click into a simulated incident in the demo and show the exact forensic log or topology map that would shave minutes off. Otherwise, that operational savings is just a spreadsheet fantasy.
Still looking for the perfect one
>If they just say "better visibility," ask them to click into a simulated incident in the demo and show the exact forensic log or topology map that would shave minutes off.
This is the gold standard. I once pushed a vendor to trace a simulated packet drop from alarm to root cause inside their demo. The engineer had to click through four different dashboards to piece it together - that told me their "single pane of glass" was actually a window with a bunch of separate curtains. The real MTTR savings vanish if your team needs to log into three places to diagnose one problem.
Your point about QoS markings getting wiped is a killer catch. It reveals if their control plane is truly unified or just a bundle of separate modules glued together.
Data doesn't lie, but dashboards sometimes do.
Spot on about the four-dashboard shuffle. That's the vendor's tell. They've built a 'platform' by acquiring three startups and bolting their UIs together under one login page.
But I think that simulated packet trace test is too easy to game. A clever engineer can have the 'correct' forensic log pre-loaded in a tab. The real filter is asking them to do it for an *adjacent* service you just invented on the spot, like a custom internal app on a weird port. If their 'single pane' can't pivot that fast, it's just a pre-rendered slide.
FOSS advocate