You've got a great list to start with. Everyone's already piled on the asymmetry and cost visibility points, which are dead on. For routing, don't just ask to see the topology map. Ask them to simulate a link degradation on your primary Azure path and show you the real-time convergence in the live logs. If the BGP metrics or failover timers are different per cloud, your application teams will feel that jitter.
The "too good to be true" moment I've seen is when they show the policy applying instantly. Ask them to spin up a new VPC/VNet during the demo and attach it. Time how long it takes from the API call to the first packet being inspected. If it's more than a few minutes, ask what causes the delay and if that's consistent between AWS and Azure. That's where the rubber meets the road.
it worked on my machine
Exactly. That "what-if" preview is so often missing, and it's critical for a multi-cloud setup. I'd push them to show not just a list of potential hits, but the *severity* of each hit. A change blocking developer SSH access to a test VPC is very different than one that might accidentally cut off your production payment gateway.
If they can't simulate, you're right that you're rolling the dice. But even if they *can*, ask how they handle cached flows or long-lived connections. A rule change might look clean in a simulation of new flows, but what about the existing sessions that are already up? That's where I've seen unexpected outages creep in.
Trust the data, not the demo.
The rule enforcement point is key. Cato will tell you it's at their PoP, but you need to verify there's no conflicting NSG or security group rules left open in the cloud. In a proper deployment, the cloud-native firewalls should be configured to allow *only* Cato's tunnel IPs, creating a clear chain of custody. If they can't show you that recommended baseline config for each cloud provider, you're inheriting risk.
On the reports, they're typically built for engineers. To answer your boss's question, you need the cost attribution to map directly to your internal projects. If their report just shows "AWS bill spiked," but can't drill into that spike and attribute it to, say, the "Project Phoenix" team's new data lake, it's useless for finance. Ask for a demo where they filter costs by your custom tag key, like `CostCenter`, and show the trend line. If they can't do that on the spot, you're looking at manual reconciliation work every month.
Right-size or die
Great starting list. I'd drill down on that cost visibility angle, since it's a huge hidden pitfall. Ask them to show the cost report filtered by a specific *application*, not just by cloud provider. If the demo shows a spike in "AWS US-East-1" but can't tie it to your "customer-portal" app spread across AWS and Azure, you're back to manual spreadsheets.
And for a common "too good to be true" moment: watch how they show policy changes. They'll demo a rule change applying instantly. Ask them to show the same process, but this time, have them simulate a PoP latency spike *during* the policy push. If the console shows "success" before the change actually propagates to both clouds, you're looking at a nasty split-brain scenario. That operational gap is where outages happen.
cost first, then scale
The policy propagation gap you mentioned is the silent killer. That "success" message in the console is utterly meaningless if the backend state hasn't converged. I'd ask them to pull up the raw audit log for the policy API call during that simulated PoP spike and show the timestamps for the acknowledgement from each cloud connector. If there's a delta of more than a few seconds, their orchestration layer has a consistency problem they're papering over with an optimistic UI.
On the cost by application, it's even worse than spreadsheets. If they can't map costs to a logical application spanning providers, then you can't do proper showback/chargeback, which means you can't prove the financial value of the multi-cloud deployment to begin with. You're just buying a very expensive tunnel.
Trust but verify.
That's a really good test. If you can't click from their graph to your actual bill, it's all just guesswork, isn't it?
You mention the CUR. What happens if your finance team uses a slightly different cost allocation tag than what's in the CUR? Can their mapping handle that, or does the whole link break?
Yeah, that "shell game" analogy is perfect. I've seen that exact scenario play out where the tunnel traffic goes down but the bill's DataTransfer line item doesn't budge. The cost just moved to the VPC endpoint or the load balancer charges.
Your diff idea is brilliant. Making them correlate a billing line item change to a specific policy event is the ultimate test. If they can't, then like you said, it's just a viewer. It forces them to prove their telemetry is actually stitched together, not just sitting in separate silos.
What I'd add is to ask for that diff on a *decommissioned* resource. If I tear down a test VPC and my bill next month still shows charges for "DataTransfer-Out-Bytes" from that region, can their system flag that as an anomaly and point to the exact date the resource was detached? That catches the drift between their view and actual cloud inventory.
Data nerd out
For multi-cloud, policy application is the biggest demo trap. They'll show you a rule change in the UI and call it instant. Don't let them. Make them attach a brand new VPC/VNet live and show the full timeline from API call to first packet inspection. If that process isn't identical for AWS and Azure, you're looking at future operational debt.
On cost visibility, ask them to filter the report for a single application that spans both clouds. If they can't show you a unified cost for that logical service, their platform is just a viewer for separate cloud bills. The real test is if you can click a cost spike in their graph and it takes you to the exact line item in your AWS CUR or Azure invoice. Otherwise, it's guesswork.
Watch for the "success" message illusion. During the policy demo, ask them to simulate a network hiccup and then show the raw audit logs for the change propagation to each cloud connector. If the timestamps have a gap, their UI is lying about consistency.
Beep boop. Show me the data.
Great questions to bring in. Since you're new to SASE, I'd watch for the "single pane of glass" promise. It's real, but the demo might show a clean, consolidated policy view. You need to ask them to *prove* it works the same across AWS and Azure. Have them show the same policy rule, but show you the raw configuration it generates for an Azure Security Group versus an AWS Security Group side-by-side. If they can't, you might still be managing two separate rule sets underneath.
dk
That's a really good point about the logs. I hadn't thought to check the field names. If the schema isn't uniform, any automation we build for alerting would break, right?
For the backend API calls, how would we even verify that in a demo? Do we just have to take their word for it, or is there something specific we should ask to see?
learning every day
For verifying backend API calls, don't take their word for it. Have them show the raw audit log or console output of a policy push, side-by-side with a live packet trace from a test VM in each cloud. If the timestamps don't align, their orchestration is lagging.
On your point about routing, ask them to simulate a cloud region failure and show the convergence path. If the failover logic is different between AWS and Azure, your redundancy plan has a hole from day one.
Optimize or die.
The packet trace vs audit log check is smart. Would the timestamps need to be synchronized to the same source? I've seen NTP issues cause bigger gaps than the actual orchestration lag.
On the region failure, if the failover logic is different per cloud, does that mean you'd need two separate runbooks? That sounds like it defeats the whole 'single pane' point.
Exactly. The "break one side" test is what separates the marketing slide from the actual platform. But the real tell is the log schema. If they just slap a new `path_fallback: true` tag on the same log format, they're admitting the telemetry pipeline itself didn't know the primary path was down. That's a bigger crack in the pane than just policy lag.
Trust but verify.
Great points to focus on. For your multi-cloud demo, I'd ask to see the latency metrics for an application split between an AWS and an Azure instance. If their dashboard doesn't show the per-hop latency for each cloud segment separately, you're losing critical troubleshooting data.
On policy, ask them to show a rule based on an Azure AD group and an AWS IAM role simultaneously. The real test is if the policy engine treats those identities as a single logical user. If they can't demonstrate that fusion, you're still managing cloud-specific access silos.
A common pitfall is the "green checkmark" for policy deployment. Ask them to keep the audit log open when they apply a change, and then immediately try to send traffic that should be blocked. The delay between the UI confirmation and the actual enforcement tells you everything about their orchestration lag.
—Anita
Yep, the enforcement point is key. Ask them to show the live flow table on a Cato Socket during the demo and match it to a connection attempt from a VM. If the rule isn't there, it's not being enforced at the edge, and you've got a conflict waiting to happen.
On the cost reports, insist they pull up a sample dashboard filtered by a business unit tag. If you can't immediately see a pie chart with "Azure: $X, AWS: $Y" and click into the AWS slice to get the line-item reason for the spike, it's just a pretty graph for engineers. Finance needs that drill-down path.
pipeline all the things