Vendors love to claim "guaranteed execution" in their demos. It's almost always a canned script running on pre-provisioned, over-sized infrastructure.
To verify, you need to move from watching a presentation to a controlled test. Don't let them drive.
* **Demand a fresh environment.** No pre-warmed containers or cached queries. They should spin it up live, from zero. Watch the clock.
* **Inject your own data.** Not their perfectly curated sample set. Use a messy export from your actual systems. Ask for the data volume and profile upfront.
* **Define "execution" in your contract/SLA.** Is it query response time under load? Is it pipeline completion time with 10TB? The demo must test the exact metric you will pay for.
* **Run it during their peak business hours,** not at 2 AM in their timezone. Ask to see resource utilization (CPU/memory) during the demo. If it's pegged at 100%, their scaling claim is fake.
If they refuse any of this, the guarantee isn't worth the slide it's written on.
show me the bill
Completely agree on the fresh environment and injecting your own data. I'd add that you should ask to see the audit trail for that environment's creation. In a proper cloud service, that spin-up from zero will be logged in their control plane (something like AWS CloudTrail or Azure Activity Log). If they can't show you a timestamped log entry for the API call that created the demo instance right then, it's almost certainly a pre-existing container.
The resource utilization point is key, but go a step further. Ask them to run a representative workload *while* also pulling those metrics. A real scalable system should show a spike and then a return to baseline as it scales out. If CPU stays pegged at 100% for the duration, you're just watching a single box sweat.
Logs don't lie.
Audit trails are theater. They can spin up a fresh "instance" that's just a namespace on an already-overprovisioned cluster. The API call logs prove nothing about isolation or true capacity.
And watching a CPU spike then drop is the oldest trick. It's usually just auto-scaling a container group they had pre-warmed. The real question is the delta between the spike and the baseline they promised you. If baseline is 50% utilization on their "demo pod", you're already buying half a system.
Your stack is too complicated.
> The real question is the delta between the spike and the baseline they promised you.
Exactly. That baseline is the whole scam. They promise you 90th percentile latency of 200ms, but that's on their demo pod sitting at 60% idle. Your real workload pushes it to 95% and latency quadruples. The performance guarantees in the SLA are only valid under their "recommended" utilization, which is pure fiction.
They never let you define the baseline. Your job is to nail that down before the demo. Make them commit in writing: "When the system is at 80% resource utilization, the query performance will be X." Then make the demo prove *that*.
Trust but verify.