I love the idea of asking for their renewal increase history. Smart. When I've done this, I'll also ask for the variance - the highest and lowest increase they applied in the same year. It shows if they have a flat policy or if it's negotiable based on account health.
You're right about the sample ticket with notes, but legal will always block it. A practical middle ground is to ask for their standard ticket template or a list of every field their SOC fills out for a severity-1 incident. If that list is thin, you know the replies will be too.
Data > opinions
Those are excellent starting points. Your focus on the day-to-day UI is key, and I'd push them further on that.
For the dashboard walkthrough, don't accept a pre-recorded video. Ask them to pull up a live, anonymized view of their global threat dashboard *during the call*. Seeing real, current data, even if it's just background noise, tells you more about the interface than any demo. If they can't or won't, it's a red flag about either the tool's maturity or their transparency.
Also, on the human element, ask about their own internal change management. How do they roll out updates to the protection service? A vendor that pushes changes without clear communication can create nasty surprises for your team's operations.
Beta tester at heart
Love the live dashboard idea, that's clever. It feels like the difference between seeing a car in a showroom and actually taking it for a drive.
I'm curious, has anyone here actually gotten a vendor to do that? I feel like they'd say it's "against policy" or something. But you're right, if they can't show you real, live data even in a limited view, it makes you wonder what's up.
CloudNewbie
Oh, I've gotten them to do it. You just have to frame it as a test drive for a specific use case. "Show me how you'd walk my CISO through an active volumetric spike in the APAC region right now."
If they balk at "policy," ask what their policy is for providing transparency to paying prospects. That usually cuts through it. Sometimes the dashboard is just a pretty front-end to a data lake with a 15-minute lag, and they don't want you to see the timestamp.
- elle
Your plan covers the basics, but you're trusting their narrative on a "real incident report." Insist they show you one. A real, anonymized case from their ticketing system, with full timestamps and the complete internal commentary. If they claim policy prevents it, ask for their standard incident template instead. The number of fields they have for analyst notes is inversely proportional to how useful those notes will be.
Data skeptic, not a data cynic.
You're spot on about the renewal history. Asking for a five-year trend is a good start, but I'd also ask for the underlying CPI or RPI index they typically tie it to. It forces them to show if their increases are truly formulaic or if they're just picking a number.
On the sample ticket, you'll almost never get one with real internal notes for liability reasons. A more effective angle is to ask for their escalation matrix and the specific SLA gates. If a ticket automatically escalates from L1 to L2 after 30 minutes, but the L2 team is only measured on 'time to first touch,' you still get shallow responses. You need to know how the *second* response is measured.
—chris
Great questions to start with. Since your focus is on long-term operational fit, I'd press them on how their dashboard works during an actual attack.
The walkthrough they give will likely be a perfect scenario. Try asking about a messy one - what happens when there's a DDoS attack and a separate critical incident at your end simultaneously? Does the UI hold up when your team is stressed and multitasking? That's the real test for intuitiveness.
Also, on the handoff process, ask who exactly is on the other end. Is it a dedicated engineer you can name, or a rotating shift in a NOC? The consistency matters more than they usually admit.
Exactly. If they won't show you the dashboard under load, ask about their logging API. The pretty UI will fall over, but you need to know if you can still get the raw data out to your own systems.
Keep it simple
Solid starting list, especially focusing on the UI for non-specialists. I'd add one more layer to your Day-to-Day Management question.
When they walk you through the dashboard, ask for their definition of "resolved" in an incident report. Does it mean traffic is back to normal, or just that their mitigation is active? That gap can cause major confusion with your own status pages.
For the handoff process, push them on the shift schedule. If it's a rotating NOC, ask how historical context about your environment is preserved between shifts. You don't want to explain your topology from scratch during an attack.
Spreadsheets > marketing slides.
I like the push on "resolved." That definitional gap isn't just for status pages - it directly impacts how your team logs its own incident hours for postmortems.
On the rotating NOC point, a good follow-up is to ask for the average tenure of their front-line analysts. A shift schedule with high turnover means even a great knowledge base won't compensate. You'll be dealing with constant re-education.
Measure twice, spend once