Hi everyone. I'm new to this AI tools space and honestly a bit overwhelmed. My team is looking at a vendor for an AI-powered cloud monitoring tool. We have our first sales call next week.
I know sales demos can be... polished. 😅 I'm a cloud admin, so I think in terms of infrastructure, security, and cost. What specific, technical questions should I ask to see past the marketing and understand if this will actually work in our AWS environment?
For example:
* "Can I see the actual API spec or Terraform provider for onboarding?"
* "How does your tool handle data residency? Show me the VPC endpoint configuration."
* "What's the exact cost model per API call or per GB of log data ingested?"
What else would you ask? I want to avoid getting a tool that locks us in or creates a surprise bill.
Good list. I'd add a few more based on benchmarking AI tools specifically.
Ask about the underlying model and its refresh cycle. "Which foundation model powers the anomaly detection? Is it GPT-4, Claude 3, a proprietary model? When was it last updated or retrained on new AWS service data?" The performance drift on older models can be significant.
Request a latency SLO for their API calls under load. "What's the p99 latency for an analysis request during peak periods, and how does that degrade with concurrent requests?" You don't want your monitoring tool to become the bottleneck.
Finally, ask for a real, anonymized benchmark report comparing their tool's alert accuracy against a standard dataset or a baseline like CloudWatch Anomaly Detection. If they can't provide one, that tells you something about their engineering rigor.
BenchMark