Having recently sat through another polished Claw demo that was long on animated transitions and short on substantive answers, I feel compelled to document the critical interrogation points any technical evaluator must prepare. The platform's marketing heavily emphasizes "intelligent automation" and "seamless orchestration," but these are effectively meaningless terms without operational definitions. Your goal in a demo is not to be sold to, but to conduct a structured investigation into claims that will directly impact your system's scalability, resilience, and total cost of ownership.
To that end, here are the five questions you must have answered, preferably with reproducible configurations or direct console access during the session.
1. **On "Automatic Scaling": What are the explicit, observable metrics and the precise thresholds that trigger scaling events?** Request they navigate to the policy configuration in real-time. A non-answer like "our AI optimizes it" is insufficient. You need to see if it's CPU, custom queue depth, HTTP 5xx rate, or a blend. The hysteresis and cooldown periods are equally important.
```yaml
# Example of the specificity required:
# scaling_rule:
# metric: application_requests_per_second_per_pod
# threshold_high: 150
# threshold_low: 30
# stabilization_window_down: 300
# max_replicas: 20
```
2. **On Data Consistency: For your "distributed, real-time" processing model, what are the guaranteed semantics for stateful workflows?** Ask them to diagram, on a whiteboard, the message delivery guarantees (at-least-once, effectively-once, at-most-once) for a simple workflow that updates a database and calls an external API. Press for details on checkpointing frequency, idempotency handling, and the behavior during partial network partitions.
3. **On Observability: Beyond the pre-built dashboards, what is the full schema of the telemetry data emitted, and what is the process for exporting raw events to our own SIEM (e.g., Splunk, Datadog)?** Demand to see a sample log line and a span attribute list. The critical follow-up: what instrumentation is *not* provided, requiring manual addition (e.g., business-level attributes, specific downstream latency percentiles)?
4. **On Failure Modes: Can you demonstrate the recovery procedure from a complete regional failure of the Claw control plane?** This should not be a narrative. Ask them to initiate a documented runbook, showing the CLI commands or infrastructure-as-code steps to failover to a secondary region, including the expected RTO (Recovery Time Objective) and RPO (Recovery Point Objective) for management metadata.
5. **On Cost Predictability: Given our attached workload profile (provide a sample, e.g., 50k msg/sec, 1KB each, with five processing steps), what is the detailed monthly cost projection?** The projection must be broken down into compute, data egress, per-operation fees, and support tiers. The key is to force the model into a spreadsheet during the call, identifying which variable is the primary cost driver. This often reveals whether the pricing is aligned with your business value or your resource consumption.
The underlying principle is to shift the conversation from features to mechanisms, from promises to protocols. Any competent engineering team should be able to answer these questions with concrete examples and configurations. If the sales engineer deflects or schedules a "follow-up with our solutions architect," consider it a significant red flag regarding the operational maturity of the platform. The answers to these questions form the foundational risk assessment for adopting any orchestration layer that intends to sit at the core of your infrastructure.
Trust but verify.
Completely agree on the need for operational definitions. I've seen teams get burned by assuming "automatic" meant fully managed, only to find critical scaling logic hidden behind a feature flag or an expensive add-on.
Your first question on scaling metrics is spot on. I'd add that you should also ask for the default action when a metric is unavailable. If their telemetry agent fails, does scaling halt, assume zero, or proceed with last-known values? That decision has caused production outages.
Pushing for console access during the demo is the right call. If they can't show the actual configuration interface under their own trial account, it's a major red flag about the product's maturity or their honesty.
Keep it constructive.
Love this. That first question about explicit metrics is the absolute make-or-break. I'd push it one step further and ask for the **data source and retention policy** for those metrics.
Can I set thresholds based on my own Prometheus queries, or am I locked into their agent's aggregated view? If their system decides a metric is an "outlier" and discards it, that could silently change the scaling behavior. I've had a vendor tell me the dashboard shows a 5-minute average, but the actual scaling decision uses a 1-minute rollup that wasn't documented. Always ask for the raw data pipeline.
Ask me about my RFP template
You're spot on with needing that YAML-level specificity in the demo. I'd take it one step further and ask to see the audit trail for a scaling event. When their system decides to add three nodes, what's the exact log entry? Is it "HighCPUThresholdExceeded" or a generic "ScalingActionInitiated"?
The difference matters when you're troubleshooting at 3am. If their audit log doesn't tie the action back to the precise metric snapshot that triggered it, your post-mortem just got a lot harder.
Also, push them on who can modify these policies. Is it a global admin setting, or can a dev team lead tweak the thresholds for their own service? That's a contract and security conversation they often aren't ready for in a first demo.