After leading the third cloud vendor security review this quarter for our ML platform initiative, I became convinced that the industry's standard "security questionnaire" is fundamentally ill-suited for evaluating the new generation of AI/ML inference runtimes and hosted model services. These services blur the lines between traditional SaaS, managed containers, and bespoke GPU-accelerated infrastructure, rendering generic checklists ineffective.
Our internal security and cloud architecture team has therefore developed a targeted assessment template. It moves beyond "Do you have SOC 2?" to probe the specific architectural and operational risks inherent in AI runtime environments. The goal is to generate actionable data for our risk register and architecture diagrams.
The template is structured around five core domains:
* **Data & Model Governance:** Focuses on data in transit, at rest, and during processing within the runtime's memory. Key questions address model provenance, training data lineage, and cross-tenant isolation mechanisms (e.g., GPU memory scrubbing, hypervisor-level separation).
* **Compute Fabric Security:** Evaluates the underlying container or serverless orchestration. We demand clarity on pod security contexts, privilege escalation controls, and the security posture of the base container images for custom runtimes.
* **Network Control Plane:** Assesses how the runtime integrates with our existing VPC, service mesh, and egress controls. This includes support for private endpoints, deterministic egress IPs, and the ability to inject Istio sidecars or similar proxies for east-west traffic encryption and policy.
* **Observability & Audit:** Goes beyond basic logging to require structured, machine-readable audit trails for model invocation, scaling events, and access to sensitive configuration. We require OpenTelemetry compatibility for traces and metrics.
* **Supply Chain & Integrity:** Scrutinizes the pipeline for delivering the runtime software itself, including signed artifacts, SBOM availability, and patch SLAs for critical vulnerabilities in the underlying stack (e.g., CUDA drivers, Python dependencies).
Below is the YAML structure we use to score responses. Each criterion is weighted based on our enterprise's risk appetite.
```yaml
assessment_framework:
vendor: "Example_AIVendor"
service: "Managed Inference Runtime"
domains:
- name: "Data & Model Governance"
weight: 0.3
criteria:
- description: "End-to-end encryption (at-rest, in-transit, in-memory)"
response: "Partial - in-transit and at-rest only"
score: 2
max_score: 5
- description: "Guaranteed tenant isolation at GPU memory level"
response: "Yes, via dedicated GPU instances"
score: 5
max_score: 5
- name: "Compute Fabric Security"
weight: 0.25
criteria:
- description: "Runtime supports non-root user execution"
response: "Yes, configurable in runtime spec"
score: 5
max_score: 5
- description: "Provides attested, signed SBOM for base image"
response: "SBOM available upon request"
score: 3
max_score: 5
- name: "Network Control Plane"
weight: 0.2
criteria:
- description: "Supports private VPC endpoint ingress"
response: "Yes"
score: 5
max_score: 5
- description: "Allows sidecar injection for service mesh integration"
response: "No"
score: 0
max_score: 5
```
The exercise of forcing vendors to respond to this specific template has been revealing. Several major providers could not adequately detail their GPU memory isolation, and many managed Kubernetes-based runtimes had default privilege settings that were unacceptable. I encourage others to adapt this template and share their findings; we need to collectively raise the bar for what constitutes a secure AI runtime deployment.
Boring is beautiful
Totally feel your pain on the standard security questionnaires being useless here. We hit the same wall last year when evaluating some hosted model services for our marketing automation predictions. Asking about SOC 2 is just a checkbox; it doesn't tell you if your customer data is being co-mingled in GPU memory during inference, which is the real scary part for us.
Your five domains look spot on, especially the **Data & Model Governance** focus. In our reviews, the model provenance question became a huge sticking point - one vendor couldn't trace the training data for a "premium" model back through its layers, which essentially black-boxed it for our compliance team. That lineage is everything.
Really curious about the **Compute Fabric Security** domain you mentioned. Did you find you needed to get into the nitty-gritty of the hypervisor or Kubernetes layer with these vendors, or were they resistant to sharing that depth? Some of the bigger players just gave us a glossy overview doc.
If it's not measurable, it's not marketing.
Oh, that GPU memory co-mingling point is exactly why we had to push so hard on Compute Fabric! For the bigger, "platform" style vendors, they were *extremely* resistant to share hypervisor or K8s details. We got a lot of "our infrastructure is proprietary" and links to their generic compliance pages.
The breakthrough was asking not just *what* they use, but *how* they enforce tenant isolation within it. We started phrasing questions like, "Can you describe the technical controls (e.g., kernel-level isolation, GPU memory partitioning) that prevent data leakage between concurrent inference sessions on the same physical hardware?" That forced them out of glossy overviews into specifics.
We found the smaller, niche inference runtime vendors were sometimes more willing to talk shop, maybe because it's a selling point. One even shared their use of specific K8s security contexts and node tainting strategies, which was gold for our diagram.
Your point about asking *how* isolation is enforced is the key tactic. We've had to do something similar, but we framed it around specific failure modes we've observed in our own on-prem GPU clusters. For example, asking "What mechanisms prevent a CUDA kernel from one tenant from performing a side-channel timing attack against another tenant's memory space on the same GPU?" This forces a conversation about MIG, MPS, or time-slicing configurations that generic docs never cover.
The smaller vendor transparency is a double-edged sword, though. While they're more willing to share, their controls are often less mature and haven't been stress-tested at scale. We documented one provider's node tainting strategy, only to find it broke down during autoscaling events, causing a mixing of workloads from different security tiers. Their technical openness was valuable, but it revealed a gap their sales deck omitted.
In our last assessment, we started mapping their technical answers to known CVE databases for the underlying tech (like specific GPU driver versions or K8s CVEs). It shifts the discussion from "we have controls" to "how quickly are patches applied to this proprietary fabric?"
data is the product
This is exactly where the standard questionnaires fail. Moving beyond SOC 2 to focus on *runtime-specific* data handling is the key shift.
Your **Data & Model Governance** domain reminds me of a problem we had with a model cache. The vendor's service cached fine-tuned models for performance, but the purge mechanism was tied to a billing cycle, not the user session ending. So a model trained on our internal data could persist in their cache longer than our contract allowed. Asking about "GPU memory scrubbing" is good, but also check for these secondary storage layers for models themselves.
Could you share a sample question from that domain? I'm curious how you phrase the provenance one.
Data is the new oil - but it's usually crude.