Just wrapped up a deep dive into AI runtime vendors (think model serving, inference engines). The pricing and scaling sections were super detailed, but the security sections? Shockingly thin.
Most just had a checkbox list:
* "SOC 2 compliant"
* "Data encryption at rest/in-transit"
* "Role-based access control"
Where's the real detail? I was looking for:
* How they handle prompt injection attempts at the runtime layer.
* Secrets management for API keys to third-party model providers.
* Detailed audit trails for model access and data flow.
* Supply chain security for base images/custom models.
Had to dig through support docs and even then, it was vague. Anyone else hit this wall? What specific security features did you end up prioritizing?
Trial first, ask later.
You're spot on about the checkbox lists. I hit the same issue when evaluating for a healthcare client last month.
The audit trails point is huge - I had to ask three vendors directly if they logged *which* internal model was accessed during a multi-tenant inference call. Two couldn't answer without engineering involvement.
For prioritization, we ended up making supply chain security non-negotiable. If they can't show me a software bill of materials for their runtime container or prove model provenance, that's an instant no now. It feels like the whole space is still treating security as a sales sheet afterthought rather than core architecture.
Your point about prompt injection handling is where the rubber meets the road for runtime security. The checkbox lists completely ignore runtime-specific attack surfaces.
Beyond just logging, I look for whether the vendor's API gateway or inference layer offers configurable sanitization rules or a pattern-matching engine for input/output. Some let you set regex filters or heuristics on payloads before they hit the model, which at least shows an architectural consideration for it. If that feature is absent, it means they're treating security as infrastructure hygiene, not an application-layer concern.
Secrets management is another glaring omission you noted. If the runtime calls external APIs (like OpenAI, Anthropic), ask where the keys are stored and rotated. Do they use a transient memory store, or are keys persisted in a vendor-managed KMS you can audit? The answer usually separates platform-thinking vendors from the rest.
It's not just thin, it's a cost red flag. Those checkbox features are table stakes they bundle into the base price. The detailed runtime security you're looking for - prompt sanitization, granular audit trails - is where the real operational expense is, and they're hoping you won't ask.
When we prioritize, we treat the audit trail detail as a direct line to future compliance costs. If they can't tell you *which* model version served a specific inference in a multi-tenant setup, you're looking at massive engineering hours later to meet any regulatory audit.
For secrets management, turn the question around. Ask who gets charged if their managed service leaks a key to a third-party model API. Their liability clause will tell you more than any security doc.
Your cloud bill is 30% too high