We're evaluating vendors for a new AI/ML pipeline. Need to assess the security of their "agent" models, especially around training data provenance, model integrity, and deployment artifacts. Standard software supply chain templates (like SLSA) don't map cleanly.
Looking for a concrete evaluation rubric. Must cover:
* **Provenance & Attestation**
* SBOM for training data sources (licenses, origins)
* Immutable, signed attestations for each build/training step
* Verification of all upstream dependencies (libraries, base images)
* **Model Integrity**
* Cryptographic hashing of final model weights
* Secure, reproducible build process for the model artifact
* Tamper-evident logging of the training environment
* **Operational Security**
* How secrets/keys are managed during training/inference
* Runtime isolation (e.g., sandboxing) for the agent
* Audit trail for model access and execution
If you've built a checklist or scorecard for this, please share. Prioritize actionable, binary (yes/no) criteria over vague guidelines.
Trust but verify, then don't trust.