I’ve been evaluating Arize AI for potential use in our ML monitoring stack, and the security documentation, particularly the audit reports they reference, is a key factor. As someone focused on compliance and penetration testing, I was hoping for more transparency.
From what I’ve gathered, Arize undergoes third-party security audits, which is a positive baseline. However, the publicly available information seems to be a high-level summary or a compliance certificate (like SOC 2). For a security-focused assessment, this leaves several practical questions unanswered:
* What was the exact scope of the audit? Were specific penetration tests against the API and web application conducted, or was it primarily a controls review?
* Are any detailed findings or remediation evidence available to potential enterprise clients under NDA?
* How does the audit address data handling in a multi-tenant SaaS environment, especially concerning model artifacts and inference data?
In my experience, a summary report is standard for marketing, but technical teams need to understand the depth. For example, does the audit cover the security of the agent installation or just the cloud platform?
I’d be interested to hear from current users, particularly in regulated industries:
* Has anyone successfully requested and reviewed a more detailed audit report or testing summary from Arize?
* Were there any specific findings related to vulnerability management or threat modeling that influenced your deployment architecture?
That's a great set of questions. The scope question is key - in my experience with cloud vendors, that SOC 2 Type II report is almost always about the *platform* controls, not penetration testing on the application itself.
You can usually get the detailed report under NDA during a serious procurement process. I'd push your sales rep on it. For our last vendor review, getting that full document made all the difference - it showed exactly what was tested (and what wasn't).
Your point about the agent is spot on. If they're pushing an on-prem collector or a Kubernetes sidecar, you need to know if that's in scope. Otherwise, you're inheriting that security burden.
Right-size everything
Totally get where you're coming from. The gap between the marketing summary and what a tech team actually needs is real.
What would you recommend asking for first? I'd think pushing for the full audit scope doc, even before the full report, would clarify a lot. Then you'd know if you even need to ask about the agent or the API tests.
Agreed, starting with the scope doc makes sense. It cuts through the noise.
But in my experience, even getting that can be tricky. Vendors sometimes treat it like the full report. Have you had any luck getting a scope document without starting a full procurement cycle?
Yeah, I've hit that wall too. It's frustrating when they bundle everything into a "security package" you only get after signing an NDA.
One tactic that's worked for me is asking for a redacted scope excerpt, maybe just the "Objectives and Boundaries" section. Sometimes they're more willing to share a couple of pages that outline what *was* tested, without the detailed findings.
Has anyone tried asking for the audit firm's name and the standard they audited against? You can sometimes infer the scope from that, like if it was purely ISO 27001 vs something more app-focused.
Infrastructure as code is the only way
You hit on exactly what I've been wondering about too. That gap between the high-level certificate and what the tech team actually needs to approve something is huge.
>The audit address data handling in a multi-tenant SaaS environment
This is my biggest question. For a tool that's handling model data, the isolation piece seems critical. I'd want to know if their audit covers logical separation controls, not just physical.
Is it common for vendors in this space to offer the full scope doc upfront, or is it always an NDA hurdle?
not a buyer, just a nerd