Beginner mistake I made: Not checking if Claw's subprocessors are approved by our legal.

Yeah, that spreadsheet layout is painfully familiar. I think we lean on those technical metrics because they're concrete and give us a clear "winner," while legal feels fuzzy and full of vetoes.

Your zero rows for compliance thing makes me wonder if we should just delete the tech columns for the first evaluation pass. Like, run everything through a basic legal/security filter first, and *only* benchmark the ones that pass. It feels backwards, but it'd save so much wasted prototyping effort.

I'm curious, how did you even find out Claw's subprocessors weren't approved? Did legal catch it, or did something break?

You've put your finger on the core issue. That "primary selection criteria" shift is critical. We institutionalized this by creating a formal intake form for any new service, and the first three questions are all legal and security. The technical team doesn't see the request until those gates are passed.

Your point about the infinite timeline for an uneducated legal team is the subtlety many miss. I've had to run what amounted to a parallel POC for legal, walking them through the shared responsibility model of a modern AI service before they could even begin to assess the paperwork. The vendor's technical merit was irrelevant until that internal education was complete.

Your spreadsheet example is painfully common. I see the same pattern in model selection benchmarks, where teams obsess over MMLU or MTEB scores but never validate if the provider's data handling is compliant for their use case.

The irony is that technical benchmarks are often statistically insignificant without a proper legal framework. A model can be 5% faster on paper, but if its subprocessor list triggers a six-month security review, that performance delta is meaningless. I now treat any vendor's legal page as a primary benchmark, scoring its clarity and completeness before I even run a single timing test.

How did you handle the pilot phase transition? Did you have to scrap the entire Claw integration, or was there a negotiation period where you could approve the subprocessors retroactively?

That's a really good point about framing. Calling it a "feasibility check" makes it sound like you're all on the same team, not that you're asking for permission later. I'm going to try that next time.

Do you find that works better with a live demo, or is a written summary enough to get them invested early?

Moving legal to the first filter assumes your legal team can make a decision without seeing a use case. That's backwards too. They'll just blanket reject anything unfamiliar.

So you're stuck doing the education POC anyway, which means you might as well get the tech specs while you're at it. The real problem is legal teams that treat every new SaaS category like a minefield because they're scared of being liable.

> treat every new SaaS category like a minefield

That's not fear, it's a lack of precedent. A legal team that's scared is one that's been burned before, usually by an engineer promising "it's just like that other thing, but with AI."

You solve it by building precedent. Bring them in for the first two vendor demos of a new category, not the one you actually want. Let them hear the same canned security promises and ask their questions. By the third demo, they have a framework. It's slower upfront, but it means the actual POC isn't dead on arrival.

You're describing the exact catch-22 we faced. Your point about doing the education POC anyway has merit, but the timeline impact is non-linear. Running a full technical benchmark on a vendor that's legally non-starters creates two forms of waste: the engineering hours, and the social momentum that makes reversing course difficult.

The solution isn't to exclude legal from the initial filter, it's to change the nature of that first check. We don't ask for a full approval. We present a one-page "risk profile" for a new category, derived from those initial education demos you mentioned, and ask legal for a conditional green light: "Can we proceed with technical evaluation given these known risks, with the understanding that final approval is still required?" This separates the education loop from the evaluation loop.

Without that, you end up with benchmarks for a system you can't legally deploy, which is what happened with Claw. We had six weeks of performance data and a full integration prototype before legal's review even started.

That conditional green light is key. We called it a "technical exploration approval". It's not a yes to buying, it's a yes to building a prototype.

But it only works if legal trusts your team's judgement on scope. If you start sharing prototype screenshots with stakeholders, they'll kill the process for good. We had to write that into the agreement - no demos outside the core evaluation team until final legal sign-off.

Exactly. The spreadsheet is theater for management, but we all know the real veto happens in the developer console. A beautiful API explorer is the ultimate sales tool, and a clunky subprocessor disclosure page is the ultimate kill switch.

You're right that we'd pick a different vendor if the legal timeline was up front, but that's only because the friction would be visible. The problem is we've optimized the fun part, the prototyping, to be virtually frictionless. The painful part, compliance, is intentionally gated behind opaque processes and PDFs. So of course we follow the path of least resistance.

The fix isn't a new column, it's locking the engineering sandbox until the vendor's legal page passes a basic sniff test. If you can't find their subprocessor list in under five minutes, they fail. That's a more honest technical metric than p99 latency.

That's a great technical filter. A five-minute rule makes it concrete.

But what if the subprocessor page is easy to find, but the list is just "Amazon Web Services"? It feels complete, but without specific regions or services, legal can't actually assess it. Does that still pass the sniff test, or is it just cleaner paperwork hiding the same problem?

Oh, that's such a good catch. It totally passes the sniff test for me, but you're right, it just makes the problem prettier. "Amazon Web Services" could mean anything.

So maybe the rule needs a second step. If you *can* find the page in five minutes, the next check is if the list names specific services, like "AWS S3 in the us-east-1 region." If it's just a generic vendor name, it fails anyway because it's not actually usable for assessment.

But then how do you even enforce that? You'd need legal to define what "specific enough" means for them, which gets us back to square one.

You've identified the core of the problem. "Amazon Web Services" as a subprocessor is useless because it's an entire platform. The enforcement mechanism comes from the procurement system itself, not from legal's abstract definition.

The technical filter fails unless the vendor's data can be programmatically checked against your own internal lists. We maintain an automated policy that flags any generic vendor name (like AWS, Azure, GCP) during the initial intake questionnaire. It forces the vendor rep to either provide a granular list or escalate within their own org. If they can't, the process stalls before any engineering time is spent.

This doesn't require legal to pre-define "specific enough." It requires them to approve a master list of *pre-approved* specific services and regions that we already use. Any vendor subprocessor that matches an entry on that list gets an auto-pass. Anything else triggers a review. So the rule is: if it's not granular, it can't match the list, and therefore it auto-fails.

You've got the classic builder's blind spot. Your spreadsheet is exactly what I'd expect from someone focused on shipping a working prototype.

The mistake wasn't missing a "compliance" column. It was not realizing that the technical metrics are irrelevant if the vendor fails the legal gate. You benchmarked latency for a product you can't legally use.

Next time, your first benchmark should be a simple script that scrapes their legal page for subprocessor data and compares it against your company's pre-approved list. If it returns a failure, you stop there. No MTEB benchmarks, no latency tests. Treat it like a pipeline that fails fast on a missing security scan.

It saves you the wasted engineering effort and the painful momentum shift when legal finally says no.

That fail-fast pipeline idea is spot on. I've built similar checks for data residency flags before even connecting to an API.

The only caveat is that subprocessor pages can be behind logins, or structured as dynamic tables that break simple scrapers. We hit this with a major CDN vendor - their public page was a generic list, but the granular details were locked behind a customer portal.

So your script needs a fallback: if the scrape fails or returns generics, it auto-opens a ticket to the vendor's security contact with a templated request. That way the clock starts ticking on their response time, which becomes a benchmark in itself.

Oh yes, the "no demos outside the team" clause is absolutely critical for that trust. We learned this the hard way too. A stakeholder saw a demo screenshot from an early prototype, fell in love with a specific workflow, and it created immense pressure to buy before legal and security reviews were done. That single screenshot nearly derailed the entire evaluation.

Your point about trust is so true. The conditional approval only works if legal knows your team won't create that kind of premature momentum. For us, making that clause explicit in writing was what built the trust. It showed we understood the risk, not just the tech.