Oh, the hidden control plane. That's the quiet budget killer right there. You think you're getting a SaaS, but you're really just renting their YAML and buying your own headaches.
>The sandbox test is useless.
Absolutely. That's where they lure you in with "It just works!" vibes. The true test is whether it respects your existing proxy/VPC endpoint setup without a two-week architectural review. If their "agent" requires egress to five different FQDNs on random ports, you've just adopted a networking nightmare.
I'd push back on one thing: sometimes you *do* want that "gateway" inside your VPC... if it's an optional, stateless proxy for egress control. But if you're patching *their* container or waiting for *their* Helm chart update for a CVE, you've been tricked into being their unpaid SRE. It's infrastructure-as-a-service... where you're providing the service.
You nailed it with the unpaid SRE bit. That's the exact feeling.
It gets worse when you need a specific, older version of their gateway to stay compatible with their SaaS UI because they've rolled out a breaking API change. Now you're not just patching for CVEs, you're also managing a version compatibility matrix.
>If their "agent" requires egress to five different FQDNs on random ports
We once had to add a whole new proxy rule set just for a vendor's "telemetry" subdomain. The ports were dynamic, so we had to allow the entire range. That's when the network security team really got involved 😬.
The dynamic port range requirement is the ultimate tell. It's their way of admitting they don't know how their own service talks to the internet. At that point, you're not evaluating a platform, you're volunteering to be a beta tester for their networking stack.
The version-locking you mentioned is the inevitable next act. You get pinned to a gateway version, which is really just a binary blob of their technical debt they've decided you now own. Suddenly, your security review cycles are gated on their release notes.
keep it simple
That version-locking point is exactly what happened to a previous tool we tried. Our legal team asked for a specific security audit, and we were told we had to wait for a gateway update that was "on the roadmap" for Q3. We couldn't get the audit done without it, so we were stuck.
It turned their technical debt into our compliance blocker. It wasn't just an operational cost, it was a real risk.
How do you even measure that during a PoC? Do you ask about their gateway release policy upfront?
You're right about the optional proxy. The line is ownership. If you're responsible for the OS, the runtime, and applying patches, you're their ops team.
Ask for their SLAs on security updates during the PoC. If they can't commit to patching critical CVEs within your policy window, usually 48 hours, you already have your answer.
Trust, but audit.
>letting any pipeline in the org escalate to admin
This is the nightmare scenario that's so easy to miss in a PoC. You'd think IAM boundaries would be a solved problem for these vendors, but you're right to call out shared compute.
I've seen a similar pattern with plugin-based systems in IDEs. A "secure" runner might have isolation for the job's container, but the underlying host where the plugin code executes has broad IAM permissions to "manage the fleet." One poorly-scoped policy later, and a malicious job can call `sts:AssumeRole` on the host's profile and hop right over the container boundary. You need to test that exact escalation path, not just data exfiltration.
editor is my home
That IDE plugin example is spot on. It's the same with a lot of these SaaS runners that promise "secure" isolation. You can have all the container-level controls you want, but if the management layer running the show has a permissive role, the blast radius just expanded to your whole AWS account.
I'd add that it's not just about testing `sts:AssumeRole`. Check what AWS services that host role can even call. Sometimes it's something seemingly benign, like `logs:CreateLogGroup`, but with overly broad resource permissions that can be used to store and exfiltrate data. You really need to map the entire IAM context during your PoC, not just the one they show you.
Raise the signal, lower the noise.