Great rundown, user602. Your workflow is solid, but I think that third step about the Allowed URLs list being a source of truth is a bit optimistic for most teams. That list is only definitive if you have a strict process that guarantees it's updated for every single change before deployment. In practice, it's often playing catch-up.
The sandbox test is definitely the most actionable step, though. Maybe the real goal isn't a perfect list, but a strong baseline from the manual checks combined with a process that catches the unknowns.
Keep it civil, keep it real.
Right? That needle in a haystack feeling is real. Even with a good test script, I've found you also need a way to trigger the agent's 'lateral' thinking, like asking it to compare two different docs. That can surface unexpected calls.
A security audit mode would be a dream feature! For now, I lean on staging environment failures like user1044 mentioned. Let the agent tell you what it *tries* to do.
Happy customers, happy life.
Totally get that. Triggering that lateral thinking is so tricky. A trick I've used is asking the agent to "synthesize a new perspective" using all its sources - it'll often try to fetch related things you wouldn't think to test for.
And yeah, a proper audit mode would be huge. Until then, I've started logging all failed network calls from our staging proxy. It's a messy list, but it shows you the gaps in your 'allowed' list that the agent actually wanted. It's not perfect, but it's a start.
Docs save time
Logging failed calls from a default-deny proxy is the closest you'll get to a real audit. It's a reactive control, but at least it's based on actual behavior, not speculation.
You're right that it's messy, but that's the point. The list of domains an agent *wants* to call is more valuable than a static list of what you think it should call. That delta is your actual risk surface.
Just make sure your proxy logs are detailed enough to tie each failure back to the specific agent instruction or knowledge source that triggered it. Otherwise, you're just left with noise.
Trust but verify — especially the fine print.