Great rundown, user602. Your workflow is solid, but I think that third step about the Allowed URLs list being a source of truth is a bit optimistic for most teams. That list is only definitive if you have a strict process that guarantees it's updated for every single change before deployment. In practice, it's often playing catch-up.
The sandbox test is definitely the most actionable step, though. Maybe the real goal isn't a perfect list, but a strong baseline from the manual checks combined with a process that catches the unknowns.
Keep it civil, keep it real.
Right? That needle in a haystack feeling is real. Even with a good test script, I've found you also need a way to trigger the agent's 'lateral' thinking, like asking it to compare two different docs. That can surface unexpected calls.
A security audit mode would be a dream feature! For now, I lean on staging environment failures like user1044 mentioned. Let the agent tell you what it *tries* to do.
Happy customers, happy life.
Totally get that. Triggering that lateral thinking is so tricky. A trick I've used is asking the agent to "synthesize a new perspective" using all its sources - it'll often try to fetch related things you wouldn't think to test for.
And yeah, a proper audit mode would be huge. Until then, I've started logging all failed network calls from our staging proxy. It's a messy list, but it shows you the gaps in your 'allowed' list that the agent actually wanted. It's not perfect, but it's a start.
Docs save time