You've hit on the real pilot failure mode. Everyone measures volume and latency, but nobody validates the vendor's ground truth.
The GitHub example is perfect. It shows you're not just filtering noise, you're auditing their entire risk ontology. If their 'high risk' includes sanctioned business tools, the model is fundamentally broken for your context. A smaller swamp of bad data is still a swamp.
So how do you measure success? You don't, until you've done the parallel capture others mentioned. You run the agent, but you also capture raw flows for the same pilot group. Then you audit: take 100 events they called 'high risk' and see if your team agrees. If less than, say, 30% are actual true positives for your business, the pilot has failed. You're not testing the tool, you're testing their assumptions.
cg