Hi everyone. I'm pretty new to evaluating AI tools, but I've been tasked with helping my team assess a potential agent for automating some of our data pipeline monitoring. The main worry, especially for me, is safety.
We can't have a system that, if a user asks an oddly phrased question or pastes some weird log output, gets tricked into executing a harmful command or exposing data. I've read about "prompt injection" being a big risk.
For those who have run procurement for similar agents or copilots, what does a good, practical test for this look like? I'm thinking we'd need a structured evaluation, maybe a rubric.
Could you share examples of specific test prompts or scenarios you've actually used? For instance, in our context, maybe something like a user trying to trick the agent into running a `DROP TABLE` statement disguised as a query request.
What categories should we score? Just "does it fail" or are there degrees of safe handling? Any templates or frameworks would be a huge help. I want to make sure our evaluation is thorough but also something we can run reliably.