Skip to content
First-time evaluato...
 
Notifications
Clear all

First-time evaluator here. What red flags should I look for in AI agent security docs?

6 Posts
6 Users
0 Reactions
3 Views
(@harperl)
Estimable Member
Joined: 3 months ago
Posts: 127
Topic starter   [#29358]

Hi everyone. I'm looking at a few AI-powered ticketing and chatbot platforms for our small support team. The sales demos look great, but I know security is important.

Since I'm new to this, what are the key red flags I should watch for when they share their security docs or answer questions? I'm thinking about things like vague data handling policies or missing compliance details. Any specific gotchas from your experience would be a huge help.

Thanks in advance 🙏
newbie


Ask me in a year


   
Quote
(@carlosm)
Honorable Member
Joined: 3 months ago
Posts: 339
 

Vague language on data retention and deletion is a huge one. If they can't give you a specific timeline for when customer chat logs are purged, that's a major flag. I'd ask, "Where are the training datasets stored, and is my support data used to train other customers' models?"

Also, watch for vague answers about third-party subprocessors. If they just say "trusted partners" without a clear list and the right to object, you're likely looking at a problem. Always ask for their most recent SOC 2 Type II report or similar. If they hesitate, walk away.

And don't forget a simple check: try to find their own privacy policy and security docs on their public website, not just the sales deck. If it's hard to find or missing key details, that tells you a lot about their priorities.


Keep automating!


   
ReplyQuote
(@eval_newbie_2025)
Honorable Member
Joined: 4 months ago
Posts: 370
 

Oh wow, the point about checking their public website is something I wouldn't have thought of on my own. That's a really simple but smart test.

The SOC 2 report tip is super practical. Is it fair to assume that if a vendor is SOC 2 compliant, they've probably got the subprocessor list and data handling stuff reasonably sorted? Or are those separate things I should still ask for directly?

Thanks for these concrete examples, by the way. It helps a ton to know what to actually ask for.



   
ReplyQuote
(@backend_builder)
Prominent Member
Joined: 6 months ago
Posts: 605
 

Great question. SOC 2 is a good baseline, but it's not a guarantee they've got subprocessors fully sorted. The report will detail controls, but you need to see if the list is in the report or a separate appendix. I'd still ask for it directly.

A caveat: SOC 2 is about their controls at a point in time. A clear, up-to-date subprocessor list on their website shows ongoing transparency. If it's only buried in the SOC 2 report from six months ago, that's less ideal.

Always verify. Ask: "Can you share your current subprocessor list and your process for notifying customers of changes?" Their answer will tell you more than the report alone.


Latency is the enemy, but consistency is the goal.


   
ReplyQuote
(@danag)
Reputable Member
Joined: 3 months ago
Posts: 303
 

Totally agree on checking their public site first. It's become my go-to move before even talking to sales. If they're serious, that stuff should be easy to find.

One thing I'd add about the subprocessor list: don't just get the list, ask about their *contractual* obligations. I've seen lists that look complete, but the vendor's contracts with those third parties don't actually enforce the same security standards they're promising you. A red flag is them saying, "We trust them to be compliant." The right answer is more like, "Our agreements require them to maintain the same controls and we audit that."



   
ReplyQuote
(@amandaj)
Honorable Member
Joined: 3 months ago
Posts: 516
 

Yes, the contractual point is critical and often the weak link. I'd take it a step further and ask to see a redacted copy of their standard Data Processing Agreement (DPA). The language around subprocessor obligations there is what actually matters.

A common red flag is a DPA that only says they use "commercially reasonable efforts" to ensure subprocessors comply. You want explicit language that binds subprocessors to the same data protection terms as your main agreement, gives you the right to object to new subs, and details their audit rights over them.

If they're hesitant to share the DPA early, that's a signal in itself. A vendor confident in their chain has it ready.


Data > opinions


   
ReplyQuote