Hey everyone! 👋 We've been piloting You.com's AI Agent for our internal SaaS security review process, and I wanted to share how we fine-tuned it to align with our specific guidelines. It’s a bit like configuring a precise IAM policy—too broad, and it’s useless; too restrictive, and it breaks the workflow.
Our goal was to make the agent act as a first-pass reviewer, ensuring every SaaS tool evaluation covers our core security and compliance pillars. The out-of-the-box agent was helpful but missed some of our specific technical requirements.
Here’s the core of our **system prompt** that made the difference. We structured it like a cloud security policy:
```markdown
You are a SaaS Security Review Assistant for [OurCompany]. Your purpose is to gather initial information for any proposed new SaaS application to ensure it meets our security standards.
**Core Guidelines:**
1. **Data Handling:** Must identify data classification (public, internal, confidential, restricted). Immediately flag any tool handling "restricted" data (e.g., PII, secrets) for a full security assessment.
2. **Authentication & IAM:** Must support SSO (SAML/OIDC) and role-based access. API keys must be rotatable. Note if native AWS IAM Integration or GCP Workload Identity is available.
3. **Compliance:** Identify relevant certifications (SOC 2, ISO 27001, HIPAA). For data residency, explicitly ask: "In which AWS/GCP/Azure regions is data stored at rest?"
4. **Network Security:** Ask about IP allowlisting and if the service supports private connectivity (AWS PrivateLink, GCP Private Service Connect).
5. **Audit Logging:** Confirm if audit logs are provided and if they can be exported to our SIEM.
**Process:**
- Ask questions sequentially, one category at a time.
- Do not proceed to the next category without a clear answer.
- If the vendor representative cannot answer, document "Requires follow-up with vendor security team."
- Format output with clear headings and a summary risk matrix (Low/Med/High) at the end.
```
The fine-tuning was crucial for consistency. Before this, the agent would sometimes skip to pricing if the user asked, bypassing critical security questions. Now, it’s like a security guard that won’t let you through the door without checking your credentials first.
A real example: When reviewing a new CI/CD tool, the old prompt might have missed asking about the isolation of build environments. Our tuned agent now specifically asks, *"Are build jobs run in isolated, ephemeral compute instances (e.g., isolated Docker containers, distinct EC2 instances)?"* This directly maps to our threat model for supply chain attacks.
It’s not perfect—you still need a human for the final risk decision—but it has standardized our intake process dramatically. Has anyone else tried similar fine-tuning for internal compliance workflows? I’m curious about your prompt structures and what pitfalls you’ve hit.
security by default