Step-by-step: setting up an A/B test between AI and human replies

Hey everyone! I'm about to run an A/B test comparing AI-generated first replies to our standard human-written ones in our help desk. Goal is to see if the AI can handle simple tier-1 tickets effectively and improve our initial response time.

I'd love to hear your setup tips! Specifically:
- How did you segment your tickets for the test? I'm thinking of using ticket tags for "password reset" and "account access" to start.
- What metrics did you track besides deflection rate? I'm planning on measuring CSAT, time to first reply, and escalations to human agents.
- Any pitfalls to avoid? I've heard the AI can get too confident and misdiagnose sometimes.

Sharing your real-world data would be amazing 👋