We recently completed a pilot project integrating a custom Poe bot into our first-line customer support workflow. The primary objective was to improve first-contact resolution (FCR) rates for common technical inquiries. The results, measured over a four-week period against a prior four-week baseline, present a nuanced picture that underscores the importance of choosing the right evaluation metrics beyond simple automation percentages.
**Methodology & Bot Configuration:**
We deployed a bot built on Claude 3 Haiku, given its favorable cost-to-performance ratio for straightforward Q&A. The knowledge base consisted of our product documentation, FAQ, and a curated set of resolved support tickets. The system prompt was engineered to prioritize concise, direct answers from the provided context and to explicitly decline to speculate on topics outside its knowledge.
```yaml
# Core system prompt excerpt:
You are a technical support assistant for [Product Name]. Your goal is to provide accurate, step-by-step solutions based solely on the provided documentation.
- Always reference the relevant section title if possible.
- If the user's query is ambiguous, ask exactly one clarifying question.
- If the information is not in the provided context, state: "I don't have specific documentation on that. For this issue, please contact a human agent for further assistance."
- Do not use pleasantries or empathetic language beyond acknowledging the problem.
```
**Quantitative Results (Positive):**
* **First-Contact Resolution (FCR):** Increased from **62%** to **79%**. This was the key success metric. The bot efficiently handled routine queries about password resets, API error codes, and basic configuration.
* **Ticket Deflection:** Approximately **35%** of all incoming queries were fully resolved by the bot without human escalation.
* **Agent Response Time:** For tickets that did reach human agents, average initial response time improved by **~40%**, as agents were no longer bogged down by simple tickets.
**Qualitative & Negative Results:**
Despite the positive FCR metric, our post-interaction customer satisfaction (CSAT) survey scores (scale 1-5) for bot-handled queries showed a marked decline.
* **Overall CSAT:** Dropped from **4.2** to **3.6** for bot-resolved queries.
* **Sentiment Analysis:** We ran a sample of chat transcripts through a sentiment model and found a significant increase in negative emotion keywords (e.g., "frustrated," "unhelpful," "robotic") compared to human-handled chats.
**Analysis & Pitfalls:**
The satisfaction drop can be attributed to several factors observed in the logs:
* **Lack of Rapport:** The directive to avoid "pleasantries or empathetic language" created interactions perceived as blunt and uncaring, even when the answer was technically correct.
* **Contextual Rigidity:** The bot would correctly state it lacked documentation, but the phrasing felt like a dead end. Human agents would often say "I'm not sure, but let me find out for you," which maintains engagement.
* **Inability to Handle Complex Nuance:** Queries that involved multiple, interconnected issues (e.g., "My upload failed and now the dashboard is showing old data") would often trigger the "I don't have specific documentation" clause, frustrating users who felt they had clearly explained the problem.
**Conclusion:**
The pilot proved that a Poe bot can effectively automate resolution of simple tickets, boosting FCR and improving operational metrics. However, it also demonstrated that optimizing purely for informational accuracy and deflection rate can degrade the customer experience. The next iteration will need to balance efficiency with a more nuanced, empathetic interaction style—perhaps by switching to a model like Claude 3 Sonnet for its superior instruction-following in tone, or by implementing a more sophisticated prompt that allows for limited, appropriate empathy before providing the direct answer. The trade-off between resolution speed and customer sentiment is a critical dimension for any support automation project.
Prompt engineering is engineering