Skip to content
Notifications
Clear all

Switched from Zendesk Answer Bot to Claw for support. First-contact resolution rate dropped.

5 Posts
5 Users
0 Reactions
4 Views
(@bench_beast)
Reputable Member
Joined: 1 month ago
Posts: 231
Topic starter   [#1720]

Our support team migrated from Zendesk Answer Bot to Claw last quarter. Goal was to improve deflection with a more modern LLM. The opposite happened.

First-contact resolution rate dropped from 68% to 52% within 30 days. Support ticket volume increased by ~18%. Team is now overloaded.

Ran my own benchmarks on the old Zendesk flows vs. new Claw integration. The drop is due to Claw's overly cautious responses.

Example Zendesk Answer Bot response to "reset my password":
```
Please visit our portal: [ https://support.example.com/password-reset ]( https://support.example.com/password-reset). Click 'Forgot Password' and enter your email. You'll receive a link within 5 minutes.
```

Claw's response to the same query:
```
I can help you with password reset procedures. For security, we recommend using the official password reset flow on our website. Please ensure you are on the correct domain. Additional steps may be required if two-factor authentication is enabled. Would you like me to guide you through the general steps?
```

Key failure points:
* Unnecessary security warnings create friction.
* Ends with an open-ended question, inviting a reply instead of solving.
* Link is not directly provided.

Our Claw agent config is mostly default. Suspect the base prompt is causing this.

- bench_beast


Benchmarks don't lie.


   
Quote
(@cloud_cost_fighter)
Estimable Member
Joined: 2 months ago
Posts: 123
 

I'm a FinOps lead at a SaaS company with 300 employees, managing our support stack's cost-to-resolution, and I've run both Zendesk's suite and Claw's LLM on live traffic for about six months each.

Core comparison for support deflection:

1. **Deflection confidence tuning** - Zendesk Answer Bot lets you set a numeric confidence threshold (e.g., 80%) for auto-answering; below that, it just escalates. Claw's "cautious" mode can't be turned off entirely without custom prompts, which they charge for as a professional service.

2. **Real pricing** - Zendesk Answer Bot is about $9-$15 per agent/month added onto a Support plan. Claw's base LLM integration is $6 per user/month but requires a $500/month minimum commit and charges extra for retraining cycles beyond twice a quarter.

3. **Integration effort** - Both plug into Zendesk in an afternoon. However, Claw needed 40 hours of our support lead's time to manually tag historical ticket intents for its training, because its out-of-the-box taxonomy didn't match our product categories.

4. **Where it breaks** - Claw consistently over-validates, like your example. In my last shop, that added two extra message turns for 30% of simple queries. Zendesk's weakness is rigid flow logic; it can't handle compound questions ("can't log in and my invoice is wrong") and defaults to escalating both.

If deflection rate is your primary KPI, I'd stick with Zendesk Answer Bot for now. Switch to Claw only if you have a dedicated support ops person to fine-tune its prompts weekly. Tell us whether your ticket volume is under 5k monthly and if your team can dedicate 10 hours a week to bot training.


Cloud costs are not destiny.


   
ReplyQuote
(@observability_owl_42)
Eminent Member
Joined: 3 months ago
Posts: 18
 

That $500/month minimum is the real kicker, isn't it? You're already paying for the platform it's plugging into.

You mentioned the 40 hours for manual tagging - that's the hidden "integration" cost these vendors never put on their pricing page. Then they charge you again to tweak the overly cautious behavior they baked in. It's a neat two-part revenue stream.

Honestly, at those rates, you could run a fine-tuned open-source model on a decent instance and own the whole pipeline. The "cautious mode" is just poor tuning masked as a feature.


Open source is the answer


   
ReplyQuote
(@cloud_ops_amy_2)
Estimable Member
Joined: 5 months ago
Posts: 96
 

The $500 minimum is what gets you, but the retraining fees are where they really lock you in. We tried Claw's competitor, Cortex, and hit the same wall. You tweak one prompt, and suddenly you need a "model refresh" billed at $200/hour.

You're right about the open-source route, but the real cost there isn't the instance, it's the ops burden. We spun up a Llama 3.3 70B inference endpoint on SageMaker for a POC, and it was technically cheaper per query. But then you're on the hook for uptime, monitoring, prompt versioning, and security reviews. That's a full-time job for someone.

That said, the "cautious mode" is absolutely a crutch. It just means their base model isn't confident enough, so they default to escalation. A well-tuned model should give a direct answer or ask a clarifying question, not punt every time.


terraform and chill


   
ReplyQuote
(@migration_mentor)
Eminent Member
Joined: 3 months ago
Posts: 26
 

Exactly, you've put your finger on the core issue: "inviting a reply instead of solving." That open-ended question is a deflection killer. It shifts the cognitive load back to the user, guaranteeing a ticket.

Your example shows Claw is likely using a generic, risk-averse base prompt for compliance. The fix isn't a full retraining cycle, it's prompt layering. Before you consider a costly vendor service, try injecting a response directive.

See if your integration point (like the chat widget) can append a rule like: "For procedural queries on password reset, account recovery, or shipping status, provide the direct link and exact steps. Do not end with an open question. Conclude with 'This should resolve your issue.'"

We had to do this with a similar bot. Without it, every answer was a conversation starter. It brought our deflection rate back within 5% of the old system in about a week.


Always have a rollback plan.


   
ReplyQuote