I’ve been benchmarking data ingestion pipelines for years, but my team recently got tasked with evaluating AI features for our shared internal help desk. This is a new domain for me. I’m looking at it like a systems performance problem: you have an input stream of tickets, a processing layer (agents + AI), and output metrics (resolution time, deflection rate).
For those who have implemented this, what's the foundational architecture? I’m thinking in terms of components:
* **Intent Classification & Routing:** Is this typically a fine-tuned model on historical ticket data, or are pre-trained NLP models good enough to start? What’s the training data scope?
* **Response Drafting:** Are you using RAG against a knowledge base, or purely prompting a foundational model? I’m curious about the latency numbers here—what’s an acceptable time-to-first-token for an agent-facing draft?
* **Deflection Measurement:** How are you instrumenting the pipeline to track deflection? Simply a `(deflected_tickets / total_tickets)` counter, or something more nuanced, like measuring downstream re-open rates?
Our stack is Jira Service Management, but I'm interested in general patterns. For example, a simple deflection tracking logic we might implement could look like:
```python
# Pseudo-logic for measuring deflection
if ticket.status == "RESOLVED" and ticket.solver == "ai_agent":
deflected_tickets.increment()
elif ticket.status == "REOPENED" and ticket.initial_solver == "ai_agent":
ineffective_deflections.increment() # Critical metric
```
What are the key performance indicators you found most revealing? And crucially, where did you see the highest initial friction—was it data quality, agent adoption, or model accuracy?
Your systems performance lens is the right one. It keeps you honest when the AI hype gets loud.
On intent classification: start with the pre-trained models, but you'll hit a wall fast. They're trained on general internet text, not your internal jargon and ticket structure. Fine-tuning on historical data is the only way to get reliable routing. Scope your training data to closed, resolved tickets from the last 12-18 months - anything older and the business context might have shifted.
For response drafting, pure prompting is a dead end for accuracy. You need RAG hooked into your actual knowledge base, Confluence pages, or past resolved tickets. Acceptable latency for an agent-facing draft is under 3 seconds end-to-end. If it takes longer, your agents will ignore it and work manually.
Deflection measurement is the trickiest part. A simple counter is naive and will be gamed. You need to track the re-open rate of tickets that were "deflected" by an AI-suggested solution. If a user re-opens a ticket within, say, 72 hours, that deflection event should be invalidated. Instrument that pipeline early or you'll have no idea if the AI is actually helping.
Been there, migrated that
Completely agree on the 3-second threshold for agent-facing drafts. That's a solid real-world benchmark. I'd add that the latency target should be p99, not average. A 5-second lag once every hundred tickets will still break trust and cause agents to disengage.
Your point on deflection measurement is critical. Beyond re-open rates, we've seen false deflection where users mark a solution as "helpful" out of politeness but then immediately submit a new, more specific ticket on the same core issue. Correlating AI-deflected ticket IDs with subsequent user activity in your service portal often reveals this.
BenchMark
The 3-second p99 target for drafting is a good start, but it misses the data poisoning problem nobody talks about. You get that latency by using a lightweight model, which then hallucinates based on your messy, outdated knowledge base. Your RAG accuracy is only as good as your worst internal wiki page from 2018.
On deflection measurement, you're right to look beyond the simple counter. We instrumented for re-open rates and still got burned. The real metric is whether the user *actually stopped working* on the issue. If they clicked your AI solution then immediately started searching the portal again, you didn't deflect anything, you just added a step. You need to track the user's next action, not just ticket status.
And you'll need to budget for constant tuning. The model that routes perfectly in Q1 will decay by Q3 as your product updates and internal slang changes. It's a live system, not a set-and-forget feature.
You're right to frame it as a performance problem, especially regarding latency. For agent-facing drafts, time-to-first-token is a red herring. You need end-to-end latency, and the p99 target of under 3 seconds mentioned is correct. However, achieving that requires careful instrumentation you haven't mentioned.
You can't just time the LLM call. You have to instrument the entire chain: the RAG retrieval from your knowledge base, the context window assembly, the model inference, and the post-processing. If your retrieval from Confluence or Jira takes 2.5 seconds, you've already lost. Start by adding detailed spans for each step in your APM to find the bottleneck before you even try to hit that target.
On deflection, a simple counter is useless. You need to trace the user's entire session. If an AI answer is served, you must tag that trace and then look for subsequent tickets or searches from the same user within a short timeframe. That's the only way to measure actual deflection versus a polite click. Datadog's RUM and session replay can be configured for this, but it's not a default setup.
null