Just spent the weekend building a POC with SuperAGI to auto-screen incoming tech support tickets. It was shockingly fast to get a working agent up! The goal was to classify urgency, route to the right team, and pull out key error codes before a human even looks at it.
The core logic is in the action. Here's a simplified snippet of the `determine_urgency` method my agent uses:
```python
def execute(self, task_input: str):
# Analyze the ticket text
analysis_prompt = f"""
Ticket: {task_input}
Analyze for:
1. Urgency (Low, Medium, High, Critical). Critical = system down or security breach.
2. Likely team (Billing, Infrastructure, App, Database).
3. Extract any error codes or version numbers.
Return JSON only.
"""
response = self.openai_utils.chat_completion(analysis_prompt, temperature=0.1)
# Parse response and set knowledge for routing
self.agent.knowledge = response
return "Analysis complete. Routing data ready."
```
It's not perfect, but for a weekend project, the accuracy on our test dataset was solid. The SuperAGI framework handled the orchestration beautifully. Next step is hooking it into our actual ticketing system via a custom tool.
Anyone else building practical, internal ops agents? Would love to swap notes on managing more complex workflows. Happy testing!
Another tool to try!
Cool POC. The OpenAI temperature=0.1 is a good call for classification tasks. Did you measure token cost per ticket? That can sneak up fast when you scale past a weekend dataset.
Also curious - how's the accuracy for the "Critical" cases? Missing a real security breach on the first pass could be expensive in a different way. Are you planning to add any fallback rules (keyword detections for "production down" or "data leak") before calling the LLM? Might cut both cost and latency if you filter out the obvious ones first.
Ask me about hidden egress costs.
Solid for a weekend project. But your classification logic lives entirely in a prompt now, which is a brittle dependency.
Accuracy on a test set is one thing. The real problem is ticket text you haven't seen yet - vague user descriptions, internal jargon the LLM wasn't trained on, sarcasm. A simple keyword filter for 'down', 'breach', or 'unauthorized' as a pre-check would give you a safety net and reduce LLM calls for the obvious stuff.
Have you measured the latency on a real ticket stream, not a batch? That orchestration overhead can kill the benefit.
You're right about the brittleness of pure prompt-based classification. I've seen this pattern fail in production when users include internal ticket IDs or project codenames that the model interprets literally. A hybrid approach with a rules-based pre-filter is the pragmatic middle ground.
However, keyword filters have their own blind spots. "Down" could be in "I sat down to write this," and you'll miss the genuine "API is returning 503" that lacks your trigger words. Better to implement a simple intent classifier (TF-IDF or a small BERT model) trained on historical tickets to catch obvious categories *before* the LLM, not just a regex keyword list. This reduces latency more predictably than a full LLM call and handles synonym variation.
Have you considered the operational overhead of maintaining two systems, the rule set and the prompt? That's where I've seen teams get stuck. The rules drift, the prompt needs constant tuning, and you end up with inconsistent classification.
Boring is beautiful