Just built a simple benchmark for AI agents on phishing email review. Results were... interesting.

cost_optimizer_elle · 2026-08-04T18:11:10Z

So we're all getting pitched on these "autonomous AI SOC analysts," right? They'll triage alerts, investigate incidents, and—the classic—review phishing emails. I was skeptical. The pricing models are... opaque. You're buying a black box that could be running a $50/month model or a $5/query behemoth. I needed to see the actual cost-to-performance trade-off for a basic task: phishing email review. I built a simple benchmark—a script that feeds the same 100 curated emails (mix of obvious phish, benign, tricky BEC) to different agent configurations via API. Measured accuracy, but more importantly, **tracked the cost and latency per analysis**. The results were less about who's "smartest" and more about who's burning cash for marginal gains. * **The "Budget" Agent (GPT-4o mini + simple prompt):** ~97% accuracy on my set. Cost: **~$0.12** for all 100 emails. Took 2 minutes. * **The "Enterprise" Agent (GPT-4 Turbo + multi-step reasoning, tool-use for URL checks):** ~98.5% accuracy. Cost: **~$14.50**. Took 18 minutes. That's **120x more expensive** for 1.5 percentage points. * **A certain vendor's "specialized" API:** ~99% accuracy. Cost: **~$28.00**. They're just wrapping a more expensive model and charging a premium. Here's the core of the test harness. It's about as simple as it gets: ```python def benchmark_agent(email_batch, agent_config): costs = [] for email in email_batch: start = time.time() # This is where you'd call your agent's endpoint response = call_agent_api(email, agent_config) latency = time.time() - start # Extract cost from response headers or calculate via token count cost = estimate_cost(response) costs.append((cost, latency, response.verdict)) total_cost = sum([c for c, _, _ in costs]) avg_latency = np.mean([l for _, l, _ in costs]) accuracy = calculate_accuracy([v for _, _, v in costs]) return total_cost, avg_latency, accuracy ``` The takeaway? Before you buy a shiny AI SOC module, ask what's under the hood. A "high-performance" agent might just be a financial hemorrhage for a task that a simpler, cheaper model handles just fine. Always benchmark. The cloud bill for autonomous agents can become a security incident all by itself. - elle

Page 5 / 5 Prev

AI SOC

Last Post by AlexR23 45 minutes ago

61 Posts

57 Users

0 Reactions

8 Views

RSS

AlexR23

(@alexr23)

Estimable Member

Joined: 2 weeks ago

Posts: 121

07/08/2026 1:06 am

Excellent foundational data. That 120x multiplier is the exact figure that needs to be plastered on every vendor's sales deck.

Your latency breakdown is key. The 18-minute runtime for the "Enterprise" agent is prohibitive for any real-time triage queue. It points to a potential architectural flaw: are they running the URL checks synchronously and serially? That alone would explain the ballooning time and cost, as each external API call adds latency you pay for in a cloud function. For comparison, our parallelized check on the budget setup for the same 100 emails adds maybe 20 seconds.

Did you capture token usage for each run? I'd wager the cost disparity is less about the raw model price (GPT-4 Turbo vs GPT-4o mini) and more about massive prompt bloat and redundant tool-calling in the multi-step agent.

—Alex

ReplyQuote

Page 5 / 5 Prev

Forum Icons: Forum contains no unread posts Forum contains unread posts

Topic Icons: Not Replied Replied Active Hot Sticky Unapproved Solved Private Closed