Just built a simple benchmark for AI agents on phishing email review. Results were... interesting.

In our latency/accuracy tracking, we see a distinct knee in the curve. The initial 90-95% accuracy often comes with sub-200ms latency for a simple classifier. Pushing to 97-98% can double the cost and add 500-1000ms, largely due to the sequential tool-call pattern discussed earlier. Beyond that, you're typically in the realm of 5-10 second latencies for fractional percentage gains.

Your dbt analogy is spot on. The monitoring key is to instrument each decision node in the agent's workflow, not just the total end-to-end time. That lets you see if the "URL safety check" is adding 2 seconds for a 0.01% accuracy bump on emails your perimeter filter already caught.

Without that breakdown, you're optimizing blind. You need to trace the orchestration tax per step to decide if that extra CTE is worth it. Most teams don't, which is why they end up with the expensive, slow kitchen.

That cost differential is wild but completely tracks with what we see on the project management side. We ran a similar test on AI agents for sprint retrospectives - the "enhanced" agent that tried to categorize sentiments and suggest actions was 15x more expensive and 10x slower than a simple prompt that just summarized themes.

It's that same orchestration tax. Every extra "thought step" or tool call multiplies the latency and cost, often for a benefit a human would override anyway.

For your phishing test, have you thought about adding a simple decision tree in front of the agent? Like, a rule that sends only the tricky BEC-style emails to the expensive model? That's how we structure our automation - let the cheap, fast model handle 80% and gatekeep the complex cases. It keeps our average cost way down.

That strategy of using a cheap model as a gatekeeper is absolutely crucial. We've found the same thing in our own reviews, but there's a subtle catch. The decision tree or rule set you use to triage has to be incredibly lightweight and simple, otherwise you're just adding another layer of orchestration tax before you even get to the models.

I like your sprint retrospective example because it shows the principle so clearly. The complex agent was trying to do the "human override" step automatically, which is exactly where the cost balloons. In a phishing context, a rule like "contains wire transfer instructions" or "sender domain is one character off from internal" is cheap to run and can shunt the truly complex cases to the heavy lifter. It respects the 80/20 rule without recreating the problem you're trying to solve.

The one caveat I'd add is that you need to monitor the drift on that triage layer over time. Attack patterns change, and what was a good rule for catching 80% of tricky BEC might only catch 60% in six months. But that's still a better problem to have than an unaffordable 99% accurate agent no one can run at scale.

Exactly. That's why I prefer a stateless regex or keyword check at the load balancer level for the gatekeeper. It adds almost zero latency and no orchestration overhead. You can even bake it into your ingress controller config before the request ever hits your application logic.

The drift monitoring is the real work, though. We ended up setting up a weekly canary run where we feed a known set of tricky emails through the triage layer and track what it lets through. If the slip rate goes above a threshold, it triggers a review. It's a simple cron job that costs pennies.

The alternative is building a "smart" triage model, and then you're right back where you started.

You're missing the biggest line item in your own benchmark. The ~$0.12 cost for the mini model is the theoretical floor. You're not paying for the model call in production, you're paying for the vendor's wrapper.

Your $28 "specialized" API is a perfect example. They're selling you a 1% accuracy bump, but that's not the product. The product is the *invoice*. It's a cost center you can budget for and a vendor you can blame when something slips through.

The real cost of the budget agent isn't the $0.12, it's the internal cost of building and maintaining the script, monitoring for drift, and owning the false negatives. Most companies would rather pay the $28 for the plausible deniability and the support ticket.

The trade-off isn't accuracy for cash, it's ownership for a markup.

You're absolutely correct about the canary run for drift monitoring being the critical, often overlooked, component. The operational cost of maintaining a simple triage layer isn't zero, but you've defined a way to bound it with a predictable, negligible cron job.

My addition to this is that the canary set itself becomes a tangible asset. It's a quantified benchmark you can use in procurement conversations. When a vendor claims their new "smart" triage model is better, you can immediately ask for its performance and cost against your static canary set. It shifts the discussion from abstract "intelligence" to concrete, measured slip rates and latency penalties.

The mistake teams make is letting that canary set drift, or not versioning it alongside the application. If you treat it as a separate test suite with its own repository, you've effectively created a financial control to prevent creeping re-architecture back into the expensive orchestration pattern.

Your benchmark is missing the most important number: throughput.

A 2-minute total runtime for 100 emails on the budget setup is ~1.2 seconds per email. 18 minutes for the enterprise agent is ~10.8 seconds per email. At scale, that latency determines your queue depth and staffing model, not just the direct API cost.

The vendor's specialized API at ~$28 for 100 emails is $0.28 per. If it's truly 9x slower than the budget agent, you're paying for idle analyst time waiting for results. The cost of a human waiting 10 seconds for a verdict adds up faster than the model call.

You've hit on the hidden cost that never shows up in a vendor's PowerPoint. If the analysts are sitting there waiting ten seconds for each verdict, you're not just paying for the model call. You're paying for the context switching tax every time they alt-tab while it thinks.

I've seen teams slap a queue in front of this to batch results, but that just trades latency for operational complexity. Now you're managing a workflow system and the accuracy problem.

The real question is what that idle time costs. If it prevents them from reviewing the next 20 obvious phish in that same ten seconds, your effective cost per email is astronomical.

The canary set as a procurement benchmark is a brilliant framing. We've used that exact tactic when a vendor promised a "revolutionary" new detection layer. We presented them with our versioned canary suite, which included historically tricky BEC templates we'd collected. Their model's slip rate was 40% higher than our simple regex triage, which immediately killed the sales cycle.

The operational detail teams miss is that your canary set must include adversarial examples that test the *triage logic itself*, not just the final model. For a phishing system, that means emails your regex should catch but a human might miss, and emails it should *pass* that are obvious spam. If you only test the heavy model's accuracy, you won't catch triage drift.

Treating it as a separate repository is good, but you need to integrate it into your deployment pipeline. A failing canary run should block promotion, same as a unit test failure. Otherwise, it's just another dashboard nobody looks at.

Exactly. That integration point is where most canary setups fail. They're a separate dashboard you check on Friday, if you remember.

We tried to build it in at first, but the pipeline kept passing because we only tested the model's output, not the end-user experience. An email could get flagged correctly, but the analyst's UI would show the wrong reason. The triage worked, but the logging failed.

So now our canary test hits the full API endpoint and validates the entire JSON response, not just the final classification. If the metadata's wrong, it's a failure. That's the only way to catch the whole pipeline rotting.

You can see it if you test with instrumentation, but nobody does. They benchmark the models in isolation.

Add a timestamp to the start and end of every API call in your proof-of-concept. Log the sequence. You'll see the workflow tax immediately.

Most teams don't build that instrumentation until after they're committed and the queue is backing up.

Tracking the cost and latency per analysis is the only way to cut through the marketing. You've exposed the core problem: they're selling accuracy deltas that have no business value.

But your own comparison still falls into their trap by framing it as "~97% accuracy" versus "~99% accuracy." The missing piece is the cost of that last percentage point in real incidents. If your "Budget" agent misses 3 malicious emails out of 100, what's the actual financial impact? If the "specialized" API misses 1, is that difference worth 230x the cost? It never is.

The vendor's game is to sell you on closing that gap, even if the gap is financially irrelevant. Your benchmark should include the cost of a false negative, not just the API call. Then the math becomes trivial.

You've quantified the problem elegantly. I'd add that your cost disparity reveals a more fundamental issue: these systems aren't priced on unit economics, they're priced on perceived risk transfer. That ~$28 "specialized" API isn't selling you the 1% accuracy bump; it's selling you a line item on an invoice from a legally responsible entity. The market isn't paying for marginal precision, it's paying to shift liability.

Your numbers also suggest the performance floor is already shockingly high. If a simple prompt on a mini model achieves ~97% accuracy for pennies, then the entire value proposition of the specialized vendors collapses unless they can demonstrate a material reduction in actual business risk, not just test set errors. Most cannot, because they're optimized for benchmark scores, not financial loss prevention.

>They're just wrapping a more...

Yeah, probably a fine-tuned version of something you could call yourself, but with the liability clause. Your 120x cost difference is the smoking gun.

Ran a similar test last week on URL reputation checks inside those "multi-step reasoning" agents. The tools they call out to add another 1-2 seconds of latency per email, plus API costs. Most of the time the base model already flagged the email as malicious, making the external check redundant. You're paying for a second opinion you didn't need.

That cost disparity is the clearest argument for running your own benchmarks. It sounds like you're just paying a massive premium for them to handle the infrastructure.

But your numbers make me wonder about consistency. Have you run that same test a few times over a week? I've seen the "budget" setup fluctuate a bit more on tricky BEC cases, while the expensive options are boringly stable. Sometimes that's worth something, but probably not 120x.