Walkthrough: adjusting your ROI model when your agent success rate is only 70%.

devops_dad · 2026-07-21T19:02:45Z

Alright folks, gather 'round. I was cleaning up some old Terraform states in the home lab last night and it got me thinking about a real "aha" moment I had a few years back at the day job. We'd implemented this fancy new automation agent for patching and configuration drift, and our initial ROI model was beautiful. Spreadsheets for days, showing a 90% reduction in manual work. We were patting ourselves on the back for months. Then reality hit. The agent's success rate on any given run was only about 70%. Not because it was bad software, but because our environment was... let's say "organic." Legacy OS, weird network segments, you know the drill. Our beautiful ROI was leaking like a sieve. The key mistake? We modeled cost savings based on 100% task completion. When your success rate is 70%, you're not just losing that 30% of the work. You're *adding* toil because now you have **two** workflows: the automated one, and the manual triage/remediation for the failures. The cost equation changes completely. Here's the adjusted framework we started using. The old model was basically: `Total Manual Time * Hourly Rate - (Cost of Tool + Implementation Time)` The new model had to account for the hybrid state: ``` Adjusted Savings = (Successful Runs * Time Saved per Run) - (Cost of Tool + Implementation Time + (Failure Rate * Manual Intervention Time)) ``` Let's make it concrete. Say an automated task saves 10 minutes of manual work per server, per week. You have 500 servers. Tool costs $20k/year. * **Naive ROI:** `500 servers * 0.167 hours * $50/hr * 52 weeks = $217,100 saved` 🤩 * **Reality Check (70% success):** * Successful runs save: `500 * 0.7 * 0.167 * 50 * 52 = $152,000` * But... failed runs need manual work, say 15 minutes each: `500 * 0.3 * 0.25 * 50 * 52 = $97,500` **added cost**. * **Net "savings":** `$152,000 - $97,500 - $20,000 (tool) = $34,500`. Suddenly, the ROI is a fraction of what we thought. The conversation shifted from "look how much we saved" to "how do we get that success rate from 70% to 90%?" The budget for improving the tooling and cleaning up the environment became much easier to justify. The lesson? Model your automation ROI with a "failure tax." It keeps you honest and highlights the real bottlenecks. Anyone else had to go back to the spreadsheet with a red face? -- Dad

Page 2 / 2 Prev

TCO & ROI Analysis

Last Post by cloud_cost_analyst_pro 2 months ago

16 Posts

16 Users

0 Reactions

104 Views

RSS

cloud_cost_analyst_pro

(@cloud_cost_analyst_pro)

Honorable Member

Joined: 6 months ago

Posts: 469

28/07/2026 4:20 pm

Yes. Track the hourly cost of your failure triage lambda and dead-letter queue processor. Multiply by average execution duration and invocations per month. That's your direct cloud cost for the failure tax.

The multiplier approach works for engineering time. Assign 0.1 hours for a simple timeout retry, 0.5 for a payload mismatch needing a script. But you need the cloud infra cost too, or you're only seeing half the bill.

If your 30% failure rate needs a dedicated engineer to manage the queue, that's the whole project ROI gone. The model shows it immediately.

cost per transaction is the only metric

ReplyQuote

Page 2 / 2 Prev