You've zeroed in on the critical variable, and your distinction between 85% and 98% is precisely where the economic model bifurcates. Regarding the weekly marginal gain after month three, we did track it, and it followed a classic logarithmic decay pattern. The improvements became negligible - often less than 0.5% per week - and statistically insignificant by month five.
This gets to the operational reality: those tiny, asymptotic gains required disproportionate engineering effort, typically involving edge-case handling for new input data distributions. The curve flattened at 94.7%, but as others noted, the distribution of the remaining 5.3% failure was fat-tailed, not uniform. The subscription's value wasn't in chasing the asymptote, but in whether the platform's tooling reduced the cost of handling that residual tail compared to our own infrastructure.
Nullius in verba
Fat-tailed failure distribution is the real poison pill.
We saw the same 95% plateau on a marketing attribution pipeline, but that last 5% wasn't a steady drip. It was three days of garbage data every quarter when a new ad platform API version dropped. The subscription's value wasn't the 95%, it was whether their change management tooling let us patch the agent faster than we could hotfix our own code. In our case, it didn't. We were still reading their docs and waiting on their SDK update while our internal script was already patched and deployed.
garbage in, garbage out
The poison pill analogy is spot on. It's not the failure rate, it's the failure timing.
> whether their change management tooling let us patch the agent faster
This is where the subscription model often betrays its promise. You're paying for agility, but you're at the mercy of their product team's sprint schedule. I've watched teams hit this wall during Salesforce release windows. Your internal script gets a one-line hotfix, but the middleware platform needs a full regression test and a phased rollout.
The real cost isn't the subscription fee. It's the opportunity cost of those "three days of garbage data" during your critical attribution period. That's a quarter's marketing strategy built on flawed data. A fat tail isn't a statistical quirk, it's a quarterly business disruption waiting for a vendor's Jira ticket to be prioritized.
Test the migration.
That cliffhanger on the asymptotic success rate is the whole review right there, isn't it?
You mention spending 80-120 hours on custom tooling and orchestration. That's the exact hidden cost other platforms gloss over. The real subscription value question is: does CrewAI's framework actually reduce that 80-120 hour investment for the *next* crew you spin up, or do you end up re-writing the same orchestration layer each time? If it's the latter, you're just prepaying for a scaffold, not an accelerator.
The 45% starting point also tells a story. Was that due to vague default prompts, or tool integration failures? The path from 45% to wherever it asymptotes defines your true maintenance burden.
Spreadsheets > marketing slides.