Our platform's new "AI-powered transaction coding" feature was rolled out with the usual fanfare. Promised to cut our manual entry time by 80%. So, like any good skeptic, I set up a controlled test batch of 500 transactions from last quarter, where we already knew the correct general ledger codes.
The AI got 425 right. That's an 85% accuracy rate, which sounds great in a press release. But in the real world, that's 75 transactions headed to the wrong GL account unless someone catches them. For us, that's a 15% error rate. The mistakes aren't subtle—software subscriptions going to "Office Supplies," cloud infra costs landing in "Marketing."
So I manually review every single one now. It takes *longer* than if I'd just coded them myself from scratch, because now I'm doing my job plus playing QA for a black box.
I'm curious if this is the norm. Are others just blindly trusting this? Or are you also babysitting the AI, effectively using it as a slightly-faster-but-error-prone first draft? What's your observed error rate, and has anyone gotten it below, say, 2% where it's actually trustworthy?
Our vendor's response was basically "the model will improve with more of your data." Cool. So I'm supposed to spend the next quarter feeding it corrections until it's barely adequate? Feels like I'm the one training their model, for free.
just sayin'
Data over dogma.
You're absolutely not alone. I've seen this exact pattern with three different "smart coding" tools we've trialed. The 85% headline number is meaningless when the 15% errors are catastrophic misclassifications.
What drives me nuts is the vendor line about "improving with more data." Sure, but that's a long-term promise, not a fix for my quarter-end close next week. We had to build a whole secondary validation layer using webhooks - every AI-suggested code triggers a call to our rules engine, which flags the obviously wrong ones (like your software-to-supplies example) for human review. It cut the manual check volume by about 60%, but it's still a band-aid.
Has your platform exposed any sort of confidence score or alternative suggestions in their API? The black box part is the real killer - if we could at least see "I'm 95% sure this is Software, but 4% it's Supplies," we could auto-approve the high-confidence ones.
null