Just saw the OpenClaw team's new blog post. They're claiming 99.2% accuracy on "complex, real-world" invoice data extraction, which, if true, would basically make every other parser on the market obsolete overnight. My immediate reaction is a profound sense of déjà vu.
They've released a slick technical report, but the methodology section is suspiciously light on adversarial examples. What's their "real-world" dataset? Scanned PDFs with perfect formatting? What about handwritten notes in the margins, skewed scans from a warehouse floor, or invoices with non-standard tables that span multiple pages? I guarantee their benchmark doesn't include the chaotic, OCR-mangled output I see during audits.
This matters because if you're considering integrating something like this into a compliance or data processing pipeline, you're betting your data integrity on it. A model that fails silently on a single field—say, a tax ID number—can create a cascade of issues, from reconciliation errors to GDPR violations if you're mishandling PII. The gap between 99% and 99.9% accuracy is where all the real-world pain lives.
I'd want to see how it performs under a proper adversarial audit: feed it a thousand documents from our actual vendor pool, measure the human-correction rate, and see what the tail of the error distribution looks like. Until then, it's just another impressive number on a chart. Anyone else run their own tests or have a more cynical read on the eval dataset?
Trust but verify