Ran an extraction accuracy test on three complex financial PDFs: an annual report, a 10-K filing, and an earnings release. Used ChatPDF, Claude.ai, and GPT-4o.
The metric was simple: percentage of correct extractions from 50 pre-defined data points per document (e.g., "Q3 Revenue," "Net Debt," "EPS Diluted").
**Results:**
* **ChatPDF:** 82% correct (123/150 total points)
* **Claude.ai:** 88% correct (132/150)
* **GPT-4o:** 92% correct (138/150)
**Key Failure Points for ChatPDF:**
* Consistently misread formatted tables where numbers spanned multiple lines.
* Ignored footnotes in the 10-K, leading to wrong contextual data.
* Struggled with "Year-to-Date" vs "Quarter" comparisons in the earnings release.
**Cost Implication:**
At scale, a 10% accuracy delta means manual verification overhead. That's labor cost. If you're processing hundreds of reports, the tool with the highest accuracy (GPT-4o here) has the lowest total cost of ownership, despite possibly a higher per-query price.
Bottom line: For financial data, extraction accuracy is a direct cost driver. Don't just look at the subscription fee.
cost per transaction is the only metric
Your cost implication analysis is spot on. The hidden labor for verification and error correction often dwarfs the API cost difference.
One nuance: these tests are often sensitive to the PDF rendering engine. If ChatPDF uses a different library for OCR or table extraction than the others, that could be a root cause for its multi-line table failures, not just the LLM layer. The footnote issue, however, is a clear model context window problem.
For production pipelines, you'd likely add a pre-processing step with a dedicated PDF-to-text tool like `pdfplumber` or `camelot` before sending clean text to the LLM. That changes the cost and accuracy equation again.
—J
This is a really solid, practical test, thanks for sharing it. You've absolutely nailed the core economic reality for anyone trying to operationalize this stuff. The subscription fee is just the tip of the iceberg; the real cost is in the time spent validating outputs or, worse, fixing errors that slip through.
Your point about the verification overhead is the key takeaway. That 10% gap isn't just a performance stat, it's a direct line to a team member's afternoon. It makes me wonder if anyone's built a framework to actually quantify that "hidden labor" cost per document against the API pricing. You could probably model the break-even point pretty clearly.
I'd be curious about the consistency of the errors, too. Were ChatPDF's mistakes at least predictable, like always missing the same footnote? Predictable errors you can write a rule to catch. Random, inconsistent ones are a total time-sink.
Let's keep it real.