So, I’m supposed to be using this tool to *accelerate* my vendor contract analysis, and instead I’m spending more time fact-checking its arithmetic than I ever did just reading the damn PDFs myself. The collective hype around ChatPDF as some sort of oracle for financial documents is, frankly, bewildering once you feed it anything with a table.
My particular gripe—and the reason I’m here instead of quietly fuming—is its utter inability to correctly pull numbers from uploaded spreadsheet excerpts. I’m talking about straightforward, cleanly formatted PDFs exported from Excel. I’ll ask something like, “What is the sum of the Q3 column on page 7?” and it will confidently return a figure that is off by orders of magnitude, or worse, invent numbers that simply aren’t present in the document. In one egregious example, it cited a unit cost of $149.50 from a column where every cell was clearly $99.50. When challenged, it apologized and then proceeded to hallucinate a *different* incorrect number.
This isn’t a minor inconvenience. When you’re using this for procurement benchmarking or contract line-item reviews, accuracy is non-negotiable. A hallucinated discount rate or a fabricated annual fee completely skews the vendor comparison. It makes you wonder what the underlying model is actually doing—is it parsing the data, or is it just performing a sophisticated, context-aware guess? For text summarization, a guess might be fine. For quantitative analysis, it’s a catastrophic failure.
I’ve tried the usual suggestions: re-exporting the PDF with different settings, using OCR on scanned statements (that was a disaster), and phrasing my prompts with robotic precision (“Using the exact values in the ‘List Price’ column on page 4, calculate the mean”). The results are inconsistently wrong. Sometimes it gets it right, which is almost more dangerous because it builds a false sense of trust.
Is anyone else using this for serious SaaS pricing or contract tear-downs and actually getting reliable numeric extraction? Or are we all just pretending it works because the alternative is admitting we spent budget on a tool that can’t do basic math? I’m genuinely curious if there’s a workflow I’m missing, or if we’re all just collectively serving as beta testers for a feature that’s fundamentally broken.
—Bella
Price ≠ value.