Showcase: My script that cross-checks ChatPDF outputs with source PDFs.

hannahr · 2026-07-29T23:50:51Z

After our team migrated a massive archive of legacy contracts to a new vendor portal, we learned a hard lesson: automated extraction tools like ChatPDF are incredibly useful, but you can't fully trust them without a verification step. We had a few close calls where subtle but critical clauses were misrepresented. To mitigate this risk, I built a simple Python script that cross-chexes ChatPDF's answers against the actual source PDF text. It doesn't rely on perfect OCR; it just checks if the key phrases and terms from the answer genuinely appear in the document's raw text. This has been invaluable for our compliance audits. Here's the core workflow my script automates: * **Input:** You provide the PDF file path and paste the ChatPDF answer. * **Processing:** It extracts text from the PDF (using `PyPDF2` for simplicity) and preprocesses both the source text and the answer (lowercases, removes punctuation). * **Verification:** It checks for the presence of significant nouns/terms from the answer within the source text. It ignores common stop words. * **Output:** It generates a report showing: * A confidence score (percentage of key terms found). * A list of any claimed "key terms" not found in the source. * The relevant source text snippets for verification. For example, if ChatPDF claims a contract includes a "liquidated damages clause of 15%," the script will flag it if "liquidated damages" and "15%" do not co-occur in the actual PDF. It's caught several instances where ChatPDF confidently conflated details from different document sections. This isn't a silver bullet for hallucinations, but it adds a crucial, repeatable checkpoint. It's especially helpful when dealing with large document batches where manual spot-checking is impossible. I'm happy to share the core logic if anyone is building similar safeguards into their workflow. - h

Page 2 / 2 Prev

ChatPDF Reviews

Last Post by AndrewH 59 minutes ago

17 Posts

17 Users

0 Reactions

6 Views

RSS

danw

(@danw)

Estimable Member

Joined: 2 weeks ago

Posts: 89

30/07/2026 12:21 pm

PyPDF2 will fail on any real contract with scanned pages or tables. Swap it for pdfplumber or a dedicated service. Your confidence score is meaningless if it's built on partial text.

The list of missing terms is useful, but you need a proximity check. Finding "indemnify" on page 1 and "Party A" on page 20 doesn't mean ChatPDF is right.

ReplyQuote

AndrewH

(@andrewh)

Estimable Member

Joined: 2 weeks ago

Posts: 106

30/07/2026 12:26 pm

Thanks for pointing out pdfplumber, I'll swap that in next. I hadn't even thought about scanned pages breaking everything.

>need a proximity check
That's a really good call. My test cases were short, so I didn't see that problem yet. How do you decide what's close enough in a long document? A certain number of words? Or looking for the same paragraph?

ReplyQuote

Page 2 / 2 Prev