I've been systematically testing ChatPDF's ingestion capabilities as part of a broader benchmark for document processing pipelines. A consistent failure mode I've identified involves scanned, image-based PDFs generated by office multifunction printers from roughly 2005-2010.
The issue manifests not with modern, digitally-born PDFs, nor with high-quality contemporary scans. It's specifically with older scanned documents where the OCR layer (if it exists) is often minimal or malformed. My test corpus includes technical manuals and academic papers from that era. ChatPDF frequently returns one of two error patterns:
* **Complete failure to parse:** The system returns a generic error about being unable to extract text, despite other OCR engines (Tesseract, Azure Document Intelligence) succeeding, albeit with lower accuracy.
* **Partial ingestion with hallucination:** It extracts only the metadata (like a filename from the printer) and then, when queried, generates plausible but entirely incorrect answers based on that scant context.
From an architecture perspective, this suggests a potential weakness in the pre-processing pipeline. A robust system should have fallback mechanisms. My hypothesis is that the service either:
1. Relies heavily on existing text layers and fails to trigger a secondary OCR process for image-only PDFs of a certain vintage/quality.
2. Uses an OCR model trained primarily on modern typefaces and clean scans, which struggles with the lower DPI, fax-like quality, and jpeg artifacts common in documents from that period.
Has anyone else conducted similar tests or encountered this issue? I'm particularly interested in:
* The specific vintage/scanning source of your problematic documents.
* Whether you found any pre-processing steps (like running them through a local OCR first) that made ChatPDF handle them successfully.
* Comparisons with other similar services (e.g., Claude for PDFs, AskYourPDF) on the same documents.
I'll be publishing a full benchmark with throughput and accuracy numbers across several services, but this particular edge case is proving significant for archival material.