Our department recently standardized on Scholarcy for literature processing and summary generation across a 15-person interdisciplinary graduate research team. The primary workflow involves ingesting a heterogeneous corpus of academic PDFs—from modern, born-digital articles to scanned chapters of 20th-century monographs. The core technical challenge we aimed to evaluate was the consistency and quality of Scholarcy's OCR pipeline, as this directly impacts the fidelity of extracted summaries, references, and key claims.
We designed a simple but methodical test to quantify the OCR-dependent output variability. We selected 20 sample documents across a spectrum:
* **Category A (5 docs):** Modern, text-based PDFs with selectable text.
* **Category B (10 docs):** Scanned PDFs from library archives, ranging from decent to poor quality scans.
* **Category C (5 docs):** Mixed-format documents containing both text layers and embedded image-based figures or equations.
We processed each document through the Scholarcy web interface and its browser extension, then programmatically extracted the plain-text summary and reference list via their API for analysis. The key metric was the accuracy of the extracted text body, which we measured by comparing Scholarcy's output to a ground truth established using a combination of:
1. Original source text for Category A.
2. High-quality commercial OCR (ABBYY FineReader) for Categories B and C.
Our findings on OCR quality are nuanced:
* **Text-layer PDFs (Category A):** Performance was flawless. Scholarcy correctly bypassed OCR and used the embedded text, resulting in 100% character accuracy for the main body. Reference extraction success rate was 98%, with failures only occurring on highly unconventional citation formats.
* **Scanned PDFs (Category B):** Here, the variance was significant. The quality of Scholarcy's internal OCR appears heavily contingent on the scan's cleanliness.
* For high-DPI, clean scans with clear typography, accuracy averaged 94-97%. This was sufficient for Scholarcy's NLP summarization models to produce coherent and accurate highlights.
* For medium-to-poor quality scans (shadowed margins, low contrast, smudged text), accuracy dropped sharply to 70-85%. The errors were not random; they clustered around specific font families, punctuation, and superscript/subscript characters. This directly corrupted the extraction of key sentences and, most critically, the reference list, rendering many citations unusable.
* **Mixed-format PDFs (Category C):** Scholarcy exhibited a deterministic but problematic behavior: it often defaulted to applying OCR to the entire page, even when a selectable text layer was present. This introduced errors where none should exist, particularly damaging mathematical notation and in-line references. The workflow required manual pre-screening to force the use of the text layer, negating the automation benefit.
**Technical Conclusions and Workflow Adjustments:**
1. **Pre-processing is Non-negotiable:** We instituted a mandatory pre-processing step for all scanned documents using an external, high-accuracy OCR engine before ingestion into Scholarcy. This improved overall summary reliability by approximately 40% for our historical document corpus. The cost is an added step, but the lift in downstream data quality is justified.
2. **API Reliability:** The OCR quality directly affects the structured JSON output from the API. An error in the text layer propagates into the `highlights`, `references`, and `publication_info` fields. We implemented a basic confidence check by calculating the ratio of successfully parsed references to the total number of reference-like strings detected.
3. **Recommendation:** Scholarcy's value proposition remains strong for born-digital literature. However, for archives or disciplines reliant on scanned material, its built-in OCR should be considered a fallback, not a primary feature. The platform's summarization and extraction logic is robust, but it is fundamentally Garbage-In-Garbage-Out (GIGO) dependent on the text input quality.
Our team now uses a conditional workflow: clean PDFs go directly to Scholarcy; scanned PDFs are routed through a dedicated OCR pipeline (currently using Tesseract with custom training for common typefaces in our field) before being fed to Scholarcy for analysis. This hybrid approach has yielded the most consistent and trustworthy results.