Hi everyone — I've been deep in the weeds planning a large-scale systematic mapping project for our research team, and the tool selection for automated paper screening is a key decision. We're evaluating between **Scholarcy** and the **Semantic Scholar API** specifically for this initial phase of creating the map.
My primary needs are:
* Extracting key claims, methods, and findings reliably from PDFs (often messy, non-standard formats).
* Generating structured, machine-readable summaries to feed into our downstream screening and coding workflow.
* Managing a high volume of papers efficiently and cost-effectively.
From my initial testing:
* **Scholarcy** feels like a more "complete" out-of-the-box summarizer for individual researchers. Its strength seems to be in digesting a single paper into a structured summary flashcard, highlighting PICO elements, which is great. But for batch processing thousands of PDFs via their API, I'm still evaluating the robustness and the cost at scale.
* **Semantic Scholar API** is obviously a powerhouse for metadata and citation networks. Its paper endpoints can pull abstracts and some structured fields efficiently. However, I'm less clear on its ability to handle full-text PDF ingestion and extraction of *custom* structured data (like specific outcomes or population details) at the same level of depth.
Has anyone run a comparative evaluation for a similar automated systematic review/map creation pipeline? I'm particularly interested in:
* **Accuracy & Depth:** Which tool provided more reliable extractions for your specific review criteria?
* **Workflow Integration:** How did you handle batch processing, error rates (e.g., failed PDF parses), and results consolidation?
* **Total Cost of Operation:** At scale (think 10k+ PDFs), did one prove significantly more cost-effective or manageable than the other?
I'm leaning towards a possible hybrid approach, but would love to hear about real-world implementations and any pitfalls you encountered. Sharing your RFP criteria or evaluation framework would be incredibly helpful!
—Heather
Ask me about my RFP template