Everyone’s raving about AI literature search, but I’m more interested in whether it can pull a specific number or outcome from a PDF without hallucinating. That’s where the rubber meets the road for actual research.
I ran a simple, unscientific test on Elicit’s data extraction. Used the same prompt template across three recent papers: one clinical trial (medicine), one materials science study, and one economics working paper. Asked for the primary outcome measure and its value. Here’s what I got:
* Clinical trial: Nailed it. Pulled the exact p-value and confidence interval from the results section. No complaints.
* Materials science: Partial credit. Got the property being measured right, but the extracted numerical value was from a table footnote, not the main result. A human would have caught the context.
* Economics paper: A mess. Confused the baseline estimate with the instrumental variable estimate. This is a major error if you’re doing a lit review.
The takeaway? Its accuracy is domain-sensitive and hinges on how standardized the reporting is. For well-structured clinical literature, it’s a decent assistant. For anything with nuanced or varied presentation formats, you’re still going to be fact-checking every output. Don’t trust it as a source of truth—trust it as a potentially flawed first pass.
I’d be curious if others have tried similar extractions, especially in social sciences or CS, and what your error rate looks like.
/charlie
Show me the TCO.
Your test reminds me of trying to extract pricing dimensions from AWS's PDF invoices. The format is *mostly* standardized, but when they change the column headers or add new service lines, the script breaks.
Domain-specific formatting is everything. Clinical trials have highly structured abstracts and results sections - it's like reading a well-formed JSON. Economics papers? They'll bury the key estimate in a footnote three pages later, or present six different model specifications.
If you run this again, try feeding it the "Methods" section first to prime it on the study design. Sometimes you gotta pre-process the unstructured data, same as cleaning a billing file before you analyze it.
- elle
This aligns with my experience testing these tools for contract review. The more standardized the source document, the more predictable the extraction cost becomes. Clinical trial reports follow ICH guidelines, which is like a vendor using a published, fixed-price API. Economics papers are more like bespoke enterprise contracts with negotiated terms buried in exhibits.
Your point about domain sensitivity is key, but I'd extend it to cost sensitivity. Using a tool like Elicit on a predictable, structured domain might be worth the subscription seat. But if you're constantly fighting format variations, the time spent verifying outputs - the manual overage fees, if you will - erases the efficiency gain. You're essentially paying for a license and then doing unpaid QA labor.
Have you considered how you'd quantify the error rate threshold where manual extraction becomes cheaper? That's the real break-even analysis for any research team's budget.
null