Hey folks, been thinking about this ever since I started using Elicit to help with some research for a side project. I'm old enough to remember the days of printing out a stack of papers, highlighting key sentences, and manually building a spreadsheet of findings. Took forever, and my highlighter hand would cramp up! 🖍️
So when I feed a bunch of PDFs into Elicit and get a summary of claims and methods in minutes, it feels like magic. But as we know in DevOps, any automation needs to be checked for drift. I've caught it missing a crucial limitation section in a paper, or sometimes mis-attributing a finding. My manual process, while slow, had a different kind of accuracy because I was forced to read every line.
Has anyone done a more formal or even anecdotal benchmark? I'm curious about specifics:
* For a set of, say, 10 academic papers, what was the **false positive rate** (things Elicit said were there that weren't) and the **false negative rate** (key points it completely missed)?
* Does it handle certain types of papers (e.g., dense theoretical vs. experimental results) better than others?
* What's the time delta? Obviously Elicit wins on speed, but if you have to fact-check every output deeply, does the net time saved still make it worthwhile?
I've been treating it like a brilliant but over-eager intern—great for a first pass, but I always do a spot-check on the important bits. Would love to hear if others have put it through its paces more rigorously.
-- Dad
it worked on my machine
I've been wondering the same thing. For a recent lit review I used Elicit on about 15 papers, then spot-checked a few manually.
> false negative rate (key points it completely missed)
This is the bigger issue for me. It skipped over nuanced methodological choices in two experimental papers, stuff I'd consider key for judging validity. False positives were rarer, maybe one per batch where it slightly misinterpreted a finding.
I'd guess it struggles with theory papers where the key claim is woven through an argument, not a single declarative sentence. Ever try it on something like that?
The manual process forcing you to read every line is the whole point. That's not a bug, it's the feature. What you're calling accuracy I'd call comprehension, and you lose it when you outsource the reading.
You're asking for a benchmark on false positives and negatives, but that assumes the tool's output and a human's should be directly comparable. They're not. The human process builds understanding through the friction. The tool provides a compressed, lossy signal. Comparing them on "accuracy" is like comparing a map to actually walking the terrain.
The real delta isn't just time, it's cognitive load. You save hours but you also offload the work that builds your critical judgment. For a side project maybe that's fine. For anything where you need to genuinely understand the material, you're just creating a different kind of work: now you have to audit the tool's output instead of developing your own.
Your mileage will vary
That's a fair distinction between accuracy and comprehension. But I think we can still benchmark the lossy signal on its own terms, against a known baseline. For example, if I manually extract every claim from the methods section of 50 papers to create a gold standard dataset, I can then run Elicit on the same corpus and measure recall (false negatives) and precision (false positives) for that specific task. The human process has the comprehension benefit, yes, but the tool's extraction performance is still a quantifiable metric.
It tells you the tool's reliability as a filter. If it misses 30% of key methodological statements in a blind test, you know you must read 100% of the methods sections anyway, negating the time savings. If it catches 95%, you can triage your auditing effort. The benchmark doesn't equate the two processes, it calibrates the tool's output so you know exactly how much "different kind of work" you're signing up for.
-- bb42
The manual process having "a different kind of accuracy" is exactly right. You're benchmarking two different systems.
On your specific questions: I haven't seen formal numbers, but my anecdotal benchmark lines up with user537. False negatives are the real cost. I've seen it miss critical caveats in discussion sections, especially where the language isn't direct. On a batch of 8 clinical study papers last month, I'd estimate a 15-20% false negative rate on key limitations. That means you're still reading most of the paper to verify, which changes the time delta calculation entirely.
It absolutely fails on theoretical or philosophy papers. The core argument is rarely in a single extractable sentence. For straightforward methods sections in empirical studies, it's more reliable, but that's the easiest part to skim manually anyway.
—JW