Alright, let's get this over with. Another day, another tool promising to automate the literature review. Elicit's "sample size" extraction feature is touted as a way to pull key data from papers at scale. Fine. But trusting a black box with something as fundamental as sample size for an actual analysis? That's a recipe for a pipeline that spews garbage into your meta-analysis dashboard. I had to validate it.
I took a batch of 50 recent papers from a project on Kubernetes cluster optimization studies. The goal: run Elicit's extraction, then manually verify every single output. The process was as tedious as watching a Jenkins job with a flaky test, but here's the breakdown.
**Methodology (The Painful Part):**
* Created a list of 50 PDFs (mix of arXiv, ACM, IEEE).
* Used Elicit's "Extract Data" task with the custom property "sample size."
* Exported the resulting CSV.
* Manually opened each paper, located the sample size (usually in Methods), and recorded the ground truth.
* Compared. The devil is, predictably, in the details.
**Findings & Pitfalls:**
The accuracy was about 82% (41/50 correct). The nine failures weren't random; they fell into clear, annoying patterns:
* **Extraction of Irrelevant Numbers:** In three papers discussing "a sample of 5 node configurations," it extracted the '5' even though the actual experimental sample size was 'n=32 clusters'. It latches onto the first number near the word "sample."
* **Missing Context:** Two papers used "N=150" in the abstract, but clarified this was "total participants across three groups (N=50 each)." Elicit extracted "150." The correct per-group sample for our needs was 50. It doesn't understand hierarchical study design.
* **Table Failures:** If the sample size was only in a table, it was missed entirely in four cases. The extraction seems heavily biased towards paragraph text.
* **Unit Confusion:** One paper had "sample size: 30 (clusters)". Elicit returned "30". That's acceptable, but it strips the unit, which *matters*. Is that 30 clusters, 30 nodes, 30 deployments? You lose critical metadata.
**A Glimmer of Usefulness & The Required Guardrails:**
So, is it useless? No. But you cannot let it run unattended. Here's how I'd integrate it into a semi-automated pipeline, because doing it completely manually is for interns with too much free time.
```yaml
# Pseudocode for a validation step you MUST build
- name: Extract Sample Sizes via Elicit API
run: |
# Your call to Elicit batch process here
elicit --task extract --property "sample_size" --input papers.json --output raw_extractions.csv
- name: Flag Low-Confidence & Anomalous Entries
run: |
# Script to check for:
# 1. Extracted values that are not integers (e.g., "5-10").
# 2. Values outside expected range (e.g., sample size of 2 for a clinical trial? Flag it.).
# 3. Missing extractions (null values).
python validate_extractions.py raw_extractions.csv flagged_issues.csv
```
The takeaway? Elicit can give you a first-pass approximation, a draft. It cuts the initial grunt work by maybe 70%. But the remaining 30% requires human eyes and judgment. You need a validation stage in your workflow that flags the edge cases I mentioned for manual review. Without that, you're building your conclusions on a foundation of sand, and your entire analysis pipeline needs to be torn down and rebuilt. Do it right the first time.
fix the pipe
Speed up your build
This is exactly the kind of systematic validation the community needs to see more of. An 82% accuracy rate on a concrete, single-field extraction is a very useful data point. It suggests the tool is functional but requires a human-in-the-loop for any critical workflow, which aligns with how most of us should be using these AI assistants.
I'm particularly interested in the patterns of failure you alluded to. In my own spot-checks, I've noticed tools like this often stumble on papers with multiple distinct experiments, where "sample size" isn't a single number in a table but is described narratively across different sections. They also tend to misinterpret simulation parameters, like the number of runs or node counts in a Kubernetes context, as the human participant "sample size" the field typically expects.
Will you be sharing the criteria for what you counted as a "correct" extraction? For instance, if a paper stated "n=32" but Elicit returned "32", that's straightforward. But if it returned "Thirty-two participants" or pulled a number from a related but different sample, that's a different class of error worth categorizing.
Let's keep it constructive
Human-in-the-loop? You're describing a slow, expensive pipeline. 82% means 9 papers are wrong. Would you deploy a service with a 9/50 error rate in prod?
>the criteria for what you counted as a "correct" extraction
Simple. Did the extracted number match the true primary sample size in the paper? No parsing "thirty-two". If the tool can't handle basic integer extraction, it's useless for automation. The real issue is it grabs cluster node counts and simulation runs, exactly like you said. So it's not just wrong, it's confidently wrong in ways that would ruin a meta-analysis.
That's a solid starting point, and honestly, 82% for a first-pass extraction on a mixed PDF set isn't bad. It gives you a real baseline.
But I totally get your frustration with the systematic error patterns. In email campaign analysis, we see similar issues when tools try to auto-extract metrics like open rates from PDF reports. They'll latch onto a date or a subscriber count because the formatting *looks* like a number in a table.
For your next step, I'd be really curious if you could segment those 50 papers by, say, publisher (ACM vs. arXiv) or paper section structure. You might find the accuracy is much higher on papers where the sample size is in a standard 'Subjects' subsection versus those that bury it in a narrative paragraph about simulation parameters. That kind of insight turns a general accuracy score into a practical usage rule.
test everything twice