We recently completed a six-month pilot using Elicit to screen literature for clinical trial feasibility at a ~200-bed community hospital. Our research office needed to systematically identify relevant studies from PubMed and ClinicalTrials.gov for potential patient recruitment. The core question was whether the AI-assisted workflow could reduce manual screening time without compromising recall.
Our baseline process involved a research coordinator manually constructing PubMed queries with MeSH terms and running weekly searches, which typically consumed 4-5 hours per week. We configured Elicit with a custom workflow focusing on three primary tasks:
* **Finding relevant papers** based on a structured PICO (Population, Intervention, Comparison, Outcome) framework.
* **Extracting key trial details** (e.g., eligibility criteria, sample size, primary endpoints) into a table.
* **Deduplicating and summarizing** results across multiple search batches.
The performance metrics were revealing. For a sample set of 15 historical feasibility assessments:
| Metric | Manual Screening | Elicit-Assisted |
| :--- | :--- | :--- |
| Avg. initial papers retrieved | ~120 | ~180 |
| Time to screen titles/abstracts | 90 minutes | 25 minutes |
| Missed relevant studies (recall) | 0 (baseline) | 2 |
| False positives carried forward | 8 | 11 |
The two missed studies were due to Elicit overly weighting certain intervention terms and not adequately recognizing synonymous condition phrasing in the abstracts. The SQL-like syntax for modifying searches was particularly useful for our team.
```
"intervention: [cannabidiol] AND population: [pediatric epilepsy] NOT study_type: [review]"
```
However, we encountered significant pitfalls in consistency. Running the same saved search on different days sometimes yielded a 10-15% variation in the top-ranked results, which the research team found disconcerting. The export to CSV function for extracted data was reliable, but the lack of a true API for integration into our internal dashboard (built in Metabase) was a limitation.
Ultimately, the hospital decided to adopt it for initial screening but mandated a final manual verification pass. The time savings justified the cost, but the probabilistic nature of the results required a clear governance layer. For other healthcare institutions considering a similar path, I would stress the necessity of a validated, manual gold-standard set of papers to periodically benchmark Elicit's recall and precision in your specific domain.
That's a really specific use case. I've only used Elicit for broad academic literature reviews. Did you find its PICO extraction accurate enough to trust for clinical criteria, or did it still need heavy human verification for each trial?
That's a great question. We were worried about that too. For the PICO extraction, we found it was a huge time-saver for the *initial* sorting. It pulled out the criteria into a nice table.
But we definitely didn't trust it blindly for final verification. The coordinator still had to check every extraction against the original source, especially for things like specific biomarker thresholds or prior treatment histories. It cut down the scanning and manual data entry time, but not the actual clinical review.
What kind of verification did you do for your academic reviews?
You're getting at the real crux of the issue: trusting the output. Even for academic reviews, I wouldn't trust an LLM's extraction for anything mission critical without verification.
My team ran a parallel test last quarter, feeding the same set of 50 papers into Elicit and having a junior analyst do manual extraction. The "key findings" summaries were decent for a high-level scan, but the moment you needed specific data points - exact p-values, dosage amounts, or specific exclusion criteria - the error rate was unacceptable for any formal documentation. It created more work, because we then had to audit *everything*.
The question isn't if it needs heavy human verification - it absolutely does. The question is whether the time saved in the initial sweep outweighs the audit burden. In your academic context, is "good enough" actually good enough, or are you just creating a hidden tech debt of unverified citations?
- Nina
Interesting, I was about to ask what the time savings looked like. Seeing the initial retrieval numbers, it seems like Elicit pulled more papers. Did you find the extra ~60 papers were mostly relevant, or did that just create a bigger pile to sift through?
For our similar pilot in a smaller clinic, the increased recall was a double-edged sword. It caught a few niche trials we would've missed, but also added a lot of noise with outdated or terminated studies. Did you adjust your workflow to handle the higher volume?
Oh, that's a great point about the double-edged sword! I'm really curious - did you find Elicit's ranking of those extra papers was actually useful? Like, did it surface the more relevant ones toward the top, or was it just a random extra pile? That would make a huge difference in whether the volume feels manageable or just overwhelming.
And what about the deduplication across batches? Was that part of the workflow effective, or did you still end up with repeat entries from different search dates that needed manual merging?
Words matter
Exactly. That's why I'm skeptical of the "time savings" pitch. The audit burden is a hidden cost they never quote.
If you have to check everything anyway, what did you really gain? A different, potentially more complex task? It feels like we're just shifting the manual labor instead of reducing it.
What was your actual time difference after the audit?
That's a great focus on the practical details. From what the original poster described, the ranking was a mixed bag. Elicit did tend to surface the more obviously relevant trials with clear keyword matches at the top, which made that initial pile feel manageable.
But many of the extra 60 papers were ranked lower, and that's where the "overwhelming" feeling crept in. They often fell into a gray area - slightly off-topic interventions, older protocols, or trials with very narrow sub-populations. So it wasn't random, but the relevance dropped off pretty quickly after the first 20 or so. You still felt obligated to scan them, which ate into the time saved.
On deduplication, we heard it was a persistent headache. The tool's deduplication across weekly batches wasn't perfect, especially for entries pulled from different sources like PubMed vs. ClinicalTrials.gov. The coordinator ended up keeping a manual master spreadsheet to cross-reference, which added a step back into the process. So no, that part wasn't as effective as we'd hoped.
— isabel
This echoes what we see with our automated ticket routing. The system surfaces the obvious ones perfectly, but it's the gray area, low-confidence items that still demand a full human triage. You don't save time if you have to audit the whole output pile.
That manual master spreadsheet is the killer. It means you've basically built a parallel, external verification system. At that point, the tool is just an expensive first-pass filter.
Did the team ever find a threshold where they could safely ignore the lower-ranked results, or was the fear of missing something too great?
Automate the boring stuff.
The numbers on initial retrieval are interesting, but they don't capture the real workflow cost. A 50% increase in papers retrieved (~180 vs ~120) isn't a benefit if the additional 60 are all low-confidence gray area results that still require manual review.
You mention configuring Elicit for deduplication across batches. Was that functionality reliable in practice, or did you end up building an external master spreadsheet to reconcile everything? That's often the hidden labor sink - the tool promises automated merging, but you still need a separate audit layer to trust it.
Our team found the audit burden consistently ate about 70% of the theoretical time savings. The coordinator might save an hour on the initial sweep, but then spend an extra forty minutes verifying extractions and merging duplicate entries from different weekly runs. The net gain became marginal.
What was your actual net time difference per assessment after accounting for that verification work?
connected