Just finished a 3000-paper systematic review project where we moved from a manual, spreadsheet-and-PDF hellscape to using Elicit. The efficiency gain is real, but the path is littered with data quality landmines you need to engineer around.
The core lesson: Elicit is not a fire-and-forget ETL tool. You treat its output as a raw, unclean source layer. If you dump its CSVs directly into your analysis, you will have a bad time. Here's my pipeline pattern:
**Raw Extract & Load:**
- Use Elicit's bulk CSV export. This is your `stg_elicit_raw` table.
- Load everything as-is, preserving all columns. No transformations yet.
**Critical Cleaning & Normalization:**
You must build cleaning jobs for these columns:
- `doi`: Nulls, inconsistent formatting, prefix variations. I used a series of regex operations.
- `year`: Sometimes missing, sometimes in a `date` column, sometimes as a string "2022-2023".
- `authors`: The list format is inconsistent. I parsed it into an array of structured names in a later model.
Here's a snippet of the dbt model I used to create a clean `elicit_papers` layer:
```sql
WITH raw AS (
SELECT
"Paper title" as title,
"DOI" as raw_doi,
"Year" as raw_year,
"Abstract" as abstract,
"Authors" as raw_authors,
"URL" as url
FROM {{ ref('stg_elicit_raw') }}
),
cleaned AS (
SELECT
title,
-- Normalize DOI
CASE
WHEN LOWER(raw_doi) LIKE 'http%' THEN SPLIT_PART(raw_doi, 'org/', 2)
WHEN raw_doi IS NOT NULL THEN TRIM(raw_doi)
END as doi,
-- Extract year as integer
CAST(REGEXP_SUBSTR(raw_year, 'd{4}') AS INTEGER) as publication_year,
abstract,
raw_authors,
url
FROM raw
)
SELECT * FROM cleaned
```
**Integration & Deduplication:**
Elicit's search results have overlap. You need to deduplicate on `doi` (or a hash of title/authors if DOI is absent) *after* cleaning. I used a window function for this.
**The Pitfalls:**
- The "Study type" and "Intervention" extractions are noisy. Good for initial screening, but you must manually verify a sample to understand precision.
- Bulk export has a limit. You'll need to manage multiple exports and merge them, which introduces its own duplication issues.
- It's easy to burn credits on poorly constructed searches. Prototype with a narrow, well-defined query first.
Bottom line: Elicit massively accelerates the initial paper collection and high-level screening. But you must build a robust data pipeline around it. Think of it as a very smart, but messy, web scraper. Your value is adding the engineering rigor to make its output trustworthy.
garbage in, garbage out
I run a research group at a mid-size university, where we've managed two systematic reviews using Elicit and other semi-automated screening tools over the past year. Our current stack is Elicit for initial abstract extraction, which feeds into a custom Airflow and PostgreSQL pipeline for deduplication and manual review.
**Real Cost:** The base researcher plan is around $10/month, but that's per user. The cost comes from researcher hours spent cleaning the data. For your 3000-paper project, I'd budget 40-50 hours of engineering time to build a reliable cleaning layer, which is the true hidden cost.
**Deployment Effort:** It's a SaaS tool, so setup is minutes. The integration effort is high, as you discovered. You must build a full post-processing pipeline. Expect 1-2 weeks of work to get a trustworthy, automated flow from Elicit export to your final screened list.
**Primary Limitation:** Data consistency is the main break point. Authors, DOIs, and publication dates are messy. In our run, about 15% of records required manual correction or lookups due to missing or malformed DOIs. It's a noisy source.
**Clear Win:** The efficiency gain in the initial screening pass is undeniable. It can process and summarize thousands of abstracts in an hour, which would take a team weeks manually. Its strength is rapidly narrowing the field for a detailed manual review.
Given your description of a large, one-off project, I'd stick with Elicit but only if you have the data skills to build that cleaning layer. If your team doesn't have SQL or scripting capacity, tell us - the alternative is a more structured but expensive dedicated review platform.
Stay constructive
So the efficiency gain is real, but now you need to build a whole ETL pipeline to make the data usable. Sounds like the efficiency just shifted from manual screening to manual data engineering. At what point does the 'saved' time on screening get completely absorbed by the new overhead of maintaining these cleaning jobs?
—DW
Great question! I think you're right to question the net gain, but it hinges on the project's scale and reusability.
The 40-50 hour cleaning pipeline mentioned upstream isn't a sunk cost for a single project. Once you've built that data normalization layer for Elicit's CSV schema, you can reuse it across every subsequent review. Your second project starts with clean data day one. That's where the real efficiency flips - the engineering overhead gets amortized.
For a one-off, small review, you're absolutely correct, it might not pencil out. But for any team doing this regularly, treating the messy API or CSV output as a "raw source" you standardize is just basic data hygiene. It's the same pattern we use with any external data source, like webhook payloads or third-party APIs that change formats.
null
Your point about treating the CSV export as a `stg_elicit_raw` table is exactly right. It's the same pattern we apply when ingesting logs from any third-party service. The schema drift on fields like `year` is predictable.
I'd add that you should also treat the `abstract` and `summary` fields with skepticism. We've seen significant variation in completeness and even occasional hallucination of details not present in the source. We built a simple validation step that cross-references the word count of the original abstract (when we have it) against Elicit's summary to flag potential oversimplification or fabrication for manual spot-check.
What was your strategy for handling the 'takeaways' or 'key findings' columns? We found those to be the most inconsistent and ultimately dropped them from our analysis model entirely, opting to generate our own from the cleaned text.
Data never lies.
Totally agree on dropping the 'takeaways' column. We tried to salvage them for a quarter but the signal-to-noise ratio was abysmal. The inconsistency wasn't just random, it was systematically bad for certain paper types, like methods-heavy or review articles, where Elicit seemed to just grab a sentence from the intro.
Your word count check is clever. We took a more blunt approach: we just don't trust any AI-generated summary field as a source of truth. They're a rough filter at best. The engineering time to validate them exceeded the time to just have a grad student skim the original abstract.
Makes you wonder if the whole 'insight extraction' layer from these tools is just premature optimization. The raw metadata extraction is where the real, non-hallucinatory value is.
I'm going to need the rest of that dbt snippet to see how you're handling the type casting on the year field, because that's where we blew up our first pipeline. We assumed it was always an integer, then hit a batch where it was "2023-01-15" from some preprint server's metadata.
Your author array normalization is critical, but wait until you run into papers where the author field is a single string like "Smith, J.; Jones, A. B.; et al". Parsing that into a reliable array requires a state machine, not just split operations. We ended up creating a separate `elicit_authors_cleaning_failures` table to quarantine those for manual fix.
Did you also find that the `journal` column is sometimes the full journal name and sometimes the abbreviated ISO 4 form? That's another one that needs a lookup table.
Show me the benchmarks.