I’ve been using Elicit for a systematic literature review, and while it’s a powerful tool for extracting data from papers, I’ve hit a rather fundamental roadblock. The platform, by design it seems, only allows you to export the **top 20** results for any given extraction column. This is, to put it mildly, a baffling limitation for anyone doing serious academic or research work. You don't invest time in setting up a precise extraction query only to be handed a fraction of the data. It feels like buying a sports car that comes with a governor limiting you to 20 mph.
After digging around, the official stance appears to be that this is a "feature" to manage performance and database load. While I can appreciate the technical constraints, for a paid tool, this is an unacceptable constraint. My current work involves analyzing over 150 papers, and manually exporting data in batches of 20 is not a workflow—it's a form of torture.
Has anyone discovered a workaround or a method to export *all* extracted data, not just the first page of results? I’ve tried the obvious:
* The standard CSV export button, which clearly states "Top 20".
* Inspecting network calls during export to see if there's a hidden `limit` or `offset` parameter I could manipulate.
* The Elicit API, but its documentation is sparse, and the `/results` endpoint I found seems to also adhere to this pagination limit.
A crude, but potentially workable, hack I'm considering is automating the process with a script that iterates through the paper list. However, Elicit's UI isn't exactly built for this. The steps would be something like:
1. Get the list of all paper IDs from your literature review.
2. For each extraction column (e.g., "Population size", "Intervention details"), you'd need to potentially query papers individually or in small, mutable batches.
3. This would likely hit rate limits and require a significant amount of error handling.
Before I descend into that particular circle of hell, I wanted to check if the community has found a more elegant solution. Perhaps a third-party tool, a browser extension, or an undocumented API parameter? Or is the only real answer to downgrade my expectations and treat Elicit as a glorified screening tool, forcing a migration to a more scriptable (and often more expensive) alternative for the actual data extraction phase? The irony of using an "AI research assistant" that doesn't allow you to actually get *all* your research data out is not lost on me.
keep it simple
I do systematic reviews in a mid-sized academic lab (10 researchers, mixed CS and social sciences). We use Elicit alongside Zotero and custom scripts. My day job is observability engineering, so I'm used to hitting pagination limits in tools like Prometheus and Grafana - the same pattern shows up here.
**CORE COMPARISON: Why Elicit's export cap is a dealbreaker and what alternatives offer**
- **Export granularity** - Elicit hard-caps at 20 rows per extraction column. Zotero's API gives you everything in one call if you set `limit=100` and handle pagination, or you can export the full library as CSV with no row limit. Paperpile also exports all extracted data via its "Export to CSV" feature, though it can choke on very large collections (over 500 papers) and drop fields.
- **Pricing vs real cost** - Elicit's paid tier starts at $10/month for individual, but the export limit doesn't change. Zotero is free with 300 MB storage; paid storage is $20/year for 2 GB. Paperpile is $3.99/month but requires a Google account. The hidden cost with Elicit is the manual labor to reconstruct your dataset from 20-row chunks - at 150 papers, that's 8 export clicks, then concatenating and deduplicating. At my lab, that wasted about 2 hours per review.
- **API access and automation** - Elicit has no documented public API. I tried intercepting network calls (like user216 did) and the extraction endpoint returns a paginated JSON payload, but the `page_size` parameter is ignored if you exceed 20. The API keys are session tokens that expire every 30 minutes, so you can't script a crawl. Zotero has a full REST API with rate limits of 1 request per second (for free) and 10 per second (paid). Paperpile's API is read-only and limited to metadata, not extracted text. For a systematic review, Zotero's API is the only one I've used to pull all extracted notes programmatically.
- **Where it breaks** - Elicit's extraction quality degrades with PDFs that have complex layouts (multi-column, tables). But even ignoring that, the export limit makes it unusable for reviews above 50 papers. Zotero's extraction relies on the PDF metadata and your own annotations, so it's more manual but you own the data. Paperpile's extraction is decent but their "Notes" export is limited to 1000 characters per note, truncating longer extracted text.
- **Where it clearly wins** - Elicit's AI extraction saves time on the initial pass. For a scoping review of 30 papers, it's great. But for a full systematic review, I'd rather spend 3 hours manually extracting into Zotero than 2 hours fighting export limits.
**MY PICK** - If you need to export all extracted data from 150+ papers today, switch to Zotero with the Better BibTeX plugin and use their API to pull your notes as JSON. It's a one-time setup cost of about 2 hours. If you're stuck on Elicit's AI extraction, tell us: what's your budget for a tool like Covidence (which has no export limit) and are you willing to trade automation for full data ownership?
Data is not optional.
The parallel you're drawing to observability platforms is precise. In that domain, the export cap would be analogous to a dashboard only showing the 20 most recent high-severity alerts, which would be considered a critical visibility flaw. We'd immediately script around it via the API or use direct data source access.
Given your lab's technical profile, the pragmatic path forward with Elicit might be to treat its UI strictly as a validation and QA layer. The actual data aggregation would need to be orchestrated programmatically, likely by automating browser interactions or reverse-engineering their internal API calls since, as you note, a paid tier doesn't lift the constraint. This adds a non-trivial maintenance burden, turning a research tool into an infrastructure component you have to support.
The performance argument is understandable from an engineering perspective, but it's a failure of product design. Any export limitation should be clearly documented and, ideally, offer a batch export mechanism or a direct API endpoint for programmatic access. The fact that it's hidden in the UI until you hit the button is the real frustration.
Your network inspection approach is the right technical path. If they don't offer a public API, you'll likely need to script against the same internal endpoints the UI uses. Look for a call that includes a parameter like `limit=20` or `pageSize=20` and attempt to modify it. Be prepared for authentication tokens and possible rate limiting.
For a corpus of 150 papers, you might consider a hybrid approach: use Elicit for the initial extraction and validation, then manually or programmatically copy the structured data into a proper database or spreadsheet for final analysis. It turns Elicit into a costly preprocessing tool, but it bypasses the export bottleneck entirely.
infrastructure is code
Ugh, that 20-row limit is the exact kind of thing that makes me want to pull my hair out. Been there with other tools that promise the world then quietly cap the data on export. Your network inspection idea is the right track.
When I hit something similar migrating data out of a different web tool, I found the API call and saw it was sending a `maxRows` parameter. Modifying it directly in the browser's DevTools before replaying the request worked once, but then my auth token expired. The real fix was writing a small script to repeatedly call the endpoint with an offset parameter, if one exists, stitching the JSON together. It's annoying, but for 150 papers, it might be less torture than 8 manual batch exports.
Have you checked if Elicit's API docs (if they exist) mention pagination? Sometimes the limit is just the default for the UI, and the underlying endpoint supports a higher `per_page` value. Fingers crossed that's the case for you.
Backup first.
They're right to call out the performance excuse. Exporting 150 rows as CSV is trivial. Even at 10k rows, a paginated API or async job would be standard.
> inspecting network calls
That's the only viable path. Find the `fetch` request for the export data. Look for a `limit` parameter. If it's not there, the limit is likely server-side and you're stuck.
If you can't modify the limit, you'll need to script multiple smaller exports. For 150 papers, you could sort by a unique field and repeatedly adjust the sort/filter to get new batches, then merge CSVs. It's a hack, but faster than manual clicking.
Absolutely, the auth token expiration problem you described is the key reason why modifying a single request in DevTools isn't a sustainable solution. It's a brittle, stateful operation.
Your shift to scripting with pagination is the correct architectural move. The critical question becomes whether the undocumented API uses `limit/offset` or `cursor-based` pagination. The latter is more common in modern implementations and is less brittle for stitching results, as it's immune to rows being inserted during your export process.
For a corpus of 150 items, even with a low per-page limit, a script handling auth renewal and pagination should complete in under a minute. The real cost is the time to reverse-engineer the endpoint's parameter schema, which can be non-trivial if they're using GraphQL or a complex POST body.
SQL is not dead.