I've been testing Elicit for a few weeks to automate pulling data from academic and technical PDFs into our CRM for lead scoring. The promise is huge, but the risk is bad data. You can't trust the extraction without a verification process.
I built a simple, manual audit routine. It's not fancy, but it catches inconsistencies.
My verification steps:
* **Sample Stratification:** Don't just check random pages. Pull a stratified sample that includes:
* The first page (often has headers/abstracts).
* A middle page with dense text and tables.
* A page with figures or images.
* The references/bibliography section (formats vary wildly).
* **Focus on Key Fields:** I'm not verifying every word. I target the specific data points I need to extract, like:
* Author names and affiliations
* Publication dates
* Specific numerical results or metrics mentioned in the text
* Contact emails or funding sources
* **Side-by-Side Comparison:** I open the source PDF and the Elicit extraction results in two windows. I scan the Elicit output for my key fields and then physically locate and read that section in the PDF. I note any:
* Omissions (data point not extracted at all)
* Truncations (partial text extracted)
* Hallucinations (data point that isn't in the source)
My initial audit of 50 documents showed an 85% accuracy rate on my target fields. The main failure points were complex tables and text within figures. This baseline is critical for deciding whether to use Elicit for a given document type and what post-processing checks we need to build.
What's your method? Are you doing spot checks or a full reconciliation? I'm particularly interested if anyone has quantified the error rate by document type (e.g., clinical papers vs. white papers).
Agreed on stratification, that's smart. I'd add a 'worst case' page to your sample - something with poor scan quality or a complex multi-column layout. That's usually where the extraction gets wobbly.
How are you tracking the omissions and mismatches you find? I started a simple spreadsheet to see if errors clustered around specific document types or Elicit's field labels. Turns out 'author affiliation' was a mess for pre-prints but fine for journal PDFs.
Good call on focusing the verification effort. You mentioned tracking omissions in a spreadsheet. Have you thought about automating that data collection? A simple script could log the page numbers and field types for mismatches when you flag them, then dump to a CSV. That would give you way better data to spot patterns.
The 'worst case' page suggestion from user416 is solid. I'd throw 'first page after a format shift' in there too, like the start of an appendix. Extractors often stumble on those boundaries.
Alert fatigue is real, but so is my rule of silence.
The side-by-side comparison is a good method. How do you handle discrepancies in numerical data, like metrics in a table, where the formatting might cause misalignment in Elicit's extraction? That seems like a high-risk area for our use case.
That side-by-side comparison is exactly where I'm at, and honestly, it's where I get stuck. Opening two windows feels like the bare minimum, but I'm already drowning just trying to manage that for a decent sample size.
How are you physically noting the omissions and mismatches? I tried a notebook, then sticky notes on my monitor... it got messy fast. Is there a slightly better manual method before jumping to a script? I'm worried if I don't systemize the note-taking, I'll never get a clear picture of the actual error rates.
That side-by-side method is the core of it. I've found the friction comes from switching context back and forth. My workaround is to print the sample pages from the PDF, then annotate the printout directly while looking at the Elicit output on screen. It keeps my place and I can scribble notes right on the source.
For your omissions list, do you track the *reason* you think Elicit missed it? Like, was the data in a footnote, a complex table header, or just plain text that got skipped? That context helps later when you're deciding if you need to adjust the source material or your extraction parameters.
What's your threshold for an acceptable error rate before you'd feel comfortable automating this into your CRM flow?