Ran the new "Data Extraction" beta on a set of 50 recent chemistry PDFs from various journals. The promise is tempting: auto-pull compounds, methods, properties into structured JSON.
My verdict: it's a fragile toy, not a tool. For anything beyond a simple, perfectly formatted table, it falls apart. Got maybe 30% usable output. The rest was hallucinated schema fields or a complete miss.
Tried the API. You feed it a schema, it returns... something. Here's a typical "success" where it just invents data:
```json
{
"extracted_data": {
"compound_name": "Catalyst X",
"yield_percentage": 95.2,
"reaction_temperature": "150C"
}
}
```
Problem? The paper never mentioned a "Catalyst X" or a yield. It just made those up because the schema asked for those fields and the model guessed. Garbage in, structured garbage out.
Sticking with my boring, reliable `pdftotext` + custom regex filters for now. At least when it fails, I know *why* and it doesn't fabricate results. This beta feels like they wrapped a general LLM with a JSON formatter and called it a feature. Not production-ready, probably never will be for complex domains like chem.
If it ain't broke, don't 'upgrade' it.
Your point about the fabricated data is crucial. Even a 90% success rate is catastrophic when the 10% failure mode is confident hallucination. In production infrastructure, we can't have Terraform inventing S3 buckets because the HCL schema expected one.
I've seen similar patterns with LLM-based log parsers - they'll invent structured fields that look plausible but contaminate your observability data. The problem isn't just the model, it's treating extraction as a pure generation task instead of a retrieval-and-validation one.
Your `pdftotext` + regex approach, while "boring", gives you a deterministic failure boundary. You can at least build a pipeline where you know which stage broke. This beta seems to swap that for a black box where correctness is probabilistic. For scientific data, that's a non-starter.
infrastructure is code
Oof, that fabricated data example is scary. It reminds me of when I first tried using a generic LLM to parse Jenkins logs for failures. It would invent error codes that looked real but totally broke the downstream alerting.
You mentioned sticking with `pdftotext` and regex. Do you run that in a pipeline, like a Git hook or a scheduled job? I'm trying to set up something similar for Docker image reports, and I'm worried about the maintenance when journal formats change.
Learning by breaking