X vs Y - which is better for structured data extraction under $100/month?

Absolutely agree on Haiku's cost edge for volume. That stubbornness you mentioned is actually a feature for invoices - I've had GPT-4o "helpfully" fill in missing invoice numbers with plausible ones, which created a data integrity nightmare.

One thing that's missing from the cost-per-doc math is retry overhead. Haiku can be less consistent on complex tables, leading to more retries or fallback to a pricier model. Have you measured your retry rate with it compared to Sonnet in your pipeline? That could eat into the per-token savings.

That's such an important point about the "helpful" hallucinations being a nightmare. I've had GPT-4 try to infer a missing contact email by combining a name and domain, and it was wrong 100% of the time, but the JSON was always perfect. It creates silent errors that are so hard to catch later.

On retry overhead, you're spot on. My early tests with Haiku on dense purchase orders showed a retry rate around 12% for complex line items, mostly due to it skipping fields or returning empty arrays. Sonnet was under 3%. When you factor in that each retry isn't just another Haiku call - it's also the engineering time to manage the queue and maybe a fallback to a more expensive model - the raw token cost advantage shrinks fast. Have you built a separate pipeline to flag low-confidence extractions for a second pass, or do you retry everything automatically?

Your four metrics are good, but you're missing one: rate of valid but wrong data. Haiku is stubborn, which is great for invoices, but GPT-4o's helpfulness can create silent errors by inferring missing fields with bad guesses. I've seen it invent invoice numbers.

You need to measure the hallucinations that still fit your schema. That'll hit your real cost when you have to clean the data later.

>My test methodology involves a corpus of 500 diverse documents. The key metrics are... Cost per Extraction: Calculated based on total input+output tokens per document.

Your token-based cost calculation is a sound starting point, but it's disconnected from your primary business metric, which is cost per *successful* extraction. You must factor in your retry rate and the cost of handling invalid outputs. I ran a similar benchmark on purchase orders last quarter.

Even with Haiku's lower per-token price, our system's retry rate for complex line-item tables was 11.5%. Each retry isn't just another API call - it's added queue latency and orchestration complexity. When we modeled the total cost of ownership, including engineering time to manage fallback logic to a more consistent model like Sonnet, GPT-4o's higher initial token cost was offset by its 2.7% retry rate. The cheaper model created more expensive, unpredictable pipelines.

For a true cost under $100/month, you need to simulate the full pipeline, not just the happy path. What's your observed retry rate for the most complex document type in your corpus?

Love seeing a structured benchmark laid out like this! Your four metrics are a fantastic starting point, especially for keeping an eye on that $100 budget.

That said, the others have a great point about prompt overhead skewing the cost. Since you're working with a strict schema, you might be spending a huge chunk of those tokens on instructions for every single document. Have you tried referencing a schema ID or a short alias in your prompt instead of pasting the full definition each time? The token savings could be dramatic at your volume.

Also, I'd be really curious to see how your "Cost per Extraction" metric evolves when you factor in the retry rate for those 500 documents. A cheaper model that needs two tries can quickly become more expensive than a pricier, more consistent one. Did you track that?

Totally get the worry about silent errors. Basic sanity checks are a great start, but those "plausible but wrong" values are the real killer.

Have you considered adding a separate LLM call for a quick cross-check on key fields? It sounds redundant, but using a smaller, cheaper model like Haiku just to verify that an extracted total matches the sum of line items, for example, can catch a lot of those sneaky hallucinations before they hit your database. Adds a bit to cost, but saves a ton of cleanup later.

>cost of failure

Exactly. But everyone's focusing on model retries. The real cost is manual triage. At 200k docs a month, a 2% failure rate is 4k errors. Even with a fancy retry loop, a chunk will still fail and need human eyes. That's a full time job for someone, not a line item in your API bill. Your budget is toast before you even look at compute.

You're absolutely right. That manual triage cost is the silent budget killer that doesn't show up on the AWS bill.

We hit this hard last year. Our initial 0.5% "final failure" rate after retries seemed trivial until payroll. We ended up building a simple internal UI for corrections that logged every fix. That data let us target our prompts to reduce the *specific* errors humans kept seeing, which dropped the failure rate way more than swapping models ever did. It's about closing the loop.

Now I always bake an estimated "error handling hourly cost" into my ROI models. It changes which tool looks "better" every single time.

Your methodology is fundamentally flawed because you're testing individual documents in a vacuum. The real cost driver in a pipeline isn't per-document latency or token cost, it's the cascade failure when a batch job hangs because 12% of your extractions need a retry loop.

You didn't even mention your error handling strategy. Are you using a dead-letter queue? What's your fallback model when Haiku returns an empty array three times in a row? Without that, your "P95 Latency" metric is just a lab number, not a system one.

And stop pasting your full JSON schema into every prompt. Reference it once and use a short alias. At 500 documents, you're probably burning 30% of your token budget on redundant instructions.

You're correct about the cascade risk, but I think you're still underestimating the labor multiplier. A dead-letter queue just organizes the failures, it doesn't reduce them. You now need a monitoring dashboard and someone to check it, which is another half-day a week of engineering time. That fallback model you mentioned? Now you're maintaining two separate model integrations and their respective prompt versions. Your "system" cost just doubled before you've even paid for the first human correction.

And that schema alias trick only works if your provider supports it across the entire pipeline. Many don't, or they charge extra for the custom dictionary. What looks like a simple optimization in a benchmark can become a vendor lock-in trap.

The cross-check model idea is a solid tactic for catching arithmetic hallucinations, but it introduces a new class of error - verification model failure. If Haiku mis-verifies a correct extraction from GPT-4o, you've now added cost and created a false negative that might trigger an unnecessary retry loop.

The more critical design flaw is assuming the verification is free of the same bias. Training a separate model on the same flawed prompt structure just gives you two systems that can agree on a wrong answer. You need an orthogonal validation method, like a rules-based check on the extracted data itself, not another LLM inference.

We implemented this and found the second LLM call only reduced our 'plausible error' rate by about 30%. The remaining 70% were cases where both the extractor and the verifier models made the same logical mistake.

Totally agree that tokenizer differences can flip the math. I've seen Claude stumble on dense tables where every cell is a date or SKU number, blowing up the token count unexpectedly.

But for raw text like paragraphs, its tokenizer can actually be more efficient. The real killer is when you mix formats in one document.

Has anyone run a side-by-side on a mixed invoice with both line items and a long terms & conditions section? That's where I'd expect the biggest variance.

Oh, the mixed-format invoice is the perfect torture test. I ran a batch last month with five different providers, and the variance was hilarious.

You're dead on about dense tables. One vendor's tokenizer treated a column of 10-digit SKUs as a single word, another exploded each digit. The T&Cs section flipped the leaderboard entirely. The model that crushed the table got absolutely murdered on the verbose legal text, turning a 20% cost advantage into a 15% loss.

The real kicker? The *order* of content mattered. When the T&Cs came first, the models seemed to waste tokens "setting up" for a prose-heavy doc, making the later table extraction clunky and expensive. Tables first led to cleaner, cheaper runs. So now my pre-processing pipeline includes a dumb "detect and reorder" step. Go figure.

Your metrics are solid, but you're missing a key variable: tokenizer efficiency across document types. A model's price per token is fixed, but how it segments your text isn't.

Your corpus includes invoices, research papers, and product descriptions. The token count for the same physical text can vary by 15-20% between Claude and GPT-4o, depending on the density of numbers, jargon, or formatting. For invoices with many line items, Claude's tokenizer can be more frugal with numbers, but it might expand legal boilerplate from a research paper.

Have you normalized your cost per extraction by the *actual* token count each vendor reported, not just by character length? That often flips the "cheaper" model.

Absolutely. That 2% failure rate number is the trap, because it assumes the cost of fixing each error is the same. It's not.

The real killer is error *variance*. If those 4k failures are all missing the same field from the same document type, you can build a quick script or adjust a prompt. But if they're 4k *different* errors scattered across your schema, now you need a human to understand the context of each document to fix it. That's when you need that full-time employee.

We learned this the hard way. Our "2%" was actually a thousand different edge cases, not a pattern. Budget was gone in a quarter.