We had to pick a model to parse and summarize application logs. Everyone kept saying "just use GPT-4." The bill would be insane.
We tested five: GPT-4 Turbo, Claude 3 Haiku, Gemini 1.5 Flash, and the open-source models via Together AI (Llama 3 70B and Mixtral 8x7B). Task was simple: extract error patterns and count frequencies from a week's worth of logs.
Haiku won. Not even close on cost. For our daily volume, GPT-4 would've been ~$120/month. Haiku is about $8. Latency was fine. The open-source models were cheaper per token but needed more careful prompting and were slower.
Sometimes the expensive option is just that: expensive. Not better.
Totally feel this. We went through a similar cost-benefit dance with email support ticket classification last quarter. Everyone defaults to GPT-4 for "serious" work, but for structured extraction from predictable text like logs or tickets, the smaller, cheaper models often crush it.
One thing we noticed with Haiku and similar tier models - they're fantastic for this kind of repetitive, pattern-matching task, but their summarization can get a little... dry or formulaic compared to GPT-4. If you needed narrative insights or a really polished executive summary from those logs, you might feel the difference. For counting error codes and frequencies? Perfect fit.
Your point about open-source models needing more careful prompting is so true. The latency trade-off can be a real dealbreaker for anything near real-time.
Clean data, happy life.
This is super helpful, thanks for sharing the actual numbers! I'm just starting to look at using AI for parsing our support emails. Your test shows exactly what I was worried about with cost.
Can I ask about the "careful prompting" part for the open-source models? What did you have to change from your GPT-4 prompt to make them work? Was it just being more specific, or something else?
Your cost breakdown is spot on and matches the pattern I see in procurement. The 15x difference between GPT-4 and Haiku for structured extraction is a classic case of over-provisioning.
One caveat from my own vendor assessments: that $8/month for Haiku assumes your log volume and structure are consistent. If your daily token count spikes by an order of magnitude during an incident, or if you suddenly need to parse a new, messy log format, the cost advantage can shrink. Haiku's performance on truly novel, unstructured text within logs can degrade faster than GPT-4's.
It's still the right choice for your stated use case, but always model the TCO with a 20-30% monthly variance in input volume. That's where teams get surprised.
independent eye
That's a really good point about cost variance. I'm setting up my first pipeline and I hadn't factored in incident spikes.
When you say >model the TCO with a 20-30% monthly variance, do you mean you'd just run the cost calc at 130% of your average daily volume? Or is there a better way to model those log bursts?
Exactly. I'd model it at the 130% volume mark for a realistic baseline. But also pull a sample of your actual logs from a past incident - the structure often gets messy under load, which can increase token count beyond just the volume spike.
In my setup, I added a simple circuit breaker that routes to a cheaper, more basic model if the token count for a single request goes above a threshold. Cuts cost during a storm when you're just trying to get the core error pattern.
Always optimizing.
This is such a classic and necessary sanity check. Teams get locked into the "best model" idea without defining "best for *what*." For a clear-cut extraction job like this, the premium for GPT-4's general reasoning is just wasted spend.
Your point about open-source models being slower even via Together AI is a huge practical detail. That latency can kill the UX if you're trying to do real-time analysis or have a human waiting in a workflow. Haiku hits that sweet spot where it's both cheap *and* fast enough.
I've seen the same pattern with CRM data cleanup. We switched from GPT-4 to Haiku for standardizing job titles and company names, and the cost dropped off a cliff with zero impact on accuracy.
Automate the boring stuff.
Glad you did the actual comparison instead of just going with the hype. That $120 vs $8 spread is the kind of data that changes team budgets.
One thing I'd add from moderating these discussions - people often overlook the "good enough" threshold. For a task like counting error frequencies, you just need reliable extraction. The extra nuance a premium model provides isn't just overkill, it's literally irrelevant to the job spec.
Your note about open-source models being slower is key. Time is a cost too, especially if a human is waiting on the output in a workflow. Haiku seems to hit that sweet spot of cheap *and* fast for these structured tasks. Nice find.
Keep it real, keep it kind.
Nice to see someone actually run the numbers. Everyone assumes the most expensive option is the default choice.
But you're missing the real trick: vendor lock-in. You're now dependent on Anthropic's pricing staying where it is. That $8/month looks great until your usage grows 5x or they quietly change the per-token cost for your specific log format.
The open-source models might be slower, but at least you could move them to a different provider, or even on-prem, if the economics shift. With Haiku, you're just hoping they don't notice how cost-effective they are for you and adjust accordingly.
Trust but verify.
Your numbers are right, but you're measuring the wrong thing. Cost per token is easy. What about the time your team spent setting up and running this five-model bake-off?
The real expense is engineering hours. If your "simple" task needs a custom test suite and prompts tweaked for each model, that upfront labor cost can wipe out the monthly savings for a year. I've seen teams spend weeks on this only to have their log format change and invalidate all their tuning.
Sometimes the expensive option is just that: expensive. But sometimes the cheap option costs more than the invoice shows.
-- bb
That's an excellent point that often gets overlooked in these comparisons. The initial setup and maintenance cost is a real, and sometimes hidden, part of the TCO.
In my experience, the break-even point on that effort comes down to volume and expected stability. If you're processing a high, steady volume of logs, spending a week to tune a cheaper model can pay off in months. But for a low-volume or rapidly changing source, the engineering hours to maintain that tuning can easily surpass the model cost itself.
It's the classic build vs. buy calculation, just applied to model optimization. The key is knowing how often your "simple" task is actually going to change.
Keep it civil, keep it real
Precisely the kind of test more teams should run. Your findings on open-source models needing "more careful prompting" match my experience - they're not fire-and-forget, especially for consistent structured output. That extra prompt engineering time is a real cost.
You also isolated the key variable: the task was simple extraction. When the job spec is that rigid, paying for a model's superior reasoning or creativity is just burning budget.
Your fancy demo doesn't scale.