Our FinOps team needed a way to quickly extract key points from quarterly earnings transcripts. Manually reading them was a huge time sink, so I took it upon myself to evaluate a few LLM summarization models on accuracy and cost. I wanted to see if the expensive, top-tier models were actually worth it for this specific, structured financial text.
I tested five models: GPT-4 Turbo, Claude 3 Haiku, Gemini 1.5 Flash, and two open-weight models—Llama 3 70B via Groq and Mixtral 8x7B. I used a set of 10 recent earnings calls from our sector, with human-written "golden" summaries from our analysts for comparison.
My evaluation framework was pretty simple but focused on what matters for us:
1. **Factual Accuracy**: Did the summary introduce wrong numbers or claims? (Binary pass/fail per call)
2. **Key Point Capture**: Did it get all three of: revenue highlight, profit margin change, and forward guidance? (Scored 0-3)
3. **Cost per Call**: Based on input+output tokens at list pricing.
Here's the quick script I used to run the evaluation and log the results:
```python
# Pseudocode of the main evaluation loop
for transcript in transcripts:
for model in models:
summary = call_model(transcript, system_prompt="Summarize key financials.")
accuracy = check_against_golden(summary, golden_summary)
key_points = evaluate_key_points(summary)
cost = calculate_cost(transcript, summary)
log_results(model, accuracy, key_points, cost)
```
The results were surprising. Haiku and Gemini Flash were within 5% of GPT-4 on key point capture for our use case, but at less than a quarter of the cost. Haiku had one factual slip (misstated a percentage), while GPT-4 had none. The open-weight models were strong on narrative but occasionally hallucinated specific figures, which is a deal-breaker for financial data.
Bottom line: For internal, non-audit summaries where a minor error can be caught by a human skim, Haiku or Flash is the cost-performance winner. If you're publishing or doing direct analysis, GPT-4's consistency is worth the premium. The open models aren't quite ready for this number-sensitive task, but I'm keeping an eye on them.
Has anyone else run similar evals on financial or technical documents? I’m curious if your accuracy thresholds led you to different conclusions.
cost first, then scale
This is a great starting point! But your "Key Point Capture" metric only looks for three things. In my experience with earnings calls, missing context like capex adjustments or one-time charges can totally change the story, even if revenue and guidance are correct.
Have you thought about adding a consistency check? I once had a model correctly state a profit increase, but its own bullet point on margins contradicted the number. Automated checks for that saved us from some weird outputs.
Keep automating!
Thanks for sharing the test setup! I really appreciate seeing practical evaluations like this, especially with the cost comparison. That's something we're also looking at.
Your key point capture metric focusing on revenue, margin, and guidance makes a lot of sense for a baseline. I wonder if factoring in the conciseness of the summary could be another useful angle. For us, a model that gives a slightly longer but perfectly accurate summary is still more efficient than having an analyst fact-check a terse one that might omit nuance. Did you notice any major differences in output length or structure between the models that impacted their usability?
still learning
Longer isn't better. It's just more text to audit. If a terse output is missing nuance, your prompt is wrong. You're optimizing for the wrong variable.
The real problem is consistency, like user846 mentioned. I've seen models that produce a perfectly accurate paragraph and then, three sentences later, contradict a key figure. Length doesn't fix that, it just buries the contradiction deeper.
For earnings calls, you need deterministic extraction, not creative summarization. Structure your prompt to output strict JSON for revenue, margin, guidance, and capex. Force it to cite the transcript line. Then the "summary" is just formatting that data. Comparing verbosity is a distraction from whether the model can reliably pull those fields without hallucination.
Don't panic, have a rollback plan.
Your evaluation framework is painfully naive for financial data. Factual accuracy as a binary pass/fail is going to mislead you. A model can "pass" by being vague and still be useless.
You need to measure precision on the numbers themselves. Did it get the exact revenue figure, or just say "revenue increased"? If it stated a 5.2% margin, but the transcript says 5.3%, that's a critical failure for our domain, but your binary metric would likely call that a pass. You're testing for "not wrong" instead of "correct."
And while I appreciate the cost angle, you're missing the real cost: the analyst's time correcting a subtly wrong figure that your evaluation deemed "accurate." The cheapest model that requires manual verification is the most expensive one.
monoliths are not evil
You're right that binary accuracy can hide a lot of problems, especially vagueness. A model that says "revenue grew" passes the test, but doesn't give the actionable number your team actually needs.
I'd push back a little on the idea that any numeric deviation is automatically a critical failure, though. For some teams, catching the directional trend correctly on a first pass is still valuable if it saves hours of reading, as long as the analyst knows to verify the exact figure. It depends on the workflow.
The core point stands: evaluating these tools requires understanding your own tolerance for error and the true total cost of review. A "passing" vague summary creates more work, not less.
Be kind, stay curious.