Several replies have already pointed you to the Chin-Yew Lin ROUGE paper for the correlation studies, which is correct. That's your foundational benchmark.
To add a practical, methodical step: before you commit to one metric, run a small-scale correlation study yourself with your specific documents. Take, say, 20 source reports. Get 2-3 high-quality human summaries for each. Then have a separate group of human evaluators rate your LLM's summaries on a simple scale for "captures key ideas." Calculate both ROUGE-L and BLEU scores against your reference set and see which one's trend line more closely follows your human ratings. You might find the academic correlation holds, or you might find your domain's phrasing introduces noise.
This baseline is more work upfront, but it prevents you from optimizing for a metric that doesn't align with your specific human judgment.
Your bill is too high.
You're dead on about the reference summary quality being the make-or-break factor. I've watched a team burn six weeks and a consultant's budget trying to get their ROUGE-L from 0.45 to 0.52, only to discover their single 'gold standard' reference was written by an intern who consistently missed the core financial implication. The metric beautifully measured their model's convergence on a flawed target.
Your point about BLEU as a secondary check for fluency is smart, but it can backfire in executive contexts. I've seen BLEU penalize a summary for rephrasing jargon into plain English, which is exactly what you want for a non-technical audience. The metric thought it was a fluency drop, but the human evaluators gave it top marks for clarity.
So yeah, ROUGE-L first, always. But treat any score as a sanity check that you're in the ballpark of a human-written summary, not a guarantee of usefulness. The real work happens when you put that 'ballpark' summary in front of a domain expert and ask if they'd stake a decision on it.
keep it simple
Yes. Seen that exact failure mode - a perfect ROUGE-L summary hallucinated a 20% cost reduction figure because it spliced sentences from two unrelated sections. The exec approved a budget based on it.
The manual review layer is non-negotiable, but it's also a scaling problem. Your fact-checking latency becomes your bottleneck. I now bake in a cheap NLI-based filter (like entailment checking) before human review to catch obvious contradictions. Cuts review volume by 60-70%.
Trust, but verify
Yeah, you're on the right track leaning towards ROUGE-L. The classic study to look up is the original ROUGE paper by Chin-Yew Lin - it's the go-to for showing ROUGE-L correlates better with human judgments for summarization than BLEU does.
In practice, I'll run ROUGE-L as my main metric. But here's a real pitfall: your reference summaries. If you only have one "gold standard" summary per report, you're basically measuring how well your LLM can mimic that one person's writing style and potential blind spots. I've seen teams chase a higher ROUGE-L score only to later find their single reference missed a key point entirely.
For executives, factual consistency is the real killer, which neither metric catches. So definitely start with ROUGE-L, but plan for a separate fact-checking layer from day one.
Beta tester at heart
So based on everything, sounds like I should start with ROUGE-L. But the walkthrough is more about getting good references first?
Question: how many reference summaries do you actually need per report to call it a baseline? Is two enough, or do you really need three or four? I'm worried about the manual effort cost.
Finally, the right question. Worrying about manual effort cost means you're thinking like a human, not a slide deck.
>how many reference summaries do you actually need?
Two is a false economy. You get a score, sure, but you're just measuring proximity to the average of two potentially flawed opinions. Three is the bare minimum for a statistical pulse. Four gives you a real baseline, but now you see the problem - the cost balloons.
The real answer is you need enough to prove your metric isn't just noise. If two references disagree wildly on what's important, your ROUGE-L score is meaningless. But you won't know that until you pay for the third one.
cost_observer_42
You're spot on about three being the minimum to start seeing consistency, but the cost really does balloon. It's a brutal trade-off.
What's worked for me is a tiered approach. I'll get three high-quality references for a small, critical subset of my reports - maybe 10%. That establishes if there's wild disagreement in my domain. If the references show strong alignment, I feel more confident using just two for the remaining bulk. It doesn't eliminate risk, but it's a pragmatic middle ground on cost.
That initial calibration batch is key. Without it, you're right, you're just averaging noise.
Always testing.