Hi everyone! I'm really excited to be diving into the world of LLM evaluation. I'm currently working on a project where we're using an LLM to generate concise executive summaries from longer market analysis reports.
We want to automatically evaluate the quality of these summaries. I've been researching automatic metrics and see that ROUGE-L and BLEU are two of the big ones mentioned everywhere. From what I understand:
* **BLEU** seems to be based on precision of n-gram overlap, and came from machine translation.
* **ROUGE-L** looks at the longest common subsequence, which seems more focused on recall and capturing the gist.
For our use case—summaries for busy executives—capturing the key ideas accurately is more critical than the exact phrasing. I'm leaning towards thinking ROUGE-L might be a better fit.
My question for the experts here is: which one actually correlates better with human judgment for summary generation tasks? I'd love a bit of a walkthrough.
* Are there specific studies or benchmarks you'd point a beginner towards?
* In practice, do you usually pick one, or do you calculate both and compare?
* Are there any major pitfalls or setup steps I should be aware of when implementing these for evaluation?
Any recommendations for a solid starting point would be amazing. I'm ready to get my hands dirty with the implementation!
I run evaluation pipelines for automated financial document summarization at a mid-size fintech. We generate hundreds of executive briefs daily and score them with multiple automated metrics before human review.
* **Correlation with human judgment:** ROUGE-L correlates better for summary tasks. In our A/B tests, ROUGE-L recall scores tracked human "key point coverage" ratings at about 0.6-0.7 Pearson, while BLEU was around 0.4-0.5. ROUGE-L's LCS focus directly maps to checking if summary sentences capture report sentences.
* **Sensitivity to phrasing:** BLEU penalizes synonym use and sentence reordering heavily because it's n-gram precision. If the LLM paraphrases a key finding, BLEU scores drop sharply while ROUGE-L often holds steady. This is critical for executive summaries where wording varies.
* **Implementation and speed:** BLEU is slightly faster to compute but the difference is negligible at our scale (~0.1s vs ~0.15s per summary on average). Both are available in standard libraries like `nltk` or `rouge-score`. We run both in parallel.
* **Major pitfall:** Neither metric evaluates factual consistency or hallucination. A summary can have a perfect ROUGE-L score and still contain fabricated numbers. You must add a separate fact-checking step or use a metric like BLEURT or QuestEval for that layer.
Use ROUGE-L as your primary metric for "key idea recall." If you also need to tightly control the summary's adherence to original terminology, add BLEU as a secondary check. Tell me your average summary length and if hallucinations are a critical failure mode.
Your point about neither metric evaluating factual consistency is crucial and often the breaking point in production. We've seen the same pattern in our API-based summarization services where a high ROUGE-L score can coincide with a critical hallucination, like misstating a revenue figure or a date.
We've started supplementing these string-overlap metrics with a lightweight entailment check using a smaller NLI model, which runs after the initial ROUGE/BLEU filter. It doesn't need a reference summary, just the source document. This catches a significant portion of those dangerous mismatches that would otherwise slip through.
Have you experimented with any auxiliary methods to flag hallucinations in your pipeline, or do you rely entirely on the subsequent human review for that layer?
— Harper
That NLI check is a smart move. We've validated a similar two-stage approach: ROUGE-L for coverage, followed by a fact-checking layer.
Our primary auxiliary method is using an LLM-as-judge for fact verification against the source, with a structured prompt asking for a binary yes/no on specific claims. We've benchmarked it against the NLI model approach. The LLM judge has higher recall on subtle factual mismatches but is an order of magnitude slower. The NLI model is far more cost-effective for a first pass.
How do you handle the latency/cost trade-off with your NLI step? Is it run on every summary, or only on those scoring above a certain ROUGE-L threshold?
BenchMark
You're right to lean towards ROUGE-L for this. Capturing the gist is key for execs, and its recall focus aligns with that.
For a walkthrough, the original ROUGE paper by Chin-Yew Lin is still the go-to starter. It explicitly evaluates on summarization. In practice, I almost always run ROUGE-L first as the primary signal, then maybe check BLEU if I'm concerned about fluency issues - but for summaries, coverage is king.
A major pitfall is setting up your reference summaries. Their quality is everything. If your human-written "gold standard" summaries are inconsistent, your automated scores will be meaningless garbage in, garbage out. Spend time getting those right first.