Help: My BLEU scores don't match my human ratings at all. What gives?

That's a great point about short-form creative text. I've seen the same thing with model-generated commit messages, honestly. BLEU really falls apart when there are many equally valid ways to phrase the same core idea.

You mentioned ROUGE-L. I think that's a smart next step, especially for looking at the longest common subsequence. It might capture reordered phrases a bit better. For brand voice, have you looked at any embedding-based metrics, like using sentence-BERT to check semantic similarity against your reference pool instead of exact word matching? It can be more forgiving with synonyms.

Thanks for sharing this, it's super helpful. Did you find the discrepancy was worse for certain types of copy, like subject lines versus longer ad body text?

Yeah, you're spot on about the single metric trap. We had the same rude awakening when evaluating automated email personalization snippets.

That 70% human acceptance vs. low BLEU score is the exact pattern we saw. It's like the metric penalizes good, varied copy. Your point about short-form text is key. For a subject line, "Don't Miss This" and "Last Chance Inside" have zero n-gram overlap but are functionally identical.

Your idea to explore semantic similarity is the right path. We got better correlation by using a lightweight model to check if the generated text and a reference live in the same conceptual neighborhood, rather than requiring them to be lexical twins. It's still not perfect, but it closed the gap a lot. Did you find the problem was worse for certain types of references, like older campaign copy versus your current brand voice?

Exactly. The brand voice point is why we stopped using old campaigns as the only reference. Our tone shifted last year, so new copy that scores high on human ratings now gets hammered by BLEU against outdated references.

We used sentence transformers for the semantic similarity check. The correlation with human ratings was better, but you need a decent-sized reference set. A single "perfect" reference still trips it up.

Ever benchmark the inference latency of those lightweight similarity models? We had to cache embeddings to keep evaluation times reasonable.

Your hypothesis is correct. BLEU is a machine translation metric built for a different problem, where deviation from a reference translation is an error. Marketing copy isn't translation.

> systemize our eval process

You're focusing on the metric, but your real problem is your evaluation framework. Using past campaigns as a static reference set is the first flaw. It locks you into historical patterns and assumes there's one "correct" output. For creative text, that's false.

ROUGE-L or semantic similarity are just different forms of the same mistake: measuring against a reference. You need to define the qualities you actually care about - fluency, brand alignment, persuasiveness - and score those directly, perhaps with a small classifier or rubric. Stop looking for a proxy metric and measure the thing itself.

Your hypothesis about BLEU's rigidity is correct, but I think you're underestimating how structural the mismatch is. It's not just about flexibility; the metric's core logic of geometric mean of n-gram precisions is fundamentally misaligned with evaluating creative variance.

> trying to systematize our eval process

You've identified the real need. Swapping BLEU for ROUGE-L or a cosine similarity on embeddings is an incremental fix that still anchors you to a reference set, which itself becomes a bias model. For brand alignment, you need a metric that evaluates adherence to a *style guide* or a set of brand attributes, not similarity to past examples. A more systematic approach would be a small, fine-tuned classifier trained on your team's thumbs-up/thumbs-down decisions to predict acceptability directly, using the text's features rather than its distance from a historical artifact.

Have you profiled the variance in your human-written reference set itself? If you calculated BLEU scores between those human references, you'd likely find they also score poorly against each other, which perfectly illustrates the problem.

The problem isn't your metric, it's that you're trying to quantify creativity like it's a system ping. You're using a translation metric for a marketing problem. That's like using a network latency check to judge a painting.

Your next idea is to jump to ROUGE or semantic similarity. That's just swapping one flawed reference-based metric for another. It's still measuring deviation from a static set of past work, which you've already identified as shifting.

If 70% of the output is acceptable to humans, you already have your evaluation system. You're trying to automate a human judgment call. Train a simple classifier on your team's thumbs-up data and skip the proxy metrics entirely.

Yep, that's the root cause. You're trying to measure "good" by checking distance from a single point, when "good" is actually a whole region.

> Stop looking for a proxy metric and measure the thing itself.

This is the real shift. We built a tiny classifier on our 'approved/rejected' labels for support ticket summaries. It's not perfect, but it correlates way better than any n-gram overlap score ever did. The trick is you have to treat the classifier score as a heuristic, not a ground truth - just like your human thumbs-up ratings are.

You still need a way to generate those labels, though. That's the operational cost everyone forgets.

The classifier approach is a solid operational step, but it's still a proxy. You've just swapped a reference text for a reference distribution of human votes. That classifier's accuracy ceiling is your labeling consistency, which for creative copy is often low.

The bigger issue is treating evaluation as a single score at all. For our marketing team, we had to break it down: a separate cheap heuristic for brand term inclusion (regex), a semantic similarity check for core message, and a thumbs-up model for stylistic flair. Each catches different failure modes. A single classifier tends to optimize for the most common failure pattern and miss the rest.

And yes, the labeling cost is real. We found it cheaper to run a small human eval on each campaign's outputs and use those as dynamic training data for the season, rather than aiming for a universal classifier.

That's a really good point about breaking it down. I've been stuck thinking I need one magic number too.

> Each catches different failure modes.

So it's more like setting up separate cloud monitoring alarms for latency, errors, and cost, instead of one "health" score. Makes sense.

But for a small team, that's a lot of separate heuristics to build and maintain. How did you decide on those three specific checks? Was it just from looking at past failures?