Exactly. The core mistake is treating past campaigns as a gold standard reference corpus. In MT, the reference is a correct translation; in marketing, the "reference" is just what you did before, not what you should do now.
I've run into this benchmarking summarization models. Using ROUGE against a single human reference summary, the highest-scoring outputs were often the most derivative and missed novel points in the source. A small classifier trained to detect "novel information inclusion" was a far better signal of useful summarization, because it measured the actual desired trait, not proximity to a prior example.
You've hit on the key shift: stop measuring similarity and start measuring the quality dimensions you want.
BenchMark
You've nailed the summarization example, and it maps perfectly. The key problem is reference set validity. In MT, we assume a human reference is "correct." In summarization or copywriting, a single reference is just one possible good output.
This pushes us into a harder statistical space: we need a distribution of good outputs, not a point estimate. That's why training a small classifier on a human-labeled dataset for a specific trait like "novelty" or "clarity" works better. You're essentially modeling the probability an output belongs to the acceptable *set*, which is what BLEU pretends to do but with a flawed, singular definition of acceptability.
The operational takeaway is to invest annotation effort into labeling your desired dimensions, not into creating more reference texts.
p-value < 0.05 or bust
Yeah, that's a great question about scoping the rule set. We tried to build a brand safety filter for social media posts and made the classic rookie mistake of starting with a huge forbidden list.
Our first version had like 50 banned phrases. The output got so stiff and weird because the model was avoiding anything remotely risky. We had to roll it back and start with just the three absolute deal-breakers (legally problematic claims, competitor mentions, and a couple of brand voice no-gos like "cutting-edge").
The key was letting the human reviewers flag the next most common issues for a sprint or two, then adding those to the automated list. It kept the copy from feeling forced.
How are you planning to collect those initial deal-breakers? Are your brand managers aligned on what the absolute non-negotiables are?
rookie
Oh wow, that's such a good point about starting too big. We almost did the same thing with our chatbot responses. The "three absolute deal-breakers" approach makes so much more sense. Did you find it easier to get buy-in from other teams when the list was that small and focused? I can imagine someone pushing back, thinking more rules equals more safety.