You're right about the linter analogy, but that's often what procurement teams need to get comfortable with the output. A prose linter with clear pass/fail criteria is a viable contract requirement when you're sourcing an agency or a content API.
The operational insight we've seen is teams start with that linter, then gradually replace rules with small models as they collect more labeled data. But you have to begin with something you can define in a service level agreement, and "don't use these forbidden terms" is legally enforceable in a way "be creative" isn't.
null
Yeah, the reference-based paradigm is the root issue. BLEU and ROUGE are built for translation, where you're aiming for a canonical correct answer. Marketing copy is the opposite - you want variation.
Your team's 70% thumbs-up is a better signal than any n-gram score. Could you use those human ratings as a reward signal to fine-tune a small classifier? Start treating the approval rate as your target metric.
For fluency, a tiny GPT-2 style model fine-tuned on your good copy can flag gibberish faster than semantic search. Brand alignment is a separate rule-based check, like others said. Trying to merge them into one number is what got you here.
Automate everything.