Alright, let's get this out there. Every vendor deck and "AI-powered" sales email tool is suddenly touting their BERTScore metrics like they're handing out gold stars. "We achieved a 0.92 BERTScore!" Great. What does that *actually* mean for my pipeline velocity or my quota attainment?
I work with sales teams. They need to know if an AI-generated email sequence is going to resonate with a VP of Procurement, not if it's semantically similar to some reference text in a way a transformer model finds pleasing. BERTScore feels like the ultimate in-house metric for ML engineers to high-five over, while leaving the business side completely in the dark.
My specific gripes:
* **The "reference" problem:** The score is only as good as the reference text you compare against. Who defines the perfect reference reply in a complex sales negotiation? A single "golden" answer doesn't exist.
* **Zero interpretability:** A sales ops manager can't look at a score of 0.87 and know *what* to fix. Is the value proposition weak? Is the call-to-action unclear? It gives a grade, not feedback.
* **Gameability:** I've seen demos where tweaking a few keywords sends the score soaring, but the resulting message loses all its natural, persuasive flow. You're optimizing for the model's bias, not for human response.
We're being asked to trust a metric that, from a business user's perspective, is a black box. It correlates with *something*, but is that something *revenue*? Or just technical similarity? I'm deeply skeptical of any framework that can't translate its results into actionable business terms.
Are we just accepting this because it has a fancy paper behind it? What are people *actually* using to evaluate if an LLM output will *work* in a real B2B sales cycle, not just score well on an academic benchmark?
Just my 2 cents
Trust but verify.
You've hit on the core issue: BERTScore measures semantic similarity, not business efficacy. A model can generate text that's semantically close to a "perfect" reference email but still fail on clarity, persuasion, or specific jargon a VP of Procurement expects.
The gameability point is critical. I've run tests where swapping out a generic phrase for a niche industry term barely changed the meaning but tanked the score because the reference corpus lacked that specificity. It optimizes for the wrong thing.
For sales sequences, you're better off with a simpler, interpretable checklist scored by the team: does it name the prospect's company, cite a recent event, include a clear next step? Those might map to pipeline velocity better than a 0.92 from a black box.
Data is the source of truth.
Totally agree, and your point about gameability is spot on. It reminds me of when we tried using BLEU for chatbot responses - optimizing for that score led to verbose, awkward replies that technically matched the references but annoyed users.
Your checklist idea is the right path. We actually built a hybrid system where a simple rule-based layer (checking for company name, next step, etc.) gates the content before it even gets a BERTScore. The transformer score then becomes just one faint signal among many, more for detecting when something goes *wildly* off-topic rather than grading quality.
I'd add one caveat: even the checklist needs careful tuning. If you make "cite a recent event" a required field, you'll get a lot of forced, irrelevant mentions just to tick the box. The human-in-the-loop scoring you mentioned is the only real fix for that.
Integration Ian
Exactly. It's an engineering metric, not a business KPI.
I've seen teams waste months tuning pipelines for a 0.02 BERTScore bump that had zero correlation with actual reply rates. The business needs a feedback loop they can act on.
If you must use it, bake it into a quality gate that's invisible to ops. Fail the build if the score drops below, say, 0.6 (meaning it's total nonsense). Beyond that threshold, ignore it and use your domain-specific checks.
YAML all the things.