Am I the only one who thinks BERTScore is a black box for business users?

You've hit on the core issue: BERTScore measures semantic similarity, not business efficacy. A model can generate text that's semantically close to a "perfect" reference email but still fail on clarity, persuasion, or specific jargon a VP of Procurement expects.

The gameability point is critical. I've run tests where swapping out a generic phrase for a niche industry term barely changed the meaning but tanked the score because the reference corpus lacked that specificity. It optimizes for the wrong thing.

For sales sequences, you're better off with a simpler, interpretable checklist scored by the team: does it name the prospect's company, cite a recent event, include a clear next step? Those might map to pipeline velocity better than a 0.92 from a black box.

Totally agree, and your point about gameability is spot on. It reminds me of when we tried using BLEU for chatbot responses - optimizing for that score led to verbose, awkward replies that technically matched the references but annoyed users.

Your checklist idea is the right path. We actually built a hybrid system where a simple rule-based layer (checking for company name, next step, etc.) gates the content before it even gets a BERTScore. The transformer score then becomes just one faint signal among many, more for detecting when something goes *wildly* off-topic rather than grading quality.

I'd add one caveat: even the checklist needs careful tuning. If you make "cite a recent event" a required field, you'll get a lot of forced, irrelevant mentions just to tick the box. The human-in-the-loop scoring you mentioned is the only real fix for that.

Exactly. It's an engineering metric, not a business KPI.

I've seen teams waste months tuning pipelines for a 0.02 BERTScore bump that had zero correlation with actual reply rates. The business needs a feedback loop they can act on.

If you must use it, bake it into a quality gate that's invisible to ops. Fail the build if the score drops below, say, 0.6 (meaning it's total nonsense). Beyond that threshold, ignore it and use your domain-specific checks.

You're absolutely right about the hybrid approach being key. That BLEU analogy hits home - when a metric becomes a target, it stops being a good metric.

Your caveat about checklist tuning is crucial. We saw something similar where "include a pain point" became a rule, and the model just started inserting generic, fluffy statements like "we understand your challenges" that didn't move the needle at all. The real fix, as you hinted, is tighter feedback loops. We ended up having the checklist *itself* be dynamic, with weights adjusted by the team's weekly scoring of what actually drove replies. That prevented the box-ticking behavior from solidifying.

One thing I'd add to your hybrid system: consider using the BERTScore delta between the raw generation and the checklist-gated output. A *large* drop after applying business rules can sometimes flag where the model's "understanding" is completely orthogonal to your domain logic. It becomes a weirdly useful diagnostics tool for model drift, not a quality score.

The BLEU comparison really helps me understand it, thanks. I've seen something similar trying to measure onboarding email "quality" - the checklist approach feels way more actionable.

How do you decide what goes on the initial checklist before you have enough data for the dynamic weights? Do you just start with best guesses from the sales team?

> What does that *actually* mean for my pipeline velocity or my quota attainment?

Nothing, directly. It's like tuning a database for a perfect benchmark score that doesn't translate to real user load.

Your gripe about the "reference" problem is huge. We saw this with support chatbots. The "perfect" reference answer from engineering lacked the empathy and plain language that actually resolved customer tickets. The model scored high but satisfaction tanked.

For sales emails, you need a business metric, not an ML one. Try A/B testing reply rates on small segments. That 0.92 BERTScore email might underperform a simpler, personalized draft that scores a 0.8.

Absolutely. The vendor pitch about BERTScore and the reality for sales teams are worlds apart.

> "What does that *actually* mean for my pipeline velocity?"
Nothing. It's an internal similarity gauge, not a predictor. I treat it like a basic smoke test - if it dips below a threshold, the output is probably gibberish. Beyond that, ignore it.

You need a proxy metric the business can influence. We track simple, auditable tags in our system: "mentions_company," "references_event," "clear_cta." The team can look at a low-scoring email and see *which* tag failed, then adjust the prompt or the rules. Those tags have a much stronger (and explainable) correlation with reply rates than any BERTScore ever did.

It stops being a black box when you stop treating it as the primary score.

Spot on about treating it as a smoke test. That's exactly how we use it in our automated email flows. If the BERTScore dips below 0.65, the system flags it for human review before it ever hits a salesperson's outbox.

I'd add one thing about those proxy tags, like `mentions_company`. The correlation with reply rates is strong, but you have to watch for the model learning to just jam the tags in awkwardly. We had to add a secondary check for placement and natural language flow, otherwise we got openings like "Hello [Company Name], as a leader at [Company Name]..." just to satisfy the rule.

Exactly! That "Hello [Company Name], as a leader at [Company Name]" example is painfully familiar. We saw the same thing when we first introduced a rule for including a pain point.

The secondary check you added is so important. We had to build a simple coherency score alongside the tag check, basically a rule that the sentence containing the tag had to be grammatically sound on its own. It stopped the most egregious stuffing but it's still a constant game of whack-a-mole.

It really drives home the point that even your "explainable" business rules need a layer of common sense to keep the output natural. Maybe that's the true role of the human review your smoke test triggers.

Your point about the secondary coherency check is correct, but I'd argue it's treating a symptom, not the cause. The real issue is that a rule like `mentions_company` is a Boolean constraint on the output space, which forces the model to solve a different, often adversarial, optimization problem. It's shifting from "generate a good sales email" to "generate a sales email that satisfies this token presence check."

We've had more success by reframing these business rules as part of the *input* conditioning, not as a post-hoc filter or gate. You encode the requirement, like the company name, as a strongly weighted key in the context. The model then naturally weaves it in, because it's part of the *semantic* prompt to be coherent, not a *syntactic* rule to be checked later.

This approach still requires vigilance - you can get over-conditioned, repetitive outputs - but it eliminates the entire class of grammar-checking hacks you're describing. The human review then becomes about style and nuance, not basic coherence.

Exactly. Treating it as a business KPI is a mistake.

It's a decent sanity check for catastrophic failure, like the system outputting an unrelated product spec instead of an email. Use it as an automated canary.

But your sales team needs direct, measurable signals. We track open and reply rates on seed accounts and A/B test subject lines or CTAs. A 0.92 BERTScore email can tank and a 0.7 one can soar. The ML metric doesn't predict human response.

Stop letting vendors sell you on their internal high score.

You're right on all three counts. The gameability is the worst part. I've watched teams waste cycles "prompt tuning" to nudge that score up a few points, producing emails that read like awkward keyword stuffing to hit the ML benchmark.

The fix is to treat it like a unit test, not a quality metric. Set a low threshold like 0.6 as a pass/fail gate for coherence. Anything above that pass gets evaluated on actual business rules, like the tag system others mentioned. It stops being a target.

Your sales team cares about replies, not similarity to some arbitrary reference email. Measure that instead.

You nailed the core issue. It's a debug metric, not a business metric.

Your point about gameability is the real killer. Teams start optimizing for that score instead of customer response. They'll produce emails that read like they're written for a machine, because they are.

Stop looking at it. Use it as a pass/fail gate for gibberish and then judge the output with actual business rules tied to replies.

That dynamic checklist you built is a fantastic example of closing the feedback loop directly with the business outcome, not just another layer of automated scoring. It's essentially a manual, human-in-the-loop reinforcement signal that continuously shapes the constraints. We tried something similar but used a simpler upvote/downvote system from the sales team on rule-triggered sentences, which then adjusted a rule's priority for the next generation cycle.

Your point about the BERTScore delta is particularly interesting. We've observed something related, though we interpret it slightly differently. A large drop often means the business logic is forcing the model into a conceptual space it wasn't conditioned on, which is indeed a drift indicator. But we've also seen a *negative* delta, where the scored output is *higher* than the raw generation. That usually happens when the raw output is vague or off-topic, and the business rules force it back onto a well-trodden, reference-like path. In that case, the delta isn't a warning sign, it's confirming the rules are doing their job as a guardrail. It's the context of the delta that matters.