We implemented Consensus about eight months ago, aiming to surface the best-performing call snippets and automate coaching insights. The initial metrics looked great: "top moments" usage was up, and managers had a new dashboard.
Then we noticed the quality of deals wasn't improving. In fact, discovery calls felt more scripted and shallow. We dug into the data.
**The Problem:** Reps learned to trigger the "good call" detection algorithms by:
* Repeating key phrases from known high-scoring snippets ("value alignment," "pain point," "ROI") even if contextually forced.
* Artificially extending call length with filler conversation after hitting a detected "moment" to create more positive data points.
* Gaming the "questions asked" metric by rephrasing the same question multiple times.
The system was measuring proxies for good sales behavior, not the behavior itself. Once reps reverse-engineered the proxies, the data became useless.
**Technical Oversight:** Our setup was naive. We relied on default scoring and didn't isolate the variables tightly enough. For example, a "top moment" should be correlated with a subsequent stage advancement within X days, not just exist in a vacuum. Our configuration was basically:
```json
{
"detection_criteria": ["keyword_density", "speaking_pace", "question_count"],
"coaching_flags": ["low_question_score", "no_value_keywords"]
}
```
We needed to tie it to actual pipeline outcomes, which requires a custom integration with our CRM stage data. Consensus can do this, but it's not out-of-the-box.
**Takeaway:** If you're evaluating this tool, you must:
* Design your scoring metrics in tight feedback loops with closed-won/lost data from day one.
* Regularly audit "top moments" for substance, not just detection confidence.
* Involve reps in calibration to ensure the system measures genuine skill, not gamable signals.
The tool isn't broken, but it's dangerously easy to create a self-licking ice cream cone of metrics that please the tool but degrade actual sales craftsmanship.
Show me the query.
Classic. You optimized for what you could measure, and now you're measuring optimization. The technical oversight is believing a "top moment" exists independent of outcome.
You're right to look at stage advancement correlation, but that's still a lagging indicator they'll learn to game. The fix is adversarial: regularly change the scoring variables before patterns solidify. Treat your own scoring model as a secret you have to rotate, like a credential.
And audit the audit trail. If a rep triggers ten "great question" flags in one call, the system should flag *that* for review, not average it into a higher score.
Trust but verify – and audit
This reminds me of something I'm learning with Terraform monitoring. If you just alert on "apply failed," people start masking errors to make it pass. The proxy becomes the goal.
How do you even design a metric that can't be reverse-engineered? Is the only way to keep changing it, like user956 said? Sounds exhausting for the engineers building it.
Makes me nervous about adding too many "smart" metrics to my own projects.
Your point about the "technical oversight" of measuring in a vacuum resonates. It's the classic correlation vs causation trap in metric design.
In our infrastructure monitoring, we faced a similar issue where engineers optimized for "mean time to recovery" by implementing superficial fixes that triggered immediate resolution flags, but caused higher long term instability. The fix wasn't rotating metrics, but forcing a causal chain. For your case, that means a "top moment" flag shouldn't be a standalone event. Its validity should be contingent on a downstream, business-critical signal that the rep cannot directly manipulate, like a qualified opportunity created by a different team member, or a positive outcome in a later deal stage that this call influenced.
Have you considered adding a negative weighting layer? If your system detects phrase repetition, filler dialogue, or rephrased questions within a short window, it should degrade the score of the associated "positive" signals, not just average them. This makes gaming require more sophisticated, potentially self-defeating behavior.
—Alex