Yes, I've seen exactly that. When scoring uses a relative normalization like log(total_engagement/avg_engagement_across_cohort), raw scores will drift downward as the cohort expands and average activity rises, even if the individual lead's activity is static. It's a scaling problem disguised as an inconsistency.
The giveaway is when new leads consistently score higher than older ones with identical event histories. You can test for it by tracking a cohort of leads over time and plotting their raw scores against the total active lead count. If you see an inverse correlation, the formula is normalizing against a moving population average.
Locking thresholds won't help here. You'd need to adjust the formula or use an absolute scoring system.
benchmark or bust
That's a classic symptom of relative scoring. The raw algorithm might be stable, but if it calculates scores based on a moving percentile of the cohort, any change in the overall population will alter a static lead's rank.
Pull the raw numeric scores for a few bouncing leads from the plugin's logs for the past week, not just the hot/cold labels. If the raw numbers are consistent, the problem is dynamic thresholding. If the raw numbers themselves are shifting, you're likely looking at either a data source sync delay, a temporal decay function, or a batch window mismatch in the scoring job.
Great initial post, that "shifting on its own" feeling is a perfect diagnostic clue. Many are pointing to batch timing, which is solid, but I'd check for a data source weighting issue, too.
We had a similar problem where the plugin was pulling from multiple systems, like a newsletter platform *and* our website tracking. If one of those feeds has a delay or a different event timestamp logic, the relative weight of a lead's activities changes daily, even if their total engagement is flat. The AI isn't shifting, the mix of inputs it sees is.
Can you confirm if all your engagement data sources are syncing on the same schedule? A lagging source can make a lead's profile look different from one scoring run to the next.
Data > opinions
Oh, that's a really good point I hadn't considered. So even if I lock the thresholds, the raw number itself could be moving because it's based on where the lead sits compared to everyone else? That makes the label jump around even with a fixed rule.
How do you usually check the raw score logs? Is that in the plugin's own dashboard, or would I need to look in our data warehouse?