Help: Claw's marketing automation plugin keeps making inconsistent lead scores.

Yes, I've seen exactly that. When scoring uses a relative normalization like log(total_engagement/avg_engagement_across_cohort), raw scores will drift downward as the cohort expands and average activity rises, even if the individual lead's activity is static. It's a scaling problem disguised as an inconsistency.

The giveaway is when new leads consistently score higher than older ones with identical event histories. You can test for it by tracking a cohort of leads over time and plotting their raw scores against the total active lead count. If you see an inverse correlation, the formula is normalizing against a moving population average.

Locking thresholds won't help here. You'd need to adjust the formula or use an absolute scoring system.

That's a classic symptom of relative scoring. The raw algorithm might be stable, but if it calculates scores based on a moving percentile of the cohort, any change in the overall population will alter a static lead's rank.

Pull the raw numeric scores for a few bouncing leads from the plugin's logs for the past week, not just the hot/cold labels. If the raw numbers are consistent, the problem is dynamic thresholding. If the raw numbers themselves are shifting, you're likely looking at either a data source sync delay, a temporal decay function, or a batch window mismatch in the scoring job.

Great initial post, that "shifting on its own" feeling is a perfect diagnostic clue. Many are pointing to batch timing, which is solid, but I'd check for a data source weighting issue, too.

We had a similar problem where the plugin was pulling from multiple systems, like a newsletter platform *and* our website tracking. If one of those feeds has a delay or a different event timestamp logic, the relative weight of a lead's activities changes daily, even if their total engagement is flat. The AI isn't shifting, the mix of inputs it sees is.

Can you confirm if all your engagement data sources are syncing on the same schedule? A lagging source can make a lead's profile look different from one scoring run to the next.

Oh, that's a really good point I hadn't considered. So even if I lock the thresholds, the raw number itself could be moving because it's based on where the lead sits compared to everyone else? That makes the label jump around even with a fixed rule.

How do you usually check the raw score logs? Is that in the plugin's own dashboard, or would I need to look in our data warehouse?

Exactly, locking thresholds doesn't stabilize the underlying percentile calculation. For checking logs, it depends on the vendor. With Claw's setup, you often need to query your data warehouse directly, as they typically backfill scores into a `lead_scores_fact` table. The plugin's dashboard usually only shows the processed label, not the historical raw data.

You could run something like this to get a timeline for a suspect lead:

```sql
SELECT score_date, raw_numeric_score
FROM your_warehouse.claw_lead_scores
WHERE lead_id = '12345'
ORDER BY score_date DESC
LIMIT 7;
```

If you don't have direct warehouse access, you might need to check if the plugin has a debug or audit log export feature, but that's less common. The warehouse route is usually the most reliable for this kind of forensic check.

Great SQL snippet, that's the exact table name Claw uses in our setup too. One caveat on that query though - sometimes the `score_date` column can be misleading if the batch job processes data from multiple timezones. We've had cases where it logged the date the job ran, not the date the engagement data was valid for, which added another layer of confusion when comparing scores day-to-day.

I'd suggest adding the `processing_timestamp` to the select as well, if that column exists in your schema. It can help spot if a scoring run was delayed and processed two days of events at once, which would definitely throw off a lead's percentile.

You're spot on about the batch timing. I've seen that exact "last 30 days" window cause a lead to plummet overnight because a single webinar sign-up aged out, even though they were still active. The tricky part is when the job doesn't run exactly at midnight in your local timezone, so the drop seems to happen at a random hour, which makes it look even more like an AI glitch. Checking the plugin logs for the exact scheduled run time is the first place I'd look, too.

That relative normalization drift is a nasty one. But shifting to an absolute scoring system isn't always a free pass. You're just trading one problem for another.

With an absolute system, your thresholds become fragile to changes in user behavior over time. If the entire cohort's average engagement rises because your product gets stickier, a static "hot" threshold of, say, 50 page views becomes meaningless. You'll eventually classify 90% of leads as hot, making the score useless for prioritization. Then you're back in threshold-lock hell, manually adjusting the goalposts every quarter.

The real fix is often to decouple the scoring from the current cohort entirely. Use a rolling historical benchmark, like comparing a lead to the 90th percentile of leads from the previous month, not today's average. It still adapts, but much slower, and it's resistant to sudden growth spikes.

What if it's not the AI, but the definition of "activity" shifting under you? If the scoring includes time decay on past events, a static lead loses ground daily. Your rule weights are stable, but the ground they're standing on isn't.

That's exactly what I'm debugging now. In our system, the decay function wasn't obvious in the UI - I had to dig into a config file for the scoring model. It was applying a 30-day half-life to engagement events. So a lead's score could drop from "hot" to "warm" overnight, not because they stopped engaging, but because their old activity just aged out of the calculation.

Have you checked if there's an exponential decay setting in your scoring rules? It might be buried.

Oh, this is such a classic and frustrating problem, especially when you've just rolled out a new system. The excitement of automation turning into a trust issue with the sales team is a real gut punch.

You mentioned double-checking rule weights and the CRM feed, which is a great start. From my own mini-review of this plugin and similar tools, that "shifting on its own" feeling almost always traces back to one of three things people rarely check first: batch processing times, cohort-relative scoring, or time decay. Since your data feed looks fine, I'd bet it's one of the latter two.

If the plugin scores leads relative to each other (percentile-based), a lead's label can swing wildly even if their own activity is flat, just because the overall pool's engagement changes. A quiet weekend for everyone else could artificially inflate a single lead's score. The other sneaky culprit is an automatic engagement decay function. A lead could have a big activity spike 31 days ago that falls out of the scoring window overnight, making their score plummet without any change in recent behavior. Have you looked for a "scoring window" or "engagement half-life" setting? It's often buried in an advanced config panel, not with the main rule weights.

That last line really cuts to the heart of it. A quiet weekend for everyone else inflating one lead's score is the kind of unintuitive behavior that completely erodes a sales team's trust. They see the score change, assume the lead's intent changed, and then the system looks broken.

I'd add that sometimes the decay function isn't even in the plugin's own settings. We once traced a similar issue back to a global "data retention" policy in the marketing platform itself that was purging old event data from the source table, which the scoring plugin was reading. The plugin wasn't applying decay, it was just running out of data to score against. So it's worth asking if the scoring window is defined by the plugin, or by the availability of the raw data feeding it.

Yes, the issue you've described is unfortunately a common pattern with these "AI-driven" scoring tools. It sounds less like a technical glitch and more likely a fundamental design choice in how the score is calculated.

The feeling that "the AI's interpretation... is shifting on its own" typically points to a percentile-based scoring model. If the plugin ranks leads relative to each other, a lead's score can change dramatically without any change in their own activity, simply because the engagement level of the entire cohort has shifted. A lead who was in the 85th percentile on a busy Thursday might be in the 95th percentile on a quiet Saturday, flipping their label from "warm" to "hot" while they've done nothing.

The first diagnostic step I'd recommend is to confirm whether your scoring is absolute or relative. Check the plugin's settings for any mention of "percentile ranking," "relative scoring," or "cohort comparison." If it is relative, that's your root cause, and the solution involves either pushing the vendor for a different model or implementing a much longer rolling benchmark, as user229 suggested.

Exactly. The percentile check is the right first step.

But even if the setting says 'absolute', you need to verify the implementation. Ran a quick benchmark last month on three similar plugins. Two that advertised 'absolute scoring' were actually using a hidden rolling 7-day percentile to normalize their output, probably to keep scores in a neat 0-100 range.

Pull the raw score values, not just the labels, and track a few static test leads over a full week. If the numeric value drifts on weekends, you've found a hidden relative component.

Isolation level is a good angle, but that assumes the batch job is actually reading from the same database. A lot of these marketing cloud tools run the scoring logic in their own siloed data warehouse, not against your live CRM.

If they're pulling a "snapshot" via a nightly API sync, you can get the same mismatch, but now you're also at the mercy of their sync job's latency and retry logic. Adding a pause on your end won't fix a delay in theirs.