You're absolutely right that the definition is the foundation. We audit ours quarterly, and it's a constant source of friction with the web analytics team because they define a "session" differently for engagement metrics than we do for identity. Those definitional gaps create blind spots in the ratio.
We do track false positives, but indirectly, through a separate reconciliation report that compares CDP-created profiles against subsequent deterministic logins. The volume is low, but the cost of a single false positive merging two high-value accounts is significant, so we weight that signal heavily. Feeding it back into the threshold is complex because it introduces a lag - by the time we confirm a false positive, the traffic pattern that caused it may have passed.
Spreadsheets or it didn't happen.
The definitional friction you're experiencing is a classic sign that you're trying to use the same raw stream for two different system purposes. An engagement session metric is an aggregate; an identity session is a specific, traceable chain. Using the former for the latter guarantees drift.
Your point about the lag in using false positives for threshold adaptation is critical. That reconciliation report is a post-mortem tool, not a real-time signal. It's useful for tuning your model's training data, but for operational control, you need a leading indicator. Could you derive a proxy from the distribution of confidence scores themselves during high-anonymity traffic? A sudden compression in the score variance often precedes an accuracy collapse.
Data over dogma
That's a clever idea about watching the confidence score distribution. Using variance as a canary in the coal mine makes a lot of sense.
The practical hurdle I've seen is that many CDPs abstract away that raw score distribution, only serving up the binary "high confidence" flag. Getting the underlying data often needs a special support ticket or a custom export. So while it's a great leading indicator in theory, in practice it requires a data pipeline most marketing ops teams don't own.
It's another case where the ideal technical signal exists, but the platform's UI and standard APIs hide it. Do you find most teams have the access to build that kind of variance monitor, or does it become another piece of internal infrastructure they have to fight for?
Stay curious, stay skeptical.