I'm planning for a migration where cross-device tracking will be a big challenge. With all the privacy changes, deterministic matching seems less reliable.
I've been reading about probabilistic models in platforms like AppsFlyer and Adjust. But how do you actually measure their accuracy in a real-world scenario? Are there any benchmarks or case studies? I'm particularly worried about overcounting users during our campaign analysis.
learning every day
Measuring accuracy in a real-world scenario is inherently tricky because you lack a ground truth. You can't know for certain if a probabilistic match is correct, so you're forced into indirect methods. I've relied on a few validation techniques.
One is cohort consistency analysis. You take a known group, perhaps users who authenticated on two devices, and see what percentage the probabilistic model correctly lumps together. Another is looking for implausible chains, like a single user "using" a phone in London and a tablet in Tokyo within five minutes. High rates of those indicate an overly permissive model.
Overcounting is a very real risk, and it often stems from over-reliance on IP address clustering post-ITP. Benchmarks are scarce as the models are proprietary black boxes, but you can pressure-test by running a parallel analysis: compare campaign attribution totals from your probabilistic provider against a separate, strict deterministic sample (like logged-in users only). A persistent, large divergence, especially in uplift reports, is a red flag.
Great point about using a deterministic sample for parallel analysis. I've seen teams take that a step further by using it as a contractual KPI during vendor negotiations. You can agree on a tolerance threshold for that divergence, say 15%, for your core campaigns.
The >implausible chains1000 km/h) more useful than a fixed time window.
Pressure testing is the only way, really, since as you said, these are black boxes. Have you ever gotten pushback from a vendor when presenting them with the results of a parallel analysis? Some are surprisingly open to tuning, others get defensive.
Ask me about my RFP template
Pushback is real. I once had a vendor claim our deterministic sample wasn't statistically significant enough, which felt like deflecting. The ones who are open to tuning are gold, though.
We got better results when we stopped asking "is your model accurate?" and started asking "can you adjust the confidence threshold for our specific use case?" Framing it as a configuration need, not a fault-finding mission, changed the conversation. Some platforms let you dial that setting up or down per campaign, which is clutch for high-stakes launches.
The contractual KPI idea is smart. We've used the divergence tolerance, but linked it to a credit on spend if it's exceeded. That gets their attention.