Hey folks, hoping to get some collective wisdom here. I’m helping our marketing team troubleshoot our CDP's identity resolution, and we're seeing a huge number of merged customer profiles that just don't seem right. Our "identity graph" is lumping together what are clearly different people, which is throwing off our segment counts and activation.
We’re using a deterministic rule set based on email and user ID, with some fuzzy matching on IP addresses and device IDs for unknown visitors. The volume of merged profiles has spiked about 300% in the last month. 😬
Here’s a simplified version of the rule configuration we’re using:
```json
{
"identity_rules": [
{
"priority": 1,
"rule_type": "deterministic",
"attributes": ["email", "user_id"],
"action": "merge"
},
{
"priority": 2,
"rule_type": "fuzzy",
"attributes": ["ip_address", "device_id"],
"threshold": 0.8,
"action": "merge"
}
]
}
```
From my observability world, this feels like an overly sensitive correlation rule creating noise. I’ve been checking the raw events feeding the graph, and I'm wondering:
* Could shared or dynamic IPs (like from a corporate network or mobile carrier) be causing the fuzzy rule to over-match?
* Is there a common pitfall in how device IDs are being generated or persisted across different platforms (web vs. app)?
* Should we be adding more negative correlation rules to prevent merges when certain high-confidence fields (like last name) don't match?
I’d love to hear from anyone who has tuned these rules before. What metrics did you track to validate your merges? Did you have to dial back fuzzy matching significantly? Any dashboard screenshots of your merge rates over time would be super helpful!
Dashboards or it didn't happen.
Your fuzzy rule threshold at 0.8 is way too aggressive for IP and device ID. That's almost certainly the culprit.
Shared corporate or public IPs, plus mobile devices that share ad IDs, will hit that similarity score easily. It'll merge whole office buildings or families into single profiles.
Drop the fuzzy threshold to at least 0.95 and check your match keys. You likely have a data quality issue where placeholder or null device IDs are being scored as identical.
Prove it with a benchmark.
user1054's point about the fuzzy threshold is correct, but the 0.95 suggestion could still be problematic. The core issue is using fuzzy matching on inherently unstable identifiers for deterministic merging.
An IP address isn't a fuzzy identifier, it's a categorical one. Two users sharing a corporate NAT have identical IPs, not similar ones. Fuzzy logic on a full IP string gives you a false sense of precision. A better approach is to separate the logic: exact match on a hashed identifier, then apply a completely different rule for shared-network scenarios based on a confidence weight, not a similarity score. Device IDs should be handled the same way - either they match exactly or they're different.
This often points to a rule ordering problem. If your deterministic email/user_id rule runs first and creates a profile, then your fuzzy IP rule runs and merges another profile onto it, you've now contaminated your graph. You need to quarantine matches from low-fidelity signals, not merge them directly.
brianh