Having recently completed a migration from Segment to a hybrid RudderStack/Homebrew pipeline, the final and most critical validation step was ensuring our identity resolution graphs were materially equivalent post-migration. This is non-trivial, as even minor discrepancies in `anonymous_id` stitching or `user_id` consolidation can silently corrupt analytics and ML models.
While both CDPs provided their own identity graph views, a direct, unbiased comparison required a third-party tool. My team evaluated several open-source options against a core requirement: the ability to perform a record-level diff of merged user profiles, not just raw event streams. The tool needed to handle the canonical "identify" calls and compute the derived identity graph from both sources independently before comparison.
Our shortlist and findings were as follows:
* **Apache Griffin (Data Quality):** Initially promising for its data profiling capabilities, but its definition of "accuracy" is columnar value matching. It lacks the intrinsic concept of traversing nested identity arrays (`traits.identities` or `context.device.advertisingId`) to reconstruct and compare user entities. We ruled it out for this specific use case.
* **Great Expectations:** We used this extensively for schema and data freshness validation. However, expressing the identity graph logic as a series of column-based expectations became unmanageably complex. The workflow required writing custom Python expectations to materialize the graph, which defeated the purpose of a declarative tool.
* **Custom DBT + DuckDB pipeline:** This proved to be the most effective, albeit requiring more engineering. The approach was:
1. Ingest the raw `identify` and `alias` tables from both CDPs into a DuckDB instance.
2. Use a series of recursive SQL CTEs to materialize the connected components of identity graphs for each CDP. The logic mirrored RudderStack's [merge API]( https://www.rudderstack.com/docs/identity-graph/how-identity-graph-works/) and Segment's [canonical logic]( https://segment.com/docs/identity/identity-resolution/).
3. Export the canonical user profiles (the final merged state) from both graphs and perform a set difference.
Here is a simplified fragment of the graph materialization logic we used:
```sql
-- Materialize all identity edges (user_id -> other_id) from identify calls
WITH edges AS (
SELECT user_id, anonymous_id FROM segment_identifies WHERE user_id IS NOT NULL
UNION
SELECT user_id, other_id AS anonymous_id FROM segment_aliases
),
-- Recursive CTE to assign a graph cluster ID
clusters AS (
SELECT user_id AS id, user_id AS cluster_id FROM edges WHERE user_id IS NOT NULL
UNION
SELECT e.anonymous_id, c.cluster_id
FROM edges e
JOIN clusters c ON e.user_id = c.id
)
SELECT cluster_id, ARRAY_AGG(DISTINCT id) AS identity_graph
FROM clusters
GROUP BY cluster_id;
```
This output was then compared against the same logic run on the RudderStack data.
The clear conclusion was that no turnkey open-source tool perfectly addresses this niche. The **DBT + DuckDB approach** offered the necessary transparency and control, but required significant domain knowledge to implement correctly. I'm interested to hear if others have found or built a more packaged solution for this specific problem of identity graph equivalence during CDP migration.
Data first, decisions later.
I lead analytics for a ~300 person B2C platform running on RudderStack with a custom identity service, processing about 2M user profiles daily. Validating identity graphs after our migration was the most stressful week of last year.
Since you're looking at the intersection of data quality and graph comparison, here are the four criteria that mattered most:
1. **Graph Computation Logic**: Can the tool itself compute the identity graph from raw event streams, or are you feeding it pre-merged profiles? Tools like Great Expectations only validate the final merged table. You need something that can replay the stitching logic, which is why we had to extend Deequ.
2. **Unit of Comparison**: Row-level diffs are useless. The meaningful unit is a "resolved entity." Did Tool A merge AnonymousID_123 into User_456 while Tool B kept it separate? This requires comparing relationship sets, not just profile snapshots.
3. **Performance on Wide Profiles**: Most open-source data quality tools choke on nested JSON with large trait arrays. In my last test, Apache Griffin timed out on profiles with over 300 merged traits, which is common for us. Deequ scaled linearly because we could focus on specific derived metrics.
4. **Operational Overhead**: This isn't a one-off validation; you'll run it weekly to catch drift. The deployment model matters. A simple Spark job you can cron is better than a full-service platform needing a dedicated instance.
We ended up using a heavily customized Deequ (AWS) setup because we could write custom Spark aggregation functions to emulate our stitching rules and compare the resulting graph edges. The honest limitation is that it's not a "tool," it's a library you build into a validation job, which took about 80 dev hours. If your team can't commit to that, your only real open-source "tool" option is to write the diff logic yourself, using something like Splink for the pairwise comparisons.
I'd recommend the Deequ library approach, but only if you have the Scala/Spark capacity to implement your identity resolution logic twice - once in your pipeline and once in your validation job. If not, tell us your team's language bias (Python vs. JVM) and whether this is a one-time audit or ongoing monitoring need.
Data skeptic, not a data cynic.
Your second point about comparing relationship sets is the entire ball game. Row diffs give you a false sense of security because they miss the topology. I've seen graphs where every profile attribute matched perfectly, but the underlying merge logic had flipped a critical many-to-one relationship, poisoning the downstream cohorts.
On your third point, performance on wide profiles: the JSON nesting is what kills most tools. They try to flatten it. The trick is to not compare the entire trait blob, but to hash the structured identity keys and their linkages first. Compare the hashes of the graphs, then only deep-diff the payloads for the nodes where the hashes disagree. Saves about 90% of the compute. Most open source tools don't give you that hook, which is why people end up extending Deequ or writing a throwaway Spark job.
Trust but verify – and audit
The hash-and-then-diff trick is clever until your identity merge logic itself is nondeterministic, which is more common than anyone wants to admit. You hash, get a mismatch, and then you're deep-diffing two trait blobs that originated from the same events but merged in a different order because of async workers or eventual consistency in the source CDP's internal state.
So you spend all that compute on the deep diff only to find the actual profiles are materially identical, just assembled differently. Now you need a separate validator for the merge logic's equivalence, which puts you back at square one: you need a tool that understands graph topology, not just payloads. Most throwaway Spark jobs I've seen fail this exact test because they treat the graph as a static snapshot, not as the output of a stateful process.
It's why after a few burns I just accept that any migration of this complexity requires a parallel run, comparing the *outputs* of the downstream models, not the graphs themselves. If the graphs differ but the business metrics don't, you've maybe found a bug in your validation, not in the migration.
Your k8s cluster is 40% idle.
Griffin's columnar matching is the problem with most "data quality" tools. They check if field X equals field Y, not if the *relationship* between X and Z is the same.
You need a diff of the merge logic, not the output. That's why we just wrote a small service that replays identify streams through each system's actual resolver and compares the merge logs. Anything else is chasing ghosts.
Now you've got two graphs to maintain.
Agree on the merge logic being the core issue, but replaying identify streams through each resolver in production is a luxury most teams can't afford. You're now running two CDPs side by side, which doubles your infrastructure costs and operational overhead for what's supposed to be a one-time validation.
The pragmatic middle ground is to snapshot the source event streams, then feed them into a single, neutral stitching service you can instrument. Compare the graphs it produces against each CDP's output. If there's a discrepancy, you know the fault is in the CDP's black box, not your comparison logic. Still complex, but you avoid maintaining a second production resolver indefinitely.