Skip to content
Notifications
Clear all

Check out what I made: a diff tool for CDP schema mappings

3 Posts
3 Users
0 Reactions
4 Views
(@auditor_abby)
Estimable Member
Joined: 4 months ago
Posts: 111
Topic starter   [#3136]

I built this because every CDP migration I've audited has had critical gaps in event data mapping. Teams assume a one-to-one field translation, then six months later their fraud detection or compliance reporting falls apart because timestamps are wrong or nested objects were flattened incorrectly.

The tool compares source and target schema definitions (Segment, mParticle, RudderStack, etc.) and highlights:
* Type mismatches (e.g., string in source, integer in destination)
* Missing required properties in the target
* Changes in enum value sets that could break downstream logic
* Implicit data truncation risks (like sending a 300-character string to a field with a 255-character limit)

It doesn't move data. It reads the spec and shows you where your mapping will fail an audit or lose fidelity.

Example output for a hypothetical `order_completed` event:
```
Source Property: `properties.total_amount` (type: number)
Target Mapping: `event.properties.revenue` (type: string)
- CRITICAL: Numeric monetary data converted to string. Aggregations (sum, avg) will break.
```

I use it during the planning phase of any migration. You need this level of validation before you start backfilling historical events, or you'll be dealing with data quality incidents forever.

The core is open-sourced. It requires a config file defining your source and target schemas. Let me know if you want a walkthrough of the validation rules—they're based on common compliance failures I see in log integrity checks.


Where is your SOC 2?


   
Quote
(@henryg)
Estimable Member
Joined: 1 week ago
Posts: 89
 

Interesting, but does it catch the real problems? Most of the audit failures I see aren't about type mismatches.

They're about what happens when you map a field that's optional in your source but is suddenly mandatory in the target because of a vendor-specific default. The tool flags the missing required property, sure. But it won't tell you that the default value the vendor inserts is some sentinel like "unknown" that then breaks your segment definitions.

You're just trading one set of assumptions for another.


Your vendor is not your friend.


   
ReplyQuote
(@henry)
Estimable Member
Joined: 1 week ago
Posts: 79
 

You're right, the hardcoded vendor default problem is a massive gotcha. A tool like this would need to map those defaults too, which are often buried in implementation docs and not the public spec.

But honestly, just *seeing* that mismatch flagged as "missing required property" is still a huge win. It forces the conversation with the vendor or your implementation team: "Hey, I see this field is mandatory on your side. What value gets auto-populated if I don't send it?"

Without the flag, that conversation often never happens until a dashboard breaks.


Cheers, Henry


   
ReplyQuote