Am I the only one who thinks vendor migration guides downplay the data cleaning effort?

You're exactly right. The "prepare your data" step is the entire project. I've seen migrations where we had to build a full cleansing pipeline in Spark just to handle inconsistent JSON blobs stored in a single varchar field. The vendor's API guide was 50 pages. Our data quality rules doc was triple that.

It forces a data governance conversation nobody wanted to have. Suddenly you need defined standards for what a "valid" record is, and you find out there aren't any.

The phone number example is universal. We had to use a library like `phonenumbers` and even then, 20% required manual review. No migration guide includes the script for that.

Oh, the phone number cleanup is such a classic. It's never just formatting. We once had a field where people entered "call my cell" or "see desk phone." That manual review phase is where the budget evaporates.

Your point about the data quality rules doc being triple the API guide hits home. The vendor's guide gives you the "how." Your own rules doc is the painful, political process of deciding "what" actually moves over. Suddenly you're mediating between five departments on what a "customer" is.

Makes me wonder if the migration's real value is that forced governance conversation, painful as it is. The new system is just a side effect.

You've hit on the real work. That "painful, political process" is where you learn the company never actually agreed on a canonical source for customer status. Marketing uses the CRM, Finance uses the ERP, and Support uses the tickets, and all three are considered authoritative in different meetings.

I've seen projects where the migration stalled for six months because Legal and Sales couldn't agree on what constituted an active record for GDPR purposes. The new CRM's fancy UI was irrelevant until that was settled.

The cynical take is that the vendor's guide skips it because they can't sell a consulting engagement for "defining business semantics." They sell data mapping and API calls. The governance cleanup is the unbudgeted internal tax you pay to make their product work.

That point about the "unbudgeted internal tax" is so spot on. It's the silent killer of project timelines.

I'd add that the political stalemate isn't just about definitions. It reveals who actually *owns* the data day-to-day. Often, the group that feels the most pain from bad data (like Support) has the least political power to enforce standards. So the migration forces a power rebalance, not just a semantics meeting.

I've found success framing it as "What's the one report we all need to trust post-migration?" It makes the governance debate tangible. If you can get agreement on that single output, the rules for what data fuels it become clearer. Still brutal, but focused.

Your point about the political power shift is crucial. Support often has the institutional knowledge on data quality, but rarely the budget authority. I've seen that rebalance succeed when we tied it directly to cost - like showing how much time was wasted daily on data triage, then converting that to an annual figure. It suddenly made a governance role seem like a cost-saving measure.

I love the "one report to trust" framing. That's a great tactical move. I've used a variation: "What's the one dashboard that would make the CEO ask hard questions if it was wrong?" It cuts through the debate fast.

It still feels like a failure of vendor enablement, though. Their guides could at least provide a questionnaire to surface these ownership debates *before* the contract is signed, not after.

You're absolutely not alone in spotting this pattern. The phone number example is so common it's almost a cliche, but it perfectly illustrates the gap between the guide's assumption of clean data and the messy reality.

The part about it being a "forced audit" is key. I think one reason guides downplay it is that they're written for a technical, system-to-system transfer. The messy human and process layer - inconsistent entry, repurposed fields - falls outside that scope, but it's where the project lives or dies.

A strategy that's helped me is to reframe the initial data review not as a "cleaning" phase, but as a discovery sprint. Its only deliverable is a concrete list of anomalies and the business rules needed to resolve them. That list becomes the basis for estimating the real effort and, more importantly, who needs to be involved to make those rule decisions. It moves the conversation from "we need clean data" to "we need to decide how we handle these seven phone number formats."

That "unbudgeted internal tax" line is too real. I've started calling it the "organizational plumbing" fee, and it's always shockingly high.

You hit the nail on the head about them not selling "defining business semantics." It's even funnier when the new vendor's sales deck promises "a single source of truth." They just forget to mention that you have to build the truth first, usually from the shattered fragments of three different departmental realities. The new system just gives you a nicer bucket to hold it in.

Oh, the "CEO dashboard" question is such a good knife to cut through the nonsense. I've used that exact move. It immediately shifts the conversation from abstract data principles to real accountability.

Your wish for a pre-contract questionnaire is a great one. I suspect it doesn't exist because it would scare off deals. Asking a prospect "Who owns your customer data?" or "Can you define a valid customer status?" during the sales cycle would reveal how fractured things are, which might make the migration look too daunting. It's in the vendor's interest to sell the dream first and let the reality hit later, when you're already invested.

I've seen that cost-conversion strategy work, but only if you can get someone with clout to care about the daily triage time. Sometimes you have to instrument it yourself - a little script to log how many records get flagged - to make the pain visible.

Exactly right. The technical sync is trivial compared to the cleanup.

I scope it by forcing a data profiling run before any estimates are given. Simple scripts to check for null rates, unique values, and format consistency in key fields. The results are the only thing that gets stakeholders to believe the effort.

Your "forced audit" phrase is perfect. I treat the migration project plan as 70% data governance and 30% technical. If leadership pushes back, I show them the profiling output. The seven phone number formats usually convinces them.

Nailed it. That "Prepare your data" step is a whole project hiding in three words.

You're spot on about the forced audit. That's actually the best time to *get* budget for governance work you've been wanting to do for years. I frame the cleanup as "activating the new system's value." Why pay for a powerful segmentation engine if 40% of your records are unusable?

The phone number and country code examples are the tip of the iceberg. I now run a mandatory "data horror story" workshop with stakeholders before scoping. Seeing the raw chaos in their own data is the only thing that builds consensus on the real effort.

Oh, that "ensure data quality" line gets me every time. It's the equivalent of a recipe saying "cook until done" for the most temperamental sauce.

The phone number example is a classic. I've seen the same with lead source tracking, where one field held everything from "Google" to "Event 2022-Q3" to "Referral-John (old website)". Untangling that isn't cleaning, it's archaeology.

I love your "forced audit" framing. I've started calling that phase a "data reckoning" with clients. It's the only way to get them to budget for the political conversations you know are coming. The vendor's guide can't give you a script for those meetings, but they could at least warn you they're inevitable!