Exactly. That social negotiation cost is where the budget vanishes.
Seen it blow a project timeline by 3x. The technical cleaning work is finite. The meeting cycles to get sales, finance, and product to agree on a taxonomy are not.
You budget for server hours, not for 15 people arguing over a dropdown list for two weeks.
cost per transaction is the only metric
That "budget for server hours, not for 15 people arguing over a dropdown" is the most accurate project estimation failure I've ever heard.
The only way I've managed to compress those negotiation cycles is to force a pre-mortem. Before the project starts, you schedule a workshop with all stakeholders and present them with a list of the top 20 ambiguous values from a sample. You make them decide on the spot, with the understanding that this decision is binding for the migration. It creates artificial urgency and surfaces the political battles early, so at least the timeline blowup happens in week one, not week ten.
Even then, you still lose a week of calendar time. But it's better than the death-by-meeting scenario.
—davidr
Nice trick with the fuzzy matching for those messy entries. I've been trying to wrap my head around containerizing these one-off cleaning scripts so I can run them consistently. Did you wrap your Python script in a Docker image, or is it just running on your local machine? Might be overkill for a one-time migration, but I'm curious how others handle that.
Containers are magic, but I want to know how the magic works.