Our organization is entering its first CRM migration, and I find the vendor evaluation process lacking in concrete, testable criteria. The discourse often centers on abstract "ease of use" or "seamless transition" claims, which are not falsifiable. My background in model evaluation suggests we need to define clear metrics and failure modes upfront.
I propose we need to interrogate potential vendors on the following specific aspects:
**Data Mapping & Integrity**
* What is your process for schema analysis and discrepancy reporting? Do you provide a diff of source and target field mappings before any migration?
* For custom fields or objects with no direct equivalent, what is the protocol? Is it a simple lift-and-shift, or do you consult on data model optimization?
* What is the validation method post-migration? Do you provide a record-level sample audit, or merely a row-count comparison?
**Technical Execution & Contingency**
* What is the estimated downtime, and how was it calculated (e.g., based on our record volume and API limits)? What is the rollback procedure if stage N fails?
* Please detail the migration's order of operations. Are dependencies (like user records before accounts) handled automatically?
* What are the most common points of failure you encounter, and what mitigations do you have in place for each?
**Post-Migration Support & Ethics**
* Is post-migration support tied to a new, separate contract, or is it included in the migration SOW? What is the defined scope and duration?
* How is our data handled during the process? Please specify data residency, encryption in transit/at rest, and personnel access controls. Will all data be purged from your systems after a verified successful migration?
I am particularly wary of vendors who cannot provide a clear, statistical basis for their downtime estimates or who treat data validation as an afterthought. What other measurable, operational questions should we be adding to this list?
prove it with data
I completely agree with shifting the focus to falsifiable criteria. Your questions on technical execution are spot on. Let me add a dimension you might consider: observability during the migration itself.
For the downtime estimate and rollback procedure, ask them what telemetry and real time progress tracking they provide. Can you see the migration rate, error rates per object type, and API consumption as it happens, or do you just get a "started" and "finished" notification? A good vendor should give you a dashboard, akin to a deployment progress in Kubernetes, where you can see if stage N is starting to drift from its expected performance profile before it fails outright.
Also, on the order of operations, push them to explain *why* that specific order. There might be valid reasons, but their answer reveals if they're just running a script or if they understand data dependencies in your specific CRM model. The rollback plan is only as good as its granularity - can they roll back a single failed object type, or is it all-or-nothing?
Prod is the only environment that matters.
Absolutely on point about the dashboard. I've been sold that particular slice of magic before. The vendor demo showed a beautiful Grafana-like panel with green checkmarks, and on cutover night it was a static HTML page that refreshed a log file. The key question is whether the telemetry is monitoring the *actual* API calls being made, or just the status of their own worker processes. If it's the latter, you won't see throttling until their entire queue grinds to a halt.
The rollback granularity is the real kicker. An all-or-nothing revert over a weekend is a disaster scenario. Ask if they can, for example, revert a botched Accounts migration while preserving the successfully migrated Contacts that have foreign key dependencies. If they can't articulate how they'd handle that, their rollback plan is a prayer, not a procedure.
Also, probe what they consider an "error." Is a malformed date that gets nulled an error, or a successful migration with data loss? Your dashboard should differentiate between critical failures and data transformation warnings.
APIs are not magic.
Your focus on validation method is critical. Row-count comparisons are dangerously insufficient for integrity. You need to ask about their method for checksum validation at the field level, not just the record count. A proper sample audit should verify that a hash of critical field values (e.g., account name, last modified date, a key custom field) for a statistically significant sample matches between source and target. Without that, you're only confirming existence, not correctness.
I'd also stress the importance of defining "success" for custom field mapping upfront. The protocol should produce an artifact, like a decision matrix, stating whether each custom field is being migrated, transformed, or deprecated, with a business reason for each. This prevents scope creep during the migration and turns a consultative process into a verifiable deliverable.