What's the best way to export contacts from one CRM to another?

I've handled these migrations for B2B SaaS companies in the 200-500 employee range using our internal Airflow + dbt stack. We run all our own pipelines to BigQuery and have moved data between Salesforce, HubSpot, and Netsuite for multiple sales teams.

1. **Fit and target audience**
Enterprise teams are forced into Mulesoft/Celigo because of existing contracts and "safe" vendor choices, paying $60k+ annually. For SMB and mid-market doing a one-time move, a dedicated migration tool like Zapier's Interfaces or Portable is actually sane. They're built for this exact job.

2. **Real pricing and hidden costs**
Vendor tools (like Fivetran/Hightouch) bill on monthly active rows, which kills you for a full historical dump. You'll see a $500 monthly connector fee plus a one-time spike of $2-3k for the initial load. Building it yourself with Python and the CRM APIs costs engineering hours but $0 in software. My last build took about 40 dev hours.

3. **Where the DIY approach breaks**
The biggest failure point is handling rate limits and backoff/retry logic. The Salesforce Bulk API will throttle you hard after 10k records if you don't batch correctly. I've had pipelines timeout after 6 hours because I didn't segment large jobs into 5k-record chunks.

4. **Where vendor tools clearly win**
They manage schema drift and API version updates. When HubSpot changed their custom contact property format last year, our vendor tool updated over a weekend. Our manual script broke and required a day to fix. For ongoing syncs, this maintenance burden is real.

I'd use Portable for a one-off, clean migration where the client has a defined budget and needs it done next week. For a company building a long-term, bidirectional sync as part of their product, I'd build a custom pipeline with proper idempotency and state tracking. Tell us if this is a single project or an ongoing need, and the approximate record volume.

This phased, validation-heavy approach you outlined is exactly what saved us in a recent migration. Could you say more about the dry run step? I'm always concerned about how a sandbox subset translates to the full dataset, especially with custom objects or unique ID dependencies.

Also, between using the source CRM's native API and a dedicated migration tool like Portable, which would you recommend for a team with moderate technical resources but no in-house data engineer? I'm weighing the reliability of the API against the speed of a pre-built tool.

Your validation step is critical but often underestimated. The sandbox subset needs to include edge cases - test contacts with missing required fields, duplicates, or special characters in custom fields.

If your script works for the 50 clean test records but fails on a malformed entry in the full 50k, your pipeline breaks. Build validation checks into the transformation phase itself. Fail early on mismatched data types or length overruns.

Also, don't forget to validate *after* the import. Check record counts and spot-check relationships. A dry run's success means nothing if the target CRM silently drops records due to API limits.

Totally agree on building validation into the transform phase. I run a pre-flight checklist in my mapping script that logs counts of records with missing required fields, invalid email formats, or custom field overflow *before* any API calls happen. It's saved me from hitting HubSpot's API limits with bad data more than once.

One thing I'd add: the "silent drop" issue is huge, especially with rate limits. Even if your record counts match, spot-checking a few random IDs post-import isn't enough. I pull a sample of, say, 200 imported records by their new ID, then verify key field values match the source. Found a date formatting bug that way once - everything imported, but all the dates were wrong.

What's your go-to method for verifying field-level accuracy after the fact?

Your point on vendor pricing is correct, but the real sticker shock for teams who don't DIY is the year two renewal. You get off the spike, think you're done, and then you need ongoing sync for a new custom object. That's when the $500/month becomes a permanent tax.

And that 40-hour build estimate is optimistic for anyone who hasn't fought a Bulk API before. Double it if you need proper idempotency and audit logs, which you absolutely do for compliance. The scripts that just move data are easy. The scripts that *prove* they moved it correctly, and can restart from failure, are the other 40 hours.

You mentioned the timeout. It's not just throttling, it's session expiration on the source side during large batch transforms. Found that out the hard way.

The phased approach is correct, but I'd emphasize a detail in your first phase: utilizing the native API. For anything beyond a trivial dataset, you must use the Bulk API endpoints, not the standard REST APIs. A standard API call for 50,000 contacts will fail due to timeout and rate limiting well before completion. The Bulk API is designed for asynchronous job submission and handling large volumes, which changes your script's error handling significantly.

Your mapping snippet is a good start, but in practice, the transformation layer needs to handle schema discovery and type coercion programmatically, not just column renaming. Salesforce `Picklist` fields to HubSpot `Dropdown` fields often have value mismatches that break imports. A simple mapping dictionary isn't sufficient; you need a fallback strategy for unmapped values.

Also, the validation phase should include a checksum audit, not just spot-checking. Generate an MD5 hash for a batch of records (concatenating key fields) from the source after extraction and from the target after import. A discrepancy points directly to a data corruption issue in the pipeline, which spot checks can easily miss. This is how you truly verify integrity at scale.

Good call on using the native Bulk API. I've seen timeouts kill a standard REST job halfway through, leaving a messy partial import.

One caveat on your mapping snippet: direct field-to-field assignment can break if the source data has nulls or mismatched types for required target fields. I usually add a simple clean-and-coerce function in the transform step. Something like:

```python
def clean_phone(raw_phone):
if pd.isna(raw_phone):
return ''
# basic stripping, keep as string
return str(raw_phone).strip()
```

Also, for the dry run, don't just test 50 random records. Pick the 50 that represent every weird custom field and relationship you have. If it only exists once in your full dataset, make sure it's in your test batch.

Spot-checking a random sample is smart, but I've found it doesn't catch systematic drift unless your sample is perfectly lucky. My go-to is a poor man's data diff: after the import, I'll run a script that pulls back a key slice of the target data - say, the 500 most recently modified contacts - and joins it on email to the source export. Then I compare a handful of critical fields programmatically.

It's less about verifying all 200k records and more about checking for patterns. If 'Industry' is blank on 30% of the imported records, you've got a mapping problem. A random sample of 200 might only include two of those, and you'd miss the trend.

The real silent killer isn't wrong data, it's *defaulted* data. Your script passes null, the target CRM's API quietly applies a default, and your 'Lead Source' becomes "Web" for 10,000 people. Spot-checking a few random IDs won't reveal that either. You need to look for unnatural homogeneity in fields that should be diverse.

Yeah, the "defaulted data" trap is so real. I've seen HubSpot silently populate missing `Country` with the account's default, turning an international list into a single-country one. That pattern analysis you mention is key.

I've started building a simple validation report right into the post-import script. It counts occurrences for each field in the target sample and flags any that are suspiciously uniform - like 95% of `Lead Source` being identical. It's not a full diff, but it catches those systematic API "helpful" overrides.

The joint on email is crucial, but I've hit issues with email casing or whitespace differences causing missed matches. Adding a cleaned/ normalized version for the join saved me there.

Your phased approach is spot on, especially hitting the source API for the extraction. I'd add one practical twist on the transformation step: building a fallback mapping layer.

That mapping snippet is a great start, but it can break if the source field names aren't consistent, which happens surprisingly often with custom fields. I always wrap the column assignment in a small function that tries a list of possible source column names before failing.

```python
def get_column(df, possible_names):
for name in possible_names:
if name in df.columns:
return df[name]
return pd.Series(dtype='object') # returns empty series if not found

target_data['firstname'] = get_column(source_data, ['FirstName', 'First Name', 'First'])
```
It's a bit more code, but it's saved me from halting a 50k record job because the export used "First Name" (with a space) instead of "FirstName".

The fallback mapping layer is a smart defensive move for schema drift. I'd extend that concept to the metadata discovery phase, not just the transform. Before I even pull a single record, I script an audit of the source CRM's object schema, logging all custom field API names, labels, and data types. That list becomes the source of truth for the `possible_names` array in your function.

This pre emptive cataloging catches another issue: custom fields that exist in production but not in the sandbox you built your mapping against. If you rely on a sandbox schema snapshot, you'll miss fields added later. The fallback function will still fail because the field isn't in the sandbox export's columns at all.

One caveat: your function returns an empty series if no match is found. That's safe for optional fields, but for required target fields, you need a deliberate decision: either fail the batch outright or apply a business rule to populate a default. Letting it proceed with empty values often triggers the silent defaulting problem discussed earlier.

Absolutely correct about schema auditing pre-pull. That practice is non-negotiable for any migration where the source is a live, evolving SaaS platform. I'd take it a step further and advocate for performing this schema discovery not once, but at two key points: once during the initial mapping design, and again immediately before initiating the final production data extraction run. This catches any fields added or deprecated in the intervening period.

Your caveat on handling missing required fields is crucial. The decision logic for that scenario should be explicit and logged. In my scripts, the transform layer for a required field will raise a hard error if the fallback function returns an empty series and no business rule exists, halting the pipeline. This is preferable to a silent default, as it forces a mapping decision into the change log.

For the sandbox discrepancy issue, this is precisely why I never develop against a stale schema dump. The pre-flight script calls the source CRM's metadata API directly in the production environment (with read-only credentials) to build the column manifest. It adds maybe 2-3 seconds to the job start time, but it guarantees the field list is current.

Precisely the kind of methodical thinking these migrations need. I'd add a quick note on the metadata API call: for some orgs, those 2-3 seconds can balloon if the schema is large or the API has high latency. I've started caching that schema manifest for the duration of the job, but with a clear invalidation trigger tied to the job start timestamp. That way you avoid repeated calls during iterative processing, but you're still guaranteed a fresh pull for each new execution.

The hard error on missing required fields is the only sane approach. It turns a data quality issue into a process issue, which is where it belongs. The alternative is those "helpful" defaults that corrupt your dataset's meaning.

Good structure, but you're missing the most expensive phase: data residency.

Your Python snippet assumes a local transform, which works for a few thousand records. If you're moving millions, you'll be running that script on a VM or in a cloud function for hours. That's a direct compute cost that never gets budgeted in these migrations.

Pull the data into an S3 bucket or cloud storage first, then run the transform with something like AWS Glue or a batch job. It's cheaper than keeping a large EC2 instance running and you can scale it down to zero after the job.