Nah, skip the "close enough" and "runbook" questions. Those assume they've actually looked at your data.
Ask them this: "Show me the exact sample dataset you used for your initial mapping estimates."
If they can't produce a real, anonymized slice of your actual data - not a toy example - you're buying a pig in a poke. I've had partners bid on migrations using a schema doc alone. Their fancy pipeline crumbles when it hits your real 10-year-old Notes field full of semi-colons and PDFs pasted as text.
If they can't show you the sample, they haven't done the homework. All the operational questions are theater at that point.
been there, migrated that
That's the single most practical question in the thread. A sample dataset isn't just a feasibility check, it's a performance contract.
If they can show you the sample, you should immediately load it and run their pipeline's mapping logic in a test environment. Time it. Measure the transformation latency per record and the memory footprint. A partner's mapping might be logically correct but computationally naive, and you'll only discover the O(n^2) time complexity or the runaway string allocations when you try to scale to the full dataset.
The sample proves they looked. The performance profile of processing that sample proves they *understood* what they were looking at.
--perf
Forget the ETL robustness question. That's putting the cart miles before the horse. You're asking about the engine's fail-safes when you haven't even checked if the vehicle fits in the garage.
The only question that matters right now is: **"What's your hourly rate for scope creep, and at what point do you stop and renegotiate the contract?"**
You said it yourself - it's all moving fast, and you're in over your head. That's the perfect environment for a partner to rack up change requests. They'll be "great at explaining their process" right up until you discover a nasty legacy field, and suddenly that robust pipeline needs a $15k "complex logic module."
Get the financial abort criteria in writing. Not the project plan - the invoice thresholds.
trust but verify
That's a vital but dangerous line of questioning. I agree you need the financial guardrails, but demanding the hourly rate upfront can backfire spectacularly.
You're basically telling them, "I expect this to go wrong, so let's agree on your price for fixing it." A savvy partner will interpret that as a green light to de-scope their initial diligence. Why spend time uncovering "nasty legacy fields" in the planning phase if they're billable discoveries later? You've just incentivized a shallow analysis.
Better to ask, "What's included in your discovery phase to *prevent* scope creep from these ambiguous fields?" Their answer shows if they're a partner or a mechanic waiting for the breakdown.
Trust the data, not the demo.
You're right that a sample dataset cuts through the sales talk. But I've found a partner's *reaction* to that request tells you more than the dataset itself.
If they get defensive or say "that's proprietary," it's a red flag. A good partner will enthusiastically share the sample and walk you through the edge cases they've already identified in it. If they can't, they're just guessing at the data model.
And you have to check that the sample is statistically valid. I've been shown a perfect, clean 100-record slice that conveniently omitted the problematic legacy tables. Ask them how they selected it.
Cloud cost nerd. No, I don't use Reserved Instances.
Absolutely agree. The defensiveness around a sample dataset is a far more reliable indicator than the dataset's technical contents.
We had a situation where a partner immediately offered a "statistically representative" sample, but their selection methodology was simply a date range from the current year. This completely excluded the legacy custom objects from our 2018 acquisition, which contained the most problematic data structures. They weren't being malicious; they were following a standard, repeatable process that failed to account for our specific corporate history. Their enthusiasm was genuine, but their methodology was flawed.
This taught me to ask two questions upon receiving a sample: first, "Can you show me the distribution of record creation dates across this sample?" and second, "Which business events or major system changes in our company history would require a stratified sampling approach beyond a simple random pull?" Their ability to engage with that second question separates a true consultant from a technician.
Check the SLA.
You're right to be nervous about the historical data quality in the pipeline. Everyone's focusing on the *technical* questions, but there's a human one that saved us: **"Who on your team gets paged if the overnight load fails at 2 AM?"**
For a small client, you need to know the actual person, not just the company. Is it a junior engineer following a runbook, or the lead who built the mappings? That answer tells you how they structure support and where their real expertise sits. We learned the hard way when our '24/7 support' ticket went to an offshore team who couldn't access our specific transformation logic. The delay was painful.
Get a name. It makes the "robustness" conversation much more tangible.
Automate all the things
That's a fantastic lens to look through - long-term maintainability often gets lost in the cutover scramble. The "source of truth" question hits right at the operational reality after the partner leaves.
I'd add one specific technical nuance to your point about dedicated mapping tools. Even if they use a tool, you need to ask: "Where is that tool's configuration stored, and how is it versioned?"
I've seen teams use a shiny UI tool that exports JSON mapping files... but then those files are emailed around or stored on someone's desktop. The tool itself becomes a black box. You want to hear that the mapping definitions are in a Git repository, with a clear CI/CD pipeline that takes a commit and deploys it to the staging environment. If they can't show you a commit history for mapping changes, you're likely inheriting a time bomb.
The review process you mentioned is key, but it's only enforceable if the artifact is actually under a proper version control system. Otherwise, you're just hoping their process holds.
Prod is the only environment that matters.
Absolutely. That version control question is the correct one, but it misses the real reason most partners hate it.
Asking "where is it stored" gets you the sales-approved answer about their corporate GitHub instance. Asking to see the *commit history* is what makes them sweat. You aren't just checking for version control, you're auditing their internal chaos.
A clean, linear history from one engineer? Fine. But what you often find is a mess of huge, un-reviewed commits labeled "final_mapping_v3_final_revised.json" from six different people in the last 48 hours before cutover. That commit log is the true post-mortem of the project's panic, and they know showing it exposes their process as a fire drill.
So yeah, demand to see it. The story is in the timestamps and the commit messages, not the tool.
Buyer beware.
That's a great point about asking for a sample dataset, but how do you actually validate the process they use to create it? I'm worried about getting a clean sample that doesn't reflect the real mess.
What if you asked them to walk you through the exact queries they'd run to *find* the edge cases, not just show you the result? Seeing their discovery method seems as important as the data itself.