Having spent the last week evaluating Claw's newly announced migration suite for a potential client migration from a legacy on-prem data warehouse to their cloud platform, I must express significant reservations regarding their CSV import constraints. While the tool's orchestration and incremental sync features are conceptually sound, the published limits on initial bulk data ingestion via CSV present a substantial, and likely expensive, operational hurdle.
The core issue lies in the per-file row limit of 500,000 and the maximum file size of 2GB. For any organization with a non-trivial historical dataset—think user event logs, multi-year transactional data, or IoT telemetry—this necessitates a fragmentation of migration jobs that introduces complexity, points of failure, and, critically, a linear increase in cost.
* **Cost Amplification:** Claw's pricing model for the migration tool appears to be based on "job units," with each job having a base compute cost. Ingesting a 500-million-row dataset would require at least 1,000 separate CSV ingestion jobs purely due to the row limit, not accounting for file size constraints which would likely push that number even higher. This transforms a single bulk operation into a massively parallel workload, incurring 1,000x the base job unit cost. This is a classic example of a hidden fee structure: the per-job cost seems low until you are forced to run thousands of them.
* **Operational Overhead:** Managing the lifecycle (create, monitor, retry, validate) of thousands of nearly identical jobs is a DevOps burden. While their API allows for automation, the failure domain is now significantly larger. A single network blip during one of a thousand concurrent jobs necessitates a pinpoint retry, whereas a single monolithic job might simply resume.
* **Comparative Landscape:** Most modern cloud data platforms (BigQuery, Snowflake, Redshift) offer either native bulk load capabilities with far higher limits or partner tools that leverage scalable cloud storage (like S3 or GCS) as a staging area, charging primarily for storage and compute scanned, not per file ingested. Claw's approach feels anachronistic, as if they are treating CSV files as discrete "tickets" rather than a data stream.
My immediate question for the community is whether anyone has engaged with Claw's engineering or sales team on this specific limitation and received guidance on a workaround or a roadmap to increase these thresholds. Is the expectation that customers will pre-process all historical data into their proprietary streaming format, which presumably has no such limits? If so, that entails additional transformation costs and tooling prior to the migration itself.
For our use case, this constraint directly impacts the total cost of migration (TCM) calculation, making the tool's advertised "simple, predictable pricing" misleading. The spreadsheet clearly shows that for datasets exceeding 50GB, the cost of the CSV ingestion phase alone would surpass the cost of a month-long, full-volume incremental sync. This creates a perverse incentive to avoid bulk loading historical data altogether, which is often not feasible.
-- Liam
Always check the data transfer costs.
You're absolutely right about the cost amplification being the hidden deal-breaker. It's a pattern I've seen with several managed service providers where the initial data transfer is artificially constrained, pushing you towards their proprietary, higher-throughput pipelines which always carry a premium.
The 500k row limit is particularly perplexing from a performance standpoint. Any modern object store coupled with a competent ingestion engine can handle streams of much larger files. I suspect this is less a technical limitation and more a business one, designed to make their "Enterprise" tier with its direct-connect options look necessary. It forces a proliferation of jobs that inflates the bill and complicates error handling, as you noted.
Have you calculated the implied ingestion rate? A 2GB file with 500k rows suggests an average row size of about 4KB. For wide tables common in legacy warehouses, you'd hit the size limit long before the row limit, making the effective constraint even tighter. I'd be curious to see how this compares to the initial bulk load limits on AWS DMS or Google's Data Transfer Service.
SQL is not dead.
Great point about the row size! That's a crucial detail I missed. If your rows are large (which they often are), you'll be chunking by size, not row count, leading to even more tiny files and jobs.
You're spot on about the business angle. It feels engineered to make the basic tier a pain point. A quick check shows AWS DMS, for all its quirks, doesn't impose per-file limits like this - you're constrained by overall task resources, which makes more sense.
Did you get a chance to look at their error reporting? My fear is that with hundreds of these micro-jobs, tracking down a failed chunk in logs would be a nightmare.
Clean code, happy life
Yeah, the chunking nightmare is a great point. I was only thinking about splitting by row count, but file size limits force you to manage two different splitting logics at the same time. That sounds like a scripting headache waiting to happen.
I'm still learning about this stuff. How do tools like AWS DMS handle error reporting for big jobs? Is it actually easier to track, or is it just a different kind of messy?
Oh, you think you only have to manage *two* splitting logics? That's cute. Wait until you have to account for duplicate primary keys across chunked files, or handle referential integrity when your customer and orders tables get sliced up differently. Your script is about to get a whole lot more "interesting".
To answer your AWS DMS question, it's definitely a different kind of messy. You get a single task with verbose logs, but parsing them for a specific bad record is like finding a needle in a haystack. At least with Claw's army of tiny jobs, the failure is isolated. You just have to find which of the 200 tiny haystacks contains your needle. Pick your poison.
But what about the edge case?
You're right, the relational integrity problem across chunks is the real beast. Even if you script a perfect split by size and row count, you're almost guaranteed to break foreign key relationships between tables unless you can somehow coordinate the chunking across every dependent dataset. That seems impossible with their tool's per-file constraints.
So it forces a decision: do you accept broken relationships during the migration and run a massive reconciliation fix-up job afterward, or do you stage everything in an intermediate location and re-chunk there first? Both add huge time and risk.
Your "pick your poison" line nails it. But with DMS, at least the messy log is for a single atomic job. With Claw, you're signing up for operational complexity before you even start, managing the army of jobs. Which type of pain is less likely to derail a project timeline? I'm leaning towards the single, messy log.