Hi everyone, I’m new to the community and honestly a bit nervous about posting, but I’ve been reading through so many helpful migration stories here. I’m currently planning our CDP switch and feel stuck on one major decision point.
We need to pull about two years of historical event data from our old CDP. I see two main paths: using a tool like Fivetran to handle the extraction and loading, or building our own scripts to use the old CDP’s API directly. I’ve read reviews for both approaches, but I’d love to hear from people who’ve actually done it.
My biggest worries are around cost surprises with Fivetran (their pricing seems complex) and whether their connectors truly handle all our custom event schemas without data loss. On the other hand, the direct API approach seems like a lot of initial dev work and ongoing maintenance, especially for error handling and rate limits. I’m also concerned about how each method impacts the backfill process—does one make it easier to resume if something fails mid-way?
If anyone has gone through this specific choice, I’d really appreciate any lessons learned. How did you weigh the setup time versus long-term reliability? Were there any hidden issues with data mapping or support from the tool vendor?
I'm a procurement lead at a mid-market retail tech company, and we've directly used both methods for CDP migrations in the last 18 months. Our prod stack moved from Segment to a custom warehouse setup.
Core comparison:
1. Real, realized cost: Fivetran starts as a predictable operational cost but inflates with volume. At my last shop, we hit a $12k surprise bill because a misconfigured sync looped and multiplied our active rows. Direct API is a fixed upfront dev cost (roughly 3-4 sprints for a senior engineer) and negligible ongoing cloud function spend.
2. Integration and schema handling: Fivetran's pre-built connectors work for standard event types, but our custom nested properties were often flattened into JSON strings, requiring additional transformation. A direct script gives you exact control over schema mapping from the start.
3. Failure recovery and backfill: Fivetran's checkpointing is reliable for resuming after network blips. With a direct script, you must build your own idempotency and state tracking. We spent a week adding a checkpoint system to our script after a 14-day pull failed on day 12.
4. Long-term maintenance: Fivetran shifts the burden to them for API updates and credential rotation. Our direct script broke twice in a year when the source CDP deprecated API fields, requiring immediate dev intervention during a migration.
My pick is the direct API, but only if you have the in-house engineering bandwidth to dedicate for a month and your event schema is heavily customized. If you're a small team with standard events and need a set-and-forget solution, Fivetran is the safer bet. Tell us your team's dev-to-analyst ratio and whether your historical data has nested custom properties.