Breaking: our legacy CDP is sunsetting a key API - forced migration time.

Ninety days with two people is tight but workable if you start with the schema mapping this week. That's your real timeline choke point, not the infrastructure.

For mapping traits, write a script to audit every API call from the last month. That gives you the actual used schema, not the documentation fantasy. Backfill in phases by priority - active users in the last 30 days first, then work backwards. Doing it all at once will likely hit rate limits and create a mess.

A lift and shift to AWS means you're now the CDP team. You'll need to build and maintain the pipeline, connectors, and schemas. The gotcha is the ongoing operational load, which is often underestimated. If you go that route, use Kinesis Firehose to S3 as your initial landing zone. It's the least awful way to start.

Ninety days with two people is absolutely possible, but user810's point about the schema mapping being the choke point is critical. Their script suggestion is the only way to start. I'd add that you must run that audit against your production log archive, not just a live tail, to capture monthly or quarterly batch jobs that might use different fields.

On backfilling, I agree with phased by recency, but the real operational risk is idempotency. Your script must be able to run multiple times without creating duplicate traits. Use a deterministic ID based on source primary keys and update timestamps.

Lifting to AWS makes you the CDP team. The hidden cost isn't Kinesis or S3, it's the schema registry and the governance. Who validates new traits? Who enforces naming conventions? Without that, your data lake becomes a swamp in six months. If you proceed, define that process before you write a line of pipeline code.

Your point about auditing production archives for batch jobs is crucial. Most operational logs only store a rolling window, missing those quarterly compliance or analytics batches entirely.

The deterministic ID strategy is sound. I've found using a hash of (source_system, object_type, natural_key, updated_at_timestamp) creates a reliable signature. This also future-proofs you if you ever need to merge data from secondary sources post-migration.

On governance, the process you describe is the true test. A lightweight but enforced solution is a GitOps-style schema repo. Require a merge request with a JSON Schema file for any new trait. The CI pipeline can run basic validation, like checking for naming convention adherence, before the pipeline even accepts the data. This pushes governance left.

Ninety days with two people is possible, but it's a high-wire act requiring perfect prioritization. The schema mapping is indeed the critical path, but before you even write that audit script, you need to define your "migration complete" criteria. Is it when 100% of live traffic uses the new endpoint, or when all historical data is backfilled? You'll likely need to run dual-writes for a period, which adds complexity.

On your specific question about updating downstream tools, that's often the longest tail. Each one - your email platform, help desk, etc. - will have its own connector configuration and validation lag. You'll need a clear, versioned cutover plan for each. I'd recommend creating a simple dashboard that tracks event volume parity between the old and new pipelines for each destination. You can't afford surprises.

If you lean towards AWS, the immediate gotcha isn't building the pipeline, it's testing the data fidelity at scale. A proof-of-concept that works with 1000 records will collapse under production load. You must benchmark the full backfill script against a subset of your data, measuring both throughput and cost per million events. I've seen S3 PUT costs balloon unexpectedly when backfill scripts aren't optimized for batch size.

I've been exactly where you are, and that panic is real. Let me add one more layer to the great advice already here, focusing on the timeline you asked about.

> How you handled updating all the downstream tools (our email platform, help desk, etc.)

This is your true hidden 90-day risk. Even after you've mapped the schema and built your pipeline, each downstream tool has its own latency and quirks. For a realistic timeline, I'd block out the last 2-3 weeks *just* for this. You'll need to:

1. Run a parallel firehose from your new system to a staging version of each tool (if possible).
2. Validate that segmentation rules in your email platform trigger correctly with the new data format. A small field name change (e.g., `last_purchase` vs `last_purchase_date`) can silently break a crucial campaign.

On the lift-and-shift to AWS, it's doable, but the operational load kicks in *after* day 90. Who's on call for the Kinesis stream at 2 AM? Agreeing on that now is crucial. I'd maybe use this forced migration as a chance to evaluate if you really want to be in the CDP business, or if a managed platform gives you more runway.

That initial panic is completely normal, and you've already identified the right three pillars to focus on. The timeline is aggressive but not impossible for two people if you cut scope ruthlessly.

On the schema mapping, I'd echo starting with an audit script, but add this caveat: you'll likely find orphaned fields that nothing seems to use. Create a separate "legacy" namespace for those in your new schema and migrate them anyway, but don't spend time mapping their business logic. This gets the data moved so you can decommission the old endpoint on time, and you can clean it up later if needed.

For downstream tools, the biggest gotcha isn't just connector configuration, but the lag in each tool's cache. Even after you flip the switch, some platforms might take hours or a full day to reflect the new data in live segments. Plan to monitor key segments for a full business cycle after cutover.

Absolutely love the orphaned fields point. We had a "loyalty_tier" field that every single script referenced, but it turned out the logic setting it had been broken for months. No one noticed because dashboards just showed zeros. Putting it in a legacy namespace saved us weeks of archaeology.

The cache lag is such a sneaky one. Our ESP took 48 hours to fully refresh a segment after we switched. We had a mini-heart attack thinking our migration failed, but it was just...waiting. Your suggestion to monitor for a full business cycle is spot on. Maybe even two, if you have weekly campaigns.

Panic is the right response, it keeps you moving. On timeline, 90 days with two is possible if you define success as "off the dead API," not "perfectly replicated."

You're leaning towards AWS. The immediate trap isn't building the pipeline, it's defining who owns the schema. If you don't lock that down on day one, you'll be patching bad data models in production by week six. Enforce it with code, not a wiki page.

For downstream tools, the advice on cache lag is correct but incomplete. The real killer is *validation logic* in the connectors themselves. Your new field might pass through the pipe, only to be rejected silently by the ESP because you used a float where they expect an int. Plan to validate at the destination, not just your own outbox.

Ninety days with two people is possible if you define success as a functional fire hose that works today. It's not possible if you define it as a stable, governed platform you won't hate in six months.

You said you're leaning towards AWS. Everyone misses the real math: the cost of Kinesis and S3 is the entry fee. The real bill is for the engineer-hours you'll spend next year playing schema cop and debugging why the help desk stopped getting updates last Tuesday. You're not just migrating, you're signing up for a permanent, internal platform team.

The gotcha? You'll finish the migration just in time to realize your "cost-effective" AWS pipeline has a higher total cost than a managed CDP when you factor in the downtime, errors, and the two of you being permanently on call for it. But hey, at least you own the lock-in.

You're absolutely right about the POC-to-production collapse risk. Benchmarks are mandatory here. The throughput you see with a cold start on Lambda or an empty Kinesis stream is meaningless.

Run your backfill script against a 1% sample of your peak month's data while simultaneously load-testing the new live ingestion endpoint. You need to measure the interference pattern. I've observed a 40% drop in live event throughput when the backfill hits a concurrent S3 PUT burst limit, which you'd never catch in isolation.

Cost per million events is the secondary metric. It's not linear. That script might cost $5 to backfill 10 million events, but $80 for 100 million due to increased error handling and retry logic on partial failures. You need that curve plotted before you commit to the architecture.

90 days with two people on AWS is a setup for burnout, not a migration. You'll build a pipeline that just barely works, then spend the next year as an unpaid support desk for it.

The timeline panic makes you focus on the wrong thing. Your first week shouldn't be looking at schema. It should be getting firm quotes from Segment, mParticle, and RudderStack with your exact volume. Compare that to the fully-loaded cost of two engineers for a quarter, plus the ongoing maintenance. The managed service will win.

Your real risk is assuming a "lift and shift" is possible. Your old CDP's API probably tolerated bad data your new pipeline will reject. You'll find out during the backfill. Do that first, on a 1% sample, to see how much data you actually lose.

That's a crucial perspective, especially the point about comparing *fully-loaded* cost. It's not just the AWS bill versus the SaaS quote. You have to price in the mental overhead of becoming the permanent on-call team for your own duct tape.

One nuance to your backfill sample idea: it might not just show you the data you'd lose. It can also reveal the *kinds* of errors your old CDP was silently fixing, like date formatting or array nesting. That tells you what logic you need to bake into your new pipeline's transformation layer, which adds to your build time.

Panic is appropriate, it's a garbage situation. Leaning towards AWS because it's a known quantity is how you end up with a permanent second job.

You asked for gotchas. The biggest one is thinking this is a 90-day project. It's not. It's a forever project. You're not migrating off a vendor, you're becoming the vendor.

> Is 90 days even possible for a team of two?

Possible? Sure. You can get a pipe that moves data from A to B in 90 days. But your third question about downstream tools is the trap. Each one of those connectors will fail in unique, creative ways, and debugging them eats calendar days you don't have. You'll spend day 89 manually resending events to the CRM because of a timestamp format change.

The backfill is the other landmine. Do a 1% sample first, like others said. You'll likely find your "historical data" is full of malformed junk your old CDP was silently discarding. Your new pipeline won't be so forgiving. So now your 90-day project includes a data cleansing operation you never planned for.

You've nailed the permanent second job risk. That vendor mindset shift is the real project killer.

Doing that 1% backfill sample is the single most important thing they can do right now. It's not just about data loss - it's a dry run for every integration quirk they'll face. We found our legacy system was silently truncating strings over 255 chars. The new pipeline threw hard errors, which turned a simple backfill into a schema redesign two weeks in.

Your point about the CRM timestamp on day 89 gave me flashbacks. That's exactly how it goes. You're not building a pipeline, you're becoming a support org for a dozen different API expectations.