Multi-week sprint, easily. The biggest time sink won't be the extraction jobs, it'll be fixing every report and dashboard that breaks because the data shape changed.
Your backfill job will need separate logic. You can't just run the new extractor on a historical date range if the old API is gone. You'll need to stage the final v2 dump and transform it to match the new v3 schema before loading.
For orchestration, pagination changes are the silent killer. Your Airflow task will succeed but pull incomplete data. Test for row count deltas and schema validation before cutover.
Totally feel you on the reports and dashboards being the real time sink. Everyone focuses on the pipeline, but the business side's "why is my KPI gone?" tickets start flooding in the second you cut over.
Your staging plan is solid. We also ran synthetic queries against both schemas to catch those silent pagination fails. It's the only way to be sure your row counts are actually right.
Oh, and for the backfill job? Don't forget to version that transformation logic separately. You'll need it again if they tweak v3 after you've already run your historical load.
Your procurement addendum is the only sane approach, and I'm mildly furious we had to learn that the hard way. Even a schema freeze isn't a full guarantee, though, unless the penalty clause has teeth.
We tried similar language, but the vendor's "queryable sandbox" was a read-only mock returning perfect, clean data. It completely failed to simulate rate limiting quirks or the new error formats our jobs actually hit in production. The contract must specify the sandbox mirrors the production service's non-functional behavior, or you're just debugging in the dark.
You end up documenting vendor instability because you have no other choice. The audit trail becomes a list of their change control tickets you were forced to open.
show me the tco