Hey folks, I've seen this panic a few times now – a legacy platform is sunsetting, and suddenly you need to get years of data out before the lights go off. It's stressful, but totally doable with a clear plan. Let's break down the usual paths.
First, always check if the platform has a native "export all data" or "data portability" feature. Sometimes it's buried in admin settings. If they offer a full database dump or a set of CSVs, that's your easiest win. If not, you'll likely be working with their API. The key here is to **document the API endpoints and data models immediately**. You don't want to be guessing the schema a week before shutdown.
Here’s a basic Python script using `requests` that can be a starting point for paginated API endpoints. You'd need to swap in your actual auth and endpoint details.
```python
import requests
import json
base_url = "https://legacy-platform.com/api/v1/"
headers = {"Authorization": "Bearer YOUR_TOKEN"}
all_records = []
page = 1
while True:
response = requests.get(f"{base_url}/objects?page={page}", headers=headers)
data = response.json()
if not data.get('items'):
break
all_records.extend(data['items'])
page += 1
# Write to a file
with open('legacy_data_export.json', 'w') as f:
json.dump(all_records, f)
```
If the API is limited or too slow, contact their support *now* and ask for a bulk data extract. Be polite but firm about your timeline. In parallel, think about where this data is going. A cloud storage bucket (S3, GCS) or a data lake is ideal for raw dumps. Then you can use a tool like Fivetran or Airbyte later to load it into a warehouse, or even run the script above to output directly to a staging database.
Don't forget about related files (user uploads, attachments)! Their URLs might be in the API, but the actual files might need a separate, parallel download process. Finally, validate your export. Do row counts match? Spot-check some recent and old records. A quick quality check now saves a huge headache later.
Hope this helps someone avoid a last-minute scramble. What legacy system are you dealing with? Sometimes there are community scripts floating around for common platforms.
ship it
ship it
That script is a good starting point but it's dangerously naive for a full historical export. It lacks any error handling or rate limit management, and blindly trusts the API's pagination structure. You'll get throttled or your connection will drop after a few thousand records, losing your place.
You need to immediately implement a resumable state. Use a local SQLite database or even a simple JSON file to track the last successfully fetched ID or timestamp for each endpoint. Wrap your request in a try/except, log failures, and have a backoff strategy. Also, never append to a single in-memory list for a large export; you're begging for a memory crash. Stream records directly to disk.
Check if the API offers any bulk export endpoints or a "modified after" filter. Sequential ID pagination is the worst case; you need to parallelize requests across ID ranges.
—davidr
Yeah, user774 is spot on about the memory crash. I've been burned by that exact thing. You start getting successful pages, the list grows, and three hours in the script OOMs and you lose everything.
Their point about parallelizing ID ranges is a lifesaver when you're against the clock. If the API uses sequential integer IDs and you have, say, 50 million records, fetching sequentially is impossible. You have to split the ID space into chunks and run multiple workers. Just watch out for API rate limits across all your workers - you'll get banned fast if you don't coordinate.
Here's a quick snippet for that ID range approach I often adapt:
```python
import concurrent.futures
import requests
def fetch_range(start_id, end_id):
# Your logic here, with backoff and state tracking
pass
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
futures = []
chunk_size = 10000
for start in range(1, total_estimate, chunk_size):
futures.append(executor.submit(fetch_range, start, start + chunk_size))
```
But honestly, if the platform is truly on its last legs, sometimes the API stability goes to hell. You might get weird gaps or duplicate IDs. That's where the local SQLite checkpoint db becomes your only source of truth 😅
ship it
Parallelizing like that might just get you banned faster if their API is already on life support. You're hammering a dying system with five concurrent requests? Good luck with those rate limits.
And the whole "estimate your total IDs" approach feels optimistic for a platform that's shutting down. You think they kept their database pristine? There will be gaps, deleted records, weird orphaned tables. Your workers will spend more time chewing on 404s than fetching data.
Honestly, this is where the premium support you paid for should be giving you a database dump. Funny how that "enterprise reliability" evaporates when they're pulling the plug.
—DW
Agreed on checking for a native export first, that's saved my sanity before. But I've also seen platforms where that "data portability" feature only dumps the last 90 days, which is a nasty surprise.
Your starting script is a good skeleton, though I'd swap that `all_records = []` for an immediate file write or a proper local database cursor. Appending to a list in memory for a huge historical pull is a surefire way to watch your script die after hours of work.
Also, while documenting the API, screenshot everything - the data model in their old docs, example responses, even the API rate limit headers. Once the platform is gone, that context disappears and you're stuck guessing at weird field values.
ship it