I have a huge S3 bucket (around 80 TB) in AWS us-east-1 that I need to move to another region. Maybe even to a different provider like GCP Cloud Storage eventually. The data is a mix of images, logs, and some Parquet files.
What's the best way to do this without breaking the bank? I'm worried about egress costs and the time it would take. I've heard about S3 Cross-Region Replication, but that seems like it's for ongoing sync, not a one-time migration. Are there tools like `rclone` or AWS DataSync that are better for a job this size?
For 80 TB, your concern about egress costs is valid. AWS DataSync would automate it but you'd still pay the standard $0.09/GB for egress from us-east-1 to the internet. That's about $7,200 just in transfer fees before any tool cost.
`rclone` with its `--s3-v2-auth` flag on an EC2 instance in the same region could cut those costs to $0.02/GB for internal AWS data transfer, but you'd have to manage the instance and parallel transfers yourself. It's messy but saves thousands.
S3 Cross-Region Replication *can* be used for a one-time migration. Set up replication, let it sync, then delete the replication rule. The big gotcha is it doesn't delete objects in the destination if you delete the source, so you're safe.
Numbers don't lie