Skip to content
Notifications
Clear all

How do I backfill historical data after adding a new cloud account?

18 Posts
18 Users
0 Reactions
3 Views
(@consultant_carl_42)
Reputable Member
Joined: 2 months ago
Posts: 175
 

"Clean slate" is the key phrase everyone ignores in these migrations, and you've nailed it. People treat cloud integrations like they have a time machine, but they're just a pipe.

Your advice to pull CloudWatch directly is correct, but I'd add that you need to sanity-check the permissions on that old account first. I've seen scripts fail halfway through because someone assumed read-only access to CloudWatch was still active, only to find the IAM role was stripped during the account transition. So before you script the pull, do a manual test for the farthest-back timestamp you need.

It's an extra step, but it beats building an entire comparison on a partial data set.


Test the migration.


   
ReplyQuote
(@infra_switcher)
Reputable Member
Joined: 2 months ago
Posts: 160
 

It's a clean slate from the moment you connect, full stop. There's no backfill button because it can't pull data that was never sent to it. The pipe only carries data flowing forward.

Everyone telling you to pull from CloudWatch directly is giving you the only real answer. But since you mentioned cost comparison, be careful about the metric dimensions you pull. CloudWatch has separate namespaces and metrics for EC2, EBS, RDS, etc. If you just grab `CPUUtilization`, you might miss the `DatabaseConnections` or `VolumeReadOps` that actually drive your bill. You need a list of every service you pay for.

And do a dry run for one metric over one day first. The volume of 30 days for all services can be massive and time out your script.


Been there, migrated that


   
ReplyQuote
(@baller_analytics)
Reputable Member
Joined: 2 months ago
Posts: 201
 

You're assuming the data exists somewhere Sysdig can reach. It doesn't. No pipe, no data.

The other replies about CloudWatch are correct for your analysis, but they're missing the core issue. You're comparing two projects now, which means you already have one account monitored. Why are you trying to backfill the second instead of just starting the clock now and doing the comparison in 30 days? That wait is the only way you get a truly apples-to-apples comparison in the same system.

Chasing a perfect historical dataset will burn more time than the analysis is worth. Set the baseline today and move forward.


If it's not a retention curve, I don't care.


   
ReplyQuote
Page 2 / 2