Hey everyone, hoping to pick your collective brains on this one. We've been using AWS Spot instances for our big, non-critical batch processing jobs (think data enrichment, report generation) to save a ton on compute costs. The strategy has been brilliant... until recently.
Lately, our instance interruption rate has shot up. Jobs that used to run fine are now getting terminated maybe 60% of the way through, and it's causing a lot of failed workflows and manual re-runs. We're using the standard Spot best practices: diversified instance families across multiple AZs, using the Spot instance advisor to pick types with lower interruption rates, and our jobs are checkpointing.
But it's still happening! I'm starting to wonder if our *particular* region or instance type has just gotten super popular. Has anyone else hit a wall like this recently?
A few things we're considering, but I'd love real-world feedback:
* **Mix in On-Demand:** Maybe a hybrid fleet? Like 70% Spot, 30% On-Demand to guarantee some capacity, but I'm worried that dilutes the savings too much.
* **Different Workload Segmentation:** Splitting the jobs into smaller, faster chunks so an interruption loses less progress. This would require some re-architecting.
* **Are we missing a tool?** I've heard about tools that can predict or model Spot interruption patterns better, or even automate the fallback to On-Demand or a different region.
What's your playbook when Spot reliability starts tanking? Have you found a particular configuration or third-party service that brought stability back without blowing the budget? Concrete examples of what worked (or didn't!) would be amazing.
Marketing ops nerd
The two-minute warning on Spot is always a tricky one. You're already doing the right things with diversification and checkpointing.
> mixing in On-Demand
This is the typical next step, but you can get clever with it. Instead of a static 70/30 mix, use an Auto Scaling group with a mixed instances policy and *capacity rebalancing* enabled. The group will proactively try to replace a Spot instance that's at high risk of interruption with a new one *before* it's terminated, which can sometimes keep your job count stable. The new instance might be Spot or On-Demand, depending on your allocation strategy. It doesn't eliminate interruptions, but it smooths out the churn.
Smaller job chunks is a solid architectural fix, but the overhead can eat savings. Have you looked at the actual Spot price history and interruption frequency for your exact instance types in the last 30 days? The Advisor gives a general score, but the raw data sometimes shows a specific type that's become a problem child, which you could then exclude from your diversified list.