Just wrapped up a month-long experiment running our nightly data processing batch jobs entirely on spot/preemptible instances across AWS, Azure, and GCP. The headline numbers are in the title, but the details are what matter. We managed to cut our compute costs for these workloads by roughly 70%, with only about 5% of jobs interrupted due to evictions. That’s a trade-off that absolutely works for us.
Our setup: mostly stateless Spark and custom Python workloads, orchestrated by Argo Workflows on Kubernetes. The key was making the application layer resilient. Here’s the core of our node pool configuration for GCP (the pattern is similar elsewhere):
```yaml
# GKE Node Pool for spot
gcloud container node-pools create spot-pool
--cluster=my-cluster
--spot
--node-taints=preemptible=true:NoSchedule
--node-labels=instance-type=spot
--machine-type=e2-standard-4
```
We used taints and tolerations to ensure only our batch pods land on these nodes. The most important tweak was setting up **graceful handling of preemption notices**. Cloud providers give you a ~30-second window (via metadata service) before termination. We captured that and made our jobs checkpoint.
Here’s the simple preemption handler we ran as a sidecar:
```bash
# Watches for termination signal and updates a shared volume
while true; do
if curl -f -s http://metadata.google.internal/computeMetadata/v1/instance/preempted -H "Metadata-Flavor: Google" | grep -q TRUE; then
touch /shared/terminate_signal
break
fi
sleep 5
done
```
The main job containers check for that file and exit cleanly if found, letting Argo retry the workflow step later.
**Key takeaways:**
* **Savings aren't linear:** The 70% came from a mix of instance types and regions. Cheapest zone != most stable. We optimized for "lowest interruption rate per dollar."
* **Interruptions clustered:** That 5% failure wasn't smooth. We had two bad days where a whole region got rocky. Diversifying across providers and families smoothed this out.
* **Warm pools help:** For our latency-sensitive final steps, we kept a small pool of on-demand nodes. The cost was negligible but prevented bottlenecks.
Biggest lesson? You have to track *why* a job failed. Our initial 15% failure rate dropped to 5% after we added resource requests that matched spot instance shapes better, avoiding internal allocation conflicts.
Would I run customer-facing APIs on this? No. But for batch, it’s a no-brainer. The engineering effort to make jobs resumable paid for itself in under two weeks.
Has anyone else run long-term spot comparisons recently? I'm curious if your interruption rates match mine, especially on Azure (which had the highest volatility for us).
-jk
Those numbers are fantastic, and I love seeing the concrete payoff for building in resilience. Your 5% failure rate is really impressive, honestly.
The piece about capturing the preemption notice is the unsung hero here. A lot of teams just accept the eviction as a hard stop, but that 30-second window is a game-changer for checkpointing stateful processes or even just getting a clean exit log. Did you find any meaningful difference in the notice timing or reliability between AWS, Azure, and GCP? I've heard anecdotes that one can be a bit less predictable than the others.
Also, curious if you experimented with any instance type diversification strategies to further reduce interruptions, or if you found the basic setup was sufficient.
Glad you caught the importance of the preemption notice capture. That 30-second window is indeed the linchpin.
On your question about timing differences, we saw GCP and AWS were pretty consistent with the full notice period. Azure... well, let's just say sometimes that window felt more like a polite suggestion. We logged a few cases where termination happened closer to 15 seconds after the metadata flag flipped. Nothing catastrophic for our setup, but it's why we also set an absolute minimum checkpoint interval unrelated to the signal.
Instance diversification helped a bit on AWS. Using a mix of similar instance families in the same node pool did seem to spread the eviction risk, maybe shaved off another half percent in failures. For GCP and Azure, sticking with the basic spot pool config got us 95% of the way there.
it worked on my machine
Great results, and that taint/toleration setup is exactly what made spot work for us too. The preemption notice handling is clutch.
One thing that bit us early on: we had to make sure our job images themselves could react to the termination signal, not just rely on the node-level hook. Had a few jobs using lightweight base images without a proper signal handler.
Automate everything.
The taint/toleration setup is so key. We had a similar journey, and your point about checkpointing during the notice window is spot on.
One extra nuance we found: for those Spark jobs, we had to also tune the `spark.kubernetes.executor.deleteOnTermination` config to false. Otherwise, even with the graceful notice capture, the driver would kill the executor pods before they could finish writing their checkpoint. Added a few more lines to our termination handler to clean them up manually.
What kind of checkpointing granularity did you aim for? We ended up with a 5-minute interval for our longest jobs, which seemed to balance overhead against the risk window pretty well.
Data is the new oil - but it's usually crude.