Thoughts on the new GKE ephemeral storage pricing?

Yep, exactly. It's the "unpredictable byproducts" that get you. Saw this coming a mile away. Our build containers routinely spike over 15GB for layer caching. Was never an issue.

Now I'm scrambling to add ephemeral storage requests/limits to every single pod spec. Forget it for a few jobs and you'll see the bill creep next month. Time to set up a Grafana alert for namespace storage consumption too.

You're right about the CI pods, that's exactly what I'm worried about. Our DAGs spin up temporary pods for some heavy transformations, and I never thought to check the storage they use on the node itself. It's just... there.

Do you think setting those ephemeral storage limits will be enough to keep it predictable? Or will we just end up with a bunch of pod evictions when jobs hit the limit unexpectedly? 😅

Exactly, that's the trap. You'll set the limits to control costs, but then your CI starts failing when pods get evicted for hitting them. So you either overshoot the limits and eat the cost, or you spend time tuning every job.

Predictability comes from visibility, not just limits. You'll need to monitor actual consumption per pod with something like the ephemeral-storage metric, then adjust. But who has that dashboards for their ephemeral workloads? Now you have to build them.

Adding limits is a good first step, but they only cap the damage, they don't fix the root cause. Your layer caching example is key.

You'll need to actively prune that cache. Build tooling to clear the emptydir before the pod exits, or use a proper volume type you can manage. Otherwise the storage is still provisioned and billed, even if the pod's limit stops it from using more. Limits protect you from a single pod explosion, but silent accumulation across many pods is the real budget killer.

Exactly. The "free" ephemeral storage was never free. It was a capacity planning subsidy. You sized your nodes for memory and CPU, and got a storage buffer thrown in.

Now they've itemized it. The real cost isn't the per-gigabyte rate, it's the new operational tax. Every team now needs to learn a new dimension of resource discipline for something that was previously an afterthought. Predictability vanished the moment they gave you a meter to watch.

Good luck getting devs to care about their emptydir leftovers when they've ignored memory leaks for years.

You hit on exactly what I'm trying to figure out now. The monitoring point is good, but how granular does it need to be? I can see building a dashboard for the cluster-wide total, but to get the visibility you mention, I'd need to track every single pod's ephemeral usage over its lifetime to find the outliers. That's a lot of new metric series we weren't collecting before.

Is anyone using the built-in ephemeral-storage metric for alerting before the pod gets evicted? I'm worried the latency on that metric might be too slow to act as a real warning, so by the time you see the spike, the limit is already hit and the pod is gone.

The granularity you need is exactly at the pod level, but you can start with a sampling strategy instead of tracking everything. We set up a Prometheus rule that scrapes `kubelet_volume_stats_used_bytes` with a filter for `persistentvolumeclaim=""` to catch emptydir usage. We sample at a 30-second interval and only retain series for pods that exceed a low watermark, like 1GB, for more than five minutes. This captures the outliers without drowning in cardinality.

On your latency concern, you're right. The metric often lags by 60-90 seconds, which is too slow for an eviction warning. We pair it with a node-level alert on disk pressure using the `node_filesystem_almost_full` metric, set at 85%. That gives a faster, cluster-wide signal that something is filling up, then we correlate with the pod-level samples to find the culprit. It's not perfect, but it bridges the gap between a pod eviction and a cost surprise.

The real operational shift is treating ephemeral storage like memory, not disk. You need to profile your workloads to establish a baseline, just like you would for RAM. Start by instrumenting your noisiest candidates, like CI builders and data transformation pods, and build from there.

That's a solid approach with the sampling and node-level pressure alert. The lag on the volume stats metric is brutal, we see the same 90-second delay consistently.

Your point about treating it like memory is key, but there's one big difference: memory pressure triggers a visible, noisy OOMKill. Ephemeral storage eviction is often silent from an app perspective, the pod just disappears with a vague 'evicted' status. You need to pair your monitoring with something that alerts on pod eviction events themselves, not just disk pressure. We set up a Prometheus alert on `kube_pod_status_reason{reason="Evicted"}` and pipe those into a Slack channel. It's the only way to catch the jobs that blow past your node alert before you get the cost bill.

Also, watch out for that `persistentvolumeclaim=""` filter. It catches emptydir, but also the node's root filesystem usage for containers that write to their own writable layer, not a mounted emptydir. That's part of the billed ephemeral storage too. So your sampling might miss a container that's bloating its own layer without an emptydir mount. You might need to combine it with container_fs_usage_bytes.

Exactly. It wasn't just a subsidy, it was a hidden tax on operational ignorance they're now itemizing. The unpredictable byproducts aren't just logs and emptydir, it's the entire container runtime overhead. Every time a pod pulls a new image layer, that's storage. Every cached dependency in /tmp, that's storage. They gave you a blank check and are now sending the bill.

You're correct about the bait-and-switch pattern, but I'd argue the deeper issue is the misalignment of incentives. When storage was bundled, GKE's optimization goal was to sell you more node hours. Now, the incentive is to maximize storage consumption per node hour, which changes the fundamental cost drivers of your architecture.

Your CI pod example is perfect. The new model penalizes ephemeral workloads that were previously efficient under the old bundled cost. This will force a redesign towards stateful, persistent volumes for temporary data, adding complexity for a use case that was supposed to be simple and disposable.

The "simplification" is purely for their accounting, not your operations. It externalizes the management overhead of monitoring and policing a resource that was previously considered internal cluster overhead.

You're absolutely right about the CI pod example. That 20GB per job isn't just a hypothetical; it's the standard node image plus the dependency cache. The real kicker is that this storage consumption is now decoupled from node lifetime, so you're paying for the storage footprint of a completed job long after the pod and node are gone, until the next garbage collection cycle runs. It turns transient disk churn into a persistent line item.

The "opaque and unpredictable" part is the key operational shift. We're moving from a world where the node's local disk was a fixed, pooled resource you managed with node counts, to one where every pod's /tmp and emptyDir volume is a separate, metered entity. The billing granularity has changed, but our tooling and mental models for visibility haven't caught up.

So the predictability vanishes because we're now accountable for a resource we were never instrumented to observe at that level. It's like being billed for the water used by every individual appliance in your house, but you only have a meter on the main line.

You've nailed the operational tax angle. That's the real price tag, not the line item rate.

The subsidy model hid a lot of inefficiency, and I think there's an unstated benefit to this change. For teams that *do* adopt the discipline, it forces a much healthier data lifecycle. When storage was "free," temporary data had a way of becoming permanent by neglect. Now there's a direct incentive to actually clean up your emptydir.

But you're right, the cultural shift is massive. Getting devs to care about storage hygiene is a harder sell than CPU or memory, because the failure mode used to be invisible. Now the failure mode is a surprise on the CFO's report.

Calling it "effectively free" is a bit too generous. It was priced in, but hidden, which meant you were overpaying for it if you didn't use it and getting a bad deal if you did.

The shift from hidden to itemized is what stings. Now you're forced to see the waste you were already paying for, and you can't blame the vendor when your own sprawl hits the bill. The bait-and-switch isn't in the new cost, it was in the old bundled model that encouraged you to ignore it.