Alright folks, let's talk about a pattern I see causing unnecessary drift and confusion in Flux-managed clusters: poorly structured Kustomize overlays.
The promise is clear—use Flux to sync a base, then patch for dev/staging/prod with overlays. But without a strict layout, you end up with teams copying entire manifests and losing the single source of truth. I've had to lock threads where the "review" was just a vendor's tutorial copy-pasted with no real-world snafus.
Here's the structure that's held up under scrutiny in our production deployments. You keep your `kustomization.yaml` for each environment directly in the repo, and your Flux `Kustomization` objects point to them. The key is that the base contains only the common, stable pieces. The overlay only has the *differences*: a ConfigMap patch, a replica count, a resource limit.
For example, your dev overlay might just patch the image tag and lower resource requests. Your prod overlay might inject a sidecar for security and up the replicas. Flux reconciles each environment independently based on its path. If you find yourself duplicating 90% of the base in an overlay, you've likely put something in the base that shouldn't be there.
Biggest pitfall? Patches that are too broad and break on future base updates. Be surgical. Use `json6902` patches for precise changes, especially for things like environment variables or annotations. And for the love of transparency, document *why* a patch exists in a comment—future you will thank present you when debugging at 3 AM.
This keeps your GitOps audit trail clean and makes environment promotions a review of the patches, not a full manifest rewrite. Seen other approaches that work? Or horror stories where this fell apart? Post your config snippets and the specific problems you hit. No vague "it's great" posts without evidence, please.
No marketing. Only receipts.
You've hit on the crucial discipline of treating the base as immutable common ground. The drift starts when teams, often under pressure, add "just one" environment-specific tweak to the base because it's easier than structuring a proper patch. I've enforced a hard rule in our repos: if a PR to the base touches anything resembling a `ConfigMap` value or a replica count, it's rejected. The base should be so generic it could theoretically deploy to a vacuum.
A related pitfall is patching at the wrong level of abstraction. It's tempting to use a JSON patch to change a single container argument, but that fragments the mental model. We mandate that overlays compose entire strategic merge patches for a resource, keeping the delta obvious in review. This also prevents accidental patch collisions when multiple overlays target the same resource.
One counterpoint: your strict separation can break down for certain stateful workloads where the storage class or node affinity *is* a core, stable piece of the application definition across all environments. In those cases, we've had to carve out a separate "base-variant" using Kustomize components, but that's a rare exception that proves the rule.
monitor first
>if a PR to the base touches anything resembling a ConfigMap value or a replica count, it's rejected.
I like that as a concrete rule. Makes the line clear for everyone. My team's still trying to get that discipline.
So for the strategic merge patches you mandate, does that mean you're basically copying the whole container spec into the patch file and just editing the one argument? That seems like it could get bulky if you have a lot of small changes.
You know, that 90% duplication test is a great rule of thumb. Been there. The one time I got lazy and let a "temporary" environment variable slide into the base, it caused a weird config bleed in our staging environment that took half a day to untangle. The overlay had to patch it *out* again. Lesson learned the hard way.
Your point about the Flux Kustomization objects pointing directly at the environment kustomization.yaml is key. It keeps the pipeline simple. I've seen folks try to make a single Flux source point at the base and then magically overlay it, which just adds complexity. This way, each env's Flux object is its own clear declaration.
it worked on my machine