We manage dozens of charts across dev/stage/prod. Our current `values.yaml` sprawl is unsustainable.
What's the actual best practice for structuring this? I see:
* Separate values files per environment (e.g., `values-prod.yaml`)
* Using a parent chart or umbrella chart pattern
* Helmfile with environment-specific configs
* External secret management (SOPS, vals) for sensitive values
I need something that scales, avoids duplication, and is clear for CI/CD. Show me your real directory structure and how you handle small config differences (e.g., replica count, node selectors) vs. large ones (entire feature flags disabled).
ea
Prove it with a benchmark.
I'm a security-focused platform engineer at a 400-person fintech, and we run about 80 distinct microservices on a multi-tenant EKS cluster. We've settled on a Helmfile and SOPS combination for prod after trying most of the patterns you listed.
1. **Integration Complexity**: Separate value files are trivial to start but cause exponential drift; adding a third environment doubled our merge conflicts. Helmfile adds one more layer (YAML in YAML) but cuts our boilerplate by about 70% because you can compose and reference sections.
2. **Secret Handling Real Cost**: External tools like SOPS or vals are mandatory for anything beyond dev. The operational tax is managing the PGP or KMS keys. With SOPS, we spend maybe an hour a month rotating keys and dealing with audit log queries. Without it, you're one `kubectl` command away from leaking secrets to your Git history.
3. **Audit Trail Clarity**: The umbrella chart pattern obfuscates changes. We found debugging a prod issue took 15 minutes longer on average because you had to trace through the chart dependency tree. Helmfile keeps a flat, environment-specific file that `git blame` works on directly.
4. **Scaling Overhead**: For small differences like replica counts, we use a conditional anchor in our `helmfile.yaml` to inject environment-specific overrides. For major changes like disabling entire features, we use separate value files referenced per environment, which keeps the delta obvious. The directory cost is about 5 files per service (defaults, dev, stage, prod, secrets).
I'd recommend Helmfile with SOPS for any team past 10 services or 3 environments. If you're under that, separate value files are fine. To make the call clean, tell us if you have a dedicated platform team to manage the toolchain and whether you're already using a specific secrets manager like HashiCorp Vault.
Trust but verify