I am currently evaluating a significant infrastructure migration for a client and would appreciate insights from anyone who has undertaken a similar journey. We are considering transitioning a large, production-grade Kubernetes configuration—approximately 120 microservices across multiple namespaces with complex interdependencies—from a pure Kustomize-based management approach to Helm, while maintaining our Infrastructure as Code (IaC) principles using Terraform for provisioning and the Helm provider for deployment.
The primary drivers for this migration are:
* **Standardization and Ecosystem:** The need to consume third-party charts directly and align with broader organizational practices.
* **Release Lifecycle Management:** Helm's native versioning and hooks offer more granular control over releases than our current `kustomize`-overlay-per-environment pattern.
* **State Complexity:** Our `kustomization.yaml` files have grown increasingly complex, with many patches and strategic merge patches, making them difficult to reason about for new team members.
The specific challenges I am trying to preemptively analyze are:
1. **State Migration:** How did you handle the existing Kubernetes resources managed by `kustomize`? Was a one-time, imperative `kubectl apply` of the new Helm releases sufficient, or did you require a more sophisticated state import strategy to avoid downtime? Did you leverage tools like `helmify` or manual templating?
2. **Structural Refactoring:** Kustomize encourages a patching approach, while Helm favors templating. Translating, for instance, a common label patch or a JSON patch for environment variables into `values.yaml` structures and `templates/` is non-trivial at scale. How did you manage this refactoring effort? Was a piecemeal, service-by-service migration feasible?
3. **IaC Integration:** We use Terraform to manage the cluster itself and other cloud resources. The `helm_release` resource will become the new interface. Did you encounter any pitfalls with Terraform's understanding of Helm state, especially during the migration phase where resources might be managed by both systems transiently?
A simplified example of the transformation we're facing:
**Kustomize (Current - `kustomization.yaml`):**
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/deployment.yaml
patchesStrategicMerge:
- env_patch.yaml
```
**Corresponding Helm (Target - `templates/deployment.yaml`):**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.appName }}
spec:
template:
spec:
containers:
- name: main
env:
{{- toYaml .Values.env | nindent 8 }}
```
The conceptual shift from patching a base to injecting values into a template is clear, but the operational logistics of executing this across hundreds of manifests are daunting.
My key questions are:
* What was your overall migration strategy (big bang, incremental)?
* Which tools or processes were most valuable for the state and template conversion?
* Were the benefits in release management and ecosystem integration ultimately worth the substantial refactoring cost and risk?
* Did you encounter any persistent limitations with the Terraform Helm provider compared to native `helm` commands for complex charts?
I am particularly interested in any quantitative data on effort or any post-migration regrets or successes.
You've pinpointed the core challenge with state migration. It's less about the tooling and more about managing the declarative state stored in the Kubernetes API server. Simply applying a new Helm chart over a Kustomize-managed deployment can cause unexpected cascading replacements or deletions, as Helm will try to reconcile its own state against the live objects.
Our approach was to treat the migration as a controlled rollout rather than a big bang switch. For each microservice, we scripted a process to first record the final, rendered Kustomize manifests for a given environment. We then used those manifests as the reference values for the initial version of the Helm chart, ensuring the chart's template logic produced an identical output. The actual cutover was handled by a two-stage apply: first, a `kubectl apply` of the static manifests generated by the new chart, then a Helm release installation using the `--replace` flag to adopt those existing resources into its management. This maintained uptime and prevented any resource recreation for stateless workloads.
For stateful components, we had to be even more meticulous, often performing the adoption during a scheduled maintenance window and verifying the persistent volume claims were correctly referenced post-migration. Did your team consider a similar adoption strategy, or were there constraints that made a more atomic transition necessary?
That's a really substantial migration you're looking at, with great reasoning behind it. The state migration piece you're starting to analyze is absolutely the crux of it.
One nuance I'd add to your planning phase is to audit how much of your current configuration is truly *release-specific* versus *environment-specific*. Helm values are brilliant for the former, but for the latter - things like external service endpoints or resource quotas that are purely about the deployment target - you might find that keeping a thin layer of Kustomize overlays *underneath* your Helm releases (using the `post-renderer` feature) actually simplifies the Terraform/Helm provider calls. It keeps the environment-specific mutations out of your values files, which can get bloated.
Also, with 120 services, you'll want a clear, incremental path for the rollout. Maybe start by converting a few non-critical, leaf-node services to prove out your process and toolchain before touching the core, interdependent ones. Good luck.
Keep it constructive.