The KEP-2395 transition from in-tree cloud providers to external cloud-controller-manager (CCM) components is now well underway, with several major providers already migrated. This has a direct and under-discussed impact on cluster autoscaler configurations, particularly for those of us running on-prem or with custom cloud integrations.
Previously, the cluster-autoscaler relied on the in-tree cloud provider logic baked into the kube-controller-manager for node group discovery and management. With the migration, this responsibility shifts to the cloud-provider-specific interfaces implemented by the external CCM. The autoscaler must now be configured to communicate with the correct external cloud provider, not the legacy in-tree one.
For example, a legacy in-tree configuration for AWS might have looked like this in the autoscaler deployment:
```yaml
spec:
containers:
- command:
- ./cluster-autoscaler
- --cloud-provider=aws
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,kubernetes.io/cluster/
```
Now, with the external `cloud-controller-manager` handling node lifecycle, the autoscaler's `--cloud-provider` flag must be set to `external`. It then relies on the CCM's node annotations and labels for discovery. The new pattern is:
```yaml
- --cloud-provider=external
- --node-group-auto-discovery=label:cluster-autoscaler.nodegroup/name=
```
The practical impacts I've observed in testing:
* Autoscalers for providers like Azure (AKS), GCP (GKE), and AWS (EKS) managed services have mostly been updated by the platform.
* For custom deployments (e.g., vSphere, OpenStack, or bare-metal with a custom cloud provider), you must ensure your external CCM implements the necessary interfaces for the autoscaler. The `cloudprovider.Interface` is now the contract.
* Missed configuration is a common failure scenario: the autoscaler pod will log errors about being unable to list node groups or cloud provider being nil.
Has anyone else begun refactoring their autoscaler deployments for on-prem or hybrid setups? I'm particularly interested in seeing real configuration snippets for vSphere or OpenStack post-migration, and any failure logs encountered during the transition.
- kelly
Data is not optional.
Good catch on the autoscaler config change. We hit this during our GCP migration last quarter.
One nuance we found: setting `--cloud-provider=external` is necessary but not always sufficient. If you're using a service account for the autoscaler pod, you need to ensure its RBAC has `get` and `update` permissions for the `nodes` resource. The external CCM model sometimes changes the required permissions slightly compared to the in-tree provider.
Also, for anyone still running a mixed setup during transition, double-check your kube-controller-manager isn't still trying to manage cloud routes. You'll get odd conflicts.