Skip to content
Notifications
Clear all

Did you see the KEP about in-tree cloud provider removal? Impact on autoscalers

2 Posts
2 Users
0 Reactions
51 Views
(@kellyh)
Trusted Member
Joined: 4 months ago
Posts: 59
Topic starter   [#9414]

The KEP-2395 transition from in-tree cloud providers to external cloud-controller-manager (CCM) components is now well underway, with several major providers already migrated. This has a direct and under-discussed impact on cluster autoscaler configurations, particularly for those of us running on-prem or with custom cloud integrations.

Previously, the cluster-autoscaler relied on the in-tree cloud provider logic baked into the kube-controller-manager for node group discovery and management. With the migration, this responsibility shifts to the cloud-provider-specific interfaces implemented by the external CCM. The autoscaler must now be configured to communicate with the correct external cloud provider, not the legacy in-tree one.

For example, a legacy in-tree configuration for AWS might have looked like this in the autoscaler deployment:
```yaml
spec:
containers:
- command:
- ./cluster-autoscaler
- --cloud-provider=aws
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,kubernetes.io/cluster/
```
Now, with the external `cloud-controller-manager` handling node lifecycle, the autoscaler's `--cloud-provider` flag must be set to `external`. It then relies on the CCM's node annotations and labels for discovery. The new pattern is:
```yaml
- --cloud-provider=external
- --node-group-auto-discovery=label:cluster-autoscaler.nodegroup/name=
```

The practical impacts I've observed in testing:
* Autoscalers for providers like Azure (AKS), GCP (GKE), and AWS (EKS) managed services have mostly been updated by the platform.
* For custom deployments (e.g., vSphere, OpenStack, or bare-metal with a custom cloud provider), you must ensure your external CCM implements the necessary interfaces for the autoscaler. The `cloudprovider.Interface` is now the contract.
* Missed configuration is a common failure scenario: the autoscaler pod will log errors about being unable to list node groups or cloud provider being nil.

Has anyone else begun refactoring their autoscaler deployments for on-prem or hybrid setups? I'm particularly interested in seeing real configuration snippets for vSphere or OpenStack post-migration, and any failure logs encountered during the transition.

- kelly


Data is not optional.


   
Quote
(@devops_dad_v2)
Reputable Member
Joined: 8 months ago
Posts: 380
 

Good catch on the autoscaler config change. We hit this during our GCP migration last quarter.

One nuance we found: setting `--cloud-provider=external` is necessary but not always sufficient. If you're using a service account for the autoscaler pod, you need to ensure its RBAC has `get` and `update` permissions for the `nodes` resource. The external CCM model sometimes changes the required permissions slightly compared to the in-tree provider.

Also, for anyone still running a mixed setup during transition, double-check your kube-controller-manager isn't still trying to manage cloud routes. You'll get odd conflicts.



   
ReplyQuote