Hey folks, hoping to get some collective wisdom here. I've been deep in a migration project at work, moving a fairly substantial Kubernetes footprint (about 500 resources across 15 clusters) from Pulumi over to OpenClaw. I'm a big fan of OpenClaw's type-safety and the fact it's just Go, but I've hit a major snag that's bringing our CI/CD pipeline to a crawl.
The problem is exclusively with `openclaw refresh`. When our state file grows beyond a certain point—I'd say around 300 resources—the refresh operation starts taking an absurd amount of time. We're talking 25-30 minutes for a full refresh. The `plan` and `apply` operations on incremental changes are fine, but that refresh is killing our pipeline's feedback loop. It feels like it's linearly processing each resource in the state, one at a time, with synchronous provider calls.
Here's a sanitized version of our core setup:
```go
// main.go - simplified
package main
import (
"github.com/openclaw/pkg/core"
k8s "github.com/openclaw/provider-kubernetes"
)
func main() {
project := core.NewProject("k8s-platform", &core.ProjectConfig{
RefreshBeforeUpdate: core.Bool(true), // Standard safety practice
})
k8sProvider := k8s.NewProvider(project, "k8s-east", &k8s.ProviderConfig{
Kubeconfig: core.String(config.GetKubeconfig("east")),
Context: core.String("prod-cluster-1"),
})
// ... 100s of resource declarations follow
}
```
What I've tried so far:
* **Concurrency flags:** I've played with `--parallel` on the CLI up to `50`. It helps a tiny bit, but plateaus quickly. The logs suggest bottlenecks in the provider's read operations.
* **Targeted refresh:** Using `--target` works, but it's a workaround. For a full compliance/sanity check in CI, we need the whole state verified.
* **State partitioning:** We're considering splitting the project by cluster or namespace, but this breaks some of our cross-resource dependencies and would be a significant architectural change.
My theory is that the default Kubernetes provider might not be leveraging efficient bulk `List` operations (like `kubectl get all`) and is instead making individual `Get` calls for each resource type/name pair. Has anyone else run into this scale issue? Did you find a provider configuration tweak, or is this a known limitation requiring a contribution to the provider's refresh logic?
I'm about to dive into the provider's source code with a profiler, but wanted to check if the community has already paved a path here. Any insights or shared experiences would be a huge help.
— francesc
— francesc