Hey everyone, I'm relatively new to GitOps and just finished a major project deploying ArgoCD to manage about 50 Kubernetes clusters for application rollouts. It was a huge learning experience.
The big win is the unified view of sync status and health across all those clusters from a single pane of glass. Declarative management is fantastic. The main pain point was managing the AppProjects and RBAC at this scale – keeping the permissions tight but manageable across so many teams was tricky. Also, handling secrets securely required some extra tooling (we used Sealed Secrets).
I'd love to hear from others who've done similar scaling. Any best practices for structuring the ApplicationSets or managing cluster discovery? Thanks in advance for any insights!
Oh, the RBAC and AppProject scaling is so real! 😅 We hit the same wall. Our "aha" moment was moving to a **centralized, templated** approach for AppProjects. We define them as Helm charts in a separate Git repo. Each team's project gets a values file with their specific permissions, destinations, and source repos. A single ApplicationSet watches that repo and generates the projects across all clusters. It turns a sprawling mess into a declarative, reviewable config.
For cluster discovery, we leaned heavily on the **cluster secret generator** pattern. We have a central service that automatically creates the ArgoCD cluster secrets whenever a new cluster is provisioned (tagged with metadata like environment, region). Our ApplicationSets then use label selectors to target groups of clusters. The key was making cluster onboarding a push model from our infra pipeline, not a manual pull into Argo.
How did you structure your ApplicationSets? Did you go with a mono-repo per cluster, or a mono-repo per app across clusters? We tried the former first but found the per-app, multi-cluster setup easier for rollbacks.
Prod is the only environment that matters.
That templated AppProject approach is brilliant, and I'm definitely stealing that for our next review cycle. It solves the drift we've seen between clusters.
We actually landed on a hybrid structure for ApplicationSets after trying both extremes. We have a mono-repo per *logical service* (which might be multiple microservices), and then inside that, we use a directory per cluster-type (e.g., `us-prod/`, `eu-staging/`). The ApplicationSet for that repo uses a matrix generator: one generator picks the clusters via label selectors (like yours), and another generator walks the filesystem for these directories. This gives us a clear, rollback-friendly path for a service across all clusters, while still letting us tailor manifests per region with kustomize or small helm value files.
The push model for cluster secrets was our savior too. We built a small operator that watches our cluster registry and reconciles ArgoCD secrets, adding labels for cost center and SLA tier. Made scaling to 50+ clusters actually sustainable.
— francesc
Sealed Secrets is a solid choice for that scale. We've used it as a stepping stone, but we eventually moved to a different pattern because managing the decryption keys across fifty clusters during rotation events became its own operational headache.
Our evolution was to externalize secret provisioning entirely. We now use a dedicated, internal secrets service (backed by Vault) that injects secrets as environment variables or volumes at runtime. The ArgoCD manifests then contain no secret data at all, only references to the secret source. This decouples the secret lifecycle from the application deployment cycle and completely removes the sealed-secrets controller overhead from the clusters.
Regarding RBAC and AppProjects, the centralized templating idea mentioned later is key. One thing I'd add is to enforce a naming convention for AppProjects that includes the team and a scope prefix (like `team-platform-core`). This, combined with a periodic audit script that compares ArgoCD's RBAC state against your source-of-truth Git repo, will save you from the "oh, we forgot to clean up that project when the team dissolved" problem a year from now.
That unified view is the killer feature, isn't it? It's the main reason we stuck with ArgoCD through the initial complexity.
On secrets, we're also using Sealed Secrets but started hitting sync latency at scale. We added a custom health check for the sealed-secrets controller to the Application definitions, which helped ArgoCD understand when a secret was truly ready and stopped some premature sync failures. Might save you some future headaches.
For structuring ApplicationSets, we found that tying them directly to our team topology in the org chart created less friction. Each dev squad owns their ApplicationSet manifest in their service repo, which feels more natural than a central repo they have to PR into.
Still looking for the perfect one