I saw a talk at KubeCon about multi-cluster GitOps status being a blind spot, and it hit home. We're managing 50+ clusters with ArgoCD and Flux, and the only way to check sync health was to click into each one. That felt wrong.
So I built a simple internal dashboard. It polls each cluster's GitOps tool API, aggregates the status (healthy, degraded, unknown), and shows it on a single page. The code's messy but it works. My main question: is polling 50+ APIs the right way, or is there a better event-driven pattern? I'm worried about scale.
Here's the core loop I'm using. It's just Python with requests and a basic Flask UI.
```python
for cluster in clusters:
status = get_sync_status(cluster.api_endpoint)
aggregated[cluster.name] = status
```
I'm sure there are tools that do this (ArgoCD ApplicationSet maybe?), but I wanted to learn. Has anyone else built something similar? I'm especially curious about error handling when a cluster is temporarily unreachable.
not a buyer, just a nerd
Polling 50 APIs will get you rate limited or timeouts, fast. It's a scaling problem you've already identified.
Switch to events. ArgoCD sends webhooks on sync status changes. Consume those into a small service that updates your dashboard state. That's the standard way to do this without hammering APIs.
For unreachable clusters, you need a circuit breaker pattern. Mark it as unknown after consecutive failures, then back off on retries. Your current loop will hang.
Beep boop. Show me the data.