Exactly. We hit the same scheduler friction. Our workaround was to use pod anti-affinity with a `preferredDuringSchedulingIgnoredDuringExecution` rule to softly spread agents across nodes, but even that sometimes backfired.
The orphaned run issue is a bigger pain than most people admit. We ended up writing a small operator that watches for `Evicted` or `Failed` pods in the sweep namespace and calls the W&B API to mark those runs as crashed. It's extra infra, but it beats manually cleaning up hundreds of unfinished runs.
pipeline all the things
The orphaned runs issue is exactly why we stopped using the vanilla operator. We built a wrapper that intercepts pod termination events and forces a W&B sync before the container shuts down, but even that's fragile.
Have you looked at setting up your own queue system instead? Like having a single agent pod that pulls jobs from a Redis queue and spawns Kubernetes jobs directly. It breaks the W&B agent pattern, but you get more control over pod lifecycle.
Still looking for the perfect one
Your config snippet highlights the core tradeoff: you're letting the W&B operator manage pod creation to get automatic experiment tracking, but you're inheriting its scaling model. The agent-per-pod architecture directly causes your first two issues.
On the **resource overhead**, the 100-pod surge isn't just scheduler load; it's a direct hit to API server and etcd. We measured it: a burst of 100 pod creations added 1.2 seconds of latency to all other pod scheduling operations in the namespace for about 90 seconds. Your mitigation needs to be at the cluster level, not just with priorities. Consider using a `Pause` schedule for the sweep config to stagger agent launches.
For **state management**, the orphaned run problem has a cost dimension often overlooked. Orphaned `running` runs in W&B still consume a seat if you're on a team plan, and they clutter the UI, increasing the time engineers spend on manual cleanup. The `preStop` hook suggestion from later posts helps, but it's a reactive fix. A proactive one is to set `WANDB_RUN_GROUP` environment variable in your template to tag all runs from a sweep, then you can script an API call to filter and kill any runs in that group that don't have a corresponding active pod.
The layer of abstraction you mention is really a vendor-specific control plane on top of Kubernetes. When it breaks, your debugging tools are split between `kubectl` logs and the W&B UI, which is where the logging rigor becomes critical. Have you quantified the time lost in this context switching during an incident? That's often the hidden operational tax.
Trust but verify.