I've run hyperparameter sweeps with W&B on Kubernetes for about a year. It works, but the agent-based model has scaling quirks you need to design around.
Key configuration for the W&B Kubernetes operator:
```yaml
spec:
sweepName: "my-sweep"
template:
spec:
containers:
- name: train
image: my-training-image:latest
command: ["python"]
args: ["train.py"]
env:
- name: WANDB_PROJECT
value: "my-project"
sweepConfig:
metric:
name: val_loss
goal: minimize
method: bayes
parameters:
learning_rate:
min: 1e-5
max: 1e-2
```
The main issues:
* **Resource overhead:** Each agent is a pod. Launching 100 parallel runs means 100 pods, which can overwhelm scheduler.
* **State management:** If an agent pod dies, its run can get orphaned. You need robust logging to track this.
* **Cost:** Each pod pulls the container image. With a large image and many parallel runs, you can hit registry pull rate limits and waste time on initialization.
Compared to a native Kubernetes batch job manager (like Kubeflow Pipelines or Argo), W&B adds a layer of abstraction that simplifies logging but introduces its own scaling bottlenecks. It's fine for moderate sweeps (<50 concurrent trials). Beyond that, you're better off managing the queue yourself and using W&B just for tracking.
Great to see someone with hands-on experience on this setup. The resource overhead you mention is real - we've seen similar issues when scaling beyond 50 concurrent agents.
You're right about the abstraction trade-off. It simplifies the experiment tracking side, but you're still managing pod lifecycle and scaling quirks yourself. Have you tried using pod affinity/anti-affinity rules to help with the scheduler load? Some teams I've talked to had success with that, though it adds another config layer 😅
The orphaned runs are a pain. We ended up adding a sidecar to the training pod for heartbeat logging, which helped trace failures.