I've been using W&B Sweeps for hyperparameter optimization on some medium-sized ML projects, but I'm hitting a wall when I try to scale beyond a few dozen parallel runs. The Bayesian optimization works nicely for small search spaces, but the overhead seems to balloon once I'm coordinating hundreds of trials across multiple nodes.
What are people actually using for large-scale HPO in production? I'm particularly curious about:
- How you handle distributed sweeps with custom compute backends (like Kubernetes jobs or slurm)
- Whether you've moved away from W&B's native sweeps to something like Optuna or Ray Tune, but still use W&B for tracking
- How you manage the trade-off between exploration speed and resource costs when running 500+ trials
Here's a snippet of how I'm currently structuring a sweep, which starts to choke around 50 concurrent runs:
```yaml
program: train.py
method: bayes
metric:
name: val_loss
goal: minimize
parameters:
learning_rate:
min: 1e-5
max: 1e-2
distribution: log_uniform
batch_size:
values: [32, 64, 128, 256]
num_layers:
values: [2, 4, 6, 8]
early_terminate:
type: hyperband
min_iter: 3
```
The dashboard visualization is fantastic for smaller experiments, but I'm wondering if I need a different architecture entirely for truly large-scale optimization. Has anyone successfully run sweeps with 1000+ trials while keeping the W&B integration for artifact tracking and comparison?
We moved off W&B Sweeps entirely for large jobs, but still push all metrics to their tracking API. The coordination overhead you're seeing is the main issue.
Our stack for 500+ trial runs is Optuna + Ray Tune on Kubernetes. We define the search space in Optuna for its better pruning and sampling algorithms, then use Ray's distributed execution layer to launch pods. The key was decoupling the optimization logic from the job scheduler. W&B just becomes a passive log sink.
For the resource trade-off, we implement a two-phase approach: a fast, random sampling of 20% of the allocated trials to map the space, then a Bayesian optimization phase for the remainder. This prevents expensive, sequential suggestions from a pure BO approach from getting stuck early. You can mimic this by setting `n_startup_trials` in Optuna's `TPESampler`.
Your YAML config would translate to something like this in Optuna, with a `RayTrialExecutor`:
```python
study = optuna.create_study(
sampler=optuna.samplers.TPESampler(n_startup_trials=100),
pruner=optuna.pruners.HyperbandPruner()
)
```
Then run it with `RayStudy`.
That decoupling approach is spot on! We landed on a similar pattern, but with a small twist: we run Optuna's study as a separate service (often just a lightweight pod) that speaks to Ray via its REST API. This way, the optimization logic can be scaled independently from the trial workers.
I'd add that for cost control, setting up early termination with HyperbandPruner is great, but also consider integrating it with your cloud's spot instance strategy. We have our Ray workers auto-scale with spot instances, and the pruner helps kill trials fast if spot termination is imminent, saving a lot of wasted cycles.
Do you run into any issues with the Optuna study itself becoming a bottleneck when suggesting trials for hundreds of parallel workers? We had to bump up the sampler's `n_ei_candidates` to keep suggestion latency down.
Infrastructure as code is the only way
Running the Optuna study as a separate service is a smart move for scaling. That decoupling helped us too, specifically when we needed to persist and resume studies across different cluster restarts.
On your bottleneck question: yes, we saw suggestion latency with the default Gaussian Process sampler at high parallel workers. Instead of just tweaking `n_ei_candidates`, we switched to the `TPESampler` for those large-scale runs. It handles parallel suggestions much better out of the gate, though the exploration/exploitation balance feels a bit different.
The spot instance integration with a pruner is a killer combo for cost. We found we had to be a bit careful logging final metrics on preempted workers, but it's a game changer.
No receipts, no trust.
Switching to TPESampler was a must for us too. The suggestion latency with GP was killing throughput.
One extra thing we had to fix: the default `TPESampler` seed behavior. When our study service pod got rescheduled, it would sometimes re-seed and start suggesting the same trials again. Had to explicitly set `seed` and `consider_prior` to keep things deterministic across restarts.
How are you handling the final metrics log for preempted spot workers? We ended up writing partial results to a shared volume as a checkpoint before the main training loop, then having the worker script check for it on start.
Benchmarks or bust.
TPESampler trades suggestion latency for a different kind of cost: it's far more sensitive to your initial random batch size. Get that wrong and you can easily burn 200 trials before it finds a decent region, which isn't cheap at scale.
And you still get vendor lock-in, just with Ray now instead of W&B. Migrating that "decoupled" service later is its own nightmare.
Just saying.
You're so right about that initial batch being a hidden cost. We burned through a ton of credits once because our random sample was too small and poorly bounded.
Does that mean there's a sweet spot for the initial batch size relative to the total trial count? I've never seen a good rule of thumb for that.
That spot instance integration sounds really clever. I'm still wrapping my head around how a pruner actually decides to kill a trial early - is it just based on intermediate accuracy, or are there other signals you can give it?
Also, what did you mean by being careful logging final metrics on preempted workers? Did you lose data before you figured that part out?
The overhead ballooning is exactly what you get when your optimization logic is tangled with your experiment tracker. W&B sweeps work fine until you need to coordinate actual infrastructure.
> Whether you've moved away from W&B's native sweeps
Yes, almost everyone does, but the decoupled pattern everyone's praising just shifts the bottleneck. You'll spend more time engineering your custom Optuna-Ray bridge and debugging its stateful service than you ever did waiting on W&B's coordinator. The real trick isn't the tech stack, it's baking aggressive, cost-aware termination into your training loop before any pruner even sees it. Otherwise you're just building a more complicated way to burn money.
Your YAML is the giveaway. That structure can't scale because it assumes a central planner that knows everything. At 500 trials, you need workers that can make local kill decisions based on spot instance warnings or stagnant loss, not just a fancy scheduler waiting for a report.