Hey everyone! I've been absolutely loving Weights & Biases for tracking my model experiments—it's been a game-changer for keeping my team organized and visualizing our progress. Seriously, the dashboard alone is worth it! 😊
But I've hit a real snag with **Hyperparameter Sweeps**, and I'm wondering if any of you have faced something similar. I'm trying to tune a moderately complex transformer model for a text classification task, and my sweeps keep suggesting hyperparameters that are, frankly, *terrible*. We're talking learning rates like `1e-1` or batch sizes that cause immediate OOM errors, or optimizer choices that diverge in a few steps. My control runs with sensible manual values work perfectly fine, so it feels like the sweep logic is just... off.
Here's my sweep configuration. I've tried both `bayes` and `random` search methods, and the issue persists. My goal is a wide but reasonable search space:
```yaml
program: train.py
method: bayes
metric:
name: val_loss
goal: minimize
parameters:
learning_rate:
distribution: log_uniform_values
min: 1e-5
max: 1e-2
batch_size:
values: [16, 32, 64]
num_train_epochs:
value: 5
optimizer:
values: ["adamw", "sgd", "rmsprop"]
warmup_steps:
distribution: int_uniform
min: 100
max: 1000
```
I expected the Bayesian optimizer to sample somewhat intelligently, maybe starting near a known-good region (I've done a few manual runs), but it seems to ignore the metric history entirely. For instance, after three runs where `lr=1e-3` gave a `val_loss` of ~0.5 and `lr=1e-1` exploded immediately, the next suggestion was *another* `lr=1e-1` with a different, equally unstable optimizer.
**My theories on what might be wrong:**
* Is the `log_uniform_values` distribution for `learning_rate` somehow being misinterpreted? Should I use `log_uniform` instead?
* Does the `bayes` method need more initial random runs to build a good model of the space? I had `early_terminate` off, but maybe I need to set a higher `num_init_points` explicitly?
* Could it be an issue with how my `train.py` logs the `val_loss` back to W&B? I'm using `wandb.log({"val_loss": val_loss})` at the end of each epoch.
* Or is my search space just too wide and chaotic for the optimizer to handle, and I need to constrain it more based on my manual experiments?
Has anyone else wrestled with this? I'd love to hear:
* Your sweep configs that actually work reliably.
* Any pitfalls in the YAML structure I might have missed.
* Whether you found better luck with `random` search and then manually narrowing down, or if the `bayes` method eventually "clicked" for you.
* How you handle interdependent parameters (like batch size affecting effective learning rate).
I really want to make sweeps work—they're such a powerful feature on paper! Let's compare notes and figure this out.
happy comparing!