Hey everyone, I've hit a snag with my experiment tracking setup that I'm hoping someone else has run into.
I'm running a hyperparameter sweep using W&B, launching multiple parallel training jobs (each in its own process, on separate machines). The runs are supposed to be distinct, logged under the same sweep. However, in the W&B UI, I'm seeing runs overwrite each other's metrics and steps. It looks like one run's data is being replaced by another's, so my charts are a mess of conflicting lines and some runs appear to vanish or merge. The runs *do* have different names in the run table, but their logged data seems to be colliding.
Here's a simplified view of my launch setup:
- Using `wandb.init()` with `sweep_id` and `group` parameters set.
- Each job gets a unique `JOB_ID` environment variable, which I'm trying to use to set `run_id` or `name`.
- I'm not manually setting `run_id` because the docs suggest letting the sweep handle it.
What I've tried so far:
- Explicitly setting `run_id` to a unique string (like the `JOB_ID`) in `wandb.init()`. This helped, but then the runs sometimes didn't link to the sweep properly in the UI.
- Setting the `name` and `group` combo uniquely. The names show correctly, but the data overlap persists.
- Checked that `wandb.finish()` is being called in each process.
My main suspicion is around how W&B associates the running process with the UI entity when jobs are launched simultaneously. Is there a specific config order or an environment variable (like `WANDB_RUN_ID`) I should be setting *before* `init` to force isolation?
Has anyone debugged this before? I'm deep in a comparison of orchestration tools and this kind of operational hiccup is a major factor in my evaluation. Missing a solid parallel run story is a big red flag for me.
Spreadsheets > marketing slides.
Oh that sounds frustrating! I've run into something a bit similar with overlapping runs, but with email campaign A/B tests, not sweeps. When you tried setting the unique `run_id`, did you also keep the `sweep_id` parameter the same in the init call? I'm wondering if there's a specific order the system expects those to be in for it to still connect them to the sweep group properly. Maybe the run ID needs to be set *before* the sweep ID argument?
Oh yeah, that run ID timing idea is interesting! I've had a similar dance with sequential vs parallel email sends in automation workflows. The order of operations can totally trip things up.
From my own mess-ups, it helped to think of the run ID as the unique record identifier and the sweep/group as the folder it lives in. If the system picks up the sweep context *after* generating an ID, maybe it gets confused.
Have you tried logging the exact combination you're using for run_id and sweep_id in each process's console output? Sometimes seeing them side by side reveals a pattern the UI is misreading.
Been there. When runs look separate but data collides, it's often a shared *step* index across processes. W&B uses step to align chart lines. If your parallel jobs are logging at the same step numbers, the UI might mush them together even if run IDs differ.
Quick test: log a unique identifier (like JOB_ID) as a metric or in the config, and check if those values are also getting overwritten in the UI. That'll tell you if it's a display grouping issue or actual data overwrite.
For sweeps, I've had better luck setting a unique `name` derived from the environment variable, but leaving `run_id` unset. The sweep controller seems to need that internal handle. Does setting `name=f"run_{JOB_ID}"` but omitting `run_id` keep them linked to the sweep cleanly?
Ask me about hidden egress costs.
Yeah, the step index overlap is a great callout. I've seen that happen with parallel prototyping sessions where everyone logs under the same iteration number - total mess.
Your suggestion to log the JOB_ID as a config field is spot on for diagnosing. If that unique value is also getting clobbered in the UI, it points to a deeper ID collision, not just a chart grouping problem. If the JOB_ID stays unique per run, then the issue is likely the visual merging like user880 said, which is a different headache.
I'd try their name-based approach first. Set a unique `name` using your env var, keep `sweep_id`, but completely omit `run_id` and `group`. Let the sweep controller manage the internal run_id. That's worked for me when I needed clean parallel logs under a single project umbrella.