Hey everyone,
I've been a heavy user of Weights & Biases for about two years now, managing large-scale hyperparameter sweeps and experiment tracking for my team. We've loved the features, but recently we've hit a serious UI performance wall, and I'm wondering if others have experienced this or if it's something in our configuration.
Our main project now has over 1,5k runs. When I load the project overview page or try to use the parallel coordinates plot for analysis, the browser tab becomes almost unresponsive. Scrolling is janky, opening a run takes 3-4 seconds, and using the compare feature often times out or crashes the tab. This happens consistently across Chrome and Firefox, on machines with 32GB RAM. It's starting to really impact our workflow.
Here's a bit of context on how we log, in case it's relevant:
```python
# Our typical run init
wandb.init(
project="model-sweep-v3",
config=config_dict,
tags=[model_type, dataset_version],
group=f"sweep-{sweep_id}",
)
# We log maybe 10-20 metrics per epoch, plus some histograms and a table at the end
```
We've tried the obvious fixes:
* Clearing browser cache and using incognito mode.
* Using the **Table** view instead of the default gallery (it helps a bit, but not much).
* Applying very aggressive filters before loading the page.
* Increasing the `polling` interval in case it was a network issue.
The strange part is that the API (`wandb.Api()`) remains snappy for fetching data. The bottleneck seems purely frontend. Has anyone else scaled to this number of runs and found a way to keep the UI usable? Are there specific views or data types (like large custom plots) that are known to cause this?
I'm considering:
1. Archiving old runs to separate projects.
2. Drastically reducing the number of metrics we log per run (but that defeats the purpose).
3. Building a custom dashboard using the API, which feels like rebuilding what W&B should do.
Any insights, similar experiences, or workarounds would be hugely appreciated. I really want to keep using W&B, but this is becoming a blocker for the team.
— francesc
— francesc
You're definitely not the only one. I'd be skeptical of any demo or case study they put out implying smooth performance at that scale. I've seen the same thing happen around the 800-run mark on a project with fairly lightweight logging. The browser's memory usage just balloons.
What's interesting is that they never seem to mention this as a known scaling limit in their docs or sales pitches. It's always "unlimited experiment tracking." The reality feels more like you're supposed to archive or delete old runs, which defeats a lot of the purpose of having a historical record. Have you tried their "artifacts" system as a workaround, or is that just as sluggish?
cg
Yeah, the Table view might feel snappier initially because it's paginated, but it's just hiding the same underlying data load. The real issue is they're dumping the entire run set into the browser's memory for those fancy plots. Over 1k runs, each with even a modest config and metric set, means you're pulling down multiple MB of JSON just to render the page.
The group and tag structure you're using might actually make it worse. Every filter or panel that needs to aggregate across those groups triggers more client-side processing. I've had some luck by avoiding the web UI entirely for analysis at that scale - use their API to pull run data into a local script and use pandas or plotly. It's a workaround, not a fix.
Did you notice if the sluggishness is worse on the project page versus a single run page? That's usually the tell.
Speed up your build