Hey everyone! I'm diving deep into a new training run with some hefty models, and I've noticed W&B is logging *so much* system data by default. While I love having visibility, I really don't need a GPU temperature reading every single second for a week-long experiment. It feels like overkill and makes the runs page a bit noisy.
I'm trying to streamline the metrics to just the essentials like GPU utilization and memory. Has anyone figured out a clean way to limit what system metrics get logged? I poked around in the `wandb.init()` settings and the environment variables but couldn't find a straightforward switch.
I'm all for comprehensive data, but for my current project, less is more. Would love to hear how others have handled this or if there's a config flag I'm missing. Maybe there's a sweet spot between monitoring and clutter? 😅
Totally feel you on the metric overload! The system metrics come from the `wandb` SDK's internal monitor process. You can actually control it via the `WANDB_SYSTEM_MONITOR_INTERVAL` environment variable, but that just changes the frequency, not the type.
The cleaner way is to set `os.environ["WANDB_SYSTEM_MONITOR_DISABLE"] = "true"` *before* calling `wandb.init()`, and then manually log just the GPU metrics you want using `wandb.log()` and pulling data from something like `pynvml` or `torch.cuda`. It's a few extra lines but gives you surgical control.
For example, you could log utilization and memory every N steps in your training loop, skipping the temp and disk I/O entirely. Makes the run UI much cleaner. Let me know if you want a quick snippet for that manual logging setup.
YAML is not a programming language, but I treat it like one.
That's a solid approach for taking manual control. Setting the `DISABLE` flag is definitely the key.
One small caveat: if you're working across multiple scripts or a codebase where you can't guarantee the environment variable is set early everywhere, you can also pass `settings=wandb.Settings(system_monitor_disabled=True)` directly into `wandb.init()`. It's just another way to skin the cat and might be cleaner in some project structures.
I'd also be curious if anyone has found a middle ground, like a way to filter the monitor's output without fully disabling it and writing your own collector. Probably not, but user277's method is the most reliable I've seen.
Stay curious, stay skeptical.
Manual logging seems like trading one maintenance burden for another. Now you're responsible for pynvml version compatibility and error handling across different GPU types.
And what's the real cost of those extra metrics? Bandwidth? Storage? Their pricing isn't that transparent. I'd bet the default logging is cheap for them, expensive for you to reimplement poorly.
read the fine print