Hi everyone, I've been using W&B to track my model trainings for a few months now, and overall I like the visualization side of things. Lately though, I've been running into a frustrating issue during really long training jobs (we're talking 12+ hours).
My runs will just... stop logging. The dashboard shows the run as active, but the metrics and logs freeze at a certain point. I'm not seeing any errors in my terminal, and the script continues to run fine locally. It's like the SDK quietly disconnects from the server. I have to manually restart the logging, which messes up the continuity of the experiment.
I'm using the standard `wandb.init()` and `wandb.log()` in a Python script, running on a remote VM. My connection seems stable. Has anyone else encountered this? I'm wondering if it's something in my setup, like a timeout setting I'm missing, or if it's a known thing with the service.
I'm particularly curious about how others handle reliable logging for extended periods. Are there best practices or config parameters (like `settings` or heartbeat timeouts) that help prevent this? I rely heavily on these logs for comparing experiment lineages, so gaps are a real problem.
Seen it. The SDK's internal heartbeat gets bored. Check your `WANDB_INACTIVITY_TIMEOUT` environment variable or `settings=wandb.Settings(inactivity_timeout=...)`. Default's surprisingly short for a 12-hour slog.
Your run stays "active" because the process holds the socket. The logs freeze when the background thread gives up.
Try bumping the timeout to 3600 or so. If that doesn't stick, wrap your training loop in a try/except for `wandb.CommError` and re-init. It's a band-aid, but it keeps the lineage.
Prove it.