Help: W&B runs are showing NaN losses, but my local logs are fine.

Oh, I've been there. It's incredibly frustrating when the data you're relying on for decisions doesn't match reality.

In our case, it turned out to be a conflict with another monitoring library that was also trying to log metrics. W&B was picking up a stale, corrupted tensor from a shared namespace before our local logger saw the cleaned-up value. Took ages to isolate. Have you ruled out any other instrumentation or hooking libraries that might be interacting with your training loop?

It's tempting to ditch it, but before you do, maybe try creating a dead-simple test run that logs a single, hard-coded scalar value from a script. If that shows NaN, then you know it's definitely a client or sync issue. If it's fine, the problem is somewhere in the interaction with your specific training framework. Good luck

That's a really good point about checking for conflicts with other monitoring libraries. I'm not running anything like that currently, but your experience makes me wonder if there's something else at the shared namespace level I haven't considered, like a custom callback or a context manager that's altering state.

Your suggestion for a dead-simple test run is logical. I suppose if that passes, the next step would be to systematically strip down the actual training loop, removing pieces until the logging mismatch disappears. It's a tedious process, but it seems like the only way to isolate the interaction.

Have you found any reliable way to inspect what the W&B client actually has in its buffer at the exact moment it calls its log function, versus what your local print statement sees? I'm thinking some kind of synchronous, blocking debug step might be necessary.