I've been digging through a lot of documentation and conference talks, and there's clearly a huge focus on using Weights & Biases for tracking supervised learning experiments. The dashboards for hyperparameters, metrics, and model artifacts look great for that.
But I'm specifically evaluating tools for a production RL pipeline. We're moving beyond toy environments to something with continuous training, where new environment interactions generate fresh training data daily. The cycle of "collect experience -> update policy -> deploy -> repeat" feels like a different beast than running a grid search on a static dataset.
My question is for teams actually running this kind of loop in production. Are you using wandb for RL, and if so, how?
I'm particularly curious about:
- How you handle the sheer volume of trajectories/rollouts. Do you log summarized metrics per episode only, or are you sampling trajectories for artifacts?
- How you track the non-stationarity of the training process. In supervised learning, a validation loss curve tells a story, but in RL, the "validation" environment itself can be a moving target.
- Whether you're using the model registry to version and promote policies, and if that workflow integrates smoothly with your deployment system.
- Any pain points with the off-policy logging, or if you've found workarounds for tracking things like exploration rate decay or value function estimates over millions of steps.
Most examples I find are for discrete, episodic tasks like Atari or CartPole. I'd love to hear about experiences with longer-running, more complex environments.
We use it, but you've hit on the real friction points. The volume of raw trajectories is too much, so we log summarized metrics per episode and only log a sampled trajectory artifact at checkpoints (e.g., every 1000 episodes). It's a compromise, but it keeps the dashboard usable.
For tracking non-stationarity, we treat our main production environment as the "validation" target and log a separate, stable evaluation metric from a simplified, frozen test environment. Seeing the gap between those two curves tells you a lot about policy overfitting to the current data distribution.
The model registry works for versioning policies, but our promotion logic lives outside wandb in our CI/CD. We use the registry more as a historical record and a way to pull a specific model artifact for a rollback. It's not the automatic promotion gate some teams might want.
automate the boring stuff