Best way to track GPU utilization alongside wandb metrics

I’ve been using wandb for experiment tracking on some deep learning training runs, and it’s great for metrics and hyperparameters. But I’m hitting a wall when trying to correlate my model performance with actual hardware utilization—specifically GPU usage, memory, and temperature.

My current setup logs to wandb from a PyTorch training loop. I know I can shell out to `nvidia-smi` or use `pynvml` to poll GPU stats, but I’m curious about the cleanest way to pipe that data into the same wandb run. Has anyone built a reliable pattern for this?

Some things I’ve considered:
- A background thread that samples GPU metrics and logs them via `wandb.log` at a set interval.
- Using a custom callback in the training framework that fetches GPU stats each epoch.
- Maybe there’s a lightweight library or wandb integration I’ve missed?

Ideally, I’d like to see GPU utilization and memory plotted alongside my loss curves in the wandb dashboard. Here’s a quick snippet of what I’m tinkering with:

```python
import wandb
import pynvml

def log_gpu_stats(step):
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
util = pynvml.nvmlDeviceGetUtilizationRates(handle)
mem = pynvml.nvmlDeviceGetMemoryInfo(handle)
wandb.log({
"gpu_util": util.gpu,
"gpu_mem_used": mem.used / 1e9,
"gpu_mem_total": mem.total / 1e9
}, step=step)
```

But I’m worried about overhead and proper cleanup. How are you all handling this? Any pitfalls to watch for, like metrics spamming or conflicts with wandb’s internal sync?

--diver