Setting up W&B offline in a locked-down environment is a pain if you don't plan ahead. Did it for a government project last year. Here's the stripped-down guide.
You need to handle two parts: the server and the client SDK.
* Server: Deploy a local W&B server (wandb/local) or use their on-prem container. Pull the image, load it into your registry.
* Client: The critical step is pre-downloading the artifacts.
Key steps for the client side:
* On an internet-connected machine, create a virtual env matching your air-gapped specs.
* Install wandb and run `wandb artifact get` on all necessary datasets/models you'll need.
* The artifacts cache (~/.cache/wandb by default) is what you need to transfer. Tar it up.
* In the air-gapped environment, set `WANDB_MODE=offline` and `WANDB_DIR` to a local path.
* Point your training scripts to the local server URL with `wandb.init(settings={"base_url": "http://your-local-server:8080"})`.
Biggest pitfalls:
* Missing artifact dependencies will break runs.
* Network proxies can still interfere even in offline mode; you might need to set `http_proxy=""`.
* Version compatibility between the local server and the Python SDK must be exact.
Yeah, "version compatibility between the local server and the Python SDK" is the silent killer they always gloss over. Their container image tags get out of sync with PyPI releases constantly.
I had a run brick itself because the local server's artifact manifest schema was a patch ahead of the SDK in the air-gapped box. The logs just spat out a generic 500 error. Took two days to trace it back to a mismatched minor version we'd assumed was safe.
All that careful artifact caching and proxy configuration, undone by a docker pull you did a week before freezing the client environment. It makes you wonder if the complexity is worth it over just logging to a local SQLite file and syncing later.
prove it to me
Your point about the artifact cache is solid, but you're understating the nightmare of getting that tar from the connected machine into the secure environment. If you're dealing with actual air-gapped systems, you're not just copying a file, you're navigating a Transfer-Of-Files (TOF) protocol that usually involves burning, verifying, and manually transporting read-only media. I've seen the artifact cache for a modest project blow past 40GB, which turns a simple `scp` fantasy into a multi-day logistics operation with checksum verification across the gap. The guide should start with: "Step zero: get your data transfer SOP signed off by security, because your .tar is now classified as 'imported data'."
APIs are not magic.