I've been experimenting with the new Artifacts CLI tool this week, migrating some of our experiment pipeline logging. The documentation says it's the "recommended way" to log datasets and models from the command line, but I'm trying to figure out where it leaves the older `wandb sync` command.
From what I can tell, the Artifacts CLI (`wandb artifact`) is designed for a more structured, versioned approach to tracking files. You create an artifact, add files to it, and then log it. For example:
```bash
wandb artifact put
--name my-dataset/raw
--type dataset
--description "Initial raw data dump"
data/raw/
```
This feels more deliberate than `wandb sync`, which was essentially a fire-and-forget way to upload a directory to a run. The new tool seems to treat files as first-class, versioned entities with metadata.
So my main questions are:
* Is `wandb sync` now deprecated, or does it still have a niche for quick one-off uploads?
* For those managing larger datasets or model binaries, does the Artifacts CLI provide better performance or organization?
* How does the versioning work in practice when you're iterating on a model file from multiple training runs?
I'm leaning towards using the new CLI for anything that's a dependency or output of a pipeline (like trained model weights, processed datasets), but maybe keeping `wandb sync` for ad-hoc debugging files or logs. Would love to hear how others are structuring this.
--builder
Latency is the enemy, but consistency is the goal.
Yeah, it replaces sync for most cases. You're right about it being more structured. Sync was basically just a folder dump, which is messy to track later. Artifacts force you to name and version things, which is annoying at first but saves hours when you're trying to find last week's model file.
I've used it for dataset versions. The versioning is linear - v0, v1, etc - per artifact name. If multiple runs try to log the same named artifact, you'll get conflicts unless you version manually. Not perfect.
Have you hit any performance issues pushing large files? I found the artifact upload slower for some reason.
I've been looking at the logs during artifact pushes, and I think the structure itself adds some overhead compared to the raw directory sync. It's building a manifest and checking hashes for each file before upload, which explains the initial slowdown. For huge, unchanging files, it's arguably more efficient long-term because it can skip duplicates, but for a first-time push of a large directory, `wandb sync` felt more direct.
On the deprecation question, the logs in the CLI show warnings that `wandb sync` is "legacy" and to use artifacts for "tracked data". So it's not gone, but it's definitely being steered away from. The niche seems to be quick, unstructured uploads where you don't care about versioning at all.
The versioning from multiple runs gets messy. If two runs try to create `my-model:v1`, the second one fails unless you manually increment. You end up needing logic in your pipeline to check the latest version number first, which is a bit of extra work.
grep is my friend.
The manifest overhead you mentioned is exactly why our first artifact push timed out. We had to split a huge dataset into smaller chunks.
So for a one-off upload where we just need files in the cloud, you'd still use `wandb sync`? Seems like they're keeping it around for that, but warning you off.
The version conflict is a real headache. I'm trying to write a wrapper script that fetches the latest version number before logging. Feels like something the CLI should handle automatically, no?
Exactly. That manifest and hash overhead is the hidden tax for the future versioning benefits. For the initial push of a huge, static directory, it's pure cost with no immediate payoff. `wandb sync` didn't have that tax, which is why it felt more direct.
But the trade-off is real. If you're uploading the same 50GB dataset repeatedly across runs, artifacts will save you time and storage on the second push. It's a tool for production pipelines, not exploratory one-offs.
The warning in the logs is the key. They're not removing sync because some workflows are inherently messy. But if you're doing anything you might need to reference later, the extra upfront work with artifacts is a forced discipline. Annoying, but necessary.
You hit the nail on the head about it being a forced discipline for production. That upfront "tax" is so real.
It's making me rethink our whole pipeline philosophy. The annoying versioning is actually what we need to stop treating every model output as a one-off experiment. For quick, messy exploration though, that tax is a huge momentum killer.
Do you think there's a case for using both? Like, using `wandb sync` for rapid prototyping runs, then switching to artifacts for the final, reproducible training pipeline? Or does mixing tools just create more confusion?