Just logged into the W&B project dashboard this morning and spotted the new artifact "streaming" toggle. This is a game-changer for anyone working with massive datasets or model checkpoints.
I was pulling my hair out last quarter trying to log a set of large pre-trained embeddings for a retrieval project. The upload kept timing out, and I ended up having to script a manual chunk-and-sync process to our own S3, which totally broke the experiment lineage. Now, with this streaming support, you can essentially register a file or directory from your cloud storage (S3, GCS, etc.) as an artifact without moving the data. The metadata and lineage are tracked in W&B, but the heavy bits stay put.
For my Salesforce integration work, this could be huge. Imagine streaming large, anonymized customer data exports directly from a cloud bucket for model training, while keeping everything versioned and linked to the training run. No more middleman copies.
Has anyone tried this in a production pipeline yet? I'm curious about the real-world performance when you have, say, hundreds of concurrent runs all trying to stream artifacts from the same bucket. Also, wondering if the pricing model around this is different from the standard artifact storage.
Hold on. "Game-changer"? You're celebrating a feature that essentially lets you use your own expensive cloud storage to solve their upload problem, and they get to keep charging you for the metadata layer. That's not innovation, that's a workaround they're monetizing.
Your real question about concurrent runs and pricing is the only sane part of this. I guarantee the answer is in the fine print of the enterprise tier, probably with a new metric like "streamed artifact references" that's exempt from the lower-tier caps. They didn't fix the cost of moving data, they just moved the cost center back onto your infrastructure bill while adding their own surcharge for the privilege.
Has anyone checked if egress fees from your S3 to their compute nodes during a run are now your problem too? Because that's the hidden trap here.
Show me the TCO.