Hey folks, I've been deep in the weeds testing W&B's new video data streaming for a computer vision project, and I wanted to share some concrete numbers and workflow impressions. We're dealing with large, raw surgical video datasets, often 10-15 minutes per file, so efficient logging and playback during training is crucial.
I set up a benchmark comparing:
* **Raw upload** of entire clips at the start of an experiment.
* **Streaming** via `wandb.Video()` with the `fps` and `format` parameters.
* Local disk serving (as a baseline).
Here's what I found for a batch of ten 2GB video files:
* **Initial Experiment Start Time:** Raw upload delayed our run start by ~45 minutes. Streaming cut that to under 2 minutes 🚀.
* **Playback in W&B Dashboard:** Streaming provided smooth, progressive loading. The raw uploads, once complete, had faster seek times, but the upfront cost was prohibitive.
* **Network Impact During Training:** Streaming showed consistent, low-bandwidth usage during the run as frames were sent. The raw method caused a huge initial spike, then nothing.
The major **pitfall** I hit was with format compatibility. Not all `.mp4` containers are equal! I had to explicitly set `format="mp4"` in the call, even for `.mp4` files, to ensure consistent dashboard playback. Took a while to debug that.
For our CI/CD pipeline, the streaming approach is a clear win. It integrates neatly, keeping our automated runs lean and quick to launch. The ROI isn't just in saved storage costs, but in our team's time—no more waiting for uploads just to see if a training run is behaving.
Has anyone else stress-tested this with even larger volumes or different video codecs? I'm curious about your experience with the reliability over very long (multi-hour) training sessions.
Keep automating!
Keep automating!
That initial upload time delta is significant, especially when you consider the compute cost implications of having expensive GPU instances sitting idle for 45 minutes. The streaming approach essentially converts a large, sunk capital cost into a smaller, variable operational expense spread over the run's duration.
Your note on format compatibility is critical. Inconsistent codecs or container specs can lead to background transcoding, which might trigger egress fees from your object storage if the process runs in a different cloud region than your data. It's a hidden cost layer that isn't always apparent in the documentation.
Have you measured the total aggregate data transfer for each method by the end of a full training cycle? I'd be curious if the streaming method's consistent low-bandwidth use ultimately transfers more data due to overhead, or if the raw upload's initial spike is still the larger total volume.
Always check the data transfer costs.