Walkthrough: Versioning datasets with W&B Artifacts and DVC.

catherine9 · 2026-08-09T01:01:45Z

A common pain point in machine learning operations is the divergence between experiment tracking systems and data versioning tools, leading to reproducibility gaps. While Weights & Biases excels at tracking model lineage and hyperparameters, and DVC provides robust versioning for large datasets, integrating them effectively requires careful orchestration. This walkthrough details a methodical approach to using W&B Artifacts as the centralized lineage registry while keeping DVC as the underlying versioned storage engine, thereby creating a unified audit trail. The core architecture leverages the concept of external artifact links in W&B. DVC manages the actual dataset files, tracking their versions via its `.dvc` pointers in Git. W&B Artifacts then reference these DVC-tracked files or directories, capturing the precise Git commit hash and DVC pipeline stage. This ensures the W&B run points to an immutable, versioned data state. **Implementation Workflow:** 1. **Version Data with DVC:** Begin by adding your dataset to DVC control. This typically involves initializing DVC in your repository and tracking the data directory. ```bash dvc init dvc add data/raw git add data/raw.dvc .gitignore git commit -m "Track raw dataset with DVC" ``` 2. **Create a W&B Artifact Linked to DVC:** Within your training script or pipeline, you create a W&B Artifact of type `dataset`, but instead of uploading files directly, you specify the DVC-tracked path and link it externally. This is done using the `wandb.Artifact` constructor with `type='dataset'` and adding the directory as a reference. ```python import wandb run = wandb.init(project="dataset-versioning", job_type="data-registration") # Create an artifact that references the DVC-controlled directory artifact = wandb.Artifact(name="raw-dataset", type="dataset") # Add the local directory, which is tracked by DVC artifact.add_dir("data/raw", name="data") # Log the artifact. This records metadata and the link, but does not upload the data. run.log_artifact(artifact) run.finish() ``` The critical detail is that `data/raw` is under DVC control. W&B reads the `.dvc` file and stores a reference to the DVC hash and the corresponding Git commit. 3. **Consume the Versioned Artifact in Downstream Runs:** A subsequent training run can use this artifact as an input, ensuring it uses the exact data version referenced. ```python import wandb run = wandb.init(project="dataset-versioning", job_type="training") # Fetch the artifact by its version alias (e.g., 'latest', 'v1') artifact = run.use_artifact("project/raw-dataset:latest") # Download the artifact's content. This will pull the data from DVC cache. dataset_path = artifact.download() # Proceed with training using data at `dataset_path/data/raw` ``` **Key Advantages & Considerations:** * **Single Source of Truth:** W&B becomes the searchable interface for all experiment lineage, including data versions. You can visually trace which model runs used which dataset artifact. * **Storage Efficiency:** Large dataset files remain in your existing DVC remote storage (S3, GCS, etc.), avoiding duplication into W&B's system. * **Reproducibility:** Any logged artifact can be fully resolved to a Git commit and DVC hash, enabling precise recreation of the data state. * **Operational Overhead:** This setup requires both DVC and W&B to be correctly configured and authenticated with their respective storage backends. Team members must have appropriate access to both systems to fully reproduce a run. * **Pipeline Triggers:** For automated pipelines, changes to data tracked by DVC can be used to trigger new W&B runs, though this requires external orchestration (e.g., using GitHub Actions, Airflow, or DVC itself). This pattern effectively decouples the metadata and lineage tracking from the bulk storage layer, leveraging the strengths of each tool. It is particularly suitable for teams already invested in DVC for data management who require the collaborative and experimental tracking features of W&B.

Page 3 / 3 Prev

Weights and Biases Reviews

Last Post by Emily K. 2 months ago

31 Posts

30 Users

0 Reactions

83 Views

RSS

Emily K.

(@emilyk99)

Estimable Member

Joined: 2 months ago

Posts: 173

14/08/2026 8:30 am

That's a solid starting workflow, but I have a practical question about the initial dvc add step. When you say "adding your dataset to DVC control," is that meant to be done manually for every new raw dataset version, or is it typically automated as part of a pipeline stage? I'm thinking about a scenario where the raw data itself is regularly updated from an external source. Would you manually run `dvc add data/raw` each time, or would you have a dvc.yaml stage that both fetches the new data and then automatically tracks it? The manual approach seems prone to human error, but I'm not sure how to bake the versioning into an automated fetch without creating a circular dependency.

ReplyQuote

Page 3 / 3 Prev

Forum Icons: Forum contains no unread posts Forum contains unread posts

Topic Icons: Not Replied Replied Active Hot Sticky Unapproved Solved Private Closed