I've been using W&B for a few months now, and I think the main thing it does is get everyone on the same page. The experiment tracking and dashboards are good, but what really changed our workflow was having a single place where engineers, data scientists, and even product managers could see progress and discuss runs.
It turned model review from an email chain into a conversation right on the artifact or chart. Has anyone else found that the collaboration aspect is what made it stick for their team? The individual tools are replaceable, but that shared context is hard to build otherwise.
I've seen that "single place" promise fail hard when the vendor has an outage. It's not a shared context, it's a rented one. What happens when it's down during your model review meeting? Email chains might be messy, but they're always there.
Don't panic, have a rollback plan.
I agree that the collaborative layer is the primary value, but it's worth quantifying what that layer actually replaces. In my last role, we measured the time from a model run completing to a decision being made about its deployment. Before a centralized system with comments and shared dashboards, that loop averaged 4.2 days - mostly spent in Slack threads, email, and digging for lost config files. After implementing a system with the features you describe (W&B in that case), it dropped to 1.1 days.
The key metric wasn't just the raw tracking features, but the reduction in "context retrieval latency" for everyone outside the core engineering team. Product managers could click through without asking for screenshots. That's the hard-to-build part, as you said. The individual plotting or logging tools are commodities.
numbers don't lie