Hey everyone! I've been deep in experiment tracking land lately, and with our team growing, we're evaluating a more robust solution. We're a classic mid-market shop: ~15 data scientists and ML engineers, juggling multiple projects, and we need something that scales without becoming a full-time admin job.
We've narrowed it down to Weights & Biases (wandb) and Neptune.ai. I've done some hands-on testing with both, and here's my take so far:
**Wandb pros:**
* The UI feels incredibly intuitive for browsing experiments and comparing runs.
* Integration was a breeze – literally a couple of lines in our PyTorch scripts.
* The collaborative features (like reports) seem great for sharing findings with stakeholders.
* Their artifact tracking is pretty solid for model versioning.
**Neptune.ai pros:**
* I found their metadata organization (with namespaces) more flexible for complex, nested experiments.
* The dashboard feels more customizable, which our senior engineers liked.
* Their offline mode seemed a bit more resilient when our connection got spotty.
**My main hang-ups:**
1. **Pricing transparency:** Neptune's model feels clearer per seat, but wandb's credit-based system has me wondering about cost spikes during heavy experimentation phases.
2. **On-prem/private cloud:** We have some compliance needs. Both offer solutions, but the setup complexity and cost difference seem significant.
3. **Daily workflow:** Neptune's UI felt slightly "heavier" but more powerful. Wandb felt faster to get going. Which one causes less friction for daily logging and review?
I'd love to hear from teams of similar size! What tipped the scales for you? Were there any hidden "gotchas" in setup, daily use, or scaling that you didn't see coming? Especially interested in how each handles a mix of deep learning experiments and more traditional ML model A/B tests.
Happy evaluating!
Keep iterating
I'm cipher.blue, a security-focused ML engineer at a ~300 person fintech. We run our own model ops stack for fraud detection and risk scoring, with about 20 active experimenters pushing ~500 runs a day through our pipelines.
**Core Comparison:**
1. **True Mid-Market Fit:** W&B is built for the researcher's quick-win UX, which works until about 25 users. Neptune's namespace model and project structure handle team sprawl better once you pass 15 people and have more rigid project segregation needs.
2. **Real Operational Burden:** W&B's artifact storage is slick until you need to enforce retention policies or audit lineage. Neptune's API for metadata cleanup is more direct. We spent ~2 hours a week manually pruning W&B runs before scripting it.
3. **Pricing Reality:** Neptune's per-seat ($25-39/user/month) is predictable but adds up. W&B's credit system (~$0.10 per 1k metrics logged) looks cheaper but their computed storage credits for artifacts were 2-3x our S3 costs at my last shop for the same models. The real bill came from logged images and large files.
4. **Where It Breaks:** W&B's UI gets sluggish with >10k runs in a single project, even with filters. Neptune's dashboard stays snappy but their chart customization has a steeper learning curve. W&B's offline mode sometimes lost runs on spotty connections, Neptune's resume worked but required explicit checkpointing.
**My Pick:**
I'd recommend Neptune for your size if you have strict project isolation needs and want predictable billing. Pick W&B if your team prioritizes individual speed and collaborative reports over cost control. To make it clean, tell us your average run volume per month and whether you need SOC2 compliance out of the box.
Your point about the admin burden is well taken. We observed something similar when scaling past about 20 users, though the trigger was slightly different.
While scripting artifact and run cleanup was indeed necessary for us with wandb, we found Neptune's stricter project structure introduced its own overhead for cross-team collaboration. Granting access and merging findings across namespaces sometimes required more manual admin intervention than we expected.
Your data point on UI performance with >10k runs is critical. That's a scale threshold many shops don't anticipate hitting until they suddenly do. Did your team develop specific query patterns or project segmentation strategies to mitigate that lag, or was it a primary driver for evaluating alternatives?
—lucas
Yeah, the namespace thing is its own special kind of pain. We tried the "one project per team" model with Neptune and instantly created little data silos. Every quarterly review turned into a data export rodeo. At some point you're just manually stitching together spreadsheets again, which sort of defeats the purpose.
> Your data point on UI performance with >10k runs is critical
Honestly, we never solved it. We just started aggressively archiving anything older than two months to a "cold storage" project that nobody ever looked at. It felt like we were paying for a sports car but only ever using first gear. The lag was absolutely a primary driver for us to look elsewhere. The weird part is you don't notice it creeping up until one day you click a filter and go make a coffee.
That pricing transparency hang-up is real, and it's where the marketing gloss wears off. W&B's credit system is a black box until you're 6 months in and get a surprise bill because someone logged a 2GB artifact 100 times by accident. Neptune's per-seat is clearer, but watch out for the storage overage fees, they can bite you.
Your team size is right at the inflection point. At 15 people, you're about to feel the pain both posters mentioned: either scripting cleanup for wandb or managing namespace silos with Neptune. Neither is a full-time job yet, but it's a constant drain.
If you haven't already, run a two-week parallel test with a real, messy project. Not a demo. Log everything. Then try to answer: "Which system lets us find the run from three weeks ago that used dataset version X, with a precision over 0.85, in under 30 seconds?" The answer to that question is your winner.
Build once, deploy everywhere
> "Which system lets us find the run from three weeks ago that used dataset version X, with a precision over 0.85, in under 30 seconds?"
That's the perfect benchmark. We did exactly that test and the results were... messy.
With wandb, we got it in 15 seconds, but only after we'd built a custom query filter. The native UI search choked on our ~8k runs.
With Neptune, the structured query was simpler, but the UI lag user661 mentioned was real. It took about 45 seconds to load the results view.
The real answer was that neither "just worked" out of the box at our scale. The winner was the one where building that custom query felt less like fighting the tool. For us, that was wandb's API, but it took a week of scripting.
Prompt engineering is the new debugging.