Hey everyone. So my team is starting to get serious about experiment tracking and model management, but there's a big catch: most of us are infra/platform engineers, not ML engineers. We're comfortable with Terraform, AWS, and CI/CD, but we really don't want to manage Python SDKs and call `wandb.init()` in every script.
We've looked at Weights & Biases and it seems powerful, but the deep Python integration feels like a barrier. We need something that can be driven more from the infrastructure side. For example, we want to log metrics, artifacts, and parameters from a shell script or a Docker container's output, maybe via HTTP API or CLI.
Is anyone using alternatives that are more "infrastructure-native"? I'm thinking about tools where the primary interface is a REST API or a CLI tool we can bake into a container, and where the tracking server can be self-hosted easily (maybe even on ECS/EKS).
Something we can call from a Makefile or a CI step would be ideal, like:
```bash
# Pseudo-code of what we imagine
EXP_TRACKING_API="https://our-tracking-server"
curl -X POST "$EXP_TRACKING_API/runs"
-H "Content-Type: application/json"
-d '{"experiment": "training-jan-15", "params": {"lr": 0.01}, "tags": ["aws-batch", "prod"]}'
```
Does this exist? Or are we trying to force a square peg into a round hole? 😅 Any experiences with MLflow in this kind of setup?
Infrastructure engineer at a mid-size fintech, we've run both MLFlow and Kubeflow in prod for about two years now.
1. MLFlow (Self-hosted)
- Fit: Designed for platform teams. The Python SDK is optional; primary API is REST.
- Real pricing: Zero license cost, but you pay for infra. Our tracking server on ECS (4vcpu/16gb) runs about $180/mo. Biggest hidden cost is the artifact store (S3) which can blow up with model churn.
- Deployment: Can be a single container. The OSS server image is ~1.5GB. We deployed via Terraform/ECS in under a day.
- Where it breaks: The built-in UI search is weak for high run counts (slows past ~10k runs). You'll need to query the SQL backend directly.
2. Kubeflow Pipelines (KFP)
- Fit: Kubernetes-native, built for orchestrating multi-step pipelines from YAML.
- Real pricing: Heavy on cluster resources. We saw ~20% overhead on a 64-node EKS cluster just for the control plane.
- Deployment/Integration: This is a platform project, not a drop-in. Took us two months to get a stable deployment. The win is you define experiments as containerized Argo workflows, zero Python SDK calls.
- Honest limitation: Horrible for rapid, iterative notebook-style development. It's for hardened, scheduled training pipelines.
3. DVC (Data Version Control) + DVC Studio
- Fit: Git-centric, CLI-first. Metrics and params are stored in flat files (YAML/JSON) committed to Git.
- Real pricing: Self-hosted DVC is free, DVC Studio is ~$15/user/month for the cloud UI. Hidden cost is managing the DVC remote storage (we use S3).
- Deployment: No server to host for core tracking. You run `dvc exp run` from CI or a container, it logs to files. Studio is optional for visualization.
- Where it wins: Perfect if your team already treats ML as code and lives in Git. No API server to maintain.
4. Neptune.ai
- Fit: Aims at enterprise teams. Has a full REST API and CLI.
- Real pricing: Quoted us ~$399/month for a team of 5 on their self-hosted option. Metering is per user, per month, plus storage overages.
- Integration effort: Their "bring your own storage" setup is straightforward, but the metadata store requires a Postgres DB you manage.
- Where it breaks: The CLI is a wrapper around the API; it's less mature than the Python client. We hit rate-limiting on bulk log ingestion under their standard SLA.
I'd pick MLFlow for your team, specifically for the use case of a simple, API-driven tracking server you can call from shell scripts and containers. The barrier to start is low. If your experiments are already defined as containerized pipelines, look hard at Kubeflow Pipelines. Tell us your average runs per month and whether you need real-time experiment dashboards or just post-hoc analysis.
Oh, I totally get the desire to avoid those Python SDK calls. I'm coming from a similar place.
When you said you want to call it from a Makefile or CI step, it made me think: does MLFlow's REST API handle *everything*? Like, can you start a run, log all your metrics, and then finish it, all with curl commands from a shell script? That would be perfect for our CI/CD pipelines.
Also, for a team that's mostly infra-focused, is the setup and maintenance of something like MLFlow's tracking server a huge time sink? The $180/mo cost user311 mentioned seems okay, but I'm worried about the hidden operational overhead.