Real experience deploying Weights and Biases in a regulated industry

Having recently completed an 18-month deployment of Weights and Biases (W&B) within a financial services environment subject to FINRA and SEC oversight, I believe there is a significant gap between the platform's advertised capabilities and the realities of operating in a regulated space. While the tool excels in agile research settings, its architectural assumptions can create substantial friction when compliance, data sovereignty, and strict change management are non-negotiable requirements.

Our primary objectives were to establish an auditable trail for model development, enforce dataset versioning, and control PII exposure, all while allowing data scientists to maintain their existing workflows. The deployment involved several key modifications and workarounds:

* **On-Premises Artefact Storage:** The default cloud storage (S3/GCP) was unacceptable. We configured W&B to use an internal, air-gapped S3-compatible object store (MinIO) for run artefacts, models, and media. This required careful network provisioning and security group configurations to allow the W&B servers to communicate with the storage layer while remaining isolated from the internet.
```yaml
# wandb server configuration snippet (helm values override)
wandb:
settings:
storage:
artifactStore: "s3://wandb-artifacts"
backupStorage: "s3://wandb-backups"
s3:
endpoint: "https://minio.internal.example.com"
bucket: "wandb-artifacts"
region: "us-east-1"
s3ForcePathStyle: true
```
* **Database Isolation:** We opted for a fully isolated deployment, hosting the PostgreSQL metadata database internally. This introduced challenges around performance tuning and high-availability setups that are abstracted away in the managed SaaS offering. Regular pg_dump backups and point-in-time recovery procedures became our responsibility.
* **Authentication & Authorization (AuthN/AuthZ):** Integrating with our corporate SAML 2.0 IdP (Okta) was straightforward. However, W&B's native team/project permission model proved too coarse-grained. We had to implement additional proxy-layer checks to enforce dataset access policies based on internal user groups, which added complexity.
* **The "Black Box" of the Sweep Controller:** For hyperparameter tuning, the sweep controller logic runs within W&B's managed infrastructure, even in a private cloud deployment. This was a major point of contention with our infosec team, as it meant metadata about our tuning jobs (parameter ranges, early decisions) traversed a boundary we could not audit. We ultimately disabled sweeps and built an internal, instrumented alternative using Kubeflow Pipelines.

**Performance and Operational Overheads:**
The self-hosted W&B server (via the provided Helm charts) is resource-intensive. Under load from ~50 concurrent active users, we observed significant memory pressure on the `wandb` pod, requiring vertical scaling. Latency for artifact logging was also highly variable, dependent on our internal object store's performance, which necessitated a dedicated performance baseline and monitoring.

**Conclusion for Regulated Environments:**
W&B can be made to function, but it is not a turnkey solution for regulated industries. The total cost of ownership extends far beyond licensing to include significant DevOps and security engineering effort to bridge the compliance gaps. The experience underscores a broader industry need: MLOps platforms designed from the ground up with a "sovereign-first" architecture, where all components, including orchestration logic, can be fully internalized without loss of functionality. For teams without a dedicated platform engineering group, this undertaking is likely prohibitive.