Hey everyone. I've been working with a couple of clients recently who have moved beyond just experiment tracking with Weights & Biases and are now looking to monitor live model performance in production. The goal is to capture drift, performance metrics on real inferences, and log prediction distributions—all flowing into the same W&B project for a unified view.
One approach that's come up is integrating W&B directly into a FastAPI inference service. The idea seems straightforward on paper: you'd initialize a run or maybe use a W&B Table within your `/predict` endpoint to log inputs, outputs, and calculated metrics. But I'm thinking about the practicalities:
* **Performance & Overhead:** How does the W&B logging call (e.g., `wandb.log()`) impact latency in a live endpoint? Should it be asynchronous or batched?
* **Architecture:** Is it better to log from within the API service, or should the service emit events to a queue and have a separate consumer handle the W&B logging?
* **Run Management:** Do you keep a single long-running "production monitor" run, or create a new run periodically (e.g., daily)? How do you handle W&B initialization in a long-lived process?
I'm leaning towards a separate background thread or a lightweight queue (like Redis) within the FastAPI app to decouple the monitoring from the immediate request/response cycle. But I'm curious if anyone has already walked this path.
Have you built something like this? What were the key design decisions and pitfalls? Did you use the public W&B API directly, or perhaps their SDK in a specific way for this non-training context?
-mike
Integrate or die
Straight logging from the /predict endpoint is a recipe for added latency and potential service disruption. That wandb.log call is synchronous by default.
Your instinct about a queue is correct. Have your FastAPI service emit prediction events to a local message bus, like Redis or a small RabbitMQ setup. A separate worker process consumes and handles the W&B logging. This decouples your serving latency from your monitoring reliability.
For run management, a single long-running run gets messy and can become unwieldy. I've seen teams script a daily run that automatically archives, keeping the project organized. You initialize the run once when the logging worker starts.
—AF
Totally agree on the async approach. We landed on using a lightweight in-memory queue with the `background_tasks` param in FastAPI. It adds the logging event to the queue and returns immediately, and then a separate thread handles the actual wandb.log call. Keeps latency down without needing a separate infrastructure piece like Redis right away.
The daily run is a great idea for organization. What we found tricky is managing the context if the worker crashes and needs to restart - you'd need to fetch that day's run ID to resume logging cleanly.
Always testing the next best thing.