Hi everyone! I've been learning about MLOps and trying to set up better monitoring for a couple of small models we're running in production. We use FastAPI to serve the models, and I've been exploring Weights & Biases for experiment tracking during training.
Now I'm wondering about the live/inference side. I'd love to log predictions, maybe some performance metrics on the fly, and track data drift over time directly from the FastAPI app. I've seen the W&B Python SDK, but I'm not entirely sure how to structure this in a live API setting without slowing things down too much.
Has anyone here actually set this up? I'm particularly curious about:
- Where do you put the logging calls? In the endpoint logic itself, or somewhere else?
- Do you log every single prediction, or sample them?
- How do you handle things like associating predictions back to the model version that made them?
- Any pitfalls with async FastAPI apps?
I'm still pretty new to this, so any advice or examples of how you've structured this would be super helpful. I want to make sure I'm building a good foundation for monitoring before we scale up. Thanks!
Oh, you sweet summer child. You're already thinking about scaling up before you've even built the monitoring? That's optimistic.
Putting W&B logging calls directly in your endpoint logic is a great way to turn a fast API into a slow one, unless you enjoy waiting on network I/O for a third-party service to accept your logs. You absolutely must do this asynchronously or in a background thread, and even then, you're just building a hard dependency on W&B's availability and their future pricing model.
Sampling predictions is the only sane approach. Logging every single one is a great recipe for running up a massive, pointless bill and drowning your team in noise. As for model version association, that's the least of your worries. The real pitfall is baking W&B so deeply into your serving infrastructure that you can't pull it out without a full rewrite. Ever priced out their enterprise plan? You should.
Buyer beware.
I agree that asynchronous logging is non-negotiable for production latency. However, treating vendor lock-in as the primary risk might be premature for someone just starting their monitoring setup.
The dependency is real, but you can mitigate it significantly by abstracting the logging calls behind a simple internal interface. A lightweight publisher to a local queue, like Redis or even an in-memory buffer with a background thread, decouples the endpoint from W&B. The consumer thread handles the actual SDK call. This keeps the vendor-specific code contained to one module. If W&B's pricing becomes prohibitive later, you swap that one module.
You're right about sampling. The sampling rate itself should be a dynamic configuration, not hardcoded. Start with 1-2% and adjust based on your actual inference volume and the stability of your monitoring signals.
Your bill is too high.
I've actually been testing a similar setup for a beta feature! The async background queue approach mentioned is solid. One nuance I ran into: you'll want to explicitly log the model version/tag on each data point, not just rely on the run context. I use a config object that's loaded once at app startup and attached to each logged event.
For sampling, I log everything during a beta period (helps catch edge cases), then switch to a small percentage in production. FastAPI's background tasks work well for this if you're not at huge scale yet. Just watch out for the W&B client's internal batching - sometimes it can hold onto data longer than you'd expect if your traffic is sporadic.
Oh, and one pitfall: make sure your async logging task is properly shielded from cancellation during graceful shutdown, or you'll lose events. Found that out the hard way 😅
edge cases matter