Hey folks! I’ve been using W&B for tracking training metrics religiously—it’s a game-changer for experiments. But I’ve hit a wall trying to adapt it for monitoring inference performance in our FastAPI services. Everyone talks about training loss and accuracy, but what about after deployment? 😅
I need to log latency percentiles (p50, p90, p99) and throughput (requests/sec) for our model endpoints, ideally in real-time. I’ve seen the `wandb.log` approach during training loops, but inference happens in a totally different environment—often across multiple containers. How are you all structuring this?
Here’s a tiny snippet of what I’m attempting in a FastAPI middleware:
```python
import time
import wandb
@app.middleware("http")
async def log_inference_metrics(request: Request, call_next):
start_time = time.perf_counter()
response = await call_next(request)
duration = time.perf_counter() - start_time
# This feels clunky—especially in production with high concurrency.
wandb.log({"inference_latency": duration, "request_path": request.url.path})
return response
```
But I’m worried about overwhelming the W&B dashboard and how to aggregate metrics properly. Should I be batching logs? Using a separate run per service instance? And how do you handle tracking throughput over time?
Would love to hear how you’ve set this up, especially if you’re using Dockerized microservices. Any pitfalls with authentication or performance overhead? Let’s share some concrete examples!
~d
Totally feel you on this! That middleware approach can get noisy fast. What I do instead is batch log inferences to a separate run.
I create a dedicated "inference-monitor" run at service startup, then log aggregated metrics every N requests or every minute. Something like this in a background thread:
```python
# Aggregate in memory, then log batch
metrics = {"p50": calculate_percentile(), "rps": requests_per_second}
wandb.log(metrics)
```
Keeps the dashboard clean and lets you see trends over hours/days instead of a flood of points.
Have you considered using W&B's system metrics alongside your custom logs? Could be useful to correlate latency spikes with container memory/CPU.
Pipeline Pilot