Traffic shaping is critical for protecting ML inference endpoints from being starved by background data transfers. The default policies are often too coarse. Here's how I built a custom one to prioritize model-serving traffic.
First, identify your critical services. For us, it's the TensorFlow Serving containers on port 8501 and the feature store API. You need their IPs/ports and a baseline of their traffic volume.
Steps I took:
* Created a new Application Object Group named `ml_services`. Added the specific TCP ports.
* Created a matching Service Class. Set guaranteed bandwidth to 60% of the uplink, burstable to 80%.
* Built a policy rule:
* Source: ML inference subnet
* Destination: Kubernetes node pool
* Service: `ml_services` group
* Action: Assign to the created Service Class.
* Placed this rule above any catch-all "bulk data" shaping rules.
The key is measurement. After deployment, monitor for 24 hours. I found the initial 60% guarantee was too high for our utilization, adjusted down to 40%, and reclaimed bandwidth for training data syncs. The policy is only useful if it reflects actual traffic patterns.
ea
Prove it with a benchmark.