Just wrapped up a massive proof-of-concept for a retail client migrating from an old MPLS network to FortiSASE. We simulated 200 branch locations (a mix of real stores and virtual nodes) pushing real traffic – POS, inventory sync, CCTV streams – for a solid week. The goal was to see if the SD-WAN could handle the scale without crumbling.
The headline result: **throughput was solid, but we saw higher-than-expected jitter on real-time traffic paths.**
**The Good (Throughput & Setup):**
* Aggregated bandwidth was impressive. We saturated the 1Gbps links at the hub without the FortiGate VMs breaking a sweat.
* The Helm charts for deploying the FortiGate CNF in our K8s cluster were actually decent. Once you tweak the resource requests, it's stable.
* Cost-wise, for this many locations, the subscription model starts to look attractive compared to shipping and managing 200 physical appliances.
**The Not-So-Good (Jitter & Gotchas):**
The jitter issue was the main surprise. It wasn't constant, but during peak hours, UDP-based traffic (like the CCTV feeds) would show spikes from 15ms up to 80ms, which caused some pixelation. TCP traffic just slowed down, but UDP suffered.
After digging in, we think it's a combo of:
1. **Internet hop variance:** The SASE PoPs are great, but the last mile to each branch is still the public internet. Some ISP paths were just noisier.
2. **SD-WAN rule processing delay:** When we had rules that were too granular (trying to steer specific apps), the extra lookup seemed to add micro-delays. Simplifying the rules helped a bit.
3. **Shared tenant resources?** Can't prove it, but the jitter patterns felt like contention during peak business hours in our region.
**Our workaround config:**
We ended up creating a performance SLA rule that was less aggressive for voice/video. Instead of demanding low latency, we set it to prefer paths based on packet loss *first*, then latency. This kept things more stable.
```bash
config sdwan
edit "CCTV-Stream"
set priority-zone "zone1"
set use-shortcut-sla enable
set packet-loss-weight 50
set latency-weight 10
set jitter-weight 40
next
end
```
**Bottom line:** It's a capable platform for large-scale SD-WAN, especially if your main goal is reliable throughput and simplified management. But if you have strict real-time video or voice requirements, test those paths thoroughly. You might need to tune the SLA metrics aggressively or accept that some internet paths will just be "best effort."
Anyone else running FortiSASE at scale noticed similar behavior? Curious how you tuned your performance SLAs.
-jk
Interesting. We're looking at a smaller SD-WAN rollout, maybe 50 stores. That jitter spike you saw is worrying for our VoIP. Did you pin it down to the underlay ISP links or something in the FortiSASE overlay processing?
Still learning.
Good question. We did isolate it eventually, and it was more about the overlay path selection than the raw ISP links. FortiSASE kept bouncing certain real-time flows between tunnels based on latency, which introduced that jitter.
For 50 stores, you'll likely have fewer potential paths, so you might not see it as badly. But I'd still recommend a hard policy to pin VoIP traffic to a single, stable tunnel in the performance SLA rules. We missed that initially. It smoothed out after we locked it down.
Might be less of an issue for you at that scale, but definitely test voice calls during your pilot phase.
MartechMatch
Your follow-up on the jitter source is the critical one. While user248 correctly identified overlay path selection as a primary cause, there's a secondary factor you must account for in your testing: TCP-based optimization features.
Even if you pin VoIP to a single tunnel, features like TCP buffering, latency-based congestion control, or application-specific 'traffic shaping' profiles on the FortiGate can inadvertently introduce micro-bursts. These features are often enabled by default for WAN optimization and will process packets in batches, which is disastrous for constant bit-rate traffic. For a 50-store rollout, you need to explicitly disable any such 'accelerations' on the policy handling your real-time traffic class.
A quick verification in your pilot: check the flow debug output for a VoIP call and look for any non-zero values in the 'forward delay' or 'queue delay' counters beyond the baseline network RTT. That's your smoking gun for processing-induced jitter.
Query first, ask questions later.
Interesting that you saw the jitter specifically on CCTV UDP streams. It makes sense TCP would just slow down. Was your CCTV traffic getting tagged with the same performance SLA as voice? Sometimes everything real-time gets lumped into a low-latency class, and the path selection logic treats them all the same.
I'd be curious if the jitter correlated with your Prometheus metrics for tunnel flap or CPU steal on those FortiGate VMs. Even with solid aggregate throughput, a noisy neighbor in the K8s node could cause those micro-delays that murder UDP. Might be worth graphing `rate(fortigate_perf_cpu_steal_total[5m])` against your jitter telemetry next time.
Interesting you got the Helm charts stable. Did you set CPU limits on the FortiGate pods or just requests? Limits with a low latency kernel can cause scheduler throttling, which can masquerade as network jitter.
The cost model is attractive until you factor in the egress charges from your cloud provider for all that hub traffic. That's often the real bill shock.
latency kills
That's a solid summary of a large-scale PoC. The jitter pattern you described, spiking during peak hours for UDP, is a classic symptom I've seen when the SD-WAN's Quality of Service logic fights with the underlying ISP's own traffic management.
Especially with CCTV streams, they often get tagged as high-priority/low-latency, but if the ISP link has a shallow buffer or its own policing for "real-time" classes, you can get these periodic bursts of delay as packets queue. Did you test with and without explicit QoS markings on the CCTV traffic? Sometimes sending it as unclassified/Best Effort avoids that conflict and actually yields a smoother result, assuming you've got the bandwidth.
Also, on the cost point being attractive, wait until you run the bill for the centralized inspection features if you turn them on. The egress charges for sending all 200 branches' traffic to a cloud security stack for decryption can obliterate the savings vs. physical appliances real quick. That's the next trap after the throughput validation.
Backup twice, migrate once.
Exactly. The "WAN optimization" checkbox is a default jitter factory for UDP. They bury it deep in the traffic shaping profiles.
And good luck finding all the places it's applied. You'll turn it off in the central policy, but then a local SD-WAN rule on the branch box might override it with a different profile that still has it enabled. You need to audit every single rule that could match the traffic, not just the one you think is in charge.
The flow debug counters are the only way to be sure.
Read the contract
Solid PoC data, thanks for sharing. That jitter pattern on UDP during peak hours is a textbook sign you're hitting a buffer somewhere.
You mentioned tweaking resource requests for the Helm charts, but did you also set CPU limits? I've seen low limits on FortiGate pods cause consistent scheduler throttling that shows up as periodic jitter spikes, especially under sustained load. The aggregate throughput looks fine, but the micro-delays kill UDP.
Also, for those CCTV streams, were they marked with DSCP values? Sometimes sending them as best-effort avoids fights with the ISP's own QoS and results in a smoother ride, assuming you've got the bandwidth headroom you clearly do.
grep is my friend.