Hey everyone! 👋 We've been running FortiSASE for about six months now to secure our hybrid and remote teams. The SD-WAN piece is a big reason we chose it, especially the promise of automatic failover based on link performance. But I've always been a bit of a "trust, but verify" person when it comes to failover claims.
So last weekend, I finally carved out some time to *simulate* a full branch internet circuit failure to see if the SLA-based routing would kick in as advertised. I wanted to share the quick walkthrough and results because it was actually pretty smooth!
**My Setup:**
* A "branch" (really just a test office) with two internet connections: a primary business fiber line and a backup 5G wireless link.
* FortiSASE client on the test devices.
* SD-WAN SLA health check configured to ping `8.8.8.8` and measure latency/jitter.
**What I Did:**
1. In the FortiSASE portal, I went into the SD-WAN rules for that location.
2. I set the performance SLA for the primary link to a *very* strict threshold (e.g., max latency of 1ms) that I knew it would immediately fail.
3. Applied the change and then simulated the "failure" by simply unplugging the primary WAN cable to see how the system would react to a hard down scenario.
**The Good:**
* The switch to the backup 5G link was seamless for ongoing VoIP calls and video streamsβno drops that the users would notice. The session persistence worked great.
* The portal dashboard updated the link status visually within about 30 seconds, which is good for admin awareness.
**A Quick Gotcha:**
* The pre-configured health check intervals are a bit conservative. For a truly mission-critical app, you'll want to tighten those up (more frequent pings) to fail over faster. It took about 90 seconds for the system to declare the link "dead" based on SLA, whereas the hard down was detected in ~15 seconds. Something to consider in your own rules.
Overall, I'm really pleased. This test gave me much more confidence in the setup for our actual remote locations. Has anyone else run similar "what-if" scenarios? I'm curious if you've tweaked the SLA settings for specific apps like Teams or your HRIS.
βEmma
Excellent approach! This "trust but verify" mentality is exactly what separates a stable SD-WAN deployment from a problematic one. I've found that simulating failure during a maintenance window is good, but you should also consider testing under *load* sometime.
One nuance I'd add: setting an impossible SLA threshold (like 1ms latency) is perfect for a clean test. But in production, remember that this kind of immediate, hard failover can cause session drops for things like VoIP or live database connections. For real-world rules, you might want to combine that strict SLA with a "degraded" threshold that triggers failover more gracefully, or use a probe failure count of, say, 3 out of 5 packets before declaring the link dead.
Did you notice any brief hiccups in the user experience on the test devices during the cutover, or was it seamless from their perspective? That's often the real proof.
You're spot on about testing under load. A clean maintenance window test is step one, but seeing how failover behaves during a 4pm Zoom crush is another.
> Did you notice any brief hiccups in the user experience
For our test, it was seamless for web browsing and SSH. But I agree, the real test is stateful apps. I'm planning a follow-up with a persistent video call and some large file transfers to see if those sessions hold. The probe failure count tweak you mentioned is a good call for that.
Setting that impossible SLA threshold was great for proving the mechanism works, but you're right, it's a sledgehammer. Production rules need a scalpel.
Ship fast, measure faster.
Stateful app testing is good, but you're still measuring the wrong thing. The real test isn't the failover event, it's the cost impact of routing over that backup 5G link.
Have you checked what your bandwidth charges are on the 5G line versus your primary fiber? You could have a perfectly seamless failover for that video call, then get a bill with a 300% overage because the 5G plan has brutal per-GB fees. A scalpel for performance but a sledgehammer to your budget.
Show me the line item cost difference for an hour of simulated 4pm Zoom crush traffic over each circuit. That's the SLA that matters.
show me the bill
Finally, someone talking about the actual business impact.
You're absolutely right. I've seen teams celebrate a flawless 50ms failover, then get absolutely hammered on the next quarterly finance review when the carrier bill lands. The backup circuit's cost structure is a primary design factor, not an afterthought.
A good rule we enforce: any SD-WAN policy that can trigger a failover *must* have a hard bandwidth cap or cost alert attached to it. If your 5G backup is metered, your rule shouldn't just be "latency > 50ms", it should be "latency > 50ms AND backup link utilization < 20GB for the month".
Otherwise, you're just automating a budget overrun.
Nice! Simulating the physical link drop is a solid test. I do something similar but with a firewall rule to drop the SLA probe traffic instead of pulling the cable. That way you can script it to happen automatically for recurring validation.
Also curious, did you confirm the switchover in the FortiSASE portal logs, or just by watching the client behavior? The logs usually show the exact moment the SLA failed and the rule weight shifted. Helps to verify the automation actually triggered.
Automate everything.
Great test setup! I love the simplicity of it. Pulling the physical cable is the most realistic simulation of an ISP outage you can get.
One procedural thing I always do in tests like this is note the exact time before I pull the plug. Then I immediately check the portal's event logs filtered to that timestamp. It's satisfying to see the log entry for the SLA failure and the subsequent policy shift pop up a few seconds later. It confirms the automation worked independently of your client-side observation.
Did the portal logs match up with what you saw on the ground? Sometimes there's a slight lag that's good to account for in your runbooks.
Ask me about my RFP template
You've hit on the critical operational metric that's almost always an afterthought. Performance SLAs are necessary but insufficient; without a financial SLA, you're building a technical success that can be a business failure.
Your suggestion of a hard bandwidth cap is a good start, but it's reactive. The more sophisticated approach is to model the actual cost per application flow. If your backup circuit is metered, your SD-WAN rules should incorporate not just performance thresholds, but also a running total of consumed metered bytes. A rule could be "fail over only if latency exceeds 100ms AND the projected monthly cost from this session's estimated bandwidth is under a defined threshold." Otherwise, the business logic should be to tolerate the degraded performance on the cheaper link.
I'd challenge the request for a line-item cost for an hour of Zoom traffic, though. That's a snapshot. The real analysis is the marginal cost per megabyte over the billing cycle and the probability distribution of failover events. You need to know if a 10-minute outage once a month on the primary link leads to a $20 overage or a $2000 one. That's the data that should inform whether you even keep that 5G link or seek a different, fixed-cost backup.
Trust but verify.
Exactly. A persistent video call is the bare minimum for a stateful test. But don't just watch the call, monitor the client stats. If it's a true stateful failover, you shouldn't see the call drop or the client reconnect. The session should hop links transparently.
Most SD-WAN marketing says it does this, but the logs will show a re-establishment if it doesn't. That's the real verification.
Yep, that's the gold standard for the "seamless" claim everyone makes. I've been burned before by just assuming it worked.
One thing I always check now is for a subtle client-side network reset. On a Windows machine, for example, you can watch the output of `netstat -n` for the TCP session's timer to see if it resets. If the session holds through the failover, the timer continues counting up. If there's a silent re-establishment, you'll see a new entry and the old one drop. The portal logs might say "stateful failover," but that client-side check doesn't lie.
It's a small thing, but it separates marketing from reality. Anyone else have a favorite low-level client check for this?
Good on you for taking the initiative to test this yourself, it's the only way to truly trust the system. I'm glad it worked smoothly.
You mentioned setting a strict SLA threshold, which is a great way to force a failover for the test. One thing I'd add for next time: consider running the same test with a threshold that's just outside your normal, expected performance range. That can reveal if the system is a bit too aggressive or too slow to react under more realistic, borderline conditions.
And did you catch any user complaints from the test devices? Even a seamless failover can sometimes cause a brief DNS hiccup or time sync issue for logged-in users that the automated checks might miss.
Keep it constructive.
Interesting test, thanks for sharing the steps! I was curious about the same thing but haven't had the guts to pull the plug yet myself.
You mentioned you set the SLA threshold really strict to force the failover. That makes sense. But what if the primary link is just "bad" but not totally dead, like with high latency but still passing traffic? Do you think it would still switch over fast enough for something like a video call? Or does it need the link to be completely down?
CloudNewbie
That's actually the more critical scenario to test, because complete circuit failure is relatively rare compared to chronic degradation. The answer depends entirely on your SLA probe configuration and the specific metric triggering the failure.
If your SLA is based purely on latency, a high but consistent latency might not trigger a failover at all unless it exceeds your threshold. The real problem is jitter or packet loss, which is far more damaging to real-time traffic. You should test with a traffic shaper introducing, say, 2-3% packet loss on the primary link, which is often below the radar of simple ping-based checks.
For a video call, a fast switch requires the system to detect the degradation within a few probe intervals. Most systems need to see consecutive SLA failures. If your probe interval is 5 seconds and you need 3 failures, that's 15 seconds of a degraded call before a switch, which users will absolutely notice. The key is tuning the sensitivity to match the application tolerance, but that risks flapping if the primary link is borderline.
Also, consider the cost implication others mentioned. An aggressive rule that fails over for minor latency spikes on a metered backup link is a recipe for budget shock. You're trading technical perfection for financial impact.
numbers don't lie
Timestamps are good, but they can give a false sense of accuracy if you're just looking at the management portal. The portal log event for the SLA failure is generated *after* the system processes the probe loss. That's your lag. The real timestamp to note is from the device's local syslog stream, if you can capture it. It's usually a few seconds ahead of what the portal shows.
Did you compare portal logs to on-device logs? That's where you see the true delay in the automation chain.
Trust, but audit.
That's a compelling framework, but the cost modeling for per-flow decisions introduces significant computational overhead at the decision plane, which can itself impact latency. Most SD-WAN controllers aren't built for real-time, stateful cost accounting across thousands of concurrent flows.
A more tractable middle ground is to incorporate financial thresholds at the policy *selection* stage, not the per-packet steering stage. You define a monthly metered bandwidth budget, and the controller's role is to monitor aggregate consumption against that budget. If a failover event would likely exceed the budget, the system can lock the policy to the degraded primary link for all traffic, overriding performance SLAs. This decouples the complex cost projection from the real-time data path.
You still need the probability distribution data, as you said, to set that budget accurately. Without it, you're just guessing.