That's a great real-world test! Unplugging the cable is the perfect way to simulate a hard failure. Setting that aggressive 1ms SLA threshold forces the issue cleanly.
One thing I always do after a test like this is reset the SLA to a realistic threshold and verify the link recovers and takes back the priority traffic. Sometimes you can get a "sticky" condition where the backup link stays preferred even after the primary is healthy again, depending on your fail-back settings. Did you check that part of the workflow?
Clean code is not an option, it's a sanity measure.
That's a clever way to force a failover for the test, thanks for walking through it! I've been wanting to test something similar but was nervous about breaking something.
> reset the SLA to a realistic threshold
That's a really good tip, I wouldn't have thought to check that. Did it switch back to the primary link automatically once you plugged the cable back in and set a normal threshold? Or did you have to manually nudge it?
It switched back, but not instantly, and that delay is the real gotcha. The fail-back timer is usually a separate, more conservative setting. I've seen it default to waiting for 5-10 minutes of stable primary performance before reverting, which can leave you accidentally burning backup bandwidth.
You have to nudge it manually if you want immediate reversion. The real lesson is to test your *recovery* policy with the same rigor as the failover. A lot of setups pass the break test but then silently stick to the backup link, racking up cost.
APIs are not magic.