What's the best way to test Cato's failover before cutting over?

I'm a platform engineer at a mid-sized fintech, we've been running Cato for about 18 months to connect our four AWS regions and a handful of remote offices, handling around 1.2 Gbps of steady-state internal traffic.

Since you're looking to test the failover yourself without a maintenance window, here's the concrete breakdown from my experience:

* **Simulated Link Failure Method:** You don't need support. In the Cato Management Application, navigate to your specific socket, find the "Test Failover" option in the connection settings, and trigger a manual test. This injects a failure signal without physically dropping the link, so it's safe for a first test. It takes about 30-45 seconds for the session to re-establish over the backup path in my setup.
* **Observed Impact During Test:** With our default settings, we saw 3-4 TCP retransmissions for existing long-lived connections (like database sessions), which caused a brief pause. UDP-based traffic (like some of our telemetry) had a small packet loss burst. New connection attempts during the failover window got a connection timeout for about 15 seconds.
* **Required Pre-Check Configuration:** The main gotcha isn't in the test button, but in your backup path configuration. Before you hit test, absolutely verify your secondary, backup Cato PoP has the correct routes advertised and that your socket's "Backup Sites" list is populated. I once triggered a test only to find the backup path wasn't fully provisioned, which extended the blip.
* **Real Limitation vs. Marketing:** The failover is "seamless" for web browsing and most HTTP/HTTPS traffic, but it is *not* transparent for any stateful, long-lived TCP connections. They will hiccup. You need to design your applications to handle reconnection gracefully. Our PostgreSQL streaming replication stalls and requires a brief catch-up cycle after a failover event.

Given you're moving real traffic, my pick is to run the manual simulated test from the console during a low-traffic period first, but plan a subsequent, scheduled maintenance window to test a physical link pull. The simulated test proves the control plane works, but only a real failure shows the full data plane impact.

To give a cleaner recommendation, tell us what percentage of your traffic is stateful TCP streams (like databases, VPNs) versus HTTP-based APIs, and whether your apps have built-in retry logic.

That's a good start. The part about TCP retransmissions is key. That brief pause is what your app teams will notice, not the Cato event itself.

But stopping at the simulated test is a mistake. You need to follow it up with a real, physical disruption test during a maintenance window. I've seen the simulated test pass, but a real fiber pull or router interface shutdown exposed a routing adjacency or BGP timer misconfiguration that kept the backup path from taking over cleanly.

Your next step should be scheduling a real link kill, even if it's at 2 AM on a Sunday. Monitor both the Cato events *and* your own BGP/OSPF sessions.

That's a really good point about the real physical test. It makes me wonder, how do you even measure that "brief pause" from the app side in a way that's useful for your maintenance window report? I'm thinking beyond just "it worked." Should we be looking at APM transaction traces, or maybe something simpler like timing a batch job that runs during the test?

You're asking exactly the right question. Validating failover means measuring its business impact, not just its technical success. Relying solely on Cato's event log or a simple ping test is insufficient.

A multi-layered measurement approach is necessary. At the application tier, APM transaction traces are ideal because they capture the full stack impact, showing you if the pause manifests as database connection pool exhaustion or increased latency in a specific microservice call. For a simpler, synthetic measure, you can use a script that performs a sequential series of idempotent operations across the tunnel, like a read-write-read cycle to a test database row, and log the total completion time. This gives you a repeatable metric.

However, be cautious about using a real batch job as your timer. The variable load and processing logic can obscure the network blip. It's better to run a dedicated, lightweight synthetic transaction alongside it. The goal is to produce a before-and-after comparison for key metrics: transaction latency, TCP retransmits from your host OS, and any increase in application-level errors.