You mentioned `mtr` runs, but relying solely on that is a classic oversight. The interval you choose for those tests directly determines whether you capture the path change or just see the new steady state. If you're running it every five minutes, you're almost guaranteed to miss the actual transition event and its characteristics.
We instrumented our applications to log the source IP of the Cato egress socket at the start of every significant transaction. That gave us a time-series of path shifts we could overlay on our application latency charts. Without that, your "regular" traceroute data is just a series of snapshots that smooth over the very dynamics you're trying to measure. The control plane's agility makes traditional network monitoring post-incident data nearly useless for causality.
Did you correlate your application performance anomalies with the specific egress PoP changes, or are you just seeing an aggregate latency improvement?
--perf
You cut the methodology off right at the auto-routing part. That's the critical piece.
The problem with comparing hop count is you need to know which path you're measuring. If the control plane is live-steering traffic, your `mtr` run at interval X might be hitting a different optimized path every time. Your baseline data gets noisy.
You need to timestamp and log the exact Cato egress IP for each test to get a real hop count comparison. Otherwise you're just averaging across multiple paths.
Benchmarks don't lie.
Your methodology's focus on continuous TCP ping tests is solid for capturing the steady-state latency envelope. However, the critical gap is the granularity of your path analysis. Relying on `traceroute` and `mtr` runs at regular intervals, as you've cut off describing, will miss the very control plane decisions that define Cato's reliability.
When their system performs an automatic reroute, your scheduled path test will only show the new, stable path. You lose all data on the transition duration, any packet loss during the failover, and the characteristics of the abandoned path. This creates a significant blind spot. You're measuring the before and after states, but not the event itself.
To truly evaluate the backbone, you need to correlate each individual latency datapoint with the egress PoP used at that exact moment. Otherwise, you can't attribute a latency spike to a specific network event versus a Cato reroute. Your empirical data will show improved averages but lack explanatory power for anomalies.
--perf
Exactly, that's the visibility trade-off. You can get the same path correlation by tagging your VPC flow logs with the Cato socket ID if you're routing all traffic through their gateway. We set up a Lambda to parse logs and join them with Cato's API data on egress PoP changes.
It won't capture the transition packet loss, but you'll at least know *when* the path changed and can correlate that to app metrics. Still leaves you blind to what happened in those milliseconds though.
Infrastructure as code is the only way
You cut off right after the word "auto." Is that auto-routing? That seems to be the entire point everyone is drilling on.
If you're only doing regular runs, how did you even know which path you were measuring for the hop count comparison? Like user518 said, if the control plane is moving traffic, your "post-migration" data could be comparing your old static path to five different Cato paths averaged together. That doesn't sound controlled.
Did you log the egress PoP for each test run?
You're correct to press on the auto-routing point and the subsequent methodology gap. Our team logged the egress PoP for each test run by correlating the test script's source IP with a table of PoP egress IPs provided by Cato and periodically verified via their API. The "regular" traceroute data you mentioned was indeed tagged with this PoP identifier.
Without that correlation, the hop count data is meaningless, as you'd be averaging across multiple optimized paths. Even with it, the scheduled nature of the tests creates the blind spot for transition events that several others have noted. Our next iteration involved socket-level logging to capture those shifts in real-time.