Alright, I've hit a wall with this one and could use some collective wisdom. I've got a pair of next-gen firewalls in an active/passive HA cluster. They're supposed to be rock solid, but we're seeing unexpected, brief failovers during what I'd consider normal business-hour traffic—no clear spikes, no DDoS, just typical user activity.
The frustrating part is the logs. The failover event itself is logged, but the *reason* given is always vague: "HA link failure" or "peer unit not detected," even though the dedicated HA links show no errors before or after. The primary just... steps down. It comes back online fine after a few seconds, but these blips are causing dropped sessions and user complaints.
I've checked the obvious: HA link interfaces are on dedicated ports, cabling is good, configurations are synchronized, and the firmware is on the vendor's recommended version. Heartbeat intervals and hold-down timers are at vendor defaults.
Has anyone else run into this kind of ghost-in-the-machine HA behavior? I'm starting to wonder if it's a subtle resource issue—maybe a brief CPU spike on the primary that the HA process misinterprets as a failure? Or perhaps an MTU mismatch on the HA data links that only manifests under certain packet sizes? I'm planning to run a packet capture on the HA links next, but if anyone has a debugging checklist or has solved a similar mystery, I'd really appreciate the pointers.
— Eric
Keep it civil, keep it real.
Ah, the classic "phantom failover"! I've chased that gremlin before with our old Palo Alto pair. Your guess about a CPU spike is a good one, but I'd wager it's something even more subtle: memory pressure. If the primary's monitoring daemon gets starved for a split second during garbage collection, it can miss a heartbeat and trigger a panic.
Check your session table size and any deep packet inspection processes. I once had an identical issue that only surfaced when the SSL decryption engine hit a certain threshold of concurrent handshakes. The logs were useless, but a Grafana dashboard for kernel memory allocation told the real story.
What's the make/model? Sometimes there's a known bug with the HA keepalive traffic getting deprioritized in the internal scheduler under specific load patterns.
it worked on my machine