Greetings, fellow community members. I am reaching out regarding a persistent issue with High Availability synchronization on a Check Point firewall pair that has resisted resolution through standard documentation and vendor support channels. My typical methodology involves rigorous, reproducible testing, but this operational instability has prevented the establishment of a reliable baseline, which is frankly unacceptable for a state synchronization mechanism.
Our test environment consists of two Check Point 6400 appliances running R81.20, deployed in a dedicated lab for performance and resilience validation prior to any production deployment. The core issue is that the state synchronization (sync) network experiences intermittent failures, causing the secondary member to enter a "Problem" state and requiring a manual `cpstop`/`cpstart` on the secondary to re-establish synchronization. This occurs despite following the vendor's deployment guide precisely for cable topology and network configuration.
I have conducted extensive packet captures on the dedicated sync interface (a direct 10GbE DAC cable) and observed the following anomalous behavior during a failure event:
* A complete cessation of sync traffic (protocol 256) from the primary for a period of 17 to 45 seconds, as measured by Wireshark I/O graphs.
* No corresponding link-down events on the switch (it is a direct point-to-point link) or interface errors (`netstat -i`, `cpstat -f all`).
* The primary appliance continues to process and forward traffic without logging any internal errors related to the sync process.
Our relevant configuration excerpts from the primary member are as follows:
```bash
# Output from cphaprob -a if
Cluster Interfaces:
Interface Number: 1
Interface Name: eth1
IP Address: 10.255.255.1
Network Mask: 255.255.255.252
Physical Location: local
Current State: OK
Interface Number: 2
Interface Name: sync
IP Address: 192.168.1.1
Network Mask: 255.255.255.252
Physical Location: local
Current State: OK
```
We have attempted the following remediation steps, each tested over a 72-hour stability window, with all resulting in a recurrence of the sync failure:
1. Replaced the DAC cable with a different manufacturer's model.
2. Swapped the physical interfaces used for the sync network on both appliances.
3. Adjusted the `kernel` and `advanced` synchronization parameters via `cpprod_util`, specifically increasing timeouts and buffer sizes.
4. Performed a clean installation of R81.20 on both units and rebuilt the cluster from scratch.
My hypothesis is that this may be related to a kernel-level resource exhaustion or a race condition within the sync driver, but the lack of definitive error logging is hindering root cause analysis. Has anyone encountered similar non-deterministic sync failures on the 6400 series or R81.20 builds? Furthermore, are there any recommended diagnostic commands beyond `cphaprob -ia list`, `fw ctl pstat`, and standard packet captures that could isolate the point of failure within the sync pipeline?
I am prepared to run any suggested synthetic workloads or monitoring scripts to gather further data. Reproducible results are key.
-- bb42
-- bb42
Oh man, I feel your pain. Watching sync fail in a dedicated lab is so much more frustrating than in production, because you know the environment is supposed to be pristine.
You mentioned following the vendor guide for cable topology, but have you tried swapping out that DAC cable? We had a weirdly similar issue last year, and after weeks of pulling our hair out, it turned out to be a faulty cable that would pass basic link tests but couldn't handle sustained sync traffic. Threw in a new one from a different batch and the problem vanished.
Also, for R81.20 specifically, I've heard some folks needed to tweak the HA-Failover setting for the sync interface under the cluster object's topology. Might be worth a quick check if a more aggressive timeout is getting triggered. Good luck
Always testing.
You're right, a faulty DAC is a classic culprit. I'd add that you should check the interface error counters on both appliances (fw ctl iflist) over a period of active sync, not just a basic link test. That can reveal transient CRC errors that point to physical layer issues.
Also, regarding R81.20, you might want to examine your cluster's synchronization settings beyond just timeouts. There's a known quirk where the "Synchronization on cluster member start" setting, if misconfigured, can cause the secondary to get stuck in a loop trying to sync an overwhelming amount of data from a dirty state.
Every dollar counts.
That's a fantastic, practical point about checking the error counters over time, not just a static snapshot. I once spent days on a similar problem where the sync would fail only under a specific, high-throughput test scenario. The standard interface status looked fine, but a tailed log of the error counters finally showed a pattern of increasing frame errors that coincided perfectly with our load test peaks.
I'm also glad you brought up the dirty state sync loop. It's a subtle one that can really waste your time, especially in a lab where you might be rebooting or resetting nodes more frequently during testing. It's easy to assume the sync should just handle it, but that setting needs to match your intended recovery behavior.
Let's keep it real.