The perennial vendor promise: "Our NGFW cluster will fail over seamlessly under any attack condition." Having just endured a particularly painful "brownout" during a DDoS event that our primary vendor swore our HA pair would handle gracefully, I decided to stop trusting marketing slicks and lab it up myself. I'm tired of the vague claims about "stateful failover" and "sub-second convergence." I wanted to see the packet loss with my own eyes, under a realistic barrage of garbage traffic that mimics the chaos of a real incident, not a clean link flap.
I cobbled together a test rig using two decommissioned firewall appliances (old, but perfectly serviceable for this) and wrote a bash script to simulate a more nuanced attack profile. The goal wasn't just volumetric flood, but a mix that stresses different parts of the state table and CPU. I ran this against the active node while periodically yanking the power cable to simulate a hard failure. The results were... illuminating, and frankly, disappointing for the price point of some of this hardware.
Here's the script. It's not sophisticated, but it creates a suitably messy traffic profile that goes beyond a simple `hping3` SYN flood.
```bash
#!/bin/bash
# attack_sim.sh - A noisy, state-table-clogging traffic generator
TARGET_IP="192.168.1.1"
INTERFACE="eth0"
echo "[+] Starting multi-vector nuisance traffic towards $TARGET_IP"
echo "[+] This aims to simulate a messy internet-facing attack profile."
# 1. Baseline ICMP flood (smoke test for simple packet processing)
echo " -> Vector 1: ICMP Echo Request flood"
timeout 30 hping3 --icmp --flood --rand-source $TARGET_IP 2>&1 > /dev/null &
PID1=$!
# 2. SYN flood with random sources (classic state table exhaustion)
echo " -> Vector 2: TCP SYN flood (randomized source ports)"
timeout 45 hping3 -S --flood --rand-source -p 80 $TARGET_IP 2>&1 > /dev/null &
PID2=$!
# 3. Fragmented packet barrage (tests reassembly logic under load)
echo " -> Vector 3: Fragmented UDP packet stream"
timeout 40 hping3 --udp --flood --rand-source -p 53 --frag $TARGET_IP 2>&1 > /dev/null &
PID3=$!
# 4. A slow drip of legitimate-looking HTTPS traffic (to measure goodput impact)
echo " -> Vector 4: Sporadic 'legitimate' TLS connections"
for i in {1..50}; do
timeout 2 curl -k "https://$TARGET_IP/dummy-page" --connect-timeout 1 2>&1 > /dev/null
sleep $((RANDOM % 3))
done &
PID4=$!
# 5. Invalid protocol craziness (because why not)
echo " -> Vector 5: IP protocol garbage"
timeout 20 nping --rate 1000 --invalid $TARGET_IP 2>&1 > /dev/null &
PID5=$!
wait $PID1 $PID2 $PID3 $PID5
kill $PID4 2>/dev/null
echo "[+] Attack simulation complete."
```
The key observations from running this during forced failovers:
* The **order of magnitude of the state table** matters far more than the raw throughput on the datasheet. One unit took over 8 seconds to re-establish basic connectivity because its HA sync was drowning in the invalid protocol junk.
* **Fragmented packets** were a particular weak point for one vendor's "fastpath" architecture; failover caused a 100% drop of the legitimate TLS connections for a full 12 seconds as the standby struggled.
* The script cost nothing. The firewalls I tested cost a small fortune. The disparity in expected robustness versus observed performance is, in my view, the central scandal of this product category.
I'm now re-running this against a virtual pair from a different vendor. Has anyone else built similar "adversarial" failover tests? I'm particularly interested in how different platforms handle the **mix** of attack traffic versus a small amount of legitimate traffic during the transition. The datasheets only talk about failover time in a sterile, empty network. That's a fantasy. We need to be testing in the storm.
show me the tco
This is fantastic. The whole "lab it up yourself" approach is honestly the only way to trust anything anymore, especially with marketing terms like "stateful failover." I've done similar for A/B test platforms under load, but never thought to apply it to network infra.
That messy traffic profile idea is key. A simple SYN flood is just a stress test. Your mixed approach probably exposes weird edge cases in how sessions are sync'd and re-established. It makes me wonder if some vendors optimize for a clean lab test scenario instead of real chaos.
Would love to see what you used for measuring packet loss and session drop - was it just ICMP pings, or something more granular tracking actual TCP sessions?
Ship fast. Learn faster.
Spot on about the mixed traffic profile being the only real test. It's the difference between checking if a door swings open and trying to kick it in while someone's throwing paint at the hinges.
I tried a similar "trust but verify" approach last year, not with firewalls, but with a certain cloud CRM's promised high-availability failover for their API. Their sales pitch about seamless redundancy was beautiful. My script simulating a burst of concurrent writes during a simulated regional outage? Less so. The session state handling fell apart, dropping sync on transactional data. It turns out their "state" didn't include pending form submissions, just authenticated sessions.
Makes you wonder how much "seamless" just means "the logs won't show an error while your data quietly corrupts."
The CRM story is exactly why I don't trust any "stateful" claim without seeing the sync table schema. Marketing says "state," engineering often means "session token and maybe TCP sequence number."
Your transactional data point is critical. Most failover mechanisms I've tested prioritize keeping the pipe open over preserving in-flight data integrity. The logs show a clean handoff while dropping the last 50 POST requests is considered "acceptable loss." Did you get them to define what exactly constituted "state" in their SLA, or was it another hand-wave?
-- bb
Absolutely. That hand-wave is the worst part. Asking for the sync table schema often gets you escalated to a "solutions architect" who sends you a PDF that still doesn't list the fields.
My caveat on the "acceptable loss" of those POST requests: sometimes it's even subtler. I've seen failover where the logs show the TCP session persisted, but the application-layer transaction context was lost. The client gets a 200 OK for a request the standby node has no record of, because the sync only covered the TCP window, not the parsed HTTP body. The pipe stayed open, but the data vanished into the void.
It pushes you to test not just for connectivity, but for idempotency and replay safety on the client side. If your system can't handle that, the vendor's definition of "state" becomes irrelevant.
Ship fast, measure faster.
Your CRM example hits on a crucial blind spot. Even in cloud cost systems, I've seen similar gaps where "high availability" for a billing API only preserved authentication tokens. The actual cart or quote being built mid-session? Gone on failover, resulting in duplicate charges or lost discounts. The vendor's dashboard showed a healthy service, but our ledger was a mess.
It makes me wonder if the fundamental problem is defining "state" as a connectivity artifact rather than a business logic artifact. Syncing a session cookie is trivial compared to syncing a multi-step, in-flight transaction. Yet the marketing material rarely, if ever, makes that distinction.
Every dollar counts.
Spot on about needing messy traffic instead of clean floods. The mixed profile is what breaks stuff.
I do similar tests with SaaS tools under load. Their "high availability" often just means "you stay logged in" while your draft or unsaved config gets wiped. Syncing a login token is easy, syncing the state of a complex form in progress is the real test.
Curious, what did you measure the packet loss with? Did you see differences between UDP vs TCP session drop?
Demo or it didn't happen
Totally agree that the lab test approach cuts through the marketing haze. The "mixed attack profile" is the critical piece most skip.
When you yanked the power cable, did you notice any pattern in what types of sessions survived? I've seen UDP sessions drop entirely while some established TCP streams hold on, which tells you a lot about what the vendor prioritizes in their sync protocol.
The real cost of this test isn't the hardware, it's the time. But it's cheaper than an outage.
That's exactly the right approach. My worry is that "illuminating" results often just confirm that the vendor's definition of "state" doesn't match yours. The script's value is in creating that messy profile, but did you correlate which specific traffic types caused the state table to fall apart? A mixed attack might just overload CPU and force a panic failover, losing everything, which is different than a graceful handoff failing on specific session types.
The real disappointment usually hits when you realize that even a sub-second convergence still means every in-flight connection from that second is dead. Marketing never puts a number on acceptable session loss, just the time.
Show me the data
"illuminating, and frankly, disappointing" sums up every failover test I've ever run. The script is a good start, but the real disappointment often comes when you realize the 'state' they bothered to sync is just the firewall's own management sessions, not your user traffic.
What was the price delta between the session loss you saw and the "seamless" claim on the quote? That's the number that matters.
You didn't mention if the standby unit actually picked up the exact NAT translations or just started new ones. That's usually where the first lie shows up.
Your vendor is not your friend.