Our organization recently completed a phased rollout of Sophos XGS 3300 appliances to replace aging edge firewalls across approximately 300 geographically dispersed retail locations. The primary objectives were to consolidate security services (UTM, SD-WAN, Zero-Trust Network Access) onto a single platform and improve centralized management via Sophos Central. The architectural baseline was standardized: each site received an identical XGS 3300 hardware configuration, connected via dual broadband links (primary/secondary) with SD-WAN policies, and synchronized to a central configuration template.
The deployment was largely successful from a connectivity standpoint, but several significant failure modes emerged post-cutover, primarily in areas where stateful inspection and complex networking logic intersected with real-world retail traffic patterns. I will detail the specific breakages below, focusing on the technical root causes we identified.
**Primary Failure Points:**
* **Stateful Inspection & Asymmetric Routing Paths:** The SD-WAN implementation, while functionally sound for failover, introduced subtle packet flow issues. Transactional traffic from point-of-sale (POS) systems, which often involves separate data and control channels to different backend IPs, would occasionally be dropped. This occurred when the reply path from the data server traversed the secondary WAN link (due to ISP latency or BGP preferences) while the initial outbound SYN packet had egressed the primary link. The XGS's state table, tied to the outbound interface, failed to match the returning packet on the secondary interface's state table, resulting in a drop. The logs showed `"Drop - No matching forward packet"` but correlating this across two WAN interfaces was non-trivial.
* **Mitigation:** Required disabling "Strict TCP State Enforcement" in the firewall policy and implementing a more aggressive SD-WAN dead gateway detection to minimize asymmetric paths. This, of course, carries a slight security trade-off.
* **Application-Specific ALG Interference:** The built-in Application Layer Gateway (ALG) for SIP/VoIP, while intended to assist, actively broke our legacy inventory management system's proprietary protocol, which incidentally used a SIP-like control port. The ALG attempted to rewrite packet payloads, corrupting the data stream.
```
# Diagnostic snippet from CLI showing ALG action
> show log firewall
... ALG: SIP - Modifying payload for session 0xabcd1234, from_port 5060
```
* **Mitigation:** We had to identify the specific traffic via a custom application signature (matching on deeper packet characteristics) and then create an exclusion rule to bypass ALG processing entirely for that traffic class.
* **Centralized Management Latency & Config Push Conflicts:** While Sophos Central provides a single pane of glass, pushing near-simultaneous configuration updates to a large subset of firewalls (e.g., updating a shared object) led to intermittent lockout conditions. The firewalls would enter a configuration merge conflict state, requiring manual intervention via local console. This exposed a weakness in the presumed linearizability of the configuration management system when dealing with a large, eventually consistent dataset of node states.
* **Hardware-Offload Inconsistencies:** For sites with higher transaction volumes, we enabled hardware offload for IPSec tunnels and basic routing. However, we observed that when an IPSec tunnel was re-keying, the hardware-offloaded session would timeout, but the software state would linger, causing a ~45-second blackhole for tunneled traffic until the session was fully re-established in software. This pointed to a lack of synchronization between the fast-path and control-plane state machines.
**Secondary Issues & Performance Observations:**
* The built-in caching DNS resolver exhibited high latency under relatively low query loads (~100 queries/second aggregate), which impacted any application relying on DNS-based service discovery. We had to bypass the firewall's resolver for critical internal domains.
* The default logging verbosity, even at "Essential" level, generated an immense volume of data to Central, obscuring critical alerts. Fine-tuning the log profiles required a per-policy approach, which was labor-intensive at scale.
* SD-WAN performance service metrics (jitter, packet loss), used for path selection, were sourced from the firewall itself. In scenarios where ISP packet shaping was present, these metrics became unreliable, causing unnecessary flapping.
The overarching theme is that the XGS platform, while robust in a general sense, exhibits fragility when deployed in a large-scale, homogeneous environment where any latent bug or design assumption is multiplied across hundreds of nodes. The issues were less about core firewall functionality and more about the complex interaction of stateful processing, hardware acceleration, and centralized orchestration at scale. I am interested to hear if other large-scale deployers have encountered similar issues, particularly around the state table asymmetry and ALG behaviors, and what their long-term remediation strategies have been.