I've been conducting a performance analysis of our WatchGuard Firebox M570 cluster over the past quarter, and I've isolated a significant latency regression that correlates directly with the APT Blocker (Advanced Persistent Threat) feature when applied to HTTPS traffic. Our environment handles a substantial volume of encrypted egress traffic to various SaaS platforms, and the introduction of full HTTPS inspection with APT Blocker has introduced a measurable and, in some cases, operationally impactful delay.
The setup is standard for deep inspection: we have a valid CA certificate deployed to all endpoints, and APT Blocker is configured with what I consider conservative settings—primarily targeting executable downloads and document-based threats over common ports. The policy is applied to specific subnet groups. However, when benchmarking with `curl` from internal hosts to external HTTPS resources and comparing a policy with APT Blocker enabled versus basic HTTPS inspection only, the TLS handshake and time-to-first-byte metrics show a consistent degradation.
Key metrics observed (averaged over 1000 requests):
* **Basic HTTPS Inspection Only:** TLS handshake completion: ~120ms. Time to first byte: ~350ms.
* **With APT Blocker Enabled:** TLS handshake completion: ~190ms. Time to first byte: ~650ms.
* **Increase:** Handshake +58%, TTFB +86%.
The configuration snippet for the relevant policy action is straightforward, but the performance hit is not:
```xml
HTTPS-Outbound-With-APT
HTTPS
enable
enable
enable
enable
```
The system resources (CPU, memory) on the M570 are well within nominal ranges during these tests, peaking at around 65% utilization on the proxy workers. This suggests the latency is not due to resource saturation but rather the intrinsic processing overhead of the APT analysis engine, which likely involves signature matching, heuristic analysis, and possibly cloud lookup integration for each inspected stream.
My primary questions for the community are:
* Has anyone performed similar quantitative benchmarking and observed comparable results?
* Are there specific tuning parameters within APT Blocker (e.g., file type exclusions, cloud-assisted vs. local analysis balance) that have proven effective in mitigating this latency without completely compromising the security posture?
* Is this overhead documented by WatchGuard in any performance sizing guides? Our calculations during procurement were based on raw throughput and connection counts, not this specific feature's impact.
I intend to present these findings to our security team, but I'm seeking corroborating evidence or alternative configurations before recommending a policy change. The security-value-to-performance-cost ratio here needs careful evaluation.
Yeah, welcome to the inspection tax. It's not just the handshake. The real fun starts when APT Blocker decides to buffer and scan that 2GB encrypted backup stream to Backblaze, thinking it *might* be a document. All that data has to be proxied, decrypted, scanned, re-encrypted. Your latency is just the visible symptom.
Also, "conservative settings" on these boxes is often a fantasy. The overhead is in the decryption/re-encryption chain itself, not just the final file scan. You pay for the full ride even if it just waves the traffic through.
Seen teams ditch this layer entirely for specific high-volume SaaS targets after similar benchmarks. Sometimes the "advanced" threat isn't the malware, it's the performance hit.
Your stack is too complicated.
The "inspection tax" is such a great way to put it. It makes me wonder about the data overhead, like, is there a measurable drop in throughput you can see on the box itself when this is active? I'm coming from more of a data analysis background, so my first thought is to try and quantify that decryption/re-encryption chain cost separately from the actual scan time.
> ditch this layer entirely for specific high-volume SaaS targets
This seems like the pragmatic move. Do you usually base those exceptions on a performance benchmark first, or just go by known high-volume destinations from your logs?
> TLS handshake and time-to-first-byte metrics show a consistent degradation.
That's the MITM proxy penalty. You're adding a full TLS termination and origination hop. Your 120ms baseline is already high, which suggests your hardware might be under-provisioned for the session rate.
To confirm, check your Firebox's proxy performance graphs under "System" for the relevant cluster member during your test window. Look for spikes in "Proxy SSL Client" or "Proxy SSL Server" latency. That's your decryption/re-encryption cost, isolated from the APT scan queue.
Your "conservative settings" don't mitigate that foundational tax. The only way to reduce it is to bypass inspection for specific FQDNs or upgrade to a box with faster crypto hardware.
Trust but verify, then don't trust.
Totally feel you on the benchmarking. I've seen similar curl results with our setup, especially on those first-byte times.
What really opened my eyes was watching the CPU on the Firebox during those scans. The crypto hardware on the M570 helps, but you're right that it can still get swamped with a lot of concurrent SaaS sessions. Have you tried comparing the latency difference between a single curl request and, say, 50 concurrent ones? The tax seems to multiply.
That made us create a "trusted SaaS" exception list pretty fast. We started with destinations from our top bandwidth logs and then validated with a quick performance test.
Always testing.
Your performance metrics align precisely with the expected overhead of a full MITM proxy architecture. The ~120ms baseline for basic HTTPS inspection is itself indicative of the hardware's crypto performance envelope, and APT Blocker adds significant processing layers on top of that.
The critical factor often missed in these analyses is the buffer-and-scan behavior for streaming data, which introduces non-deterministic latency. Even with "conservative" file-type targeting, the proxy must buffer enough of the stream to perform protocol-level inspection and file-type identification before the scan decision is made. This buffering alone, prior to any actual APT scan, can cause the time-to-first-byte increases you're observing, especially on smaller, latency-sensitive API calls common in SaaS integrations.
Have you instrumented your tests to distinguish between the initial protocol handshake penalty and the added latency from the inspection buffer? Isolating those two components in your data would validate whether the issue is the foundational proxy tax or the specific APT inspection pipeline.
—BJ
Your focus on TLS handshake and time-to-first-byte is exactly where the instrumentation needs to be. I've run similar synthetic workload tests on an M470, and the consistent degradation pattern you're seeing is the hallmark of the MITM proxy's packet buffer-and-inspect cycle, even before the APT engine engages.
One nuance from my benchmarks: the latency distribution isn't normal. You'll see a tight cluster around your ~120ms baseline for basic inspection, but with APT Blocker enabled, the tail latency (P95, P99) balloons disproportionately. This suggests the system is occasionally waiting for a threshold amount of data to be buffered for file-type detection, which can stall those initial packets on certain transfer patterns. Have you plotted a histogram of your 1000 samples?
This makes exception lists tricky. A destination like update.microsoft.com might have small, fast XML responses 99% of the time, but that 1% large payload triggers the buffering scan and ruins your percentile metrics.
-- bb42
Great question about quantifying the separate costs! I've actually tried to measure this by running iperf3 tests through the same policy, both with and without the APT Blocker scan enabled. You can see a clear throughput drop just from the MITM proxy, and then another, smaller hit when the APT scan is added on top. The decryption/re-encryption chain is the real bandwidth eater.
> base those exceptions on a performance benchmark first, or just go by logs?
We started with logs to identify the top bandwidth consumers, but you really need the benchmark to justify the change. The logs might show high volume, but only a performance test will show if that traffic is actually latency-sensitive enough to warrant an exception. Sometimes the high-volume stuff is background backup traffic where a delay is acceptable.
Clean code is not an option, it's a sanity measure.
I like that you isolated the proxy tax from the scan tax with iperf. That's the smart way to build a business case for exceptions. I've done similar tests but used `openssl s_time` to measure just the handshake/re-encryption cost more directly, and you're right, that's the bulk of the penalty.
Your point about needing benchmarks, not just logs, is crucial. We found a major CDN was our top bandwidth user, but the latency impact was negligible for those large asset downloads. The real pain came from a low-volume, but business-critical, API endpoint to a SaaS app. The logs alone would have missed it. You really have to map performance sensitivity, not just volume.
One caveat: be careful with iperf over a fully inspected path. The buffer-and-scan behavior can make the throughput results a bit wobbly for short test durations. I found running the test for a good 2-3 minutes gave more stable numbers to compare.
— francesc
I appreciate you sharing these specific numbers. That ~120ms baseline for basic inspection is your key reference point. It's the unavoidable floor due to the hardware's crypto performance and proxy architecture, as others have noted.
When you see the jump from there with APT Blocker, you're paying for that buffer-and-identify step before a file is even deemed scannable. Your conservative file-type targeting helps, but the system still needs to examine enough of the stream to make that decision. That's likely the source of your extra delay, especially on those initial API packets.
Have you been able to correlate your latency spikes with the Firebox's "Proxy SSL Server" or "Scan Manager" queues in the performance graphs? That can help confirm if the bottleneck is the initial identification or the actual APT scan engine.
—daniel
You're right about needing both logs and benchmarks, but I think the premise is backwards.
Benchmarks shouldn't just *justify* exceptions, they should *identify* them. By the time you're looking at logs to build an exception list, you're already playing defense on a problem you built into your architecture.
The smarter move is to invert the model. Start with a default-deny inspection policy for everything, then use benchmarks to prove which specific traffic flows *need* the inspection layer. You'll end up with a much smaller, more justifiable "to-scan" list than a bloated "to-except" one.
Trust but verify.
This inverted model is conceptually elegant but assumes the operational overhead of a default-deny policy is zero. In practice, managing that initial policy baseline is complex - you're essentially building and maintaining a global allow list of known-safe traffic patterns, which is its own ongoing cost.
A hybrid approach often works better: start with inspection broadly enabled, but use those initial benchmarks to aggressively cull the *known-expensive* flows first. That gives you quick wins while you build the data to justify a more restrictive baseline over time.
Less spend, more headroom.
>managing that initial policy baseline is complex
Absolutely true. We tried that "inspect-nothing-by-default" model a few years back for a client, and the maintenance overhead was brutal. Every new internal tool or SaaS app launch meant someone had to file a ticket to get it benchmarked and added to the policy *before* it could work properly. It slowed down legitimate business changes.
Your hybrid approach is the pragmatic middle ground. You get the immediate security coverage while hunting for those low-hanging, high-impact exceptions. I'd add that once you have that initial list of known-expensive flows, you can automate the policy updates based on your benchmark results, which helps tame that long-term operational cost.
Happy testing!