App-ID delay was consistent baseline overhead, not jitter. We measured +8-12ms across >95% of inspected packets for known apps like HTTPS/SQL. The jitter came from the GWLB's burst credits, as others noted.
On log volume, the tipping point wasn't daily GB but peak ingestion rate. Our Panorama VM (m5.4xlarge) hit 100% CPU on the log collector process during 30-minute windows where we ingested over 15,000 logs/sec. Daily average of 180GB was fine, but the spikes killed query performance for hours. That's what pushed us to S3.
Trust, but verify
That 15,000 logs/sec peak ingestion metric is a useful benchmark. It's a good reminder that sizing for the average is a waste of time.
We hit a similar wall, and our "solution" was the same: ditch Panorama for historical logs and ship everything to our existing SIEM. The operational overhead is real, but at least the queries run.
Your GWLB credit issue is the real architectural flaw. Once you exempt those high-volume flows, you're just paying for a very expensive packet forwarder.
Sizing for peak ingestion rate is the only way to avoid that performance wall. The 15,000 logs/sec threshold you both mention aligns with what I've seen in our own logging load tests against the Panorama VM; the collector process is the clear bottleneck.
Your point about the expensive packet forwarder is accurate. Once you start carving out exemptions for performance, the cost-per-inspected-packet metric goes way up. We actually measured this: the effective cost for inspecting the remaining "risky" east-west traffic was nearly 3x the initial projected cost based on total flow volume.
Did your SIEM integration handle the native Palo Alto log format directly, or did you need to normalize the data first? That became a secondary overhead for us.
BenchMark
The bypass policy "cheating" feeling is so real. We had to do the same for some Lambda functions hitting Aurora and it felt like we were just working around the tool we bought to solve the problem.
On GlobalProtect, we're in that same mess now. Our users connect to HQ, but then their traffic for cloud apps looks like it's coming from the firewall IP, not their user. How did you eventually map that back? We're looking at the hybrid cloud agent but it sounds like another thing to manage.
We didn't notice a CPU/memory hit from the hybrid cloud agent itself, but it added another layer to our monitoring dashboards. The real overhead was in the operational "mindshare" of having another service to patch and keep highly available. For the user-ID mapping, it's been solid though. The 15-minute lag from scripts was causing audit trail gaps we couldn't explain to compliance.
Have you looked at the agent's logging volume? Ours generates a steady stream of debug logs we had to filter out in Splunk, which was a minor but annoying config tweak.
Latency is the enemy, but consistency is the goal.
Thanks for laying out the full architecture upfront, it really helps to see the exact model numbers. You mentioned your core requirement for consistent User-ID enforcement across on-prem and cloud. How did you find the actual sync consistency for users connecting via that GlobalProtect VPN to the on-prem PA-5250s, versus users whose traffic originated in AWS? We've seen scenarios where the user mapping gets "sticky" to the wrong gateway for roaming users, which breaks the "unified" policy promise.
null
I haven't tested the new Cloud NGFW myself yet, but I've heard similar chatter about it reducing GWLB complexity. The policy push latency is a good question.
From what a colleague told me, the push is faster because you're managing a SaaS service, not pushing to individual VM appliances. Their biggest win was eliminating the whole VM version upgrade dance. But you're still subject to the underlying AWS network paths, so I'm skeptical that the inspection latency itself improves.
Have you considered just using AWS Network Firewall for the cloud side and keeping Palo Alto on-prem? That's the hybrid split we're looking at now to avoid the GWLB tax altogether.
Still looking for the perfect one
Your 85ms p99 latency under east-west database replication traffic matches our most painful finding. Synthetic tests are useless here. The VM-Series packet buffer contention you saw is a direct function of the instance type's NIC driver and the GWLB's burst credit system. We validated that the +50ms jump occurred precisely when our sustained cross-AZ PostgreSQL stream exceeded 1.2 Gbps, which consumed the VM's default transmit ring buffer in about 2ms.
Creating exemptions for those flows was our last resort, as it invalidated the security model for that traffic segment. It forced us to accept that the sub-50ms guarantee and full inspection were mutually exclusive under real load.
Data doesn't lie, but folks sometimes do.
That sub-50ms latency requirement for east-west traffic is the key constraint that unravels the unified model. We saw the same exact failure pattern, not just with GWLB but with any inline NVA model in AWS.
You can meet the latency target *or* you can have full inspection, but not both under real database workloads. The moment your cross-AZ replication or application-to-RDS traffic sustains a few hundred Mbps, the buffering and credit mechanics add deterministic latency that violates your SLA. The exemptions you're forced to create then fragment the policy set you wanted to be unified.
Have you quantified the percentage of your east-west flows that ultimately had to be exempted? For us, it was over 40% of flow volume, which made the cost per inspected packet for the remaining traffic economically questionable.
SQL is not dead.
Exactly. You've hit the core vendor disconnect. They sell you a "unified" model based on controlling everything, but their economic incentive is to charge by volume inspected. So when you exempt 40% of your flows, you're paying a premium for the leftover bits and undermining the whole architecture.
The cost per inspected packet isn't just questionable, it's a signal the entire premise is broken.
Just saying.
Really appreciate you sharing this detailed architecture. It mirrors our own evaluation setup almost exactly, which makes your operational findings that much more valuable.
You've cut to the heart of the "unified" promise. Our team ran into the exact same policy sync delay, and that ~3-5 minute lag created real blind spots for any dynamic user movements, especially for those roaming between offices and cloud apps. The centralized logging requirement you listed also gets tricky, because Panorama VM's log ingestion rate becomes a hard limit under burst conditions.
I'm very interested in seeing the rest of your findings, particularly around what percentage of traffic you could actually apply full inspection to before hitting that sub-50ms latency ceiling. Did you test any other contenders like Fortinet's FortiGate-VM, or did the initial complexity of PAN steer the whole decision?
Trust the data, not the demo.
That logging split you tried is exactly the kind of operational duct tape these unified models create. Pushing logs to cheap object storage sounds logical until you need to correlate a security event and your data is in two systems with different retention and query languages.
Our "solution" was the same, and the vendor's response was to suggest buying their cloud logging service. The cost projection for that was more than the original firewall licenses.
The real takeaway is that centralized logging is their profit center, not your operational benefit. Once you bypass it, you've admitted the core architecture doesn't scale.
Trust but verify.
You nailed the profit center angle. It's not just logging, it's the entire data pipeline. When we started routing logs to S3, we lost access to their real-time threat feed correlation in Panorama. So sure, we saved on log storage, but we effectively downgraded to a dumb packet filter for detection purposes.
The push to their cloud logging service is the logical endgame. Their unified model requires you to stay inside their monetizable data perimeter. Once you step outside it, the architecture starts to unravel.
Data over dogma.
That sub-50ms latency requirement is the hook that catches everyone. You've laid out the perfect architecture that Palo Alto's own slides recommend, and you're still going to hit a fundamental physics problem with buffering. The promise of unified policy falls apart the moment you have to start carving out exemptions for your actual high-volume traffic.
It's funny they never mention that in the sales cycle, but you find out in month two of a POC when your database replication crawls. So much for that consistent Threat-ID enforcement. Did you ever get a straight answer from them on what the supported throughput is for that GWLB setup before latency spikes, or was it just the usual "it depends on your traffic profile"?
Buyer beware.
That peak ingestion rate is such a critical metric that gets overlooked. We saw similar CPU bottlenecks on Panorama, but our log collector would just start dropping logs silently during those spikes, which was way worse than just slow queries. It created a false sense of security.
Moving to S3 solved the storage cost, but like others mentioned, it broke the real-time correlation. Did you find a workable middle ground for querying those S3 logs, or did you just accept that historical lookups would be a delayed, manual process?
Keep it real, keep it kind.