Hey everyone! I've been trying to learn more about how network performance impacts data pipelines, especially when moving files to/from cloud storage. My company recently rolled out Appgate SDP for zero-trust access, and I was curious if it would be a bottleneck for some of our bigger data transfers.
So, I ran some basic iperf3 tests from my laptop (through the SDP client) to an on-prem server inside the perimeter. I wanted to see what the throughput hit was like compared to our old VPN.
The results were... okay? Not amazing, but not terrible either. With the SDP tunnel up, I was consistently getting between 85-92 Mbps on a 100 Mbps connection. The latency add was pretty minimal, only about 2-3 ms on top of the base 25 ms. It's definitely usable for the daily ETL syncs we run, which are usually in the 1-10 GB range. I was worried it would be way worse.
But here's my newbie question: this was a simple, one-off test. In a real data engineering scenario, with concurrent connections pulling from S3 or a data lake, could things degrade a lot? I don't really know how SDP handles multiple sustained streams or if there's any tuning needed on the client/agent side for data workloads.
Has anyone else here run into this or done similar benchmarks? I'd love to know if there are any gotchas or config tips to get the most out of it for data transfer heavy tasks. Still trying to connect the dots between networking stuff and my usual Python/SQL world!
-- rookie
rookie
Yeah, that's a really practical question. Your single-stream test is a great starting point, but it's true that the behavior can change with multiple parallel connections, which is exactly what a lot of cloud storage clients use to maximize throughput.
From what I've seen with similar setups, the overhead from the encryption and policy checks can start to add up under concurrency. Your 85-92 Mbps on a 100 Mbps pipe is actually pretty solid for a single stream - that overhead is mostly fixed. But if you have, say, 10 parallel streams all hitting the SDP gateway at once, you might not get 10x that speed. The gateway's CPU and how it's configured for session handling become the new bottlenecks, not your raw bandwidth.
Have you checked if your Appgate controller has any specific settings for 'high-throughput' or 'data-intensive' connections? Sometimes there are different cryptographic profiles or tunables for the agents. It might be worth simulating a more realistic load with a tool that can open multiple iperf3 connections at once, just to see where it plateaus.