I'm looking at iboss for securing outbound traffic from a Kubernetes cluster (EKS). I've heard mixed things about latency and throughput when it's deployed as a sidecar or at the node level.
Has anyone actually run this in production with a decent traffic load? I'm particularly worried about:
- Added latency on each pod's egress calls.
- Resource overhead on the nodes.
- Any tuning you had to do in the iboss agent config or network policies to make it perform acceptably.
A real-world example of your setup and observed metrics would be incredibly helpful. We're trying to avoid a big cost for something that might bottleneck our services.
We tried the sidecar approach for a few test services last month and saw latency jump by 150ms on average for external API calls. It was a deal-breaker for us.
Have you looked at using the iboss node-level agent instead? That's what we're piloting now, but I'm still trying to figure out the resource limits for its DaemonSet. Did you run into any CPU throttling on the nodes?
150ms extra latency from a sidecar doesn't surprise me at all. Every packet taking a detour through another container's network namespace is going to hurt. The node-level DaemonSet is the right path if you're stuck with this tool.
On CPU throttling: yes, you'll see it if you let the vendor's default resource requests go unchecked. They tend to ask for the moon. We set the DaemonSet with a tight limit like 250m CPU and 256Mi memory, then monitored actual usage. It barely scratched that on our nodes handling moderate east-west traffic. The real cost is in the iptables/NFTables rules it inserts for interception, which adds some overhead on every network call, even internal ones.
Did you have to tweak the conntrack table sizes? That's where we saw some drops under high throughput before we bumped them up.