Skip to content
Notifications
Clear all

My results after stress-testing Appgate with 500 concurrent users

5 Posts
5 Users
0 Reactions
50 Views
(@backend_latency_queen)
Honorable Member
Joined: 6 months ago
Posts: 613
Topic starter   [#7529]

I've been evaluating Appgate SDP for a potential deployment where we expect significant concurrent user loads. To move beyond spec sheets, I built a test harness to simulate 500 concurrent users performing a typical auth + resource access workflow. My primary interest was in the backend performance characteristics: latency under load, database impact, and the efficacy of its internal caching.

The test simulated users logging in, fetching policy decisions, and accessing a sample application gateway every 30 seconds over a 15-minute period. I used `k6` for load generation and instrumented both the test clients and monitored the Appgate Controller's resource usage.

**Key Findings:**

* **Connection Establishment:** The initial TLS handshake and tunnel setup for all 500 users created a predictable CPU spike on the Controller (pegged at ~85% on a 4-core VM). After establishment, CPU settled at a steady ~45%.
* **Policy Decision Latency:** This was the most revealing metric. Under load, the median decision time was 120ms, but the 95th percentile jumped to 450ms. This points to some underlying datastore (likely PostgreSQL) query contention.
* **Caching Behavior:** Appgate's internal policy cache is effective but seems to have a conservative invalidation strategy. After a forced policy change, latency spiked for ~30 seconds as the cache repopulated, suggesting the cache warming mechanism isn't aggressive.

Here's a snippet of the test scenario that caused the highest load:

```javascript
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
stages: [
{ duration: '2m', target: 500 }, // ramp-up
{ duration: '10m', target: 500 }, // sustained load
{ duration: '3m', target: 0 }, // ramp-down
],
};

export default function () {
// Simulate auth token acquisition
const authRes = http.post('https://sdp-controller/api/auth', authPayload);
check(authRes, { 'auth status was 200': (r) => r.status === 200 });
const token = authRes.json('token');

// Use token for policy/access request
const accessRes = http.get('https://sdp-controller/api/access', {
headers: { 'Authorization': `Bearer ${token}` },
});
check(accessRes, { 'access status was 200': (r) => r.status === 200 });

sleep(Math.random() * 10 + 20); // Random sleep between 20-30s
}
```

**Recommendations for Production:**

1. **Database Tuning:** Isolate Appgate's PostgreSQL instance. Our tests suggest the default configuration may not be optimized for high-concurrency `SELECT` statements with joins across policy tables. Consider adjusting `max_connections` and `work_mem`.
2. **Controller Sizing:** The Controller is clearly the bottleneck. For 500+ concurrent users, a horizontal scaling strategy with multiple Controllers behind a load balancer is likely necessary, not just larger VMs.
3. **Cache Warm-up:** If policy changes are infrequent, investigate if there's a way to pre-warm the policy cache post-update to avoid those periodic latency spikes.

Overall, Appgate SDP handled the load but revealed scaling is more about architectural planning (multiple Controllers, tuned DB) than just throwing hardware at a single node. The policy decision latency under peak load is the metric to watch most closely.

-- latency


sub-100ms or bust


   
Quote
(@jacksonw)
Estimable Member
Joined: 4 months ago
Posts: 63
 

The 95th percentile jumping to 450ms is interesting. Did you see any pattern in those slower policy decisions, like were they clustered in time, or was it more random? Wondering if it's a thread-pool queue issue or something specific to certain policy complexity.


not a buyer, just a nerd


   
ReplyQuote
(@latency_king_2)
Estimable Member
Joined: 6 months ago
Posts: 78
 

The 95th percentile policy decision latency is a classic symptom of lock contention on a shared resource, not necessarily thread pool queues. The jump from 120ms median to 450ms p95 suggests the decision engine is waiting, likely on serialized access to the policy cache or a database connection pool. When you have 500 concurrent users hitting the same cache keys or trying to check a global condition, the unlucky requests at the tail end of that queue get punished.

You can verify this by correlating your high-latency policy decision timestamps with PostgreSQL active session count or the Controller's JVM garbage collection cycles. A sudden GC could also cause a batch of requests to stall, creating that clustering effect. Did you monitor the Controller's JVM heap pressure during the run? That delta between median and p95 often disappears when you shift from a local, on-box PostgreSQL instance to a dedicated, optimized cluster with a properly sized connection pool.



   
ReplyQuote
(@liam4)
Trusted Member
Joined: 4 months ago
Posts: 35
 

Interesting you chose to test with a 4-core VM. That's a common vendor default, but it's almost always the bottleneck in these scenarios. The jump in policy decision latency is textbook database connection pool exhaustion.

You'd likely see the p95 improve significantly just by scaling the controller's CPU. The real question is whether the license model lets you do that without jumping into a completely different, more expensive tier. That's where the cost of "good enough" performance gets hidden.


Every cloud has a dark cost.


   
ReplyQuote
(@emmaf)
Reputable Member
Joined: 4 months ago
Posts: 297
 

That's a great point about the license model, it's always the sneaky variable. I've seen vendors where scaling the VM within the same license tier is fine, but the moment you need to add a second controller for HA you're forced into an "enterprise" package with a 300% cost bump.

I'm also curious if the database connection pool is the only culprit. I wonder if user497's point about cache lock contention could be happening *on top of* the pool exhaustion. A 4-core box might be hitting both limits at once. Did you see if the connection pool wait time metrics in PostgreSQL aligned perfectly with those p95 spikes, or were they slightly offset? That could point to a two-part bottleneck.


If it's not measurable, it's not marketing.


   
ReplyQuote