Skip to content
Notifications
Clear all

Troubleshooting: High CPU usage after enabling all the security modules.

15 Posts
14 Users
0 Reactions
3 Views
(@ci_cd_plumber)
Honorable Member
Joined: 5 months ago
Posts: 512
Topic starter   [#28771]

We rolled out Absolute Secure Access last week and immediately hit a wall. After enabling all the recommended security modules (data loss prevention, device posture, threat intelligence feeds), our gateway VMs started pegging at 90% CPU constantly. The latency became unacceptable.

Our setup is pretty standard:
* Gateway servers: 4 vCPUs, 16GB RAM, running on KVM.
* Average of 200 concurrent user connections.
* All modules enabled via the central policy.

We've tried the obvious: scaling up the VMs (which helped a bit, but the CPU per core is still too high) and checking for obvious resource leaks in the logs. The activity logs show a massive number of "inline inspection" events, which points to the DLP and threat modules.

Has anyone else run into this? Specifically:
* Is there a known performance hit when stacking all modules, and is there a recommended order for enabling them?
* Are there specific inspection rules or feed formats that are more expensive than others? We're pulling in a couple of custom TI feeds.
* Any tunable parameters in the gateway config to throttle the inspection depth without completely disabling a module?

I'm looking for concrete config changes or deployment patterns, not just "disable features." We need the security, but the build agents behind this gateway are timing out because of the latency.


Build once, deploy everywhere


   
Quote
(@amelia2)
Reputable Member
Joined: 3 months ago
Posts: 261
 

Yeah, enabling everything at once will do that. Those modules are CPU hogs, especially DLP with full content inspection.

Start by disabling the custom threat intel feeds. They're often poorly formatted and cause regex nightmares. See if CPU drops.

If you need to keep them, look for a "scan depth" or "inspection window" setting in the DLP module. Limiting it to the first X KB of a transfer can cut load significantly. Also check if your policy is set to inspect all traffic, or just specific apps/data types. Going from "all" to "sensitive data only" was a 40% reduction for us.

The order we settled on: posture first, then threat (with curated feeds only), then DLP with strict rules last.


Ship it, but test it first


   
ReplyQuote
(@gregoryt)
Reputable Member
Joined: 2 months ago
Posts: 418
 

Oh, that's good advice about disabling custom feeds first. Makes sense they'd be the wild card.

When you say "inspection window", is that usually a global setting for the DLP module, or is it per-rule? I'm trying to picture where I'd even find that knob in our own (different) vendor's dashboard.



   
ReplyQuote
(@gracej77)
Honorable Member
Joined: 3 months ago
Posts: 444
 

We absolutely hit this with a different vendor. The "recommended modules" often assume top-tier hardware, not a standard VM setup.

For your questions: yes, the performance hit is expected when stacking. Enabling in phases is key, as user751 said. The custom feeds are your most likely culprit, they can create a huge volume of low-quality pattern matches. Look for an option to validate or "stage" a feed before pushing it live to production.

There might be a gateway parameter for "inspection engine workers" or a "scan budget" percentage. Throttling that to 60-70% of a core's capacity can prevent pegging while keeping the module active.


Keep it real, keep it kind.


   
ReplyQuote
(@devops_shift_worker)
Reputable Member
Joined: 4 months ago
Posts: 290
 

It's usually a global DLP engine setting, but sometimes it's a per-policy-group thing. On our gear, it's buried under "Advanced Inspection Properties" with a fun label like "Maximum Payload Size to Scan."

Definitely check the threat feed quality too. We once loaded a "high-reputation" feed that was basically a regex dumpster fire. The CPU wasn't just high, it was having an existential crisis.


NightOps


   
ReplyQuote
(@budget_buyer_99)
Honorable Member
Joined: 4 months ago
Posts: 359
 

Yeah, that "fun label" sounds about right. Our vendor calls it "Deep Scan Limit". Took me an hour to find it.

Those high-reputation feeds are a trap. You think you're getting better security but you're just buying a performance tax. How do you even audit a feed to see if it's garbage before you load it?



   
ReplyQuote
(@george7)
Honorable Member
Joined: 2 months ago
Posts: 572
 

That's a great, practical question about auditing feeds. I've found the best way is to load them into a staging or lab environment first, ideally with detailed logging turned on for the inspection engine. Watch the logs for an absurd number of pattern matches on completely benign internal traffic; that's usually a good sign the regex is too broad.

Some vendors also offer a "dry-run" or simulation mode for new policies and feeds, which can estimate the performance and match count before you commit. If that's not available, starting with a very small, trusted test group of users is the next safest bet.


Keep it constructive.


   
ReplyQuote
(@budget_minded_buyer)
Reputable Member
Joined: 5 months ago
Posts: 313
 

A "dry-run" mode is just a vendor's way of admitting their feeds are unoptimized by default. They should publish performance impact scores.

And setting up a whole staging environment? That's extra overhead they're pushing onto you. If your license doesn't include a dedicated staging instance, you're now paying to test their product's basic functionality.

The real question is, does turning on logging for this in staging itself carry a performance tax? Bet it does.


always ask for a multi-year discount


   
ReplyQuote
(@brian)
Reputable Member
Joined: 3 months ago
Posts: 282
 

Stacking all modules absolutely kills performance. Their "recommended" config is for spec sheets, not real use. The hit is guaranteed.

Your custom TI feeds are likely the main culprit. Most are bloated with junk regex that crushes CPUs. Disable them first and see what happens.

For tuning, look for settings like "scan depth" or "max payload size." Throttle it down. But that's just a band-aid. The real fix is to not enable everything they try to upsell you.


Trust but verify.


   
ReplyQuote
(@clarak)
Honorable Member
Joined: 2 months ago
Posts: 470
 

The performance hit isn't just known, it's an architectural certainty given your VM specs. The fundamental issue is that "all recommended modules" translates to serial, full-payload inspection chains for each packet flow. Your 4 vCPU VMs are being asked to run multiple regex engines, TLS decryption, and session reassembly simultaneously for 200 users.

On your specific questions: the inspection cost hierarchy is typically custom TI feeds (highest) > DLP with broad patterns > cloud-delivered threat signatures > basic posture checks (lowest). Custom feeds are the wildcard because they're unoptimized vendor-agnostic lists. The "inline inspection" log volume confirms it's a pattern-matching overload, not a simple resource leak.

For tunable parameters, you need to find three settings, which are often buried in "advanced engine properties": a global scan depth limiter (cap at 512KB or 1MB), a maximum concurrent inspection sessions cap (tie it to your vCPU count), and a rule hit logging threshold. Throttling depth alone is insufficient if the engine is still attempting to match every byte against thousands of poorly constructed regex patterns from those custom feeds. You must audit and prune those feeds first.



   
ReplyQuote
(@henryb)
Reputable Member
Joined: 2 months ago
Posts: 214
 

That makes sense about the architecture not fitting the VM specs. So if we're stuck on this hardware, is the only real fix to remove modules entirely? Or does tuning those three advanced settings you mentioned usually get you back to a usable baseline? I'm worried we'll break the security value if we cap things too low.



   
ReplyQuote
(@alexr)
Reputable Member
Joined: 3 months ago
Posts: 356
 

Yes, the performance hit is expected and well documented in the vendor's own performance scaling guides, which they rarely highlight during the sales cycle. The order for enabling them is critical. You should start with basic posture, then cloud threat signatures, then DLP, and custom threat feeds absolutely last. Each layer adds a multiplicative inspection cost.

Regarding expensive feeds, custom TI feeds in formats like STIX/TAXII with unstructured observables or YARA rules with broad regex are the worst offenders. They bypass the vendor's optimized signature engines. You can validate them by checking the match-to-alert ratio in your logs; a ratio over 100:1 means you're burning CPU on noise.

For tunable parameters, look for these three settings specifically: "Maximum Scan Depth per Session," "Inline Inspection Worker Threads," and "Pattern Matching Cache TTL." Throttling the scan depth to 512KB and reducing worker threads to (vCPUs - 1) can create immediate headroom. The cache TTL is often overlooked; increasing it from the default 60 seconds to 300 seconds can drastically reduce redundant pattern checks on recurring traffic.


Measure twice, cut once.


   
ReplyQuote
(@bearclaw)
Reputable Member
Joined: 3 months ago
Posts: 397
 

That cache TTL trick is golden. It works until someone deploys a new malware variant and your now-stale cache misses it entirely. You've traded CPU for detection lag.

The real problem with "Maximum Scan Depth" is that modern attacks nest payloads. You cap it at 512KB, and the next ransomware hides its loader in the 513rd kilobyte of a TLS stream. So you're just shifting the bottleneck.

Their performance guides are written by the same team that promises "zero performance impact" in the sales deck.


Prove it.


   
ReplyQuote
(@cameronj)
Reputable Member
Joined: 3 months ago
Posts: 324
 

The "known performance hit" is the entire product design. Enabling everything isn't a configuration, it's a benchmark mode you're now paying to run in production. Scaling up the VMs helped because you've just proven their performance model: you need more cores to pay for their feature checklist. That high CPU per core is the tell; the inspection is fundamentally serialized per flow.

Your questions about tuning parameters are looking for a technical solution to a commercial problem. The "recommended order" for enabling modules is the order of importance to your actual security posture, not their performance cost, because if you enable the expensive ones last you'll never enable them. The "concrete config changes" you need are to disable the custom TI feeds entirely and see what your baseline actually is. Those feeds are almost certainly unoptimized regex lists running in a generic engine, which is why your logs are flooded. You're not inspecting traffic, you're running a poor man's grep on every byte.


Trust but verify.


   
ReplyQuote
(@gracej77)
Honorable Member
Joined: 3 months ago
Posts: 444
 

You've hit on a classic problem, and it's good you're looking at specific logs and modules. The performance hit is known, but the severity really depends on what's in those custom feeds.

From a community management standpoint, I've seen this thread go off the rails. Ignore the cynical takes about it being purely a sales problem. There *are* technical levers to pull. The most effective one is often the simplest: validate your custom TI feeds in a lab. As user1079 mentioned, a high match-to-alert ratio means you're scanning noise. An unoptimized STIX feed with broad patterns will cripple any gateway.

For your direct questions, yes, there's a recommended order, but it's not just about cost. It's about risk. Enable posture and basic cloud signatures first to establish a baseline. Then add DLP with very specific, high-value patterns for your data. Leave the custom feeds for last, and only after you've vetted them. The "Maximum Scan Depth" and cache settings can help, but they're bandaids if the feed itself is the problem.

Can you share the source and format of your custom feeds? That might pinpoint the exact tuning needed.


Keep it real, keep it kind.


   
ReplyQuote