Skip to content
Notifications
Clear all

Help: Agent keeps failing on our EKS nodes with 'permission denied' errors.

2 Posts
2 Users
0 Reactions
0 Views
(@avag2)
Reputable Member
Joined: 3 weeks ago
Posts: 155
Topic starter   [#23305]

We're evaluating Lacework alongside a couple of other CSPM tools for our environment, which is about 85% Kubernetes (EKS) with the rest on EC2. The core promise of agent-based visibility is what drew us in, but the implementation is hitting a major roadblock.

The Lacework agent DaemonSet is consistently failing to initialize on a subset of our EKS worker nodes. The pods enter a `CrashLoopBackOff` state, and the agent logs show a recurring pattern of permission errors when trying to access certain kernel data structures. The most common error is:

```
level=error msg="collector: failed to initialize ebpf programs" error="failed to load collector: failed to load ebpf programs: permission denied"
```

We've followed the standard EKS installation guide to the letter. Our nodes are running Amazon Linux 2 (Kernel 5.10) and we're using the IAM role-based service account as prescribed. The DaemonSet is running privileged, as required.

Here's a sanitized snippet of our agent configuration for the DaemonSet:

```yaml
securityContext:
privileged: true
capabilities:
add:
- SYS_ADMIN
- SYS_RESOURCE
- SYS_PTRACE
- IPC_LOCK
seccompProfile:
type: Unrestricted
```

The issue appears to be intermittent and node-specific, which points to a host-level configuration conflict, not the base Kubernetes permissions. We've ruled out:

* SELinux (it's disabled on the node AMI).
* Node-level AppArmor profiles (none applied).
* Conflicting eBPF-based tools (we run Cilium, but the affected nodes are in a separate cluster without it).

My working theory is that Lacework's eBPF probe is attempting to hook a kernel function or access a `/sys/kernel/` resource that is blocked by a kernel security feature we haven't identified, or there's a subtle kernel version mismatch between what the pre-compiled agent module expects and what's running.

Before I dive into building a custom node image with full kernel headers just to debug their agent, I wanted to see if anyone else has torn this apart.

* Has anyone performed a `strace` or `bpftool` analysis on the failing agent container to see the exact syscall/permission that's denied?
* Are there known conflicts with specific EKS-optimized AMI families or kernel parameters (e.g., `kernel.yama.ptrace_scope`, `kernel.kptr_restrict`)?
* Is the official troubleshooting step still just "disable `unprivileged_bpf_disabled`" (which is a non-starter for our security compliance)?

The lack of granular, actionable error logging from the agent itself is a significant hurdle for diagnosis. I need concrete data points to either fix this or to build a case against using an agent this fragile in production. Any detailed logs, host-level checks, or kernel parameter comparisons from similar environments would be invaluable.


Show me the benchmarks


   
Quote
(@integrations_jane_new)
Estimable Member
Joined: 4 months ago
Posts: 146
 

That `permission denied` on eBPF programs is a classic headache with certain EKS node configurations, even with the privileged mode and capabilities you've set. The kernel lockdown or security module on the underlying host can still block it.

A couple of things to check on those failing nodes:
* Run `cat /proc/sys/kernel/kptr_restrict` and `cat /proc/sys/kernel/perf_event_paranoid` on the node itself (not in the pod). Some AMI hardening scripts set these restrictively.
* Verify the node's instance type. Some older generation types (like t2/t3) or those with constrained networking can have issues with eBPF's memory requirements.

Could you share the output of `uname -r` from one of the problematic nodes? Sometimes a kernel patch level mismatch, even on the same major version, can trip up the agent's probe.



   
ReplyQuote