Alright, let's cut through the usual vendor fog and get to brass tacks. I'm dealing with an Elastic Agent deployment across a fleet of several hundred Linux servers (mix of RHEL 8 and Ubuntu 20.04/22.04) and we're seeing intermittent connection drops. The agent status flips to `degraded` or `offline` in Fleet, then recovers on its own after a few minutes. No discernible pattern across regions, instance types, or load averages.
We're on Elastic Stack 8.12, self-managed, with the agents talking to a dedicated Fleet Server cluster. The usual vendor support script of "check your network" has been exhausted—we've had network engineering trace packets and confirm no drops, timeouts, or firewall issues on the intermediate hops. The traffic is clean. This screams agent or Fleet Server instability, but the logs are infuriatingly unhelpful without spending a week correlating timestamps.
Before I go down the rabbit hole of instrumenting this thing with eBPF to see what it's *really* doing, I want to see if anyone else has torn this apart. The vendor's documentation is, predictably, a collection of best-case scenarios.
Key details of our environment:
* Agents are deployed via a custom Ansible playbook, not the vendor's install script.
* We've hardened the OS following CIS benchmarks, but have validated that none of the relevant sysctls (e.g., `net.ipv4.tcp_keepalive_*`) are overly restrictive.
* The `elastic-agent` service runs under systemd with default restart policies.
* We've already ruled out:
* DNS resolution issues (everything is IP-based in the policy).
* Resource contention (CPU/Memory is well within limits, no OOM kills).
* The obvious Fleet Server scaling issues (our load metrics are at 30% capacity).
What I'm looking for are the *actual* failure modes you've observed. Not the theoretical ones from the KB articles.
* Are there specific, non-obvious systemd limits (`LimitNOFILE`, `LimitNPROC`) that will cause a silent choke, not a crash?
* Has anyone seen the agent's internal queueing mechanism for logs/events back up and cause a TCP disconnect, rather than a graceful backpressure?
* Are there known, unresolved race conditions in the heartbeat mechanism with particular versions of the endpoint TLS stack?
* Most importantly: what are the definitive, smoking-gun log lines from the agent's internal logs (`elastic-agent-*.ndjson`) or the Fleet Server audit logs that you've used to pin this on a specific component?
I'm less interested in "update to the latest version" (we're on the latest GA) and more in the actual root cause dissection. This isn't a hobbyist setup; the TCO of chasing these ghosts is starting to rival the license cost, which is saying something.
show me the tco