I've been conducting a rigorous, long-term performance analysis of several CI/CD platforms for a large-scale data processing project, using Buildkite as our primary contender due to its flexibility with self-hosted agents. However, I'm encountering a persistent and statistically significant issue that's skewing my reliability metrics: our Buildkite agents are consistently disconnecting during jobs with a duration exceeding 45-60 minutes.
This is not a sporadic network blip. I've instrumented the agents and the jobs to log heartbeats, system resource utilization, and network connectivity. The disconnection pattern is predictable and leads to job failure, requiring a full restart and causing substantial resource waste. We are running version `3.x` of the agent on AWS EC2 instances (c5.2xlarge) with Amazon Linux 2.
My current agent configuration, stripped of sensitive details, is as follows:
```yaml
# /etc/buildkite-agent/buildkite-agent.cfg
name="data-processor-%n"
meta-data="queue=data-heavy,os=linux,role=processor"
token="REDACTED"
endpoint="https://webhook.buildkite.com"
no-pty=true
no-ssh-keys=true
no-command-eval=false
disconnect-after-job=true
disconnect-after-idle-timeout=300
health-check-addr="0.0.0.0:8080"
tracing-backend="datadog"
```
The jobs that fail are orchestrated pipelines that execute a series of Python and JVM-based data transformations. The disconnection always manifests as a sudden cessation of log output in the Buildkite UI, followed by a "Agent lost" status. The on-instance process, however, sometimes continues running for several more minutes before terminating.
Key observations from my data collection:
* The agent's internal `buildkite-agent` process remains active post-disconnect, but the TCP connection to Buildkite's backend is terminated.
* There is no corresponding spike in CPU, memory, or network I/O on the instance at the moment of disconnection.
* The instance's security groups and NACLs are permissive for outbound HTTPS on port 443, and we see no `TCP RST` packets in VPC flow logs.
* The issue is exacerbated, but not exclusively caused by, periods of high stdout/stderr log volume from the job.
I have reviewed the agent debug logs (`BUILDKITE_AGENT_DEBUG=true`), which show a normal heartbeat pattern until an unexpected EOF or a "connection closed" message from the server side.
My primary hypotheses are:
1. An undiagnosed timeout in the agent's long-polling HTTP connection to the Buildkite backend, potentially configurable but not documented.
2. A server-side (Buildkite-controlled) limit on job connection duration for agent sessions.
3. A TCP-level timeout or proxy issue in our AWS environment that only surfaces after an extended period of sustained, but not heavy, traffic.
Before I proceed to packet capture analysis across the entire job lifetime—a non-trivial undertaking—I wanted to query the community for empirical data.
**Specific Questions:**
* Has anyone performed a controlled experiment to determine the maximum stable connection duration for a Buildkite agent under load?
* Are there known `buildkite-agent` configuration directives (beyond the idle timeout) that directly govern the keepalive behavior for active jobs?
* Could this be related to the `disconnect-after-job` logic firing incorrectly on a perceived "stall" in log output, even though the process is busy?
p-value < 0.05 or bust