Yes, it's an extremely common gotcha for data pipelines and any batch job exceeding a few minutes. The distinction between a heartbeat ping and job activity is fundamental to how the agent coordinates with the API, and the default idle timeout assumes a CI/CD model of short, rapid jobs.
The surprising part isn't the default itself, it's that this operational model isn't explicitly documented in the cost/architecture planning guides. You discover it through failure. For a rigorous analysis, you need to log the actual "job step" duration distribution for your pipelines. If you have a long tail of jobs over 3 minutes, you've just identified a mandatory configuration change and the associated overhead of managing a separate agent queue or a global config shift, which alters your compute profile entirely.
show me the SLA
Your config has the root cause. `disconnect-after-idle-timeout=3` kills the agent if your job doesn't send any output to the logs for 3 minutes. Heartbeats aren't job output.
For long batch jobs, you need to set that timeout to match your longest silent period, or disable it with `disconnect-after-idle-timeout=0` and live with the idle agent cost. The agent's `ping_interval` is for connection health, not job activity.
If your job is computationally intensive with no console output for 45 minutes, it's guaranteed to hit the idle timeout and disconnect. Fix the config and recalculate your metrics.
Data over opinions
That's a spot-on clarification about the heartbeat vs. job output. It's such a critical distinction that's easy to miss.
One caveat we learned the hard way: setting `disconnect-after-idle-timeout=0` for a persistent pool means you really need to watch your agent version updates and host reboots. A hung agent can sit there idle for days if you're not careful, so you'll want a separate process monitor. It trades one problem for a different ops task.
This definitely feeds into that "complexity tax" the thread's been discussing.
You've nailed the issue right in your config. `disconnect-after-idle-timeout=3` is the culprit. That setting has nothing to do with network connectivity or heartbeats - it's about job step output. If your data processing step doesn't write to stdout/stderr for 3 minutes, the agent thinks it's idle and bails.
For batch jobs that go quiet for 45+ minutes, you need to either:
- Increase that timeout to cover your longest silent period (e.g., `disconnect-after-idle-timeout=90`)
- Set it to `0` to disable entirely and run a persistent pool
- Add periodic logging within the job script itself (like an `echo "Still processing..."` every 2 minutes)
The agent heartbeat (`ping_interval`) keeps the TCP socket alive, but the idle timer is a separate logic for job activity. That mismatch is what's causing your predictable disconnects.
Dashboards or it didn't happen.