We've encountered a persistent and troubling issue with our Elastic Agent fleet management in a strictly air-gapped, offline environment. Our deployment architecture utilizes a dedicated Fleet Server instance within the isolated segment, configured to pull agent and integration policy updates from a local artifact repository we maintain. The agents successfully enroll and report basic heartbeat data, but all attempts to upgrade the agents to newer versions fail without actionable errors in the Fleet UI or agent logs.
The core of the problem appears to be a silent failure in the artifact distribution chain. We have meticulously followed the documentation for offline artifact hosting, mirroring the ` https://artifacts.elastic.co/downloads/` directory structure locally and updating the `fleet.agent.download.sourceURI` Fleet Server setting accordingly. The Fleet Server itself seems to recognize the new agent version target, but the rollout never progresses.
Here is a simplified view of our Fleet Server configuration relevant to the artifact source:
```yaml
fleet:
agent:
download:
sourceURI: "http://internal-repo.corp.local/elastic-artifacts/downloads/"
outputs:
- id: fleet-server-output
type: elasticsearch
hosts: [ "https://local-es:9200" ]
```
In the Fleet UI, the rollout operation shows as "Pending" indefinitely. The most revealing log entries from a target agent are these, which indicate a failure to fetch the upgrade specification, but crucially, no HTTP status or detailed error:
```
{"log.level":"error","@timestamp":"2024-01-15T10:23:45.123Z","log.logger":"fleet.agent.upgrade","log.origin":{"file.name":"fleet/agent_upgrade/upgradeapi.go","file.line":187},"message":"Failed to fetch upgrade specification","ecs.version":"1.6.0"}
```
Our hypothesis is that the agent's internal upgrade manager is attempting to construct a download URL that either deviates from our local structure or is attempting an unauthorized egress check. We have verified network connectivity and path accessibility from the agent hosts to the internal repository.
Key questions for the community:
* Has anyone successfully orchestrated a multi-version Elastic Agent upgrade path in a fully air-gapped environment, and what were the critical configuration nuances for the artifact repository?
* Beyond the `sourceURI`, are there other Fleet Server or agent policies that must be explicitly configured to prevent any external URL fallback behavior?
* What is the exact expected directory structure and artifact naming convention for the agent packages? Our mirror includes the `.tar.gz` packages and the `beats/` manifest files, but we are uncertain about the required `latest-version` or `.json` manifest files that the upgrade API might be seeking.
We are currently analyzing packet captures to reverse-engineer the HTTP calls the failing agent is making, but any prior experience or insight into the silent failure modes of the offline upgrade process would be invaluable.
Data is the source of truth.
Yeah, that silent failure in the artifact chain is the worst part. Been there. One thing I'd double-check beyond the directory structure is the specific agent artifact file names themselves. Sometimes the expected filename in the internal manifest differs slightly from what you've mirrored, especially around -rc or -SNAPSHOT suffixes that might have been in a public release candidate but not the final. Have you compared the exact `.tar.gz` filename the Fleet Server logs say it's requesting against what's in your local repo? It's often a one-character mismatch causing a 404 that doesn't bubble up clearly.
Stay curious, stay skeptical.