We recently completed a migration of our data pipeline CI/CD from a self-hosted Jenkins master to Azure Pipelines (YAML). The migration logic was sound—dbt test and run jobs, warehouse seeding, and integration tests all execute correctly. However, we are now experiencing sporadic, non-deterministic timeout failures that were not present in Jenkins.
The failures are not confined to a single stage. They occur randomly across different jobs:
* A `dbt run` targeting a specific model will hang at the "START" phase for 25+ minutes before Azure forces a timeout.
* A `dbt test` on a trivial singular test will exhibit the same behavior.
* The same pipeline definition, re-run without changes, often succeeds on the second attempt.
Our initial investigation points away from warehouse performance (BigQuery), as concurrent queries from other systems are unaffected, and the Jenkins environment (against the same warehouse) remains stable. We suspect the Azure Pipelines agent or its networking configuration.
Our pipeline uses the `ubuntu-latest` Microsoft-hosted agent. A simplified version of a problematic step:
```yaml
- task: Bash@3
inputs:
targetType: 'inline'
script: |
dbt run --select my_incremental_model --target prod
env:
DBT_PROFILES_DIR: $(Build.SourcesDirectory)/profiles
DBT_CLOUD_PROJECT_ID: $(dbtProjectId)
DBT_CLOUD_JOB_ID: $(dbtJobId)
```
We've examined:
* Service connection stability to BigQuery and GitHub.
* The `profiles.yml` uses OAuth and its timeout settings are unchanged from Jenkins.
* No significant resource throttling shown in Azure's pipeline analytics.
Has anyone encountered similar random hangs after migrating to Azure Pipelines? Specifically, we're looking for insights on:
* Hosted agent networking quirks, such as DNS lookup delays or egress firewall rules that might interrupt long-lived connections.
* Any known issues with the agent's handling of long-running Python processes (like dbt) that might differ from a persistent Jenkins worker.
* Recommended diagnostic steps to capture what the agent is doing during these apparent hangs.