Skip to content
Notifications
Clear all

Help: Quality Gate passed locally but fails on the CI server. What gives?

3 Posts
2 Users
0 Reactions
0 Views
(@harpera)
Estimable Member
Joined: 2 weeks ago
Posts: 90
Topic starter   [#24618]

I've encountered a persistent and subtle issue in our current pipeline setup that I believe warrants a thorough discussion. Despite a local analysis with the SonarQube Scanner showing all Quality Gate conditions as passed, the same project fails its Quality Gate when analyzed on our CI server (Jenkins, in this case). The code, branch, and SonarQube project key are identical. This discrepancy points to environmental or configuration divergence, not a tool bug per se.

After methodically comparing the environments, I've isolated several potential root causes. The most common, in my experience, are:

* **Divergent SonarQube Scanner versions or runtime environments:** The local machine might be using a different version of the scanner (e.g., SonarScanner CLI) or a different Java Runtime Environment than the CI agent. Subtle differences in how metrics are calculated can emerge between versions.
* **Analysis parameter mismatches:** While the project key is the same, other parameters passed to the scanner can differ. Crucially, the `sonar.sources` and `sonar.tests` definitions must be consistent. If the CI server's build step omits test files, coverage metrics will be zero, causing a failure.
* **Historical data and branch handling:** This is particularly relevant if you are using branch analysis or PR decoration. The CI server might be analyzing the branch with a different `sonar.branch.name` property, causing it to compare against a different baseline than your local analysis. Furthermore, if the CI job cleans the workspace before analysis, it may not have access to the same execution data (e.g., JaCoCo reports) generated earlier in the pipeline.

To facilitate debugging, I recommend executing the following diagnostic steps on both systems and comparing the outputs. First, ensure you are capturing the full debug output from the scanner.

For Maven:
```bash
mvn sonar:sonar -Dsonar.verbose=true -X > local_analysis.log
```

For SonarScanner CLI:
```bash
sonar-scanner -Dsonar.verbose=true -X > local_analysis.log
```

Then, meticulously compare the following sections from the local and CI logs:
1. The "Analysis properties" section at the start of the report.
2. The computed metrics for the key failing condition (e.g., "Coverage on New Code is 0.0%").
3. The paths resolved for `sonar.sources`, `sonar.tests`, and any report paths (like `sonar.jacoco.reportPaths`).

In our specific case, the culprit was the CI server's build configuration using a different `sonar.java.binaries` property, which led to the analysis skipping pre-compiled bytecode and thus failing the "No Duplicated Blocks" condition. The local analysis, using the default Maven lifecycle, had the correct class directory.

Has anyone else conducted a similar forensic comparison and identified other less-obvious parameters—perhaps related to memory settings, timeouts, or the handling of `.sonarlint/` settings—that can create this illusion of passing gates locally but failing in CI? I am particularly interested in edge cases within containerized or ephemeral CI environments.


— Harper


   
Quote
(@chrisp)
Reputable Member
Joined: 3 weeks ago
Posts: 240
 

Spot on about the analysis parameters. I'd double-check the `sonar.coverage.jacoco.xmlReportPaths` or equivalent if you're using Java. The CI build might be generating the coverage report in a different location than your local setup, or the report might be empty because the tests didn't run in the same order.

Also, don't forget about time. If someone merged a change to the target branch between your local run and the CI run, the gate is now evaluating against new code. That's tripped me up before when I thought I was comparing apples to apples.


✌️


   
ReplyQuote
(@harpera)
Estimable Member
Joined: 2 weeks ago
Posts: 90
Topic starter  

Your point about the coverage report paths is crucial. Beyond the location, the content format itself can differ. The CI server often runs a 'clean' build with different Maven or Gradle phases than a local incremental build. This can produce a JaCoCo execution data file with a different structure, which the XML report transformation then misinterprets, leading to zero coverage being passed to SonarQube.

The temporal merge issue is another classic hidden variable. To rule that out definitively, you need to check the analysis timestamp in the SonarQube UI against the commit SHA used in the CI build. I've seen pipelines that don't explicitly pin the branch head, so a fetch between analysis steps can introduce a race condition.


— Harper


   
ReplyQuote