I am attempting to integrate Black Duck into our Azure DevOps pipelines to automate vulnerability and license compliance scanning for our C# and Node.js microservices. The integration is functional in that it executes the scan and uploads the BDIO file to the Black Duck server. However, the process consistently fails during the post-processing phase with an error related to "duplicate findings." This is causing our release gates to block deployments, which defeats the purpose of automation.
After reviewing the logs and the Black Duck project BOM, I have isolated the issue to components that exist in multiple locations within our solution's dependency tree. For example, consider `Newtonsoft.Json v13.0.1`. It is a direct dependency of several of our services and is also a transitive dependency brought in by various Azure SDK packages. The pipeline scan appears to be creating multiple, seemingly identical entries for the same component version, which the server then flags as duplicates, causing the pipeline step to terminate with a failure.
My current pipeline configuration for the Black Duck scan step is as follows:
```yaml
- task: BlackDuckSecurityScan@4
inputs:
Connection: 'BlackDuck_Service_Connection'
ProjectName: '$(Build.Repository.Name)'
VersionName: '$(Build.BuildNumber)'
ScanMode: 'Intelligent'
FailureMode: 'FAILURE_ALWAYS'
```
I have attempted to mitigate this by:
* Setting `ScanMode` to both `Intelligent` and `Rapid`. The duplicate error persists in both, though the composition of the duplicates differs slightly.
* Running the `detect` tool locally with the `--detect.blackduck.scan.mode=INTELLIGENT` and `--detect.blackduck.scan.mode=RAPID` flags, which completes successfully. This suggests the issue is specific to the Azure task's workflow or its interaction with the BDIO upload phase.
The core questions I have for the community are:
* Is this a known issue with the Azure DevOps task's handling of nested dependency graphs, particularly in polyglot projects?
* Are there specific configuration properties for the Azure task (such as `AdditionalArguments`) that can be used to deduplicate findings at the client level before BDIO upload?
* Has anyone developed a reliable workaround, such as a script to prune the dependency tree or to filter the BDIO output, that can be integrated into the pipeline prior to the Black Duck upload step?
I am seeking a reproducible and stable configuration. My next step is to conduct a controlled benchmark, comparing the output of a local `detect` run against the pipeline task's output, but any prior insights would significantly reduce the investigative overhead.
-- bb42
-- bb42
Your problem isn't the duplicates themselves, it's how the scanner is matching files. The default behavior in the pipeline is to treat each project or solution as a separate scan, and when you've got the same component pulled into multiple places, it looks like new findings each time.
You need to set the `--detect.tools` argument to exclude the package managers causing the overlap, or use a unified `--detect.source.path` that points at your entire source directory so it builds one coherent dependency graph. More importantly, use the `--detect.blackduck.scan.mode` property. Set it to `INTELLIGENT` or `RAPID`. The default `STATUS` scan is what's choking on your duplicate BOM entries.
Also, check your `--detect.project.codelocation.unmap` and `--detect.project.codelocation.delete.old` settings. If you're not cleaning up old code locations from previous scans, they're sitting there causing match conflicts.
You're right about the root cause being how the scanner handles the individual projects, but switching to `INTELLIGENT` scan mode is often a band-aid. It just tells the server to try and deduplicate after the fact, which still chokes up the pipeline step while it processes.
The real fix is to stop scanning each microservice in isolation if they're part of the same release. Consolidate the scan into a single run at the repo or top-level directory. That unified `--detect.source.path` approach you mentioned is the only way to get a correct, singular BOM. The pipeline task's default pattern of "scan everything in $(Build.SourcesDirectory)" is lazy and creates this exact mess.
Also, be wary of `--detect.project.codelocation.delete.old`. In a pipeline, that can delete code locations from other branches or prior builds if you're not using unique project/version strings, which breaks historical tracking. It's a sledgehammer.
null
That's a really practical warning about the historical tracking. I hadn't considered how `--detect.project.codelocation.delete.old` could wipe out data from other active branches if the project and version naming isn't distinct per pipeline run.
When you say to consolidate into a single scan at the repo level, how do you handle the scenario where the microservices are in separate repositories but still deploy together? Is the expectation then to have a separate, orchestrator pipeline that clones all the needed repos into a single working directory for the Black Duck scan? That seems like it would complicate the build artifact process.
Your config snippet got cut off, but it doesn't matter. The other replies are overcomplicating this.
You said the error is during post-processing. That's a server-side deduplication failure, not just a scanner overlap issue. The duplicate BOM entries are the symptom, not the root cause. Your scan is probably generating multiple BDIO files for the same project/version, and the server is rejecting the upload.
Check the exact error code. If it's a 409 CONFLICT, you're hitting a known bug in some versions of the pipeline task where it doesn't handle incremental scans correctly. Update the task. If that's not it, force a single BDIO output by setting `--detect.output.path` to a specific file and see if the failure stops.