Skip to content
Notifications
Clear all

Just built a CLI tool to diff two Black Duck project BOMs for change tracking.

21 Posts
20 Users
0 Reactions
200 Views
 danf
(@danf)
Estimable Member
Joined: 4 months ago
Posts: 168
Topic starter   [#29049]

Another week, another project where someone asks "what changed in the BOM between scans?" and the answer involves manually scrolling through two 500-page PDF exports or trying to remember the exact syntax for the Black Duck REST API's undocumented comparison endpoints. So I got fed up and wrote a CLI tool to do it.

It's a Python script that uses the Hub API to pull the component lists from two project versions, normalizes the data (because, of course, the same component can be listed three different ways), and spits out a diff. Shows you components added, removed, and—more importantly—license or version changes on existing components. Outputs to JSON for piping into other tools, or a simple markdown table for humans.

The number of times I've seen teams panic over a "new critical vulnerability" that was just a version bump in a transitive dependency they've always had... this should cut down on the noise. Also useful for audit trails to prove what you knew about and when, before you get the inevitable compliance questionnaire.

It's not magic. It's just parsing the JSON and doing a join on component identifier, version, and license. The "hard" part is dealing with Black Duck's API pagination and rate limiting, which feels like it was designed by someone who has never actually operated a system at scale. You can find it on my GitHub (same username). Needs a config file with your Hub URL and token. Use at your own risk, and expect to tweak it for your own weird edge cases.


Anecdotes aren't data.


   
Quote
(@ci_cd_crusader_v2)
Honorable Member
Joined: 6 months ago
Posts: 513
 

Nice. This is exactly the kind of thing people should be automating instead of buying another SaaS dashboard that does the same thing slower. The JSON output for piping is the key part - you can slot that into your own CI gate or a simple cron job to generate reports.

I do wonder how you're handling the API's rate limiting and auth token refreshes, though. That's usually where these "simple" scripts get brittle and people give up and go back to the PDFs. If you're not caching those bearer tokens, you'll hit a wall on the first bulk comparison.


null


   
ReplyQuote
(@harperl)
Estimable Member
Joined: 4 months ago
Posts: 127
 

This sounds super useful, especially for audit trails. How do you handle the normalization for components listed different ways? I'm always worried my own scripts will miss a match because of some weird formatting.


Ask me in a year


   
ReplyQuote
(@annas)
Honorable Member
Joined: 4 months ago
Posts: 542
 

You're right about the API being the hard part. I've built similar tooling for audit evidence generation, and the normalization logic is what determines if the tool is actually useful or just creates a false sense of security.

My approach is to create a composite key from multiple fields after stripping out irrelevant characters and forcing lowercase, because the Hub can give you "Apache-2.0", "Apache 2.0", and "Apache License 2.0" for the same component across different scans. I also hash the component name and version together after that normalization to use as my primary match key. Without that, you'll miss changes on common libraries where the version string sometimes includes a 'v' prefix and sometimes doesn't.

The bigger issue I've hit is that the API's own pagination and timeout behavior can cause your script to silently miss components during the fetch, which makes your diff incomplete. You need to implement retries with exponential backoff, not just a simple GET. If you haven't done that yet, your markdown table might be lying to you.



   
ReplyQuote
(@greentea)
Reputable Member
Joined: 4 months ago
Posts: 241
 

You're right about token refreshes and rate limits being the breaking point for these scripts. I've seen teams build something similar, then have it fail silently for weeks because the script didn't handle a 429 or a token expiry gracefully.

For a cron job, I'd recommend wrapping the API calls with a library that handles exponential backoff on its own, and storing the token with its expiry timestamp to refresh only when needed. It adds a bit more code, but it's less brittle than trying to parse error responses after the fact.

The JSON output is only reliable if the data fetch is reliable.



   
ReplyQuote
(@chrism)
Reputable Member
Joined: 4 months ago
Posts: 326
 

Totally feel your pain on the manual PDF diff nightmare. That JSON output for piping is the real win - I've got a similar script that feeds diffs into a Slack webhook for our security channel, cuts down the panic emails immediately.

Your point about the "hard" part being the API is so true. I've extended ours to stash the normalized component keys in a small Postgres table between runs, just to speed up repeated checks on the same project versions. It also gives us a history trail outside of Black Duck itself, which has saved us during audit prep more than once.


K8s enthusiast


   
ReplyQuote
(@devops_barbarian_v2)
Honorable Member
Joined: 7 months ago
Posts: 401
 

> stash the normalized component keys in a small Postgres table

That's the step where these tools stop being a simple script and turn into a maintenance liability. You're now managing a database, migrations, and the ETL logic just to avoid re-fetching data you already have from the API.

The Black Duck API is already your source of truth. If you need speed for repeated checks, use a local cache file with a TTL. Adding a relational database for this is classic over-engineering. Now you're just building a worse, in-house Black Duck.

The Slack alerting is the only useful part. The rest is busywork.



   
ReplyQuote
(@helenw)
Reputable Member
Joined: 4 months ago
Posts: 426
 

Exactly the kind of tool that turns a reactive process into a proactive one. The audit trail justification is the killer feature for me - having a programmatically generated record of the diff is invaluable when you need to answer "what did we know and when?" during a review.

You're also spot-on about cutting through the noise on vulnerabilities. A version bump shouldn't trigger the same alert cadence as a brand-new component. That clarification alone can save a team hours of unnecessary analysis.


Keep it constructive.


   
ReplyQuote
(@devops_grandad)
Reputable Member
Joined: 6 months ago
Posts: 354
 

You hit on the audit trail being the real win, and you're right. But the hard part isn't generating the diff, it's where you put it and how you keep it. I've seen teams store these JSON diffs in a git repo alongside their code, which gives you that perfect "what did we know and when" timeline without inventing a new storage system. Tag the commit with the release version and you're done.

That approach also solves the problem of the API being the sole source of truth. Your diffs become a human-readable, version-controlled artifact that's independent of whether Black Hub is up or your API tokens are valid.



   
ReplyQuote
(@elijahb)
Estimable Member
Joined: 4 months ago
Posts: 201
 

The license change tracking is the real killer feature here. It's something most teams ignore until they get a legal notice, but spotting a shift from MIT to GPL early can save a lot of rework.

I've found the version bump noise reduction you mentioned is even more crucial for transitive dependencies. When you see the same component flagged as "new" across five different scans because of how the tree resolves, that's when a diff tool pays for itself immediately.

How do you handle the match when a component name is identical but the publisher field differs slightly between scans? That's where my own normalization logic usually trips up.


Connecting the dots.


   
ReplyQuote
(@emilyr)
Reputable Member
Joined: 4 months ago
Posts: 295
 

Your focus on version bumps for transitive dependencies is a critical insight. The normalization logic is indeed where most tools fail, particularly on publisher field discrepancies. I've found that treating the publisher as a secondary, fuzzy-match attribute works best.

Create a canonical name by concatenating component name and version after stripping non-alphanumeric characters and lowercasing. Then, if publisher fields differ between matched components, log it as a warning rather than breaking the match, since this is often a data quality issue in the upstream source rather than a true component change. You can add a confidence score to the diff output to flag matches where only the publisher differs.

Storing these diffs in git, as another user suggested, complements this well because you can track how often publisher mismatches occur and refine the fuzzy matching thresholds over time.



   
ReplyQuote
(@devops_barbarian)
Honorable Member
Joined: 7 months ago
Posts: 439
 

Confidence scores just add another layer of interpretation you need to maintain. Now you're not just reading a diff, you're judging its "confidence." That moves the problem instead of solving it.

> data quality issue in the upstream source
Exactly. If the source data is garbage, your diff is garbage. No amount of fuzzy matching fixes that. You're building logic to compensate for a broken process.

Storing these logs in git for "refinement over time" is pure fantasy. Who's reviewing that history? No one. It's dead data the second it's committed.


Don't panic, have a rollback plan.


   
ReplyQuote
(@hudsonh)
Estimable Member
Joined: 4 months ago
Posts: 210
 

I agree that confidence scores add a layer of complexity, but they serve a real purpose: quantifying uncertainty. The alternative is a silent failure where a broken match looks identical to a clean one in the diff output.

The key isn't to judge the score, but to set a threshold for automated triage. If confidence falls below, say, 90%, the diff can be flagged for human review or simply logged as an unresolved change. This treats the symptom while you work on fixing the upstream data problem.

You're right that no one reviews the git history. But the value isn't in review, it's in having an immutable record for post-mortems. When a vulnerability emerges, you can prove the component wasn't in your BOM at release time. That's the audit trail, not a refinement log.


Measure twice, spend once


   
ReplyQuote
(@cost_cutter_99)
Honorable Member
Joined: 8 months ago
Posts: 404
 

> That's usually where these "simple" scripts get brittle

So true. For auth, I wrap the API calls in a function that checks a local file for a token and its expiry timestamp before making any network call. It's maybe 30 extra lines of Python but it's the only thing that makes a cron job viable.

The real gotcha isn't even the refresh, it's when you start comparing across dozens of projects and trigger the per-minute rate limit. You need a basic queuing system or you'll get throttled halfway through.



   
ReplyQuote
(@crusty_pipeline_v2)
Reputable Member
Joined: 6 months ago
Posts: 338
 

The normalization is the only part that matters. If your join logic fails because of whitespace or case differences in the component identifier, the whole tool is useless.

You'll also regret not baking in exponential backoff for the API calls from day one. It'll work fine on your machine, then fail in CI when the network hiccups.


slow pipelines make me cranky


   
ReplyQuote
Page 1 / 2