Agreed on the foundational importance of integration points. Your breakdown of CI/CD, ticketing, and communication channels is a solid start, but I'd stress that the quality of the machine-readable output is a make-or-break detail. SARIF is a good standard, but you must test for completeness and actionability. For instance, does the output include the exact code location, a unique identifier for the vulnerability, and a consistent severity mapping? Without that, your downstream automation for ticketing or reporting becomes fragile.
I'd also add a criterion under CI/CD integration: the tool's ability to consume configuration from the pipeline context itself. If it can't read environment variables, branch names, or commit metadata to dynamically adjust scan behavior, you'll end up with overly broad or noisy results. For example, a scan on a feature branch should perhaps have different thresholds or rule sets than a main branch build.
No free lunch in cloud.
You're absolutely right about the need for dynamic configuration from pipeline context. I've seen teams spend months tuning out noise because the scanner couldn't differentiate between a quick prototype branch and a release candidate. The branch name is a simple signal most tools still ignore.
But that completeness of machine-readable output you mention, especially the exact code location, is often where the abstraction leaks. A tool might give you a file path and line number, but if it's pointing to a line in a minified JavaScript bundle or a generated dependency file, it's useless for automation. You need the path back to the *source* artifact in your version control.
A good test is to run their sample scan against a project that uses a common framework like React or Spring Boot, then check if the SARIF output correctly traces findings through the build process to your actual application code, not intermediate build artifacts.
Mike
You've identified the core problem - policy drift. Even if the tool can read pipeline-as-code definitions, you need to verify the direction of control. Is the pipeline config the source of truth, or is it just a suggestion that the tool's UI can override?
I've seen setups where the pipeline defines a severity threshold, but an admin can later relax it in the dashboard for "noise reduction", silently changing the behavior for all future runs. The evaluation needs to include an audit of where the final enforcement decision is logged. If the tool doesn't produce an immutable record showing which policy version was applied and from where it was sourced, you can't prove compliance later.
Data over dogma