That forced funnel idea is really smart - it ties the metric directly to the policy artifact. It reminds me of our product analytics where we track feature flag changes. A metric like 'flag overrides by role' is useless unless it forces you to document the decision in the flag's comment history, which is what gets audited.
So for SAST, maybe the metric isn't 'remediation rate,' but 'decisions without documentation.' Track any finding state change from 'open' that doesn't include a link to the approved risk acceptance ticket or the commit hash that fixed it. That way the dashboard gap *is* the compliance gap.
It does make me wonder, how do you handle the noise for genuinely irrelevant findings? Like a Java shop getting a bunch of Node.js vulnerability alerts? Do you have to document ignoring each one, or can you justify turning that whole rule off at the policy level?
That audit log integration is the killer feature for SOC 2, no doubt. It saves you from the nightmare of trying to correlate events across systems. I'd add one practical caveat, though: you need to verify your GitHub audit log retention settings align with your compliance policy's required retention period. The defaults might not be long enough, and if an auditor asks for proof from 14 months ago and it's gone, that's a problem.
Also, the `runs-on:` line in your snippet is empty. That'll break the workflow and your entire compliance gate.
You're absolutely right about the retention mismatch being a silent failure point. Our policy required 24 months, but GitHub Advanced Security's audit log default was only 6. We had to explicitly set it via the API during provisioning, and we documented that configuration step in our control narrative.
It also creates a secondary reporting issue. If your internal compliance dashboard pulls from an audit log that's on a shorter retention cycle, your historical reports will suddenly have data gaps when the logs purge. You need to architect your reporting to either mirror the logs locally for the full period or clearly annotate when data is sourced from a truncated stream.
The empty `runs-on` is a perfect example of how a tiny, non-security bug can completely break a critical compliance control. It underscores the need for peer review on these workflow files, not just the application code.
Data > opinions
Totally agree, especially about remediation time being a key metric. In my experience, you've got to separate "time to merge a fix" from "time to acknowledge and ticket" for that metric to be honest. If you only track the PR block to merge, teams get tempted to just click 'merge' with a temporary suppression to keep things moving, which defeats the whole point.
The dashboard showing PRs blocked is gold, but you need to filter for repeats. If the same library vuln is blocking 50 PRs because it's in a shared package, that inflates your "gate failure rate" but doesn't really reflect new risk being introduced. It just shows you have a widespread problem you haven't prioritized to fix yet.
ship it
That's a solid setup, and I agree audit logs are the foundation. Building on your CI/CD gate example, the 'secret sauce' for SOC 2 is often in the failure path. What happens when this workflow breaks, like a timeout or a flaky runner?
You need an alert and a documented process for that. If the SAST scan doesn't run, your PR shouldn't be mergeable. We had to add a secondary status check that validates the CodeQL analysis job itself completed successfully, not just that it passed. Otherwise, a silent failure in your workflow creates a gap in your evidence.
—Anita