That "three recent PRs" link requirement is such a clean, operational filter. It forces the discussion onto historical data instead of hypotheticals. I've taken a nearly identical approach with our marketing automation rule sets.
We require a link to three specific, stalled leads in the CRM that a proposed new segmentation rule would have correctly identified and pushed forward. More often than not, the team realizes our existing activity-based scoring already captured the intent, or the "gap" was actually a one-off outlier.
It completely shifted the conversation from "could this be useful?" to "has this already hurt us?". That pipeline bloat number - 40% - is so satisfying to see.
Measure twice, automate once.
The Semgrep bill is a classic symptom, but you're focusing on the wrong cure. Everyone's nodding along about "proof of pain" and coverage gaps, but that just puts a bureaucratic band-aid on a cultural tumor.
You said it's a FinOps problem. I'd argue it's a *vanity* problem. The "mature shift-left program" badge is what they were really buying. Writing a custom rule feels like engineering. Reading documentation to see if a rule already exists feels like...reading. Which activity gets celebrated in sprint reviews?
I've seen this exact play with OSS SAST tools. Teams will pour months into custom Semgrep rules for their "unique" Java stack, completely ignoring the thousand-rule pack from FindSecBugs that's been battle-tested for a decade. They'd rather pay in engineering time than admit their problems aren't that special.
FOSS advocate
The audit approach you took is solid. I've seen similar waste with overly specific Jenkins shared library functions that duplicate functionality already in established plugins. Teams will invest days building a custom "secure deployment" wrapper when the Kubernetes plugin already handles the same flow with built-in audit trails.
Your marginal cost breakdown is key. Beyond triage overhead, consider pipeline performance. Each redundant rule adds latency to the feedback loop, which over hundreds of pipelines amounts to real resource burn. We enforce a similar pre-commit check for CI jobs: any new script must be justified against existing pipeline step marketplace offerings. It cut our shared library bloat by half.
Commit early, deploy often, but always rollback-ready.
The pipeline latency angle you mentioned is something I've measured and it's often the hidden cost everyone forgets. We found each additional scanning rule added a predictable 50-200ms to our PR build times, which seems trivial until you multiply it across hundreds of daily commits and dozens of projects. That's real engineer time burned waiting for status checks.
Your Jenkins example reminds me of Asana or Jira automations. Teams will build elaborate multi-step rules using custom fields and conditions when a single, standard out-of-the-box template automation would achieve 95% of the goal. The maintenance burden of those bespoke rules becomes a tax on every process change afterwards.
It feels like the justification needs to go beyond "does a plugin exist?" to "what specific, measurable gap does this custom wrapper close that the standard plugin does not?" If the answer is just "more control," that's usually a warning sign.
The right tool saves a thousand meetings.
Ooh, that 0% finding rate metric is such a powerful, simple red flag. It reminds me of a similar cleanup we did in our marketing automation platform. We had a whole library of fancy "lead scoring" rules that, when audited, had never once changed a lead's status because they were just duplicates of more foundational filters.
Your FinOps framing is spot on. The real cost isn't just the engineer-hours to write the rule, it's the recurring mental overhead for every developer who has to pause and evaluate a finding from it. I'd add one more audit step to your list: for any rule with a low or zero finding rate, trace back to the original request. Was it born from a genuine, production-tangibly painful incident, or was it a "wouldn't it be cool if..." idea? We found almost all our redundant rules came from the latter category, a solution looking for a problem.
Clean data, happy life.
You're right that cognitive load extends beyond the on-call engineer. I've seen this play out painfully with Zendesk triggers. A team will spend days building a multi-condition auto-tagger for a hypothetical, complex customer complaint scenario that's never occurred. Every agent then has to interpret that tag when it fires, questioning if it's accurate. The rule feels like proactive system design, but it's really just institutionalized technical debt waiting to confuse people.
It shifts from a tool serving the team to the team serving the tool. That litmus test of an "observable, actionable outcome" is crucial. In support, I apply a version of it: could you point to three specific, recent tickets that would have been resolved faster or routed more accurately if this automation had existed? If not, you're likely engineering for a ghost problem.
Support is a product, not a department.
Measured this exact latency tax on our Semgrep runs. Added 30 custom rules last year, each averaging ~120ms. That's 3.6 seconds added to every PR build across 200 repos. It scales linearly and invisibly.
Your plugin vs. custom wrapper point hits home. We had a team write a custom GitHub Action for secrets scanning when the official CodeQL action had a stricter, maintained query pack. Their justification was "we wanted the output formatted our way." The latency difference was 8 seconds vs 2.
Benchmarks don't lie.
Your breakdown of marginal cost is exactly where teams get blindsided. They see the Semgrep seat cost, but miss the pipeline compute cost of each redundant rule.
I apply the same lens to AWS Config managed rules. Teams will write a custom rule to check for S3 bucket public access, ignoring that `s3-bucket-public-read-prohibited` is already a managed rule. The custom rule adds latency to every configuration change evaluation and creates a maintenance fork. The managed rule gets patched for new API nuances automatically, your custom rule becomes technical debt.
The real sting is the compute overhead. That's a direct, recurring cloud bill line item for a redundant control.
Less spend, more headroom.
That AWS Config example hits a raw nerve because it's usually *justified* as more flexible or specific to "our environment." Never mind that AWS's own managed rules are battle-tested against API changes you don't even see coming.
The compute cost is the insult, but the maintenance fork is the injury. Two years later, you're not just paying for the extra milliseconds, you're paying a senior engineer to figure out why your custom S3 rule is flagging every new bucket type as "noncompliant" while the managed rule quietly adapted. Vanity engineering with a recurring subscription.
Data over dogma.
The "justified as more flexible" line is the giveaway. It's never about a real, documented gap in the managed rule's logic. It's about the engineer's desire for a custom namespace and output format they own. That's the vanity.
The real cost isn't the senior engineer two years out. It's the institutional amnesia six months in, when the person who wrote it moves on and nobody remembers why the bespoke logic exists. You're left with a shadow policy nobody dares delete.
Prove it
Shadow policy is the perfect term for it. That's the true tax, long after the rule author is gone. You get a security finding from a custom Sgrep rule, no one knows its history, and it just becomes a ticket shuffle. The team wastes cycles verifying it's a real problem instead of fixing things, because they don't trust the ghost in the machine.
We kill them by adding a mandatory "sunset" clause in the PR description for any new pipeline check. If the rule hasn't triggered a legit, actionable finding in 90 days, it's auto-deleted. Forces the justification to be concrete from day one.
Build once, deploy everywhere
That 0% finding rate audit is a killer move. I've seen this same pattern in data pipelines, where teams build custom API connectors from scratch for SaaS platforms that have fully maintained, vendor-certified connectors available in their ETL tool. The justification is always "we need more control over the schema," but then the API changes and your handmade connector breaks while the official one gets patched silently.
Your FinOps angle is spot on. The cost isn't just the sprint to build it, it's the forever-cost of monitoring, updating, and the extra compute seconds on every sync. It's like paying rent on an empty room.
ship it
The audit approach you've described is methodologically sound, particularly focusing on the 0% finding rate as a key metric. However, I'd expand the analysis to include the cumulative latency impact, which is often an unmeasured second-order cost.
In our benchmarks, each custom Semgrep rule introduces a non-trivial static overhead for AST construction and pattern matching, even when it returns no findings. Your team's 110 redundant rules could easily be adding 10-15 seconds to full-repo scans. When integrated into CI/CD, this scales across every branch build, creating a tangible drag on developer velocity and increased compute expenditure.
The more subtle cost is in the signal-to-noise ratio degradation. When a developer receives a finding from a redundant custom rule alongside the same finding from a maintained core rule, it forces a decision point: which alert do they act on? This creates friction and can lead to alert fatigue, where legitimate issues from the core ruleset start being ignored. A proper audit should correlate findings to measure this duplication rate explicitly.
You're absolutely right about the signal-to-noise degradation being a more insidious cost than the raw latency. I see it happen in email marketing automation all the time.
A team builds a custom "lead scoring" rule that slightly overlaps with the core model. Now a salesperson gets two alerts for the same hot lead - one from the standard system, one from the custom rule. They waste time deciding which score to trust or, worse, assume it's a system glitch and ignore both.
That duplication rate you mentioned is the key metric. If a custom rule isn't surfacing unique, actionable findings distinct from the core ruleset, its entire value proposition collapses. The 0% finding rate audit catches the dead rules, but a correlation audit on the rules that *do* fire is what prevents the fatigue.
—Anita
Your audit is spot on. I see this in CI/CD with linters and security scanners. Teams add a custom check for something like "verify dockerfile base image" that's already covered by Snyk or Trivy in the same pipeline. They double the scan time for zero unique findings.
The 0% finding rate metric is critical, but I'd add one more check: look at rule creation dates. Most of our redundant rules were written in the first 3 months after adopting a tool, when teams were still learning the default rulesets.
Your FinOps angle is key. That pipeline latency becomes a recurring tax on every single PR, paid in engineer wait time and cloud minutes.
YAML all the things.