After a six-month evaluation and planning period, we completed a full-scale rollout of GitHub Advanced Security (GHAS) to our engineering organization of approximately 100 developers last quarter. The primary objectives were to shift security left, establish a consistent baseline for code scanning and secret detection, and leverage the code scanning codeQL functionality for variant analysis. The technical implementation itself, guided by the official documentation, was relatively straightforward.
However, the operational and cultural integration revealed several significant friction points that I believe are critical for any organization of similar scale to consider. The issues were not with the tool's capabilities per se, but with its interaction with existing development workflows and system constraints.
**What Broke (or Nearly Broke)**
* **CI Pipeline Latency and Cost:** Our average pipeline duration increased by 40-70%, depending on the repository's size and language composition. The default `github/codeql-action` configuration, which builds the codebase for analysis, often duplicated work already done by our existing build and test steps. This led to:
* Increased queue times for other CI jobs due to resource contention on our self-hosted runners.
* A direct and substantial increase in compute costs. We observed that the automatic "build mode" detection was not optimal for our polyglot monorepo setup.
We mitigated this by transitioning to explicit, manual build steps for CodeQL to reuse artifacts. The configuration snippet below illustrates the shift:
```yaml
# Initial, problematic approach (reliance on autobuild)
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: 'java, javascript, python'
- name: Autobuild
uses: github/codeql-action/autobuild@v2 # This would often trigger a full rebuild
# Revised, optimized approach
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: 'java, javascript, python'
- name: Perform Java Build
run: mvn compile -DskipTests # Reuses same goal as earlier CI step
- name: Perform JavaScript Dependency Installation
run: npm ci
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
```
* **Alert Fatigue and Triage Overload:** The initial rollout, with default query suites, generated over 2,000 alerts across our repositories in the first week. A substantial portion were:
* **Historical Findings:** In existing, dormant code paths.
* **False Positives or "Best Practice" Violations:** Particularly from security-quality queries (e.g., `js/clear-text-logging`). While valuable, the volume overwhelmed teams.
* **Duplicate Variants:** The same pattern repeated across multiple files.
This led to developer disengagement, as the signal-to-noise ratio was too low. We had to implement a phased, prioritized rollout:
1. Enabled only security severity queries (Critical/High) for the first month.
2. Used the `github/codeql-action/init@v2` `queries` parameter to add a custom, smaller suite of precision-tuned experimental queries.
3. Aggressively used the `github/advanced-security-feedback` repository to mark false positives and improve query precision.
* **Secret Detection in Developer Workflows:** While catching a committed secret is the goal, the default push-time blocking created unexpected workflow disruptions. Developers accustomed to using temporary, non-sensitive placeholder values in feature branches for configuration testing found their pushes blocked. This necessitated a cultural shift and clearer documentation on using local environment variables or development-only secret managers, rather than committing placeholders.
* **Dependency Review in Large Monorepos:** The dependency review graph generation for pull requests with hundreds of changed dependencies could time out. We had to strategically split large dependency update PRs and adjust timeout settings, which added manual process overhead.
The rollout was ultimately successful, but the path was defined more by addressing these emergent system behaviors—the latency costs, the alert volume management, and the workflow impedance—than by the initial security configuration. The key lesson was treating GHAS not as a simple tool enablement, but as the introduction of a new, resource-intensive subsystem that required its own performance tuning and operational controls.
brianh
Six months of evaluation and you still ended up with the default config that redoes the entire build? That's not a tool problem, that's a reading comprehension problem. The `github/codeql-action` has a `build-mode` option and you can point it at existing build artifacts. Or you could stop feeding the hosted runner machine and run CodeQL on your own self-hosted runners where you control the caching and the pipeline cost isn't measured in per-minute Azure markup.
I'm genuinely curious: did anyone actually test the CI impact before the full rollout, or was the "six-month evaluation" spent on PowerPoint decks about shifting left? Because a 40-70% increase in pipeline duration on a 100-person org isn't a friction point, it's a bill you're about to find out about when the next invoice arrives. The real question is how many of those 100 devs are now waiting an extra 10 minutes per push and blaming the tool instead of the people who decided to run two full builds in sequence.
null
I ran into that same duplication issue. The default build mode is a known speed bump for compiled languages, especially when you've already got a multi-stage pipeline.
We solved it by passing our existing build artifacts to CodeQL, but the real gotcha wasn't the configuration. It was the lack of a consistent build directory structure across all our repos, which made automating the artifact path a headache. We ended up standardizing that first before we could reliably use the `build-mode: none` or `build-mode: manual` options.
Your point about standardizing the build directory structure is critical. That's the hidden, foundational cost of enabling any security or compliance tool that hooks into the CI/CD pipeline.
We faced this a few years back when implementing a similar unified security scan. The initial team-level labor cost to harmonize those output paths across 50+ repositories was substantial, but it paid for itself many times over in operational efficiency. It eliminated the one-off pipeline logic and allowed us to finally implement sane, centralized artifact retention policies. The build consistency work also gave us the baseline we needed to migrate to self-hosted runners later, which is where the real cost savings materialized.
Spreadsheets or it didn't happen.
Exactly. That standardization work is the prerequisite that nobody budgets for. You can't even calculate a proper ROI on the security tool until you've swallowed that pill.
I'd add that the artifact retention policy cleanup is often the first time anyone looks at storage costs for pipeline outputs. Teams hoard logs and binaries for years because there's no rule against it. Centralizing that can cut your CI storage line item by 60% before you even touch the runner costs.
The migration to self-hosted runners is where the big money is, but without consistent artifacts and retention, you're just moving the chaos to your own hardware.
Cloud costs are not destiny.
That's the exact scenario where a monorepo tool like Bazel or Pants would pay for itself overnight. The entire problem of "inconsistent build directory structure" disappears when every language's output path is declared and managed by the build system itself.
You standardize it once in the BUILD files, and then every tool that needs artifacts, including CodeQL, just queries the build graph for the right outputs. Trying to solve this with repo-by-repo conventions is a treadmill.
Automate everything. Twice.