Everyone's raving about CodeQL being the "future" of SAST. Seen any real comparisons for Java shops?
We run Fortify on the legacy monolith. Checkmarx on newer microservices. Management wants to consolidate on GitHub because it's "integrated." I'm skeptical. I need concrete numbers, not marketing slides.
Ran a quick test on a known vulnerable sample app. CodeQL found the SQLi and path traversal. Missed a deserialization vuln that Fortify flagged. Checkmarx report was, predictably, a mess of false positives.
Typical CodeQL query for the SQLi finding:
```ql
import java
from MethodAccess call, Expr arg
where
call.getMethod().hasName("executeQuery") and
arg = call.getArgument(0) and
not isSafe(arg)
select call, "Potential SQL injection"
```
Anyone done a proper bake-off? False positive/negative rates? How's the custom query writing compared to Fortify's rules?
SQL is enough
Your quick test mirrors what I've seen in enterprise rollouts. CodeQL's out-of-the-box queries for common issues like SQLi and path traversal are excellent, but it often misses the more complex vulnerability patterns where Fortify's deep language packs have years of tuning.
> How's the custom query writing compared to Fortify's rules?
That's the critical difference. CodeQL's custom query language is a real programming environment. You can model data flow with a precision Fortify's custom rules can't match, but the learning curve is steep. A Fortify rule is often a few lines of XML matching a pattern. In CodeQL, you're writing a full data flow analysis. For that deserialization vuln it missed, you could likely write a custom query that tracks unsanitized data into `ObjectInputStream.readObject()`, but you need the expertise to do it.
The consolidation push for "integration" is a real risk if your team lacks the cycles to develop that in-house CodeQL competency. You'll get great coverage on the OWASP Top 10 and a cleaner report than Checkmarx, but you might lose depth on your specific, inherited codebase oddities. Have you looked at the ability to import Fortify's results into the GitHub code scanning SARIF format for comparison? That's been a useful middle ground for some teams I've advised.
- Mike
Your quick test's latency profile, between scan initiation and first actionable result, is the critical metric CodeQL struggles with on large monoliths. Fortify's incremental analysis cache means subsequent scans on a stable codebase can complete in minutes, while CodeQL rebuilds the entire database. For a 500k-line Java monolith, I've seen initial CodeQL database creation take over 45 minutes on a 32-core runner, versus Fortify's translation phase completing in under 15. The integrated GitHub runner's time limit becomes a real constraint.
On the custom query point, yes, CodeQL's data flow precision is superior, but you pay for it in analysis time. Writing that deserialization query involves defining additional taint steps through reflection or custom wrappers, which adds significant overhead. A Fortify XML rule for the same pattern might be less precise but executes in a fraction of the time during the regular scan cycle.
Have you measured the total wall-clock time for a full scan pipeline, including database creation, for your codebase sizes? That's often where the "integrated" argument falls apart.
Every microsecond counts.
Your test lines up almost perfectly with the bake-off I ran on our services last quarter. The deserialization miss is a known gap in the standard query pack, but like user425 said, you can absolutely write a custom query for it. The challenge is the time investment.
The more concrete number you should pressure-test is the false positive rate on a *large* codebase. On our 300k-line monolith, CodeQL's default Java suite had a false positive rate around 22%, which was better than Checkmarx's 40%+ mess but not as good as Fortify's tuned rules at 15%. However, CodeQL's false positives were *different* - often more nuanced logic flow issues that were harder for junior devs to triage.
The custom query comparison is night and day. A Fortify rule for that deserialization vuln is maybe 10 lines of XML. The equivalent CodeQL query modeling taint through the reflection layer was over 100 lines of QL for me. More powerful, yes, but it feels like you're paying for that precision with developer hours. Is your team prepared to become part-time security researchers?