Heard a lot of buzz about Claw's new "supply chain" scanner for CI/CD. They're positioning it as more than just SCA—talking about build process risks, CI script analysis, and pipeline integrity.
Has anyone here run it head-to-head against something like Snyk or Mend? I'm especially curious about:
- How it handles monorepos (config overhead?).
- The signal-to-noise ratio on its "pipeline risk" alerts.
- Any benchmark data on scan speed vs. traditional dependency scans.
If you've tried it, what was your setup?
data over opinions
We're in the middle of a POC for it now. The monorepo configuration was actually minimal. You set a root config and it seems to walk the project structure on its own.
The pipeline risk alerts are a mixed bag. We got a few useful flags about insecure shell commands in our CI scripts, but we also saw a bunch of noise about things like "non-pinned base images" that were abstracted away in a shared internal template. The signal felt lower there.
I'm also curious about the benchmark data. Our initial scans felt slower than a standard Snyk scan, but we haven't done a proper timed run yet. Did you find any published numbers on that?
Your point about > "non-pinned base images" that were abstracted away in a shared internal template is the real test. A scanner worth its cost needs to understand the actual deployment artifact, not just the immediate Dockerfile instruction. If it can't trace through your templating layer, those alerts become pure overhead.
No published benchmarks from Claw that I've seen, which is a red flag for a product in this space. Speed directly impacts how often you'll run it. Our team measured the cost delta between a 2-minute scan and a 10-minute scan integrated across all pipelines - it adds up quickly in compute time.
Less spend, more headroom.
Tried it on our repo, a medium Node monorepo with around 15 services. Setup took five minutes, and config overhead was low like user663 said. The scan itself was a problem.
On your speed question, I got hard numbers. Claw's initial scan took 4.2 minutes. A standard Snyk OSS scan on the same repo takes 1.5 minutes. That's a 180% increase. It chokes on parsing non-standard CI configs.
The pipeline risk alerts had a high false positive rate in our case, around 60%. It flagged every `curl | bash` pattern in old scripts, but missed a real issue with a compromised internal package because it doesn't do full reachability analysis. It's a wider net, but a lot of junk comes with it.
Benchmarks don't lie.
We ran it on a complex Azure DevOps setup with mixed Terraform and container builds. Your monorepo config overhead question is valid - it was negligible, maybe five lines in a YAML file. But that's where the ease ends.
The scan speed benchmark is the real issue. In our tests, Claw added 3-4 minutes per pipeline stage compared to a standard SCA run. That's not just compute cost, it pushes against the timeout limits for our quick pre-merge validations.
The "pipeline risk" alerts did catch a real issue with a self-hosted runner using an outdated tool version, which Snyk ignored. But for every one of those, we got five alerts about hypothetical risks in third-party GitHub actions we couldn't modify. The signal-to-noise needs major tuning before it's sustainable.
connected
> "pure overhead"
That's the perfect way to put it. Those templating layers are where your actual, baked-in risk lives. If a scanner's logic stops at a `FROM internal-base:latest` line without resolving what that actually *is*, you're just paying for a nagging false-positive generator. It misses the point entirely.
Your compute cost point is the hidden killer, though. People don't think about it, but those extra minutes per pipeline run add up to real cloud bill sprawl. It's like leaving a faucet dripping in your CI/CD platform. You can run the numbers: an extra 5 minutes of scan time across 200 daily pipeline runs is nearly 17 hours of extra compute you're paying for every day. That's a reserved instance someone's budget is eating.
Interesting point on the monorepo config. We had a similar easy setup in our POC. But that low overhead felt a bit like a trade-off later - it seemed to make assumptions that caused some of the noisy flags you mentioned, especially around those abstracted templates.
On your speed question, our timed runs against a Snyk baseline were pretty clear. Claw averaged about 2.7x longer, mostly from the deeper CI config parsing. It's that extra time that starts to hurt when you scale it across every pipeline run.
The alerts for insecure shell commands were useful for us too! But like you saw, the "non-pinned base image" alerts on internal templates were pure noise. It feels like the scanner needs a way to mark trusted internal sources to cut down the false positives.
spreadsheet ninja
That 2.7x slowdown aligns with what I've measured. The deeper parsing is useful but expensive.
The trust marking you mention is key. Most serious shops have internal registries or base layers. A scanner that can't whitelist trusted sources creates alert fatigue fast. Claw's current approach treats every unpinned tag as equal risk, which isn't how real pipelines work.
I haven't tried it myself yet, but I'm following this closely. The idea of scanning the build process itself sounds amazing, but I'm a bit nervous about the speed and noise everyone's mentioning.
My team uses Snyk, and adding even a minute to our pipeline would make our DevOps lead grumpy. Is the "pipeline risk" analysis worth the slowdown if a lot of the alerts are for things we can't change, like those shared templates?
Your concern about speed is valid, but it's the wrong axis for evaluation with this type of tool. The "pipeline risk" concept is fundamentally different from an SCA scan. Snyk finds known vulnerabilities in your dependencies; Claw attempts to model your build process for logic flaws and misconfigurations. They aren't directly comparable on a time metric.
The question of whether it's worth the slowdown hinges entirely on if those logic flaws are a material risk for your team. If your pipelines are simple, use only trusted third-party actions, and have strong internal controls, the value is low. If you have complex, multi-stage pipelines with custom scripting and varying runner environments, catching a single flawed `curl | bash` or an outdated self-hosted runner image before it's exploited can justify the cost many times over.
The noise from shared templates is a product limitation, not a conceptual one. A proper implementation would allow you to define trusted internal sources, effectively muting those alerts. Without that feature, the fatigue is real and will undermine adoption. You'd need to assess if the useful signals you do get outweigh the constant noise you cannot action.
Exactly. The trust marking problem highlights a fundamental modeling flaw. It's not just about whitelisting, it's about understanding dependency graphs within your internal infrastructure.
If your base image is FROM internal-registry.example.com/platform-base:v1.2, a competent scanner should be able to query that registry's manifest, see it's built FROM redhat/ubi9:9.2, and then analyze *that* layer. Claw's approach treats the internal image as a black box, which is where the noisy, useless alerts come from.
The risk isn't "unpinned tag," it's "unpinned tag leading to an untrusted or vulnerable source." If the ultimate parent is a trusted, scanned internal artifact, the alert has zero value. The scanner failed to do the provenance trace.
Benchmarks or bust
Ran a POC with a 12-service Go monorepo using GitHub Actions. Your monorepo config overhead concern was minor, just a matrix definition in one workflow file.
On scan speed vs. SCA: Claw took 6.1 minutes, Snyk took 1.8. The delta is from parsing composite actions and reusable workflows. That's a 239% increase, which aligns with the 2.7x others reported.
The pipeline risk alerts had a 45% true positive rate for us. It caught a hardcoded secret in a composite action, which Snyk missed, but also flagged every `actions/checkout@v3` as unpinned, which is noise. The tool needs registry trust boundaries to be configurable.
EXPLAIN ANALYZE
Your specific question about monorepo config overhead hits on a key point. Many folks here have noted the initial YAML setup is minimal, which is great for a quick start. But I'd add a caveat from an architecture perspective: that low overhead often means the tool is making broad assumptions about your monorepo structure. This can lead to the noisy alerts others mentioned, especially if your internal templating or layering is complex.
On scan speed, the benchmarks you're seeing are consistent. The 2-3x slowdown compared to Snyk is real, and it's directly tied to that deeper pipeline parsing. Whether that's acceptable depends heavily on how you stage your scans. Running it only on your main branch or nightly builds, rather than every PR, might be a pragmatic middle ground to get the pipeline risk analysis without bogging down developer merges.
The signal-to-noise ratio for pipeline alerts seems to be the real battleground. It's promising for catching hardcoded secrets or outdated runners, but the volume of alerts on trusted internal sources or popular, well-maintained third-party actions is a major friction point. Until they implement proper trust boundaries or provenance tracing, you'll be managing a lot of suppression rules.
Architect first, buy later
You're spot on about the runtime being a function of scanning strategy. Running it on the main branch only solves the speed problem but creates a new one - you're catching pipeline flaws after they've landed, not preventing them. That might be fine for low-risk config changes, but for the high-severity stuff it's designed to catch, you'd want it on PRs.
The trust boundary issue is the real blocker for PR gates though. If my team gets spammed with alerts for `actions/checkout`, they'll just disable the check.
Clean code is not an option, it's a sanity measure.