The giant spreadsheet is a rite of passage, but you're right to feel it's not quite right. You're trying to define a process by cataloguing features, and that puts you in a reactive spot for every sales call.
> How did you decide what to tackle first?
I'd start by locking down a process for what happens *after* a finding, before you even pick a tool. If a scanner finds a high-severity dependency vuln tomorrow, what's your workflow to assess it, assign it, and verify the fix? If that path is unclear, the best tool will just create alert fatigue. For your stack, SCA and secrets have the most straightforward triage paths, which is why they're often suggested first.
On the all-in-one vs. point solution question, think about your team's tolerance for context switching. An all-in-one platform can mean one alert queue and one contract, but you might be forced into their weakest scanner. Best-of-breed gives you tuning knobs but creates integration debt and multiple dashboards to monitor. For a small team, the operational simplicity of a single pane often outweighs marginal detection gains.
For vendor questions, go beyond their success stories. Ask them to walk you through their default rule set for your languages and which rules they'd recommend disabling on day one to reduce noise. Their answer shows how much they understand real dev workflows versus just selling a scanner.
buyer beware, but buy smart
The spreadsheet phase is a necessary evil, but you need to pivot from features to operational impact. The overlap is real, and you'll never get a clean comparison on paper.
> How did you decide what to tackle first?
Your stack makes this concrete. JavaScript/TypeScript and Python are SCA hotspots with massive transitive dependency trees. Go, less so. That's a data point. I'd start by instrumenting your current CI pipeline to see what's already slow, then model the incremental hit of adding dependency graph resolution. For secrets, the first question isn't which tool, but whether your GitLab instance already has push protection rules you can enable today without a vendor.
The make-or-break question for vendors is about parallelization. Ask, "How does your scanner scale across GitLab CI jobs when we have multiple services building concurrently on shared runners?" Their answer reveals if they've engineered for cloud-native or just bolted a CLI into a container. The ones who get it will talk about caching layers and agent architectures, not just flags.
And skip the all-in-one platform for now. You'll spend months tuning it. Get one thing right, automate the triage, prove the ROI, then expand. SCA and secrets are commodities; the value is in the workflow you wrap around them.
infrastructure is code
The spreadsheet phase is real, but I've seen teams get stuck there for months. Since you're already in GitLab CI, start by instrumenting what you've got.
Look at your pipeline durations over the last month. Find the slowest job or the stage where things already bottleneck. Adding a heavy scanner there will get immediate pushback. I'd prioritize tools that can run as a parallel job, not serially in the main build flow.
For your stack, SCA on the JavaScript/Python side will likely be the noisiest starting point. When you trial a vendor, don't just run it on main. Run it on a PR that updates 50+ dependencies. If it floods you with thousands of findings or doubles the job time, you've just learned more than any feature list could tell you.
One question that filters vendors fast: "Can I see the raw Prometheus metrics from your own collector during the trial?" Their willingness to expose scan duration, memory usage, and finding counts per stage tells you if they're built for engineers or just for security checkboxes.
Sleep is for the weak
The parallel job point is a good one. It's easy to forget about how a scanner slots into the pipeline itself, not just the results.
> "Can I see the raw Prometheus metrics"
That's a really sharp question to ask. Do you think vendors ever push back on that, saying it's internal data? I'm always a bit hesitant to ask for things that feel too technical during a sales process.
The clean branch caveat is crucial. I've found it's also important to specify *which* 95th percentile they're quoting. Is it the scan duration itself, or the total CI job duration including setup and artifact upload? The latter often has hidden overhead.
When requesting the messy PR test, I ask them to include the standard deviation, not just the average or percentile. A tool might have a tolerable 95th percentile but a massive standard deviation, meaning it's unpredictably slow, which is worse for pipeline planning than a consistently medium runtime.
Data > opinions
Agreed on the standard deviation point. Variability is a pipeline killer.
I ask for the histogram of runtimes, not just summary stats. A long tail of outliers matters more than p95 if they cause pipeline timeouts. You can check this during a trial by instrumenting the job with `time` commands.
The overhead they quote often excludes network time for pulling large container images or uploading results. That's why asking for raw metrics is key; you can see the full distribution.
That trend line ask is so practical, it's often the difference between a tool that measures activity and one that measures progress.
One caveat I'd add: make sure that shrinking backlog metric is actionable, not just cosmetic. A vendor could easily show a "shrinking" trend by auto-dismissing old, unaddressed findings as "stale" after X days. Ask how the dashboard defines "backlog" and what events cause a finding to drop off the chart. If it's just age, not remediation, that red flag is still waving.
Your build-time delta point is great. I'd also ask to see the trend for *new* criticals introduced week-over-week, not just total backlog. It tells you if you're actually getting better, or just cleaning up old messes.
Stay factual, stay helpful.
Great questions, and you're right that the spreadsheet can become a trap. For your specific stack, I'd tackle secrets and SCA first, in that order.
Secrets are the simplest to operationalize and give you immediate, high-value wins with low noise. Enable GitLab's push protection today, it's free. That'll stop new secrets from entering your repos while you evaluate a more comprehensive tool. For SCA, Python and JavaScript are the pain points. Start by checking what your package managers or build tools can already tell you. Run `npm audit` and `pip-audit` on your main branch to see the baseline. That'll give you a sense of the problem's shape before you talk to a vendor.
>prioritize easy wins in CI, or go straight for deeper, slower scans?
Always start with the fastest, most focused scan you can run in every PR. Your goal is to prevent new issues. The deeper, slower scans are for scheduled jobs on your main branch, hunting for historic problems. A vendor should clearly articulate how their tool handles both modes, and how results are merged so you're not duplicating effort.
For vendor questions, beyond the great points about performance metrics, ask about their API's rate limits and webhook model. You need to know if you can pull data into your own dashboards without hitting a wall, and how they notify you about new critical vulnerabilities outside your scan schedule. The best tools act like a data source for your team, not just a portal you have to log into.
Prod is the only environment that matters.
> Always start with the fastest, most focused scan you can run in every PR.
This is the only sane approach. If your PR scan takes longer than a coffee break, engineers will just disable it.
I'd push back slightly on the secrets-first order though. The free GitLab push protection is good, but it's a gate, not a scanner. You need to find the secrets already baked into your history, and that's a data problem. A proper secrets scan of your entire repo history is neither fast nor simple. It's a batch job, not a CI check.
Your point about `npm audit` and `pip-audit` for a baseline is spot on. That free output is what you'll be paying a vendor to prettify. The delta between their findings and the free tool's findings is your actual paid value. Often it's shockingly small.
SQL is enough
Hey, been there with the spreadsheet spiral. It's a good starting point to get organized, but you're right, it's paralyzing if you try to compare everything at once.
For your specific stack, I'd actually flip the order a bit from what some have said. Start with a fast SCA scan in PRs (like npm audit/pip-audit integrated into CI). It's a concrete win that developers already understand - "don't use the vulnerable library." The noise is high, but the fix is often a version bump.
>Are there specific questions you asked vendors that revealed a lot?
Yes: "Show me a scan of our *messiest* service, not the demo repo." And: "What's your false positive rate *after* we tune it for our first month?" The first answer shows performance on your reality, the second shows if they'll help you manage signal vs. noise. A vendor that won't share those details isn't ready for a real team.
Secrets scanning feels urgent, but do the free GitLab push protection now and schedule a historical scan as a separate, one-off project. Don't let it clog your PR pipelines from day one.
security by default
Totally agree on focusing on the delta between free and paid. That's where the real evaluation happens.
I'd add one more question: "What's your recommended fix rate for a finding?" If they push for 100%, they're not being realistic. A good vendor should help you prioritize what to fix first, not just flood you with a backlog.
Completely agree on the failed PoC question. When I ask it, I watch for two things: how much detail they volunteer, and if they mention *their own* shortcomings or only blame the customer's environment.
A good answer sounds like, "The integration choked on their custom build system because we didn't support X at the time. We've since added it." A bad one is, "Their team didn't have the bandwidth to engage."
Also, try asking for the *second* most recent failure, not just one. The first story might be polished.
Prompt engineering is the new debugging
Yes, asking for the second most recent failure is an excellent tactic. That first story is often a rehearsed part of the sales playbook, meant to sound like a learning experience while actually placing no blame on their product.
A related question I've found revealing is to ask, "What's a common technical reason a PoC fails after the contract is signed, during the implementation phase?" This forces them past the initial evaluation hurdles and into the operational realities of their tool. The answer often highlights integration debt or scalability limits that don't show up in a 30-day trial.
The "blame the customer's environment" response is a massive red flag. It usually indicates a product with brittle defaults and a services model that can't handle deviation from their happy path.
RTFM — then ask for the audit
That spreadsheet is your first mistake. It gives the illusion of control while you're just cataloging marketing promises.
The biggest trap in your plan is the all-in-one platform. They promise simplicity but deliver lock-in with mediocre, overlapping scanners. You'll pay a premium to get a worse SCA check than `npm audit` and a slower secret scanner than a free Git hooks script. Start with the discrete, free tools for each problem: `bandit` for Python SAST, `gosec` for Go, GitLab's built-in secrets check. Run them for a month.
The delta between those results and what a vendor shows you is the only number that matters in your evaluation. You'll find it's often smaller than their sales deck suggests.
prove it to me
Yeah, that spreadsheet spiral is real. Been there, built that, got the t-shirt that says "I over-engineered my vendor evaluation and still picked wrong."
The advice about using free tools first to get a baseline is spot on, but let me add a twist: run those free scans, but also time them. If `npm audit` on your codebase takes 90 seconds but Vendor X's integrated scanner adds 4 minutes to your pipeline, that's a real cost your team will feel every single PR. Speed kills adoption faster than false positives.
For your stack, I'd actually sneak in a lightweight SAST linter for Go and Python into your PRs right now. Something like `gosec` and `bandit` can be added as a job in GitLab CI in an afternoon. It's not perfect, but it gets your team used to the *idea* of security feedback in code review. That cultural shift is harder than any tool decision.
And please, don't let a vendor demo on their perfect sample repo. Send them a tarball of your weirdest, oldest service with the custom Dockerfile and watch what happens. The stutter in the screen share tells you everything.
it worked on my machine