Having been through numerous SAST and code analysis platform evaluations, I find the most common mistake first-time evaluators make is focusing solely on feature checklists. The real differentiator for a platform like Checkmarx, especially for a performance-minded team, lies in its operational characteristics. You need to treat this like a performance benchmark of a critical backend service.
Here are the key metrics you should instrument and demand during your trial. I'd recommend setting up a dedicated, representative codebase (or a snapshot of your own) and running these tests systematically.
**1. Scan Latency & Throughput**
This is your P99 latency and requests-per-second equivalent. Don't accept "it depends." Measure it.
* **Initial Full Scan Time:** Clock the wall time from scan initiation to results being available for your representative codebase. Record the lines of code (SLOC).
* **Incremental/Delta Scan Time:** How long does it take to scan a small, representative change (e.g., a merged PR diff)? This is critical for CI integration.
* **Concurrent Scan Capacity:** If you have multiple teams/repos, can you run N scans simultaneously without linear degradation in scan time? Test with 2, 5, and 10 parallel scans.
**2. Resource Consumption Profile**
The tool's efficiency directly impacts your build infrastructure costs and stability.
* **Agent/Scanner Footprint:** What are the CPU, memory, and I/O demands on the build machine during a scan? Capture this using `time`, `/usr/bin/time -v`, or monitoring tools. A memory-hungry scanner can evict other critical services from cache.
* **Network I/O:** How much data is transferred to/from the Checkmarx central server? Is the scanner sending full source trees or compressed diffs? This matters for distributed teams and cloud egress costs.
**3. API Responsiveness & Integration Points**
The UI is one interface, but the API is what you'll use for automation. It must be predictably fast.
* **Results Retrieval API Latency:** Time from issuing a call for scan results (especially for a partial/incremental scan) to receiving a complete, parseable response.
* **Webhook/Notification Delay:** If configured to notify on completion, what's the delta between scan completion and webhook firing?
* **Authentication/Token Refresh Overhead:** Does the CI plugin or CLI tool require frequent re-authentication, adding dead time to short CI jobs?
**4. Query Performance & Triage Efficiency**
This is the "database optimization" angle. The speed at which your team can work with findings is a multiplier.
* **UI Filter/Query Response Time:** With a large results set (e.g., 10k historical findings), how quickly does the dashboard respond when filtering by severity, file path, CWE, or state (e.g., `status:not-false-positive`)?
* **Bulk Operation Performance:** How long does it take to triage (mark as not-exploitable, confirm, etc.) a batch of 100 similar findings? Slow triage creates backlog.
**Example Measurement Log Structure:**
Create a simple table to track your trial data.
| Metric | Test Repo A (50k SLOC) | Test Repo B (5k SLOC) | Delta Scan (200 SLOC change) |
| :------------------------------- | :---------------------: | :--------------------: | :--------------------------: |
| **Full Scan Wall Time (s)** | | | |
| **Peak Memory (GB)** | | | |
| **CPU Time (s)** | | | |
| **Results API Latency (ms)** | | | |
| **Findings Load in UI (ms)** | | | |
Without this data, you're buying based on spec sheet claims, not operational reality. The difference between a 4-minute and a 40-minute incremental scan will dictate whether you can feasibly run it on every PR or are forced to run nightly scans only, drastically reducing the value. The overhead of a slow, resource-intensive scanner can become a significant drag on developer velocity and infrastructure spend.
--perf
--perf