You're evaluating a SAST/SCA tool like Veracode, so you're thinking about security findings. That's good, but from an observability perspective, you're missing the point if that's all you track. You need to instrument the *process*, not just consume the output. Your PoC metrics should tell you if this tool can be integrated into your workflow without killing velocity and if the signal is actionable.
Track these three categories:
**1. Integration & Performance Metrics**
These are your SLOs for the tool itself.
* Scan duration: from submission to results. Track p50, p90, and p99. A p99 that's 10x your p50 means unpredictable pipelines.
* API reliability & error rates: Failed scans or timeouts during your normal usage patterns.
* Resource consumption: How much CPU/memory does the agent use on your build nodes? This is a direct cost driver.
**2. Signal Quality Metrics**
This is where you separate noise from actionable alerts.
* Precision: Percentage of findings that are true positives upon triage. Sample a few hundred.
* Burn-down rate: How many findings are fixed vs. how many new ones appear over the PoC period. A flat line is a bad sign.
* Breakdown by severity and type: Raw counts are useless. You need a histogram. Example Prometheus query for your dashboard:
```
sum by (severity) (veracode_findings_total{application="your_app"})
```
**3. Operational Impact Metrics**
If your team can't or won't use it, it fails.
* Mean Time To Triage (MTTT): How long from a finding appearing to a developer labeling it (e.g., true positive, false positive, accepted risk). This measures clarity.
* Developer feedback: Simple survey score (1-5) on two questions: "Was the finding easy to understand?" and "Was the fix guidance useful?"
* Pipeline blockage frequency: How often did a scan failure or a critical finding (misconfigured) stop a deployment that shouldn't have been stopped?
Don't just collect a PDF report at the end of the PoC. Build a basic Grafana dashboard from day one. Feed data in manually if you have to. If you can't measure the tool's performance and impact, you're just taking a vendor's word for it.