Hey folks, just had to share some results from a recent exercise we ran. We've been running a public-facing demo of OpenClaw (that new open-source e-commerce platform) for a few months. Before considering it for a more critical internal use case, we decided to commission a 30-day penetration test from a reputable third party. The goal was to see what would get caught by our existing AppSec pipeline (SAST/SCA in CI, plus some basic DAST) versus a dedicated human tester.
The tl;dr is: our automated tools caught about 70% of the low-hanging fruit, but the pen tester found the really gnarly stuff. Our monitoring and observability stack (Datadog APM, tracing, and logs) was absolutely crucial in triaging the findings and validating fixes. Here are some of the key metrics we tracked during the test period:
* **Total Findings**: 42
* **Critical/High from Automated Tools (SAST/DAST)**: 8
* **Critical/High from Manual Pen Test**: 6
* **Mean Time to Acknowledge (using PagerDuty + Datadog alerts)**: 18 minutes
* **Mean Time to Remediate (from detection to deployed fix)**: 4.2 days
The most interesting find was a business logic flaw in the checkout flow that allowed cart tampering. Our SAST and DAST completely missed it because it required a specific, stateful sequence of actions. The pen tester exploited it, and we could see the anomalous request flow instantly in Datadog's APM traces for the `checkout-service`. Here's a snippet of the trace that flagged the issue—note the unusual sequence of `update_cart` calls after the payment intent was created:
```json
{
"service": "checkout-service",
"operation": "POST /update_cart",
"resource": "CartController#update",
"duration_ms": 45,
"meta": {
"cart_total_before": "199.99",
"cart_total_after": "1.99",
"trace_id": "abc123"
}
}
```
We set up a new synthetic test in Datadog to mimic the malicious flow post-fix, and now have a dedicated dashboard for AppSec metrics. It pulls in findings from our SAST/SCA tools, correlates them with deployment events, and tracks open risk over time. For anyone getting started, I can't stress enough the value of **integrating your security findings into your observability platform**. It turns abstract vulnerabilities into tangible, measurable events you can prioritize and track to zero.
The whole process really highlighted the need to treat pen tests not just as a compliance checkbox, but as a fantastic source of data for your monitoring systems. We're now feeding the pen test report's IOCs (Indicators of Compromise) into our log detection rules. Has anyone else done something similar? Would love to see how you're visualizing AppSec post-mortem data in Grafana or Dashboards.
Dashboards or it didn't happen.