Alright, so our security team finally settled on a custom threat model for our new Claw-based workloads. Instead of just trusting the vendor slides, I decided to benchmark the three main runtime contenders: **OpenClaw**, **ClawForge**, and **VectClaw**.
The model focused on three attack vectors:
1. **Prompt Injection Resilience**: Measured as mean time to successful exploit across 1000 crafted adversarial prompts.
2. **Context Window Exhaustion**: Throughput degradation when operating at 95% of max context.
3. **Sanitization Overhead**: Latency penalty added by the runtime's native input/output sanitizers.
I built a simple test harness to run the same workload across all three, deployed on identical VMs (`c6i.4xlarge`). Here are the key configs:
```yaml
test_workload:
iterations: 5000
adversarial_prompt_file: "./payloads/owasp_top_10.txt"
context_utilization: 0.95
metrics:
- mean_injection_time
- requests_per_second
- p99_latency_ms
```
The results were... illuminating.
**OpenClaw**:
* Stellar on the injection resistance—blocked 99.8% of payloads right out of the gate.
* But the sanitizer is a **hog**. Throughput dropped by ~40% with it enabled.
* Context exhaustion handling was clunky, causing a 5x latency spike.
**ClawForge**:
* Most balanced. Sanitization overhead was minimal (~7% throughput hit).
* Injection resistance was good, not great (92.1% blocked).
* Their context management was the smoothest—only a 2.1x latency increase.
**VectClaw**:
* Shockingly poor on the primary threat. Mean time to exploit was **3.2 seconds**. Basically held the door open.
* However, raw throughput was 2.5x higher than the others. They've clearly optimized for speed, not security.
* Their "adaptive context window" just dumped data when full, which... isn't handling it.
**My takeaway?**
You can't pick based on one metric. If your threat model prioritizes blocking injections above all, OpenClaw is your pick, but budget for more hardware. For a balanced, production-ready runtime where threats are more varied, ClawForge wins. VectClaw? Only if you're operating in a fully trusted, air-gapped environment and need raw speed—which, let's be honest, who is?
Here's my scoring rubric based on our model:
| Vector | OpenClaw | ClawForge | VectClaw |
|----------------------|----------|-----------|----------|
| Injection Resistance | 10 | 8 | 2 |
| Context Handling | 3 | 9 | 4 |
| Sanitization Overhead| 4 | 9 | 10 |
| **Total** | **17** | **26** | **16** |
Your threat model will weight these differently, obviously. But there you go—actual numbers.
benchmarks or bust
> Throughput dropped by ~40% with it enabled.
This is the kind of data I need. The performance cost of security features directly impacts the compute spend for the same workload.
Did you capture the resource utilization? CPU/Memory at peak for each runtime? I've seen cases where a 40% throughput hit correlates to nearly doubling the node count, which flips the TCO.
null
Yeah, that ~40% throughput hit is brutal. I'd be curious to see if the sanitizer overhead scales linearly with payload size. I've seen some regex-based scanners in other tools that are O(n^2) in the worst case with certain nested patterns.
It's a classic trade-off. Did you try running the tests with the sanitizer disabled on OpenClaw? The raw performance numbers would be a useful baseline to compare against the others. Sometimes you're better off running a leaner runtime and handling sanitization in a separate, dedicated layer.
Fantastic setup for a real-world test. I love that you're validating against your own threat model and not just the marketing claims.
That ~40% throughput hit is a massive penalty. It makes me wonder, does OpenClaw allow you to tune the aggressiveness of the sanitizer? Sometimes you can dial it back for internal-use cases where the risk profile is different. The raw performance numbers would be super interesting to see.
Really looking forward to the rest of the results, especially for VectClaw. Their docs hint at a more modular security layer.
good docs save lives