Check out my script that benchmarks Claw against GitHub Copilot on real PRs

Hey everyone, I've been seeing a lot of buzz around the new AI coding assistants, especially the ones like Claw that promise deeper codebase context. As someone who lives in pull requests and CI logs, I wanted to move beyond toy examples. The marketing claims are one thing, but how do they perform on the messy, real-world PRs we deal with daily? Specifically, the kind that involve infrastructure-as-code, CI pipeline tweaks, and Kubernetes manifests.

So, I spent last weekend building a script to benchmark Claw (using their CLI beta) against GitHub Copilot Chat (via the CLI as well) on a set of actual, anonymized PRs from our internal repositories. The goal was to simulate a realistic "assist" scenario: given a PR diff and the surrounding code context, how useful and accurate are the suggestions for completing or reviewing the change?

The script works by feeding a chunk of the PR context—the diff, relevant files, and the PR description—into each tool with a standardized prompt set. I measured a few key things:
* **Code Completion Accuracy:** For a partially written function in the diff, did the suggestion actually compile and fit the pattern?
* **Review Insight Quality:** Asking "are there any security or performance issues in this diff?" and grading the specificity.
* **Infra-as-Code Understanding:** How well it handled Terraform module changes or GitHub Actions workflow suggestions.

Here's the core of the test harness (Python, using the OpenAI SDK for Claw and the GitHub CLI for Copilot context):

```python
def benchmark_pr(pr_context_file, test_type):
"""
pr_context_file: JSON with 'diff', 'related_code', 'pr_description'
test_type: 'completion', 'review', 'refactor'
"""
with open(pr_context_file) as f:
context = json.load(f)

prompt = build_prompt(context, test_type)

# For Claw (simulated as it's not public API)
claw_response = query_claw_simulated(prompt, max_tokens=500)
# For Copilot Chat
copilot_response = query_copilot_chat(prompt)

return {
'claw': claw_response,
'copilot': copilot_response,
'eval': evaluate_response(test_type, context, claw_response, copilot_response)
}
```

**Initial Findings on 15 PRs (mostly Terraform & GitHub Actions):**

* **GitHub Copilot** was consistently faster and more concise in its suggestions. It excelled at line-by-line completions and knew the latest GitHub Actions syntax inside out. However, when the PR involved a larger architectural change, like refactoring a Terraform module to use a new AWS service, its suggestions were shallower.
* **Claw** demonstrated a noticeable advantage on PRs that required cross-file understanding. For example, when a PR changed a Kubernetes deployment, it correctly referenced the related Service and Ingress files without me explicitly providing them. The trade-off was significantly slower response times and more verbose output, which isn't always ideal.
* **The "So What?" for Our Stacks:** If your workflow is heavily based on quick, inline completions within your IDE, Copilot remains king. But if you're dealing with complex, distributed systems changes and you want an AI pair that can reason across your `kustomize` or Terraform graph, Claw's approach seems promising, assuming they can improve latency.

I'm planning to extend this to include GitLab CI and possibly some cost-optimization scenarios (like suggesting cheaper machine types in a GKE nodepool). Has anyone else run similar comparisons? I'm particularly curious about how these tools handle Helm charts or monitoring mixins.

Automate everything