Has anyone benchmarked ChatGPT's code suggestions against GitHub Copilot in VS Code?

Everyone's so eager to crown a winner in the "AI pair programmer" wars, but I haven't seen a single benchmark that matters. They're all measuring token-per-second or how many lines of boilerplate it can vomit. That's not the metric. The metric is how many production incidents it helps you create.

I ran my own unscientific, deeply cynical test over the last month. Same project, same tasks: writing Kubernetes operator logic, debugging service mesh (Istio) configuration, and crafting Terraform for a new cloud service. I forced myself to use Copilot for one sprint and ChatGPT (via the API in a custom VS Code extension) for another.

The difference wasn't in speed. It was in the *character* of the mistakes.

* Copilot's suggestions are like a overeager intern who's read all the docs but never been paged at 3 AM. It'll perfectly replicate the bad patterns already in your codebase. It once suggested a Pod spec with no resource limits or readiness probe because that's what the surrounding, terrible legacy code had.
* ChatGPT, when given the full context, tries to be "correct" in a textbook sense, which is often dangerously naive. Ask it to write an EnvoyFilter for Istio and it'll give you a syntactically valid YAML block that implements a circuit breaker with defaults so aggressive it would drop half your traffic on a single slow dependency.

Here's the kind of "help" I mean. Asked both to generate a simple Go function to parse a duration from a ConfigMap string for use in a Kubernetes controller.

ChatGPT's typical output, prone to ignoring production realities:

```go
func parseDuration(configMapValue string) time.Duration {
dur, err := time.ParseDuration(configMapValue)
if err != nil {
// Default to a safe value
return 5 * time.Minute
}
return dur
}
```

A "safe default"? In a controller? That's a silent failure mode waiting to happen. Now your operator applies a configuration nobody intended.

Copilot, trained on more real-world garbage, will likely just mimic the existing error handling in the file—which might be logging and returning zero, or worse, panicking.

So, benchmark completion accuracy if you want. I'm more interested in which tool requires less cognitive load to *vet* before its code touches a commit. Right now, that load is still crushingly high for both. They're both fantastic at creating the illusion of productivity while quietly planting time bombs. Has anyone else done a comparison that looks at the *quality* of the wrong answers, not just the quantity of the right ones?