Just built a test suite of 50 buggy Python snippets to compare fix rates.

Hey folks! I've been deep in the benchmarking rabbit hole again. This time, I wanted to move beyond simple "write this function" tasks and test something more practical: **bug-fixing capability**.

I assembled a suite of 50 small, buggy Python snippets. They cover common gotchas:
* Off-by-one errors in loops
* `NoneType` and `AttributeError` issues
* Incorrect list/dict operations
* Scope and mutable default argument problems
* Basic logic errors

I ran them through three assistants, all using their latest generally available versions (as of this week), with the same system prompt: "Fix the bug in the following Python code. Only output the corrected code."

Here's a quick example of one task:

```python
# Buggy input given to each assistant
def calculate_average(numbers):
total = 0
for i in range(len(numbers)):
total += numbers[i]
average = total / len(numbers)
return average

result = calculate_average([])
print(result)
```

The raw **fix rate** results (code runs without error and produces correct output):

* **Assistant A:** 42/50 (84%)
* **Assistant B:** 38/50 (76%)
* **Assistant C:** 45/50 (90%)

What I found most interesting wasn't just the pass/fail count, but *how* they failed. Assistant C, for instance, aced most logic bugs but stumbled on a few tricky `None`-handling cases. Assistant B often produced *working* but oddly non-idiomatic fixes, like adding unnecessary `try-except` blocks.

Has anyone else run similar comparative tests? I'm thinking of adding a layer that measures not just if it runs, but if the fix follows Python best practices. Might set up a GitHub Actions workflow to re-run this batch on every new model release 😅

-pipelinepilot