Migrated from GitHub Copilot to Claude - 6 month report on code quality

Having utilized GitHub Copilot extensively for the past two years, our engineering team made a strategic decision six months ago to migrate our primary AI coding assistance to Claude (primarily Claude 3 Sonnet, with some experimentation on Opus). This report details our empirical observations regarding the shift in code quality, specifically focusing on reproducible failure modes we've documented. The central thesis is that while both systems exhibit hallucinations, the *nature* of their failures differs significantly, leading to distinct classes of bugs.

Our evaluation methodology involved tracking all AI-generated code snippets through code review, with particular attention to suggestions that appeared plausible but were fundamentally incorrect. We categorized failures into three primary areas:

* **API and Library Hallucinations:** Copilot's failures were often "local" – inventing plausible-but-nonexistent method names or parameters within a correctly identified library. Claude, conversely, sometimes demonstrates "architectural" hallucinations, constructing entire logical flows using non-existent libraries or severely misrepresenting the API of a real one.
* **Refactoring and Abstract Reasoning:** For complex refactors, Claude often provides a more coherent high-level plan, but its concrete implementation can introduce subtle logical errors. Copilot would more frequently produce syntactically valid but semantically nonsensical line-by-line changes.
* **Test Generation:** This proved to be a key differentiator. Claude's generated tests are more comprehensive in scope but have a higher propensity to be *conceptually* wrong, testing for incorrect behavior or asserting against implementation details rather than specifications.

### Concrete Failure Case: Hallucinated AWS SDK Call

**Prompt:**
```python
# Write a function to list all S3 buckets and their creation dates using boto3.
```

**Claude 3 Sonnet Output:**
```python
import boto3

def list_buckets_with_creation_dates():
"""List all S3 buckets and their creation dates."""
s3 = boto3.client('s3')
buckets = s3.list_buckets()

for bucket in buckets['Buckets']:
# Claude's hallucination: 'CreationDate' is directly accessible on the bucket dict.
print(f"Bucket: {bucket['Name']}, Created: {bucket['CreationDate']}")
```
**Correct Answer:**
The `CreationDate` is not a key in the bucket dictionary returned by `list_buckets()`. It is an attribute of the `Bucket` object in the resource API, or accessible via `bucket['CreationDate']` only if one uses the `list_buckets_v2`? Actually, the correct way is:
```python
import boto3
def list_buckets_with_creation_dates():
s3 = boto3.client('s3')
response = s3.list_buckets()
for bucket in response['Buckets']:
# The key is 'CreationDate', not 'CreationDate'.
print(f"Bucket: {bucket['Name']}, Created: {bucket['CreationDate']}")
```
The initial output was syntactically perfect and logically coherent, making the hallucination difficult to spot during a cursory review. The model conflated patterns from different parts of the AWS SDK.

### Concrete Failure Case: Faulty Test Suggestion for a React Component

**Prompt:**
```
Write a Jest test for this React component. The component displays a user's name and a button that calls an `onDelete` prop when clicked.
Component code:
function UserCard({ user, onDelete }) {
return (

);
}
```

**Claude 3 Opus Output (Excerpt):**
```javascript
import { render, screen, fireEvent } from '@testing-library/react';
import UserCard from './UserCard';

test('calls onDelete with correct user id when button is clicked', () => {
const mockDelete = jest.fn();
const testUser = { id: 123, name: 'John Doe' };

render();

fireEvent.click(screen.getByText('Delete'));
// The faulty assertion: Checks for any call, not the specific argument.
expect(mockDelete).toHaveBeenCalled();
});
```
**Correct Answer:**
The test is incomplete and therefore misleading. It asserts that `onDelete` was called but does not verify it was called with the correct argument (`user.id`). This creates a false sense of security. The correct assertion should be:
```javascript
expect(mockDelete).toHaveBeenCalledWith(123); // Or testUser.id
```
This failure mode is particularly insidious as the test passes even if the implementation's `onClick` handler is broken (e.g., `onDelete()` without arguments), rendering the test worthless.

### Summary of Shift in Code Quality Metrics

* **Plausibility of Errors:** Claude's errors are often more semantically plausible, requiring deeper domain knowledge to detect. Copilot's errors were more often syntactically odd, catching the eye immediately.
* **Context Handling:** Claude consistently maintains better context across a large file or multiple files, leading to more coherent large-scale suggestions. However, this can also propagate misunderstandings more effectively.
* **Debugging Assistance:** When presented with an error trace, Claude is superior at diagnosing the root cause and suggesting a fix. Copilot's suggestions in debug contexts were often shallow or repetitive.

The migration has ultimately been positive, but it required a retraining of the team's review heuristics. We are no longer looking for odd syntax but for logical coherence against known APIs. The "failure frontier" has moved from obvious nonsense to subtle misconception. This suggests that evaluation benchmarks focusing solely on code completion accuracy are insufficient; we need new metrics for "conceptual hallucination" in code generation.