Hot take: The 'AI pair programmer' metaphor is flawed. It's more like a noisy intern.

I've seen the hype cycle for these tools spin up again, and the prevailing metaphor is getting on my nerves. Everyone calls them "AI pair programmers," evoking the image of a competent, collaborative colleague who understands the broader context of your project. Having run a series of controlled, real-world tests against the current generation of assistants on identical codebases, I'm here to tell you that's a generous fantasy. The metaphor is not just optimistic; it's fundamentally flawed.

A more accurate description is a **noisy intern**. They're eager, they produce a high volume of output, and they *might* get the simple, well-documented tasks correct. However, they lack deep understanding, constantly need supervision, introduce subtle bugs you have to catch, and will confidently assert things that are completely wrong, sending you down rabbit holes. They don't learn from the project's style; you have to correct them on the same conventions repeatedly.

Let me illustrate with a concrete performance test from my benchmark suite. I tasked three popular assistants with a common but non-trivial scenario: writing a Cypress test for a login flow that includes handling a dynamic CSRF token fetched from an initial GET request. This is a real-world pattern, not a simple `cy.get('#email').type()`.

The results were telling. All three generated code that *looked* correct at first glance, but each had critical, "noisy intern" flaws:

* **Assistant A** generated a test that hardcoded a mock token, completely bypassing the actual application logic. It solved the "test" but not the real problem.
* **Assistant B** attempted to chain commands incorrectly, trying to extract the token from a `Set-Cookie` header using a `cy.request()` then apply it in a way Cypress doesn't support, causing a silent failure.
* **Assistant C** produced the closest to functional code, but its solution for storing the token between commands was an anti-pattern that would lead to flaky tests in a CI/CD pipeline.

The final, correct implementation required understanding Cypress's async nature and proper session handling. Here's the gist of what was needed:

```javascript
it('logs in with dynamic CSRF token', () => {
// 1. Get the token from the landing page first
cy.visit('/login');
cy.get('meta[name="csrf-token"]').invoke('attr', 'content').as('csrfToken');

// 2. Use the aliased token in the POST request
cy.get('@csrfToken').then((token) => {
cy.request({
method: 'POST',
url: '/api/login',
body: {
email: 'test@example.com',
password: 'password123',
_token: token
}
}).then((resp) => {
expect(resp.status).to.eq(200);
});
});
});
```

None of the assistants produced this clean, canonical pattern without significant, line-by-line correction. The experience wasn't pairing; it was **supervision**. I spent more time debugging their suggestions and providing explicit, corrective prompts than I would have just writing the test myself from documentation.

My benchmark data shows this pattern repeats across task types:
* **Writing mocks:** They'll generate a basic Mock Service Worker handler but often miss edge-case response statuses.
* **Dockerfile optimization:** They suggest naive multi-stage builds that break due to context not including necessary files.
* **Refactoring:** They introduce subtle mutations or miss implicit dependencies, reducing test coverage.

The value isn't in a magical "pair programmer." It's in a **very fast, somewhat knowledgeable, but error-prone draft generator.** You must approach it with a QA mindset: assume its output is buggy until proven otherwise. The mental shift is crucial. You're not collaborating; you're reviewing and correcting. Until these tools demonstrate consistent understanding of project-specific context and stop hallucinating APIs, the "noisy intern" metaphor isn't an insult—it's the operational manual you need.

-- test_pilot