Migrated from Claude to Gemini for a 200-user shop - test generation failures

We recently completed a migration of our primary AI-assisted development contract from Claude (specifically, Claude 3 Opus via the API) to Google's Gemini 1.5 Pro. Our organization is a 200-person software shop, and the decision was driven by a combination of licensing cost predictability and our broader strategic alignment with the Google Cloud ecosystem. The migration was largely smooth for boilerplate generation and documentation tasks. However, we have encountered a significant and reproducible degradation in the quality of unit test generation, which directly impacts our team's velocity and code reliability.

The core failure mode is not a simple hallucination but a systematic misunderstanding of the scope and isolation required for a proper unit test. Gemini consistently generates tests that are, in essence, integration tests—they invoke large sections of the dependency tree rather than mocking or stubbing external services. This defeats the purpose of a fast, isolated unit test suite.

**Example Failure Case:**

* **Prompt:** "Generate a comprehensive Jest unit test for the following TypeScript function. Mock all external dependencies, including the `apiClient` and `logger` modules."

* **Function Under Test:**
```typescript
import { apiClient } from '../lib/api-client';
import { logger } from '../utils/logger';

export async function processUserTransaction(userId: string, amount: number): Promise {
try {
const user = await apiClient.fetchUser(userId);
if (!user || user.balance {
it('should return false for invalid user', async () => {
(apiClient.fetchUser as jest.Mock).mockResolvedValue(null);
// ... rest of test
});
});
```

Beyond this structural issue, we've documented other consistent failure patterns in test generation:

* **Incorrect Mock Assertions:** Gemini frequently suggests asserting on the mocked function's *return value* rather than using `toHaveBeenCalledWith` matchers to verify the interaction contract.
* **Poor Edge Case Coverage:** When prompted to "generate tests for edge cases," it produces variations of the same logical path (e.g., different numeric values) rather than true edge cases like network timeouts, malformed API responses, or `null` vs. `undefined` distinctions.
* **Async Test Flakiness:** Suggested implementations often neglect to handle async/await properly in test setup or assertions, leading to false passes.

From a product management and ROI perspective, this is a critical deficiency. The time saved by generating boilerplate tests is now consumed by senior engineers reviewing and rewriting these tests to meet our quality gates. The reliability of the suggested code is low, which increases rather than decreases cognitive load. We are currently evaluating whether to:
1. Invest in creating extensive, internal prompt engineering guides specifically for test generation with Gemini.
2. Re-architect our approach to use a different model (potentially a fine-tuned, smaller model) exclusively for test generation.
3. Revisit the cost-benefit analysis of maintaining a multi-vendor approach for different development tasks.

I am interested in whether other mid-size shops have encountered similar regressions in specific task categories during model migrations. More specifically, has anyone developed a robust prompting strategy or template that forces Gemini to adhere to proper unit testing isolation principles? Vendor stability and predictable performance across task types are becoming as critical as raw benchmark scores.