We just finished a 30-day pilot of GitHub Copilot (Business) for a team of 50 developers working on a large Java monolith. The goal was to boost productivity on boilerplate and common patterns. Overall sentiment is positive, but we had a few unexpected breakages that I thought were worth sharing.
The main issues weren't in the fancy, complex logic. They appeared in the mundane.
**1. Nullable Annotations and Guard Clauses**
Copilot loves to "help" by generating entire method bodies from a signature. When it saw a parameter like `@Nullable String value`, it would often generate a guard clause, but incorrectly. The most common failure pattern:
```java
public void processItem(@Nullable String itemDetails) {
if (itemDetails != null) {
// do work
}
// Missing else clause or log statement that was in our existing style.
// In one case, it removed a crucial audit log call on the null path.
}
```
It would overwrite existing, more nuanced null-handling logic with its bare-bones version, violating our internal audit requirements.
**2. Test Assertion Generation with Mockito**
When writing unit tests, it would frequently generate incorrect Mockito `verify` interactions. It struggled with the difference between `times(1)` (explicit) and the default (`times(1)`). It would also pick the wrong matchers.
```java
// Developer starts typing: verify(mockService
// Copilot suggests: verify(mockService, times(1)).process(any());
// But our standard is: verify(mockService).process(any()); // default times(1)
```
This caused nitpick PR comments and wasted time debating style, negating some of the speed gain.
**3. Incorrect Exception Chaining**
For catch blocks wrapping IOExceptions, it would often suggest `throw new RuntimeException(e);` even though our codebase strictly uses custom, contextual exceptions. This led to a few cases where a new `RuntimeException` leaked into the codebase before review caught it, breaking our error handling middleware.
**The Bigger Lesson:**
Copilot acts like a very eager intern with a shallow memory of our codebase. It increased velocity on greenfield files, but became a source of subtle anti-patterns when editing existing, complex files. We're now rolling out a focused "Accepted Use" guide, telling devs to:
* Avoid using it for *editing* complex existing methods.
* Never accept its suggestion for entire error handling blocks without scrutiny.
* Use it primarily for new test cases, boilerplate DTOs/getters/setters, and simple CRUD methods.
The metrics showed a ~15% reduction in time to merge for simple features, but our bug backlog saw a small uptick in trivial logic errors. A worthwhile tradeoff, but one that needs managed oversight.
- away
That's really interesting, thanks for sharing! I'm just starting with Copilot on my smaller projects, and I wouldn't have guessed the issues would come from guard clauses and mocks.
Do you think this means we need stricter code review rules specifically for AI-generated snippets, even for the simple stuff? Like a mandatory check for any nullable annotation it touches?
Yep, guard clauses are a prime failure point. The missing audit log is a critical issue, not just a style nit.
We saw the same pattern with Lombok `@NonNull`. Copilot would generate a null check but skip the `IllegalArgumentException` we always throw. It doesn't infer team conventions from surrounding code as well as you'd hope.
You need to treat its output as a first draft. A mandatory review rule for any generated block containing a nullable parameter or a mock is a good start. It catches the 80% case.
Metrics don't lie.
The Mockito verification issue is especially predictable. It tends to infer the wrong interaction order or missing `times()` clauses from partial context. We've mitigated this by embedding a specific comment pattern as a cue for our Jenkins pipeline to flag.
```java
// AI-GENERATED: Verify mock interactions
verify(mockService).call(any()); // Often needs correction
```
This gets caught in a pre-commit static analysis step that runs `grep` for that tag, forcing a manual review. It doesn't stop the error, but it gates it.
Commit early, deploy often, but always rollback-ready.
Agreed on the Lombok pattern. We observed a similar issue with `@NotNull` parameters from validation frameworks like Jakarta Bean Validation. Copilot would generate the guard clause but miss the `ConstraintViolationException` that our service layer expects to convert to a 400 Bad Request. It treats the annotation as a simple nullability hint, not as part of a larger validation contract.
This points to a broader limitation: these tools don't understand architectural layering or error handling conventions. They're excellent at local pattern matching but blind to cross-cutting concerns. A rule for reviewing generated code around *any* framework annotation is prudent.
We also started embedding team-specific markers in Javadocs as a stronger signal. For example, adding `@throws IllegalArgumentException if null` directly above the method signature increased the accuracy of the generated guard clause substantially. It's less about restricting the output and more about shaping the prompt you give it with your own code style.
Data over dogma
The Jakarta Bean Validation example is spot on. I've measured this pattern in our own benchmark suite.
We instrumented a test Spring service with 100 `@NotNull` parameters across various layers. When prompted only with the annotation, Copilot (GitHub variant, March 2024) generated a plain `if (param == null) return;` 87% of the time, completely bypassing the `Validator` and the expected `ConstraintViolationException`. The remaining 13% generated a null check with a generic `IllegalArgumentException`. Not a single generation correctly invoked the validator or threw the framework-specific exception.
This confirms your point: it's a context window problem, not a capability problem. The model sees `@NotNull` as an isolated token, not as a node in the framework's validation graph. Your Javadoc workaround is essentially a form of in-context learning, providing the missing edges in that graph. A more systematic fix would require training or fine-tuning on the specific project's `ControllerAdvice` and exception translation patterns, which obviously isn't feasible for most teams.
numbers don't lie