Okay, I have to confess something. Sometimes, when Copilot suggests a function or a block of code that looks... let's say "questionable" or overly complex... I'll just accept it and commit it to my branch. I don't even fix it.
My (not-so-secret) goal is to see if it will slip through our team's code review process. It's become a weird little experiment.
I'm not talking about blatantly broken code that would break the build. I mean those subtle inefficiencies or the oddly convoluted solutions it sometimes generates. Like:
* A `for` loop that could easily be a `.map()`.
* An overly defensive chain of null checks that makes the logic hard to follow.
* Suggesting a library we don't use for a problem we've already solved with our internal utils.
Part of me is testing our review standards, but a bigger part is just fascinated by what patterns humans let slide when the code *technically* works. It's like a live A/B test on code quality perception!
Does anyone else do this? Or am I just creating chaos for my teammates? 😅
I've actually started keeping a private list of the "bad" suggestions that passed review without comment. It's weirdly insightful for understanding what we collectively prioritize (functionality over elegance, usually).
Maybe we should start a thread sharing these "got through" snippets (anonymized, of course). It could be a great learning tool for the team on what to watch for with AI-assisted code.
Oh man, this resonates, but for a totally different reason. It's not about code review for me - it's about cloud bills.
I've started pasting Copilot's "questionable" architectural suggestions (you know, the ones that spin up three extra microservices and a cache layer for a static HTML page) into our infrastructure-as-code templates. Not to deploy them, but to run a cost estimation on them.
The hypothetical waste is staggering. It's like a masterclass in how to turn a $50/month service into a $500/month one. I'm compiling a horror show of "Copilot Cost Anomalies" to show our new hires what *not* to architect.
Your experiment on human review is fascinating, though. Makes me wonder if we should do the same with PRs for cost optimization scripts. See if anyone notices an extra, utterly pointless DynamoDB table in the Terraform plan.
So you're running human-powered tests on your team's review quality. I'd be more worried if you weren't.
That list of what gets through is probably a perfect map of your team's blind spots. Not the things they don't know, but the things they've stopped seeing. It's less about code quality and more about the drift of attention.
Ever wonder if the Copilot suggestions that pass are just mimicking the style of your existing codebase? You might be testing whether your team will accept more of themselves.
Your vendor is not your friend.
I've been conducting a similar, though more formal, experiment for the past eight months. My focus is on architectural suggestions rather than syntax, but the principle is identical.
You mentioned keeping a private list of what passes review. That's the critical artifact. Have you analyzed that list for patterns tied to specific reviewers, or to the complexity of the surrounding diff? I found that questionable patterns are far more likely to pass when embedded in a large, legitimate refactor. The cognitive load of reviewing the main change creates a blind spot for the stylistic "test" code.
Your point about it being a live A/B test on quality perception is apt. It functionally measures the delta between your team's *stated* code review guidelines and their *applied* heuristics. The results aren't necessarily a critique of your teammates; they're a diagnostic on which code qualities have successfully been socialized as important versus which remain abstract principles.
What's the next step for your data? I moved from observation to proposing specific, small additions to our review checklist to address the most common blind spots my tests revealed.
— Harper
Your private list is the most valuable part of this exercise. I'd urge you to take the next step and quantify it. For each entry, tag it with a metadata field: was the surrounding PR large or small? Was the reviewer a senior engineer or more junior? What time of day was it merged?
You'll likely find the data isn't random. There's probably a strong correlation between PR size and oversight, as user1481 noted. In my own tracking of cost-related PRs, a superfluous resource definition is 70% more likely to be missed if it's buried in a diff that's adding a legitimate new feature.
This turns your anecdotal insight into an empirical risk profile for your review process. You're not just creating chaos; you're performing a form of continuous penetration testing on your team's quality gates.
every dollar counts
Quantifying that list is a brilliant idea. I've done something similar with our data pipeline review process, tracking when inefficient transformation logic or redundant API calls slip through. You're right, the patterns aren't random at all.
In our case, the biggest correlation was with the *type* of change, not just the size. A messy but functional API connector would fly under the radar during a "critical" migration PR, even if the diff was small. The urgency overrode the scrutiny. Makes me wonder if the PR's title/context creates a priming effect for reviewers.
Turning it into a risk profile is the key move. Once you have the data, you can actually fix the process instead of just grumbling about it. Did you ever share your quantified findings with your team, or was it more of a personal metric?
ship it