Okay, so I've been trying to use one of the popular coding assistants to write unit tests for a simple B2B SaaS API client class. I'm new to testing, so I was really relying on it.
But I think its suggestions are creating a false sense of security. It gave me a test that looked perfect, but it was actually testing the *mock* instead of the actual code. Like, it set up a mocked response, called the method, and then asserted that the mock was called with the right parameters. But the assertion was on the mock's internal call list, not on the actual result returned to my test.
It passed with flying colors, but if the real API client's logic for handling the response was completely broken, the test would still pass. Isn't that the opposite of what a test should do?
Has anyone else run into this? I'm starting to doubt every test it writes for me now. How do you guys vet its test suggestions?
You're right. Testing the mock's call list is verifying the test's own setup, not the unit under test. That's a mock verification test, which can be useful for checking interactions, but it's not a functional test.
The tool is just pattern-matching from public code, and that pattern is unfortunately common. You have to assert on the *output* of your method, not the mock's internal state.
Always ask: "What is the observable behavior or returned value for the caller?" Assert on that.
Five nines? Prove it.
You've hit on the fundamental limitation of these tools: they're great at generating code that *looks* correct, but they have zero understanding of intent. The pattern you describe is everywhere in training data, so that's what it regurgitates.
Worse, this breeds a generation of developers who think a passing test suite means something. You're right to doubt every test it writes. I'd extend your point: it's not just harmful for newbies. It lets seasoned devs get sloppy, outsourcing critical thinking to a statistical parlor trick.
How to vet? Treat its output like code from an intern who has read a lot of StackOverflow but never shipped anything. You have to review it with pure skepticism. Ask yourself, "What is this actually proving?" If the answer is "that we wrote a mock," delete it.
cg