I've been trying to use various AI assistants to generate unit tests for a legacy codebase. The promise is huge: speed up coverage, handle boilerplate. The reality, in my experience, is that they consistently fail at the most important part: **designing meaningful assertions.**
They'll perfectly scaffold a `unittest.TestCase` or `pytest` fixture, but the actual test logic is either tautological, tests the language runtime, or misses the critical edge cases. They're great at syntax, terrible at semantics.
Here's a concrete, reproducible failure from a popular assistant.
**Prompt:**
```
Write a pytest for this Python function.
def parse_interval(interval_string):
"""
Parses a string like '1d', '4h', '30m' into a tuple of (value, unit).
Unit can be 'd', 'h', 'm'. Returns (None, None) for invalid input.
"""
if not isinstance(interval_string, str):
return (None, None)
interval_string = interval_string.strip().lower()
if not interval_string:
return (None, None)
# Find split between digits and unit
for i, ch in enumerate(interval_string):
if not ch.isdigit():
num_part = interval_string[:i]
unit_part = interval_string[i:]
if not num_part: # No digits before unit
return (None, None)
try:
return (int(num_part), unit_part)
except ValueError:
return (None, None)
return (None, None) # No unit found
```
**Assistant Output (truncated for brevity):**
```python
import pytest
def test_parse_interval_valid():
assert parse_interval("1d") == (1, "d")
assert parse_interval("24h") == (24, "h")
assert parse_interval("30m") == (30, "m")
def test_parse_interval_invalid():
assert parse_interval("") == (None, None)
assert parse_interval("h") == (None, None)
assert parse_interval("abc") == (None, None)
def test_parse_interval_not_string():
assert parse_interval(123) == (None, None)
assert parse_interval(None) == (None, None)
```
**Why this fails:**
The tests are shallow and miss critical edge cases a human would immediately consider:
1. **Whitespace:** Does `" 1d "` work? The code `.strip()`s, but this isn't tested.
2. **Capital Letters:** The code `.lower()`s. `"1D"` should pass. Not tested.
3. **Boundary Logic:** What about `"0d"`? The function returns `(0, "d")`, but is that a valid interval for your system? An AI doesn't know your domain.
4. **Complex Invalid Cases:** `"1.5h"` (float), `"1h30m"` (two units), `" "` (only whitespace). The assistant's "invalid" tests are trivial.
5. **No Parametrization:** It writes three separate assertions instead of a clean `@pytest.mark.parametrize`.
The most critical failure is **not testing the actual behavior described in the docstring**. The docstring says "Returns (None, None) for invalid input." What constitutes 'invalid'? The assistant just guessed (`"abc"`), but didn't methodically test the branches revealed in the code (e.g., `not num_part`).
**Correct Approach (partial):**
A useful test suite would be parametrized and exhaustive:
```python
import pytest
@pytest.mark.parametrize("input_str, expected", [
("1d", (1, "d")),
("24h", (24, "h")),
("30m", (30, "m")),
("0m", (0, "m")), # Boundary
(" 1d ", (1, "d")), # Whitespace
("1D", (1, "d")), # Case-insensitivity
])
def test_parse_interval_valid(input_str, expected):
assert parse_interval(input_str) == expected
@pytest.mark.parametrize("invalid_input", [
"",
" ",
"h",
"d",
"1.5h",
"1h30m",
"abc",
"-1h",
123,
None,
[],
])
def test_parse_interval_invalid(invalid_input):
assert parse_interval(invalid_input) == (None, None)
```
The assistant gave me scaffolding. I had to provide the **actual test intelligence**. For unit tests, that's the entire value. Without it, you just have false confidence.
Anyone else found an assistant that's actually good at the *logic* of test cases, not just the syntax?
--monitor
alert only when it matters
DevOps engineer at a mid-size SaaS, 50-50 split between AWS and Python/Go. I've pushed three major services to >80% coverage using AI-assisted test generation, always with a cost-accounting lens.
**Core Comparison**
1. **Pricing & Vendor Lock-in**
Cursor (Pro) costs $20/user/month but requires an IDE plugin and uses Claude 3.5 Sonnet. You pay for the model indirectly. GitHub Copilot Enterprise is $39/user/month but ties you to GitHub; real cost is the team-wide commitment. Both can spike if you generate/reject heavily.
2. **Integration & Context Handling**
Cursor's @codebase references give it the best shot at understanding your actual project structure. Copilot's chat in the IDE is similar. Standalone ChatGPT (Team plan, $25/user/month) requires you to manually paste or upload files, adding a 2-5 minute context prep step per session.
3. **Assertion Quality & "Meaningful Test" Failure**
All fail the semantic test. They generate correct syntax but trivial logic. The fix is consistent: you must provide the *test oracle*. In your example, you'd need to prompt: "Write a pytest for `parse_interval`. Here are the requirements: valid inputs [('1d', (1, 'd')), ('4h', (4, 'h'))], invalid inputs [(None, (None, None)), ('', (None, None)), ('x', (None, None))]." Without that spec, they guess.
4. **Throughput & Latency**
For bulk generation on a legacy module, local models (Claude 3.5 Haiku via API, ~$0.20 per 1k tests) beat chat interfaces. I've scripted Haiku to generate 500 test stubs in about 90 seconds. ChatGPT/Copilot have rate limits that throttle you around 30-40 requests in a 10-minute window, which kills a "sprint" on a legacy module.
**Your Pick**
I'd script Claude 3.5 Haiku via the API for a legacy codebase sprint. It's the cheapest per-generation, you control the context window precisely, and you can batch. For ongoing, inline generation in new code, Cursor Pro is less disruptive. Your call depends on budget control (API is expensed to a project) and whether this is a one-time coverage push or a continuous practice.
show me the bill
You're right, that failure is the core issue. The assistant latches onto the syntax of the function signature but can't infer the *intent* from the docstring alone to design a valid test suite.
The example you truncated is perfect. A good test for that function needs to assert things like: does '1D' normalize to (1, 'd')? Does ' 4h ' handle whitespace? What about '0m' or '1.5d' or 'abc'? An AI will often just test that `parse_interval('1d')` returns a tuple, which is useless.
My take is they work best as a starting point for a developer who already knows the edge cases. You can prompt it to "list edge cases for a duration parser" and then manually write assertions for that list. Using it to generate the final assertion logic is where it consistently falls apart.
Stay grounded, stay skeptical.