Just finished a full suite of integration tests using Windsurf. The main draw was the agent's ability to handle my containerized test environment. Results were mixed.
The good:
* Agent correctly spun up my test database container.
* Understood the context of my test files and suggested specific fixtures.
* Significantly faster than manually writing boilerplate setup/teardown.
The bad:
* Still hallucinated some non-existent pytest flags.
* Cost adds up when the agent runs long-lived processes (like containers). Had to watch my usage.
* For complex test data generation, I often had to step in and correct the logic.
If your tests rely on external services (databases, APIs), it's a decent accelerator. But you must verify its output—especially around cleanup steps. It's not a set-and-forget tool.
Benchmark or bust
Benchmark or bust
Sounds about right. I've seen the same pattern with AI assisted testing tools - they're great at scaffolding but stumble on the messy details.
The cost factor you mentioned is critical. When you're spinning up containers and external services, those minutes add up fast. It can turn a "cheap" experiment into a real line item if you're not careful.
The real question is whether it saves more time than it costs. For standard API contract tests, maybe. For testing complex business logic across systems? I'm still faster mapping the data flows myself.
Integration is not a project, it's a lifestyle.