Having extensively evaluated multiple AI-powered coding assistants, including Cursor, Windsurf, and various locally-run models via Continue.dev, I've developed a nuanced perspective on their capabilities and limitations. Aider's promotion of "autonomous" code changes is a concept that warrants critical examination, as it potentially misrepresents the current state of the technology and sets users up for frustration.
My primary contention is that the term "autonomy" implies a level of independent, reliable decision-making that large language models simply do not possess in complex, real-world codebases. In controlled benchmarking scenarios, I've observed that even the most advanced models (GPT-4o, Claude 3.5 Sonnet) struggle with tasks that require deep, consistent project context over multiple iterations. The model's performance is critically dependent on:
* The precision and completeness of the user's initial prompt.
* The quality and structure of the context provided via the chat history and file mapping.
* The absence of ambiguous or conflicting requirements.
For example, requesting a "refactor the user authentication module to use JWT" might yield an initial seemingly correct diff. However, without explicit, step-by-step guidance, the assistant is prone to:
* Missing interconnected dependencies in other files (e.g., session management in middleware, updates to frontend token handling).
* Introducing subtle security flaws or logic errors due to a lack of true understanding.
* Failing to update related documentation or tests unless explicitly commanded to do so.
This is not autonomy; it is sophisticated, context-aware automation. The cognitive load of ensuring correctness, completeness, and architectural coherence remains firmly on the human developer. The tool excels as a powerful co-pilot for translating intent into code when the human maintains a tight feedback loop and rigorous oversight, but it cannot be entrusted with the "why" or the holistic impact of changes.
My benchmarks suggest that the most effective use of Aider is in a highly iterative, conversational style where the user breaks down large tasks into a sequence of verifiable, atomic operations. The marketing narrative, however, often leans towards a "set it and forget it" ideal that is not yet achievable. This creates a mismatch between user expectation (full autonomy) and operational reality (guided co-pilot), leading to dissatisfaction when the tool inevitably requires significant human intervention for non-trivial tasks.
Prompt engineering is engineering
I've been testing Aider as well, and your point about "deep, consistent project context" rings true. I ran into this trying to update a series of related API endpoint files. The assistant made a change in one file that seemed correct in isolation, but it broke a pattern used in three other files it didn't consider.
So I'm curious, in your controlled tests, how did you quantify the dependency on "the precision of the user's initial prompt"? Was there a threshold where slightly vague instructions fell apart completely, or was it more of a gradual degradation?
Great example with the API endpoint pattern. That's the exact kind of cascading failure I've seen.
On prompt precision, it wasn't a clean threshold. More like a reliability curve. A slightly vague prompt might work 70% of the time on a fresh, simple codebase. But in a legacy system with established patterns? That same prompt's success rate plummets. The degradation is steep when context gets complex.
It reminds me of sprint planning in Jira - a vaguely written ticket almost guarantees rework. The AI needs that same level of specificity to avoid breaking things it can't "see".