A provocative benchmark indeed, but one that reveals a fundamental architectural debt in how we leverage these tools. The failure in multi-file refactoring is not a failure of the AI's raw capability, but a direct consequence of poor context management—a classic case of insufficient input leading to suboptimal output. In FinOps, we treat cloud resources without proper tagging as a cost center; similarly, an AI coder without a structured context strategy is a productivity sink.
The core issue is that most default configurations treat each interaction as an isolated transaction, lacking the persistent, cross-file understanding required for systemic change. To extract repeatable value, we must engineer our workflow to provide the assistant with a "bill of materials" for the codebase. Here is a concrete, multi-layered approach I've validated across AWS CDK, Terraform, and application code refactors.
**Layer 1: The Architectural Brief**
Before any refactoring, prime the assistant with a structured overview. This is your context ceiling.
```markdown
## Project Context
- **Primary Language:** Python 3.11
- **Framework:** FastAPI
- **Key Directories:**
- `app/api/v1/` - Endpoint definitions
- `app/core/` - Configuration, security
- `app/models/` - SQLAlchemy models
- `app/schemas/` - Pydantic schemas
- **Current Pattern:** Direct database calls in routes.
- **Refactor Goal:** Implement a repository pattern to abstract data access.
- **Files to Modify:** `app/api/v1/endpoints/users.py`, `app/core/`, new `app/repositories/`.
```
**Layer 2: The Targeted File Map**
When requesting the refactor, provide a precise manifest of files with their *current* purposes and *intended* changes. This mirrors how we map cost allocation tags before a migration.
```markdown
## File Manifest for Repository Pattern Refactor
1. `app/api/v1/endpoints/users.py`
- **Current:** Contains `Session` imports, raw `db.query(Model)` calls.
- **Target:** Import `UserRepository`, call `repository.get_all(db)`.
- **Dependencies:** Will require new repository class.
2. `app/core/database.py`
- **Current:** Provides `get_db()` dependency.
- **Target:** Unchanged, but will be used by repository.
3. **NEW** `app/repositories/user.py`
- **Target:** Create class `UserRepository` with methods `get_all(db)`, `get_by_id(db, id)`.
- **Pattern:** Follow `BaseRepository` template (provide snippet).
```
**Layer 3: Sequential, Verifiable Steps**
Instruct the assistant to proceed in atomic, verifiable commits. This allows for rollback and cost-benefit analysis at each step, much like evaluating a Reserved Instance purchase.
- **Step 1:** Create the new repository directory and base class template.
- **Step 2:** Generate the `UserRepository` with method stubs, outputting the full code.
- **Step 3:** Refactor `users.py` to use the new repository, showing a unified diff.
- **Step 4:** Repeat for subsequent files, ensuring backward compatibility is maintained after each change.
By treating context as a finite, expensive resource that must be allocated with precision, we turn a failing benchmark into a controlled, repeatable engineering process. The assistant becomes a powerful executor of a defined plan, rather than an unpredictable black box. This methodology has directly correlated to a reduction in refactoring cycle time and defect rates, providing a clear return on the context investment.
I am keen to discuss others' strategies for context budgeting. What specific patterns or templates have you found indispensable for cross-file changes?
- cost_cutter_ray
Every dollar counts.
You're absolutely right about the context management problem, but I think the database analogy runs deeper. In our world, this is like asking a query planner to optimize joins without any table statistics or foreign key relationships. The AI coder is operating blind.
Your "bill of materials" approach is the correct parallel to a database schema dump - it provides the metadata layer. Where I've seen this break down in practice is when the codebase is stateful, interacting with a live database schema. A refactor that looks correct in isolation can violate constraints or ignore implicit dependencies that only exist in the production data model. You need to feed it the schema definition, not just the application code.
I've had more success by including a simplified ER diagram in the architectural brief when refactoring data access layers. Without that, you're asking for a cascade failure.
SQL is not dead.
Yeah, feeding it the schema makes sense. But that's just more manual work I have to do. The whole point was supposed to be saving time, not creating a new documentation step.
Where does this end? Do I need to include the API spec for every third-party service next? The config files? It feels like we're just shifting the complexity burden.
How much context is actually realistic to provide before it's easier to just do the refactor myself?
You've hit on the real operational cost that gets ignored in these discussions. The "shifting the complexity burden" is exactly right. The break-even point isn't about raw context volume, it's about amortizing the setup cost.
In my team's workflow, we only provide extensive context for a defined class of "template" refactors we plan to repeat across multiple services. For a one-off rename across five files? It's faster to just do it manually. But for standardizing our ingestion layer patterns across thirty data pipelines, creating a one-page context sheet with the core interfaces and the target database schema let the AI produce a 90% correct first pass. The manual work was justified by the repetition.
If you're not planning to apply the same refactoring pattern at least a few times, the overhead of curating context will almost certainly negate the time saved. The tool shifts from a general-purpose assistant to a specialized automation for batch operations.
data is the product