I've been conducting a systematic evaluation of Claude Code's capabilities with a legacy PHP 5.6 / 7.0 monolith we're attempting to modernize, and the results have been... suboptimal. The codebase is characteristic of that era: procedural patterns mixed with early OOP attempts, heavy reliance on `mysql_*` functions, and extensive use of global variables. While I didn't expect perfect modernization, the tool's performance metrics against this specific workload are concerning.
My benchmark methodology involved selecting 50 representative files (~15k LOC total) and running them through Claude Code with three distinct prompts:
1. **Security Hardening:** Replace deprecated `mysql_*` with PDO prepared statements.
2. **Syntax Modernization:** Update array syntax from `array()` to `[]` and fix old `list()` constructs.
3. **Structure Refactoring:** Encapsulate global variables into a simple dependency injection container.
The failure rate was significant. For task #1, Claude Code consistently produced syntactically incorrect PDO bindings. Observe a typical input and its flawed output:
**Original vulnerable code:**
```php
$result = mysql_query("SELECT * FROM users WHERE id = " . $_GET['id']);
$row = mysql_fetch_assoc($result);
```
**Claude Code's suggested "fix":**
```php
$pdo = new PDO($dsn, $user, $pass);
$stmt = $pdo->prepare("SELECT * FROM users WHERE id = :id");
$stmt->bindValue(':id', $_GET['id']); // CRITICAL: No parameter type specified for integer
$stmt->execute();
$row = $stmt->fetch(PDO::FETCH_ASSOC);
```
This introduces a type coercion vulnerability, as `$_GET['id']` remains a string bound to an integer column, potentially bypassing query logic. The tool missed the necessity of `PDO::PARAM_INT`.
The more subtle issues were architectural:
* It attempted to "modularize" tightly coupled scripts by creating classes, but consistently broke implicit execution flows dependent on global state.
* When asked to add simple observability (logging execution time), it injected `microtime(true)` calls but placed them incorrectly, measuring only file inclusion time, not business logic.
* It demonstrated poor understanding of PHP's include path semantics for legacy apps, suggesting absolute paths that would break portability.
My current hypothesis is that Claude Code's training data is skewed towards modern, well-structured frameworks (Laravel, Symfony) and lacks sufficient token context for the idiosyncratic, "spaghetti-coded" PHP that powers a non-trivial portion of the web. The cost of validating and correcting its suggestions often exceeds the time required for manual refactoring.
Has anyone else performed quantitative analysis on Claude Code with similar legacy ecosystems? I'm particularly interested in:
* Comparative benchmarks against other AI-assisted tools (e.g., GitHub Copilot, Cursor) on the same codebase.
* Successful prompt engineering strategies to constrain its output to safe, incremental changes for procedural code.
* Whether fine-tuning on a corpus of legacy PHP patterns is feasible or if the architectural gap is too wide.
The potential for ROI on this tool is high, but the current error rate makes it a net time sink for our specific workload. I'm compiling a more detailed failure mode analysis to share with the team.
—chris
—chris
Oh, that PDO output sounds painfully familiar. I tried a similar path with a legacy WordPress plugin codebase last month and hit the same wall with the bindings - it kept trying to use positional parameters (?) but then mapped them incorrectly, creating a total mismatch. The security implications of a botched PDO conversion are actually worse than the original `mysql_*` code.
For what it's worth, I found a more successful, if tedious, workflow: use Claude Code for the initial syntax updates (task #2) which it usually gets right, then switch to manual, file-by-file for the database logic. The tool just doesn't grasp the tangled control flow in those old scripts well enough to inject a proper database connection object.
Have you tried breaking your prompt into even smaller steps? Like, "First, just identify all the mysql_query calls and their surrounding variable scope" before asking for the rewrite? I got slightly better accuracy that way, but it's still not production-ready without heavy review.
null
That benchmark methodology sounds really solid. Did you find the failure rate was evenly distributed, or did it cluster around specific patterns? For example, we have a lot of nested conditionals with inline SQL, and those seem to break completely.
I've also been trying to modernize some older Freshdesk integration code, and the global variable refactoring task is where it falls apart for me. It'll create a container class but then misses half the references in function scopes.
That benchmark approach is solid, but you're hitting the core problem. These tools, including Claude Code, fail on tangled procedural flows because they can't trace state.
> syntactically incorrect PDO bindings
This isn't a syntax error, it's a logic failure. The tool sees a `mysql_query` with interpolated variables and tries to template a prepared statement. But it can't reconstruct the data flow to determine where the connection object lives, or if the variable used in the bind is even in scope after the rewrite. I've seen it create a `$stmt->bindParam('id', $id)` where `$id` is undefined because the original variable was `$user_id` from three lines up in a nested `if`.
Your task #3, encapsulating globals, is where this becomes impossible for automation. It'll create a pretty `Config` class but miss the fifty-seven places where `global $dbconn;` is hidden inside a function that's called from a switch statement in another file. Manual triage is the only way forward for anything touching global state.
Migrate once, test twice.
Exactly. The logic failure stems from a missing semantic layer. These models operate on token adjacency and pattern matching, not on building an actual symbol table or control flow graph. When you have a variable like `$user_id` that's only in scope because of a preceding `if` block that calls `extract()` on a request array, the tool has no chance.
I'd add that the problem compounds with include-heavy architectures. The tool processes a single file in isolation, so a `global $db;` declaration might be in `config.inc.php` three includes deep. It can't resolve that dependency, so it either omits the connection or creates a new one, potentially duplicating resources.
The only semi-reliable automation path I've found is to use static analysis tools like PHPStan or Psalm first, on the raw legacy code, to generate a baseline report. Then, you feed those findings - the actual variable traces - into the prompt as concrete constraints. Even then, it's more of a guided refactoring aid than an automated solution.
Your benchmark mirrors what I've seen. The PDO binding failures aren't random; they're predictable. The model can't handle implicit scoping from `extract()` or `global` statements that pull variables from outer scope. It rewrites the query string mechanically, then binds whatever variable names are most frequent in the surrounding tokens, not the ones actually used.
> syntactically incorrect PDO bindings
Precisely. It'll often produce valid PHP that will even run without throwing an immediate error, but the bound values are null or incorrect. That's a silent failure, which is worse than a syntax error.
You didn't mention includes. Does your codebase have a lot of `require_once`? If so, that's another failure vector where it creates redundant PDO instances because it can't resolve the shared connection declared in a header file.
Your fancy demo doesn't scale.
Your methodology is good, but your expectation is wrong. Claude Code is a pattern-matching tool, not a static analyzer. It can't trace variable scope across includes or globals.
The syntactically incorrect bindings are a symptom. It sees the variable names in the original SQL string and tries to bind them, but doesn't account for scope changes after rewriting the query structure. You'll get null binds.
You need to run a static analysis tool like Psalm first to understand the actual data flow. Then you can give Claude Code smaller, scoped prompts based on that map. It can't create that map itself.
Your point about static analysis as a prerequisite is correct, but you're underestimating the scale of the problem. Running Psalm on a 5.6 codebase with `extract()` and `global` dependencies often fails to even produce a parseable graph. The baseline report is a map of errors, not a functional symbol table.
The real failure vector isn't just duplicated connections. It's the creation of *different* PDO instances with separate connection pools because the tool can't identify the shared singleton pattern from the include chain. You'll pass a CI check but silently exhaust your database connection limit in production.
I've found you need to run the static analyzer, then manually write a spec file for the analyzer itself to understand the architectural intent, and *then* use that as prompt context. It's a three-step manual process just to get a reliable input for the tool.
—davidr
Exactly right about Psalm failing on the parseable graph. The global and extract() patterns are essentially impossible for static analysis until you've already manually fixed them, which defeats the point.
Your three-step process mirrors what my team ended up doing on a Drupal 7 migration. We had to write stub declarations for the global database object just to get Psalm to stop yelling, feed *that* output into Claude, and even then we'd get subtle connection mismatches. The CI check passing while exhausting connections is the perfect example of a "successful" but catastrophic modernization.
It feels like we're using the tool to automate the middle 20% of a job where the first and last 40% are still completely manual.
Ship fast, measure faster.