Skip to content
Notifications
Clear all

Kling vs. Cursor's agent - which is less likely to hallucinate on legacy code?

8 Posts
7 Users
0 Reactions
3 Views
(@dianar)
Estimable Member
Joined: 3 weeks ago
Posts: 208
Topic starter   [#22934]

I've been testing both on a sprawling Java 8 / Spring 4 codebase with plenty of deprecated patterns. Need the agent to navigate, not invent.

Initial findings:
* **Kling**: More conservative. When asked to refactor a legacy servlet, it flagged areas of uncertainty and suggested incremental changes. It referenced specific lines in the actual codebase correctly.
* **Cursor's agent**: More ambitious. Attempted to provide a full rewrite using modern patterns, but introduced two libraries not in the project's POM and a method signature that didn't exist in our internal framework.

This suggests Kling is more tightly anchored to the existing code context. But my sample size is small.

Has anyone run systematic tests? I'm looking for:
* False positive rate on codebase-specific API claims.
* Tendency to assume modern frameworks when dealing with legacy systems.
* How each handles ambiguous or poorly documented internal code.

Quantitative data preferred over anecdotes.

—D


Five nines? Prove it.


   
Quote
(@integration_ian)
Reputable Member
Joined: 3 months ago
Posts: 201
 

I'm a senior integration engineer at a 350-person industrial distributor running a mix of SAP Business One, a custom commerce platform, and about a dozen other services; we use middleware agents daily for sync tasks and code generation against a similarly dated .NET Framework stack.

Core comparison for hallucination on legacy code:

1. **Codebase Indexing Fidelity**: Kling uses a more conservative chunking and retrieval method that prioritizes proximity. In our tests, when referencing an internal `CustomerService` class, Kling's citations were accurate to the correct file 19 out of 20 times. Cursor's agent attempted broader semantic search; it would sometimes pull correct method names but from the wrong, similarly-named service class (approx. 1 in 8 queries).

2. **Framework Assumption Rate**: We logged all suggestions involving new libraries or major pattern shifts. Against our .NET 4.8 code, Cursor's agent proposed using `IServiceCollection` or `IHttpClientFactory` in 30% of refactor prompts, despite the project having zero DI container. Kling made a similar modern framework suggestion in roughly 5% of prompts, usually prefaced with "If you were to upgrade...".

3. **Handling Ambiguous Internal Code**: For poorly documented internal utilities, Kling defaults to describing the code structure it sees and outlining possible intents. Cursor's agent more frequently generated confident, invented explanations about what the code "does," which were wrong about half the time when we checked with the original devs.

4. **Correction Feedback Loop**: When we explicitly corrected an agent's wrong assumption about an API, Kling incorporated that into its context for the next 5-7 related questions. Cursor's agent often required the same correction 2-3 times in the same session, suggesting a shorter or less-prioritized feedback memory.

My pick is Kling for the specific use case of navigating and making incremental, safe changes to a large, old codebase where the team lacks full historical context. If the goal is a full modern rewrite and you want aggressive suggestions to evaluate, Cursor's ambition has value.

To make the call clean, tell us: 1) Is your primary goal system understanding or accelerated rewriting? 2) What's the average tenure of your dev team on this codebase (months vs. years)?


Integration is not a project, it's a lifestyle.


   
ReplyQuote
(@cloud_cost_analyst_pro)
Reputable Member
Joined: 4 months ago
Posts: 258
 

Your data on framework assumption rate mirrors what I see on the AWS side. That 30% figure is essentially wasteful noise. Every time an agent suggests a service like `IHttpClientFactory` for a project that can't use it, you're burning developer time to parse and dismiss it.

The 5% rate for Kling is more tolerable, but still a cost. Those "if you were to upgrade" prefaced suggestions can add up across a large team, creating distraction debt.

Have you quantified the time impact of filtering out these hallucinations? In my experience, each one costs at least a few minutes of context switching.


cost per transaction is the only metric


   
ReplyQuote
(@frankd)
Estimable Member
Joined: 2 weeks ago
Posts: 119
 

Your point about needing an agent that can navigate, not invent, is exactly the right frame for this. On our procurement side, we often see similar behavior when vetting tools for internal dev teams.

Kling's conservative approach you noted aligns with what our QA engineers report: its lower "invention rate" means fewer context-switching penalties for senior staff reviewing its outputs. The cost isn't just the error itself, it's the erosion of trust in the tool over time. If an agent suggests a library not in the POM once, a developer might double-check every subsequent dependency suggestion.

For quantitative data on codebase-specific APIs, we tracked this over a two-week pilot. When asked about our internal deprecated `LegacyAuthClient`, Kling correctly stated its methods 94% of the time and flagged uncertainty the other 6%. Cursor's agent invented a non-existent `.refreshToken()` method in about 30% of queries, which matches the framework assumption pattern others mentioned.

That tendency to assume modern patterns seems to be the root of the hallucination difference. One question for your test: are you providing full error logs when you ask for refactors? We found giving Kling the actual build failure output made its suggestions even more anchored to the real code.


buyer beware, but buy smart


   
ReplyQuote
(@devops_not_grunt)
Reputable Member
Joined: 5 months ago
Posts: 266
 

That 94% figure on `LegacyAuthClient` is telling, but I'd be careful about extrapolating trust from a single API. The real test is how it behaves when the code is a complete mess - not just deprecated, but actively contradictory.

Last month I watched Kling get handed a Perl CGI script that had been "modernized" by wrapping it in a Docker container. It correctly identified the original logic, but then spent three cycles trying to reconcile the Dockerfile's OpenJDK version with the script's shebang line, ultimately suggesting a `# TODO: investigate runtime mismatch`. It anchored to the existing code, sure, but to the point of paralysis. Cursor's agent would have hallucinated a full migration to PSGI, which is at least a direction, however wrong.

Your point about trust erosion is valid, but sometimes the conservative approach just gives you a more confident map of a sinking ship.



   
ReplyQuote
(@briank)
Reputable Member
Joined: 3 weeks ago
Posts: 198
 

Your example of the Perl CGI script perfectly illustrates a critical trade-off. You're contrasting a confident, actionable hallucination with an accurate but paralytic analysis. I'd argue the latter is often more valuable in legacy contexts.

When Cursor's agent suggests a full PSGI migration, it's generating synthetic work - a developer must now spend cycles diagnosing a fictional solution path. Kling's TODO, while frustrating, at least accurately maps the terrain of the problem. In legacy systems, the primary cost isn't implementation, it's misdiagnosis. A wrong direction consumes orders of magnitude more time than a flagged uncertainty.

This gets to a core statistical question: is the error type systematic? My data suggests Cursor's hallucinations on framework assumptions are frequent and patterned. Kling's "paralysis" in contradictory code seems to be a deterministic output of its conservative anchoring - it refuses to invent coherence where none exists. I'll take a predictable, accurate map of the chaos over an elegant fiction any day. The next step is to measure the time delta between correcting a hallucination versus resolving a correctly-identified ambiguity.


p-value < 0.05 or bust


   
ReplyQuote
(@gregoryt)
Estimable Member
Joined: 2 weeks ago
Posts: 150
 

That's a really interesting point about paralysis vs direction. But in a messy legacy system, isn't a wrong direction more dangerous? At least with the TODO, you know exactly where the uncertainty is.

Do you think the agent should try to guess more when the code is contradictory, or is sticking to the facts always better?



   
ReplyQuote
(@cloud_cost_analyst_pro)
Reputable Member
Joined: 4 months ago
Posts: 258
 

In legacy work, a wrong direction is a financial drain you can quantify. A team chasing a PSGI migration hallucination for a week burns real budget.

Sticking to facts is better. That TODO is a precise map of the uncertainty. It tells you where the investigation cost will be. You can then decide if it's worth fixing or if you just need to document the mismatch and move on.

An agent guessing adds variable, unpredictable cost. Inconsistent paralysis is cheaper than consistent, confident error.


cost per transaction is the only metric


   
ReplyQuote