I tried Kling's PR review feature on a set of 50 recent pull requests from our projects. The goal was to see how reliable it is for catching issues automatically.
Out of the 50 reviews, Kling flagged 23 potential problems. After our team manually checked each flag, 12 of them were false positives. Most common were misidentifying standard error handling as "unnecessary code" and flagging approved library methods as "security risks." It did catch 11 real issues we had missed, mostly minor style inconsistencies and one actual bug. For now, I don't think I can trust it to run unsupervised. The noise level is a bit high for my taste. Has anyone else run a similar test? I'm curious about your false positive rate, especially in accounting or billing related code.
Thanks for sharing those numbers, really helpful to see a real-world test. I've noticed the same with false positives on error handling patterns, especially around try/catch blocks in our codebase.
The catch rate on minor style issues is interesting though. We've found it works best as a first-pass filter, but someone still needs to triage the flags before they hit the dev team. Otherwise, the noise just burns review cycles.
Haven't run it on billing-specific code, but I'd be extra cautious there. A tool flagging "security risks" where none exist could really slow down critical fixes.
Docs save time
That's a really useful test, thanks for posting the numbers. The false positive rate you saw lines up with my limited experience, though I'm newer to Kling.
I'm curious about the 11 real issues it caught. Were the style inconsistencies just formatting, or did any relate to actual logic patterns? And was the bug it found something a linter would miss? Trying to gauge if the value is in catching different things, or just more things.
That's a good question. I've been wondering the same thing about the value proposition. If it's just catching extra whitespace or quote style, that's what a formatter is for, and you'd expect near-zero false positives there.
It makes me think the real test is whether it finds logic problems a static analyzer would miss, like a potential null reference in a rare code path, or a state mutation that could cause a race condition. That's where the trade-off with false positives might be worth it. But if it's mostly stylistic, the noise seems hard to justify.
Did the original poster mention if any of the real catches were in areas like data validation or payment flow? That's where I'd be most nervous about missing something, but also most annoyed by false alarms.