Hello everyone,
I’ve been lurking here for a while, reading through the fantastic discussions on workflows and automation. As someone who spends most of my day in spreadsheets and financial systems, I’m always looking for ways to bring that same level of rigor and compliance checking to other parts of my work, especially code. I've been evaluating Cursor for potential use in some internal automation scripts, but I needed to be sure its suggestions wouldn't introduce issues that would fail our internal code quality gates, which are largely based on SonarQube rules.
So, I decided to build a methodical comparison tool. I wanted to move beyond just "does this look right?" and actually measure Cursor's output against a concrete, rules-based standard. The goal was to see if accepting its suggestions would generally improve or degrade a codebase's SonarQube score.
Here’s a high-level overview of what the script does:
* **Input & Processing:** It takes a code snippet (either a function or a small module) and runs it through SonarQube's static analysis locally using the SonarScanner CLI to establish a baseline. I capture issues by rule key (e.g., `python:S1481`, `java:S1172`).
* **Cursor Interaction:** It then uses Cursor's API to generate a suggested refactor or improvement for the same snippet. I prompt it with fairly generic instructions like "Improve this code for clarity and efficiency."
* **Analysis & Comparison:** The script analyzes the Cursor-suggested code with the same SonarQube ruleset. The core function compares the two reports, tracking:
* **New Issues:** SonarQube violations introduced by Cursor's suggestion.
* **Fixed Issues:** Baseline violations that were resolved in the suggested code.
* **Persistent Issues:** Violations present in both versions.
* **Output:** It generates a summary table and a detailed diff of the issues. I log everything to a CSV for trend analysis over many runs.
Some early, anecdotal findings from testing on Python and JavaScript snippets (mostly small finance-related helper functions):
* Cursor is consistently good at fixing **code style issues** (unused variables, commented-out code) and some **basic bug detection** (empty catch blocks).
* However, it occasionally introduces **maintainability and complexity issues**. For example, in an effort to make code more "Pythonic," it might create a complex nested dictionary comprehension that triggers a cognitive complexity warning (`python:S3776`) where the original, more verbose loop did not.
* There seems to be a pattern where it prioritizes conciseness over clarity in certain contexts, which is a fascinating trade-off to document.
My next steps are to run this against a larger corpus of code from our legacy ERP integration scripts. I'm also considering adding a simple cost-benefit scoring mechanism, perhaps weighting critical issues like bugs more heavily than style ones.
Has anyone else attempted a structured, metrics-based evaluation of AI coding assistant suggestions? I'm particularly curious if there are other static analysis rule sets (like those for security or performance) that I should incorporate into the comparison. Also, if you have ideas for standardizing the prompts to Cursor to make the test more repeatable, I'd love to hear them.