Having just completed the phased rollout of SonarQube (Developer Edition) to our entire engineering organization, I feel compelled to document the operational realities we encountered. The common tutorials and vendor documentation often gloss over the inflection points where a POC for a team of 10 meets the crucible of 300 concurrent developers. Our architecture was a containerized deployment on Amazon EKS, backed by an RDS PostgreSQL instance (db.r5.4xlarge) and an Elasticache Redis cluster for cache, with the main SonarQube service running as a StatefulSet with persistent volumes for data. We used Terraform and Helm for the entire provisioning lifecycle.
The primary failure modes were not in the core analysis engine, but in the supporting infrastructure and configuration scaling. Here is a breakdown of what we observed.
**1. Database Contention and the `issues` Table Bloat**
The most immediate and severe bottleneck was the PostgreSQL database. With all developers triggering analyses (both CI pipeline and local IDE scans), the rate of row insertion into tables like `issues`, `project_branches`, and `ce_queue` was staggering. We observed lock contention and query timeouts, primarily during the pruning tasks. The default cleanup configuration was utterly insufficient.
We had to implement aggressive, custom cleanup policies and optimize the schema.
```sql
-- Example: We increased the frequency and reduced the window of kept data
UPDATE properties SET text_value = '30' WHERE prop_key = 'sonar.dbcleaner.cleanDirectoryPeriod';
UPDATE properties SET text_value = '30' FROM internal_properties WHERE prop_key = 'sonar.dbcleaner.cleanDirectoryPeriod';
UPDATE properties SET text_value = '7' WHERE prop_key = 'sonar.dbcleaner.cleanBranchesAndPRs.period';
UPDATE properties SET text_value = '1000' WHERE prop_key = 'sonar.dbcleaner.maxAgeForClosedBranchToKeep';
```
Additionally, we had to schedule a dedicated maintenance window to add several critical indexes that are not created by default, significantly improving `DELETE` performance on the `issues` table.
**2. Elasticsearch Heap Pressure and GC Pauses**
We operated with the embedded Elasticsearch (a decision we are now revisiting). The default heap size was far too small for our volume of code and analysis history. This manifested as periodic UI unresponsiveness—the "Loading..." spinner on the project homepage would persist for minutes. The root cause was frequent, long Garbage Collection pauses in the Elasticsearch JVM. We resolved this by adjusting the SonarQube `sonar.search.javaOpts` and provisioning higher-memory nodes for the pods.
```
sonar.search.javaOpts=-Xms4g -Xmx4g -XX:MaxDirectMemorySize=1g -XX:+UseG1GC
```
**3. The "Zombie Scanner" Problem and CE Worker Scaling**
Our CI pipelines are built on Kubernetes pods. We observed that when a pipeline job was abruptly terminated (e.g., due to a timeout or manual cancellation), the corresponding scanner job in SonarQube's Compute Engine (CE) queue would not always be cleaned up. These "zombie" tasks would hold a worker slot indefinitely, leading to a backlog. The CE queue would show hundreds of pending tasks while the workers were ostensibly busy. We mitigated this by implementing a mandatory `sonar.scanner.dumpToFile` property in our shared scanner templates and setting a stricter `sonar.ce.worker.timeout` (default is 1 hour, we reduced it to 30 minutes for our codebase profile).
**4. Permission Schema Performance Degradation**
Configuring permissions for 300+ developers across 1000+ projects via groups was a necessity. However, the UI for managing project permissions became unusably slow. Any operation (adding a group, updating a template) would trigger a timeout. The issue was traced to the underlying permission evaluation queries performing full table scans on the `perm_templates_groups` and related tables. We had to script all permission updates via the SonarQube Web API to bypass the UI, and we are now evaluating a move to a project-level auto-assignment strategy to reduce the ACL complexity.
**5. Misconfigured Quality Gate Impact on CI/CD**
Developers initially praised the integration until their pipeline durations increased by 300%. The culprit was not SonarQube analysis itself, but the default Quality Gate check behavior. The `"Wait for Quality Gate"` step in our Azure DevOps and Jenkins pipelines was configured with the default timeout and polling interval. During periods of high CE queue load, the analysis processing delay meant the Quality Gate status was "pending" for longer than the pipeline timeout. This caused hard CI failures unrelated to code quality. We had to:
* Increase the timeout to 45 minutes.
* Educate teams on the difference between analysis report completion and CE task completion.
* Implement a dashboard to monitor CE queue depth as a critical operational metric.
**Cost Implications:**
The RDS and ElastiCache costs were as projected. The surprise came from the EBS volumes for the persistent data: the `data` volume for Elasticsearch indices grew at a rate of ~50GB per week, necessitating a move to larger, provisioned-IOPS volumes to maintain index performance. Our monthly AWS bill for the SonarQube infrastructure alone settled at roughly $5,200, which was 40% higher than our initial estimate, primarily due to the database instance tier and EBS optimization.
In conclusion, the rollout was successful in that the platform is now actively used, but the first month was a firefight of database tuning, JVM optimization, and process adjustment. The key lesson is that SonarQube at scale is a *data platform* first and a code quality tool second. Its constraints are database throughput, memory management, and queue processing. If you are planning a similar enterprise rollout, allocate at least 20% of your project timeline exclusively to performance tuning and scaling exercises post-deployment. Start with a pessimistic sizing for your database, assume you will need to customize all cleanup jobs, and design your CI integration with explicit, generous timeouts and clear failure modes.
Ah, the classic database bloat surprise. You'd think with all that enterprise pricing they'd mention the `issues` table becomes an unmanageable log file.
We ran into the same scaling wall, but on a self-hosted GitLab runner setup. The problem wasn't just the insert rate, it was the cleanup, or lack thereof. The default settings for purging old analyses are laughably conservative for that kind of volume. You end up needing to schedule aggressive, out-of-band SQL purges on `project_branches` and `ce_activity` or the whole thing just seizes.
Did your team also get hit by the analysis queue deadlocks, or did the Redis cache actually help there?
null
Oh, the cleanup defaults are absolutely a trap for the unwary. We were bit by that too, but our bigger immediate pain was the `analysis_properties` table. With 300 devs pushing commits, that thing grew at an astronomical rate, and the auto-vacuum on RDS just couldn't keep up, leading to table bloat and slow queries on the webhooks.
The Redis cache *did* help prevent queue deadlocks for the Compute Engine tasks, but it introduced a new problem: cache invalidation. When we needed to bounce the pods or had a deployment, the lost cache state caused a thundering herd of re-analyses for in-progress scans. We had to implement a sidecar that would pre-warm the cache from a backup on pod startup.
> Did your team also get hit by the analysis queue deadlocks
We sidestepped the worst of it by sharding the `ce_queue` worker threads across multiple app pods and tuning the `sonar.ce.workerCount` property aggressively. The deadlocks seemed to happen most when a single worker got stuck on a massive monolithic repo analysis. Splitting the workload helped isolate the damage.
What was your final strategy for the out-of-band SQL purges? Did you script something against the API, or go direct to the database? I'm always nervous about the latter with vendor apps.
Prod is the only environment that matters.