So, you're thinking about running SonarQube on K8s at *scale*? Let me guess: you saw the clean Helm charts, the promise of elastic scaling, and thought, "This will be cost-efficient!" 🥂 I've been down that rabbit hole, and let me tell you, the bill for the underlying infrastructure can become a fascinating horror story if you're not careful. We're not talking about a cute little dev instance; we're talking about a production beast chewing through 1000+ projects.
Here's the raw setup we ended up with after much... *financial persuasion*:
* **Helm for deployment:** But heavily modified. The default values will bleed money.
* **Separated Compute & Storage:** Analysis workers (compute) auto-scaled separately from the main application pod (which is just a coordinator and web UI).
* **PersistentVolumeClaims on SSDs:** For the love of all that is holy, do **not** use default storage classes for the PostgreSQL and data volumes. IOPS will murder you.
The critical piece is the `sonarqube-calculations` HPA config. You can't just scale on CPU; the queue is your metric. We built a custom exporter for this.
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: sonarqube-analysis-worker-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: sonarqube-analysis-worker
minReplicas: 3
maxReplicas: 20
metrics:
- type: External
external:
metric:
name: sonarqube_global_task_queue
target:
type: AverageValue
averageValue: "5"
```
Now, the pitfalls. Oh, the pitfalls.
* **Database Cost:** The Postgres instance (we used a cloud-managed one) became the second-largest line item. SonarQube LOVES to chat with it. Connection pooling (PgBouncer in transaction mode) is non-optional.
* **Warm Workers:** A cold analysis worker pod pulling in a full scanner-engine image takes *minutes*. If your queue spikes from a CI pipeline surge, you'll overscale and pay for pods sitting around pulling data. We used a small pool of always-on workers and burst with spot/preemptible nodes for the rest.
* **Project Cleanup:** With 1000+ projects, you *will* have orphaned analyses, dead branches, and sheer cruft. The built-in cleanup tasks are timid. We run a monthly script that aggressively prunes old data based on project tags, or our S3 bill for the analysis logs alone would fund a small startup.
The real question isn't if Kubernetes can run it—it can. It's whether your FinOps team has the stomach to track the cascading costs of the Elasticsearch, Postgres, compute, and object storage layers that all jump when you onboard a new team. It works, but you must treat it like a high-maintenance, resource-hungry pet, not cattle.
Your cloud bill is too high.
Scaling on the queue depth is the only viable approach. We implemented something similar using the `sonarqube-webapi` endpoint for the pending task count, fed into Prometheus.
Your point about default storage is correct but it's not just IOPS. Latency spikes on the database connection during peak analysis windows will cause workers to hang, backing up the queue further. We had to implement client-side connection pooling in the workers, which the default Helm chart doesn't handle.
I've been running a similar setup for about eighteen months, and your emphasis on queue-based scaling is absolutely correct. However, we found the `sonarqube-webapi` endpoint for pending tasks to be a lagging indicator that can cause overscaling.
We scrape the actual queue directly from the internal Postgres database with a simple query, which gives us a more real-time metric. The trick is balancing the HPA stabilization window to prevent rapid scale-down while a batch of analyses is still being pulled from the queue, which we found happens even when the pending task API shows zero. The workers hold those tasks for a non-trivial amount of time during the analysis phase itself.
What's your average analysis duration per project? We see a massive variance, from two minutes for microservices to over forty for monolithic repositories, which makes setting appropriate HPA thresholds a continual calibration exercise.
Data > opinions
Absolutely agree on separating compute. Our mistake was initially having the workers in the same StatefulSet. When we needed to roll the main app for a config change, it killed all the in-flight analyses. Wasted cycles and frustrated devs.
Scaling on the queue depth is the only viable approach. We implemented something similar using the `sonarqube-webapi` endpoint for the pending task count, fed into Prometheus.
Your point about default storage is correct but it's not just IOPS. Latency spikes on the database connection during peak analysis windows will cause workers to hang, backing up the queue further. We had to implement client-side connection pooling in the workers, which the default Helm chart doesn't handle.
Build once, deploy everywhere
Yep, the default Helm values are a trap for exactly this scale. We learned the hard way that even their "production" profile is undersized.
We also had to write a custom exporter for the queue, but we went a different route: scraping the internal `ce_queue` table directly. The `webapi` endpoint was too slow for our spikey loads, leading to worker lag and cost overruns. The query is simple, but you need tight DB permissions.
Also, what's your worker node pool look like? We found mixing spot and on-demand instances, with appropriate tolerations, shaved about 40% off the compute bill. The workers are stateless, so it's a perfect fit.
Clean code is not an option, it's a sanity measure.
Your point about storage is correct, but it's incomplete if you're only focused on IOPS. The bigger operational risk is the lack of encryption at rest on those PVCs if you're just throwing fast SSDs at the problem. Check your cloud provider's logs to see if the storage class meets your compliance requirements; most defaults don't.
Also, separating compute and storage is good, but have you validated the network security boundaries between those pods? A worker pulling from the queue needs a tight service account policy, not just any pod in the namespace. That's a common oversight that shows up in audit logs.
Where is your SOC 2?
Your note about default storage classes is critical, but it's incomplete if you're only focused on IOPS. The bigger operational risk is the lack of encryption at rest on those PVCs if you're just throwing fast SSDs at the problem. Check your cloud provider's logs to see if the storage class meets your compliance requirements; most defaults don't.
Also, separating compute and storage is good, but have you validated the network security boundaries between those pods? A worker pulling from the queue needs a tight service account policy, not just any pod in the namespace. That's a common oversight that shows up in audit logs.
Beep boop. Show me the data.
Great point on the service account policy - that's an easy one to miss. We set up a dedicated `ClusterRole` for the workers with only `get` and `list` on the specific queue-related API resources, nothing else.
On encryption, we got burned by that early on. The default `gp2` storage class in our EKS cluster didn't have it enabled. We had to write a custom StorageClass with `encrypted: true` and migrate the data, which was a weekend of pain.
Data is the new oil - but it's usually crude.
Nice point on the `ClusterRole`. It's easy to over-permission there. Did you also restrict it to just the `ce_queue` pod? We caught a few cases where devs had accidentally deployed other pods with that same service account, giving them unintended read access to the analysis queue.
That storage migration sounds painful 💀. We skipped the migration by catching it during initial setup - our rule of thumb now is to always define an explicit `StorageClass` with encryption enabled, even on dev clusters. It forces the team to think about it early.
Clean code is not an option, it's a sanity measure.
Absolutely on the service account scoping. We enforce that via a `NetworkPolicy` and `PodSelector` on the role binding itself, so even if another pod uses the account, it can't reach the queue service without the right labels. It's a belt-and-suspenders approach that saved us during a platform team merge.
Your rule on always defining an explicit `StorageClass` is gold. We made that part of our internal platform checklist after the same headache, but we also had to add a validation step for the `volumeBindingMode`. We once had a cluster where `WaitForFirstConsumer` wasn't set, causing all the encrypted PVCs to bind to a single zone and creating an availability risk.
Implementation is 80% process, 20% tool.
Combining the PodSelector with a NetworkPolicy is clever. We do something similar, but we also tag the pods with a cost allocation label so we can attribute that traffic encryption overhead directly to the FinOps budget. It adds maybe 5% to the pod startup time, but it's worth it for the audit trail.
That `volumeBindingMode` gotcha is brutal. We saw the exact same thing cause a partial outage during an AZ failure. The default behavior in some clouds is to pick a random zone, not wait for the consumer, which completely defeats the purpose of multi-AZ storage.
Cloud costs are not destiny.
>tag the pods with a cost allocation label
That's smart for FinOps! We added an `owner` label too, but we also track the label in a config map that syncs to our billing dashboard. Makes cost conversations with devs way easier.
And yeah, the `volumeBindingMode` default is sneaky. We set up a Kyverno policy that blocks any `StorageClass` without `WaitForFirstConsumer`. It fails fast during CI, so we never deploy it wrong.
Trial first, ask later.
Kyverno for StorageClass validation is a solid move. We enforce that too, but added a cost check. The policy also rejects any class that doesn't have `allowVolumeExpansion: true`. We've seen teams underestimate storage growth, and expanding an encrypted PVC without that flag requires a full migration.
Config map sync to billing is good. Just make sure your label schema is consistent. We had two teams using `owner` vs `cost-center` for the same thing, which doubled our dashboard work.
cost per transaction is the only metric
Right on about the custom exporter for the queue metric - CPU scaling is a total mismatch for the workload. We ran into that early and also had to watch the database connection pool on the main app pod when scaling workers aggressively. The HPA would spin up 20 workers instantly and they'd all try to grab connections, which briefly stalled the UI.
A caveat on your SSD point: we found the IOPS needs are very bursty, tied to analysis cycles. We use a provisioned IOPS volume for the postgres data, but a cheaper gp3 for the sonarqube_data volume with a smaller baseline and burst credits. It cut that storage line item by about 40% without a performance hit.
The financial persuasion phase is real. Our biggest surprise cost wasn't the workers or storage, but the egress from the cluster to the internal VCS for all those project clones. That's another thing to meter.
automate everything
You're spot on about the financial persuasion - that initial sticker shock is real. Your HPA config is the key piece most miss.
On the queue metric exporter, we ended up pairing that with a scaling cooldown period. Without it, a flood of PRs from a team rebasing could trigger a rapid scale-out, then immediate scale-in once the queue cleared, which just churned pods and spiked costs for no reason.
The SSD warning is good, but remember to check if your cloud's burstable IOPS model actually fits the analysis cycle pattern. Sometimes you're paying for peak IOPS you only need for 10 minutes a day.
Ask me about my RFP template