That monitoring script is a good start, but you're right to suspect it's not the whole picture for Jenkins. You need to see what's *causing* those CPU spikes, not just that they exist.
Correlate your existing CPU log with Jenkins executor usage. Install the Metrics plugin, then run this next to your cron job:
```bash
# Add this to your script
executors_in_use=$(curl -s http://localhost:8080/metrics | grep 'jenkins_executor_in_use_value' | awk '{print $2}')
echo "$timestamp, CPU: $cpu%, Mem: $mem_used%, Executors: $executors_in_use" >> /var/log/jenkins_correlation.log
```
If those "big pipeline run" spikes line up with non-zero executors, your problem is workflow, not instance size. Downsize after you've moved that load to agents.
Good call on adding that curl command to the existing cron job. It's a cheap way to get the correlation data without another polling interval.
One nuance with that grep pattern, the metric name can sometimes include the node label if you're in a controller/agent setup. The exact key might be `jenkins_executor_in_use_value{node="master"}`. Your command will still work, but if it ever returns empty, checking the raw metrics endpoint first will show you the full name.
Also, once you have that CSV-style log, a quick pivot in a spreadsheet can be eye-opening. Sort by executor count descending and you'll instantly see which high-CPU events are actually workload-related.
> an m5.4xlarge is an expensive way to run build slaves
That really drives the point home. I hadn't even thought about it like paying for a giant server just to act as a build node.
I'm going to set up that Metrics plugin correlation first. But I have a question on the executor zero setting. If I set it to zero before all jobs are migrated, won't the whole system just break and fail everything? Or is the idea to do it gradually, like a slow roll?
Still learning.
Absolutely, that manual check of the /metrics endpoint is a perfect starting point. It's low-effort and builds intuition before you invest in dashboards.
One thing I've seen trip people up is that the default master executor count is 2, not zero. So if you see a consistent `executor_in_use` value of 1 or 2 during idle periods, that's normal baseline overhead, not active workload. The spikes you need to watch for are when it climbs past that built-in allowance.
Your question about agent labels hits the nail on the head. If most pipelines use `agent any`, they'll happily schedule on the controller whenever agents are busy. Moving them to explicit labels is the first config change, but you can also use the "Prevent jobs from running on the master" option in the node configuration as a temporary safety net while you do the migration.
Cloud cost nerd. No, I don't use Reserved Instances.
An m5.4xlarge is nearly $600/month on-demand. Your script is fine for seeing *that* you're wasting money, but you need to confirm *why* before you touch anything.
Everyone's telling you to correlate with the executor metric, and they're right. But you need the actual bill screenshot for that instance ID first. Otherwise you're just guessing at savings. What's the monthly cost shown in Cost Explorer right now?
Once you have that, set the master's executor count to zero. If jobs fail, you've found the workloads causing the spikes. That's your migration list. No need for gradual rollouts, just do it during a maintenance window.
show me the bill