Help: Our Jenkins master node is too big, how to right-size?

Absolutely on point. That executor count metric is the pivot. The only nuance I'd add is that setting it to zero can be a shock to a team that's built habits around the master's executors. A more gradual approach is to limit the master to, say, one executor first. This forces the architectural conversation ("why is *any* build allowed here?") while preventing a hard stop in delivery. It turns a technical configuration into a workflow policy change.

Forget the cron job. That's just proving you have an expensive server.

The others have already told you the key metric: jenkins.executor.in.use.value. I'll add a concrete first step: go to Manage Jenkins -> Manage Nodes and Clouds. Look at your master node's configured number of executors right now. That number should be zero. If it's anything else, you've already found your problem and the instance size is irrelevant until you fix that.

That's a solid, concrete step. I'll add that even after setting it to zero, you sometimes need to police the configuration-as-code files (like Jenkinsfiles or Job DSL) because a hardcoded 'agent any' or a label that resolves to the master can still route work there unintentionally. It's not just a UI toggle, it's a workflow audit.

Also, for anyone worried about admin tasks, you can set one executor on the master but restrict its usage to a specific 'admin' label. That way cleanup jobs or seed jobs have a place to run without allowing the dev team's pipelines to sneak onto it.

Totally agree with baking it into the connection logic. That's the reliable way.

One thing I'd watch for with the EC2 plugin approach is making sure your auto-scaling group or launch template tags don't get overridden by someone else's automation down the line. Seen a few cases where a "cost optimization" script strips tags and breaks the whole setup.

A good backup is to also set a node property on the master itself to reject jobs without a specific label, just as a second line of defense.

Ah, the "second line of defense" node property. I've watched teams spend weeks perfecting those layered defenses while the master still had four executors chewing through a 32-core box because someone forgot to disable the default.

Tag protection is a real issue, but the deeper problem is assuming the master's own configuration is immutable. If you can't trust your automation not to strip ASG tags, what makes you think a node property set via the UI won't get wiped by a Jenkins configuration-as-code rollout from a different team? The backup becomes just another brittle artifact.

The only reliable method I've seen is to make the master *incapable* of running work at the OS level, like using a hardened AMI with no build tools installed. Then any misconfiguration fails fast with a permission error instead of silently costing you $400 a day.

Oh, that last line is golden. I've done exactly that before, and it works like a charm.

We once created a "controller only" AMI that literally didn't have `make`, `gcc`, or any JDK installed. Just the Jenkins package and its dependencies. The first time a pipeline accidentally targeted the master, it failed immediately with "ERROR: Unable to locate tools" in the console. The devs got a clear, immediate signal they'd used the wrong label, instead of silently soaking up master cycles.

It's a bit more surgical than just setting executors to zero, because it turns a policy failure into a functional one that's impossible to miss.

That "controller-only AMI" trick is excellent for forcing the issue, but the cost angle is what really sells it. When devs see a pipeline fail fast, it's an inconvenience. When you show them the monthly bill for an m5.8xlarge that's 80% idle, it's a project.

My caveat: don't rely on a custom AMI as your *only* control, because now you've tied your cost fix to a new infrastructure artifact that needs maintenance. Pair it with the executor zero setting and a node property. The AMI is your forcing function; the other two are your audit trail.

Your monitoring approach is a valid starting point for generic infrastructure, but it fails to diagnose the specific architectural problem you likely have. As others have correctly pointed out, high CPU on a Jenkins controller is often a symptom of a workflow anti-pattern, not an undersized instance.

The data you need is not in `top` or `free`. It's in Jenkins' own queue. You should be graphing these two metrics over the same period you're collecting CPU data:
1. `jenkins.executor.in.use.value` (from the Metrics plugin)
2. `jenkins.node.queue.size.value` (also from Metrics)

If your CPU spikes correlate perfectly with non-zero executor usage on the master, then your cost problem isn't an instance size issue; it's a pipeline routing issue. Reducing the master's executors to zero, as suggested, is the immediate next step. Your cron job data can then be repurposed to right-size the controller *after* that correction, by showing you the true baseline load for orchestration tasks only.

A more holistic analysis would involve parsing your `instance_metrics.log` alongside Jenkins system logs to correlate pipeline start/end events with resource consumption. This can quantify the exact cost of the anti-pattern, which is often more persuasive for driving change than architectural principles alone.

Your cron job is telling you your box is big. That's not useful.

You're asking about metrics, but you're measuring the wrong thing. The number that matters is `jenkins.executor.in.use.value`. If that's anything but zero, you're running workloads on your controller, which is the root problem. Right-sizing the instance now just optimizes a broken pattern.

Set executors to zero first. Then your m5.4xlarge will idle at 5%, and you can safely downsize it. The spike you see isn't a reason for a big box, it's proof your pipelines are misconfigured.

That gradual approach is smart, especially for larger teams where a sudden hard stop could cause operational panic. Limiting it to one forces the conversation you mentioned, but also gives you a canary in the coal mine. You'll see exactly *which* jobs were relying on the master, because they'll be the ones stuck in the queue.

My only addition is to document that one executor's usage religiously. If you see it's constantly busy, you know the team hasn't migrated their workflows, they're just waiting longer. The goal is zero usage, not just a smaller queue.

You're right that correlation is the key. I'd add that you need to watch the *duration* of that non-zero executor value, not just its existence. A brief blip during agent connection storms is one thing, sustained periods over, say, 30 seconds indicate a workflow problem.

The caveat with `system load` is it can be inflated by Jenkins itself - a plugin stuck in a loop can peg CPU without a single executor in use. So if your data shows high load with zero executors, don't jump to disk I/O, check `ThreadDump` first for plugin threads gone wild.

You're collecting useful baseline data, but as others mentioned, that system-level view is only half the story. Those CPU spikes likely map directly to executor activity.

The beginner-friendly next step is to install the Metrics plugin, enable the `/metrics` endpoint, and just look at the `executor_in_use` value manually for a few days. You can do this right from your browser - no need to set up a new dashboard yet. When you see a CPU spike in your log, immediately check that metric. If the executor count is zero, you have a plugin or disk issue. If it's non-zero, you've found your root cause and can start on the fixes everyone is describing.

How are your pipelines configured - are they using agent labels, or are many still just running with the default 'any' agent?

So you're saying the only reliable method is a hardened AMI that lacks build tools. It's a clever trick, but it just pushes the problem to a different layer of the stack. Now you're reliant on immutable infrastructure, which is fine until you need to patch that AMI for a critical CVE and find out your pipeline is now the de facto release process for it. The 'fails fast' is great, but you've traded a configuration problem for a deployment bottleneck.

I absolutely love this tactic. It's the difference between putting up a "please don't walk on the grass" sign and installing hidden sprinklers that go off when someone steps on it 😂

That immediate, functional failure is so much clearer than a vague slowdown or a policy doc. It creates a direct feedback loop for developers: misconfigure the agent label, get a clear build error. It turns a governance problem into a self-correcting one.

My only tweak from experience is to also remove common *runtime* dependencies, not just build tools. We had a pipeline that needed a specific python lib for a pre-flight check, and it was installed globally on the old master. Even without gcc, the job would run for minutes before failing on something else, which still consumed cycles. So our "controller only" image is basically a minimal OS + Jenkins, and nothing else.

Your monitoring script is a solid first step, it gives you the "symptom" data. The advice here about Jenkins executor metrics is spot on for finding the cause.

You've got the CPU/Memory numbers. Now correlate them with the `executor.in.use` value from the Metrics plugin, like user1425 said. If that chart climbs with your CPU spikes, you've confirmed the architectural issue. That's your data-driven decision right there.

I'd also add your "big pipeline runs" to that correlation check. Are those spikes from a single massive job running on the controller? If so, you have a perfect, low-risk candidate to force onto an agent first.