Has anyone benchmarked etcd performance on RKE2 vs kubeadm?

I'm a senior platform engineer at a fintech company processing several million transactions daily; we migrated from a fully managed k8s service to self-managed clusters last year and currently run over 200 nodes split between kubeadm-provisioned clusters (for core services) and RKE2 clusters (for newer, isolated workloads), giving me direct operational experience with both etcd deployments.

* **Operational Overhead and Default Tuning:** RKE2's embedded etcd is managed as a static pod, and its configuration is abstracted behind RKE2 server arguments. This reduces the surface area for misconfiguration but also limits fine-tuning. In my environment, the default `--quota-backend-bytes` was overly conservative, and we had to explicitly increase it via the config file to avoid performance degradation during large snapshot operations. With kubeadm, you have direct access to the etcd manifest, allowing you to apply etcd-specific tuning flags like `--max-request-bytes` and `--snapshot-count` from day one, but this requires deeper etcd expertise.
* **Baseline Throughput Under Steady Load:** For a consistent workload of 1KB `Put` requests, a three-node control plane on AWS `c5.large` instances showed comparable performance. Both setups sustained roughly 2,100 to 2,400 writes per second per cluster before average latency exceeded 10ms. The variance was greater in the kubeadm cluster, however, as its performance was directly tied to our manual etcd parameter tuning, while RKE2's performance was more consistent but plateaued earlier.
* **Latency During Burst Scaling Events:** This is where a material difference emerged. Simulating a configuration burst by applying 5,000 secrets in rapid succession, the kubeadm-managed etcd (with tuned `--max-concurrent-requests`) maintained p99 write latency under 120ms. The RKE2-managed etcd saw p99 latency spikes to 350-400ms in the same test. This suggests RKE2's bundled etcd favors stability and recovery semantics over raw throughput during bursts, a trade-off that matters during large-scale rollouts.
* **Infrastructure Cost and Sizing Implications:** Because RKE2's etcd is less tunable, you may need to vertically scale the control plane nodes sooner to achieve the same burst performance as a finely tuned kubeadm deployment. In our case, to handle our peak event loads, we would have needed to move from `c5.large` to `c5.xlarge` instances for RKE2 servers, a roughly 40% cost increase per node. The kubeadm cluster remained on `c5.large` due to granular tuning, representing a significant long-term cost saving.

Given your focus on right-sizing for cost and performance, I'd recommend kubeadm if you have in-house etcd operational knowledge and anticipate high-frequency configuration bursts. If your priority is reduced management complexity and you have a more consistent configuration change rate, RKE2 is defensible. To make a clean call, tell us your expected peak writes per second and whether your team has prior experience tuning etcd's memory and concurrency parameters.

That "direct access" with kubeadm is exactly what turns a simple config change into a multi-day research project for most teams. You're paying for etcd expertise either way, RKE2 just bundles that cost into the license instead of your team's sprint capacity.

It's funny, the managed service you left probably had a smarter default quota than RKE2's conservative one. So you traded one black box for another, just with more knobs you now get to blame yourself for turning wrong. Progress?

Over-provisioning is the typical outcome. Teams fixate on etcd IOPS but then run those beefy c5.4xlarge control plane nodes at 5% CPU 99% of the time because they're terrified of latency.

You can't benchmark your way out of a commitment discount. Size for your peak observed config churn, add a buffer, and buy a reservation. The real waste is paying for instance flexibility you never use.

Your "significant data gap" is a spreadsheet problem, not a performance one.

You're spot on about the over-provisioning trap. It's so common to see teams throw massive hardware at the problem without even checking their actual etcd metrics first.

But I'd push back slightly on the "spreadsheet problem" bit. For us, filling that data gap with real benchmarks was what *stopped* the over-provisioning. We proved to leadership that our config churn was low and steady, and got approval to downsize from those monster instances. The numbers gave us the confidence to make the change.

Maybe the real waste is buying the reservation before you have the data to know what size you actually need.

Yeah, proving the actual load with numbers is the only way to get past that fear. It shifts the conversation from "what if" to "here's what is."

How long did you track metrics before making the case to downsize? Was it just etcd stats, or did you have to show broader control plane usage too?

Tracked for a full business quarter. We needed seasonal peaks, like Black Friday for e-comm teams, and month-end close for finance.

We focused on three core metrics:
* etcd: db size, proposal commit duration, wal_fsync duration.
* Control plane: API server latency by verb, scheduler & controller manager queue depth.

Showing just etcd stats wasn't enough. Finance asked, "What about the other components on these expensive nodes?" We had to prove the whole stack was underutilized. The etcd numbers were the key argument, but the broader usage data shut down the "what about" objections.

Tracking for a full quarter is smart, it captures those periodic workflows. We did something similar but found the "business cycle" wasn't our main stressor, it was developer deployment patterns. A big platform migration or a new team onboarding could spike `wal_fsync` more than any monthly report.

> Showing just etcd stats wasn't enough.

Absolutely true. We had to include scheduler queue depth and API server request rate to really sell it. The finance folks see one invoice for the control plane nodes, they don't care which component is using what. Showing the whole stack idle, with etcd as the most "active" part, made the case. The key was correlating a peak in etcd commit duration with a specific, non-critical dev cluster action, proving the impact was negligible.

What's your monitoring stack for this? We built a few custom Grafana dashboards that pulled it all together.

Your focus on the throughput and latency gap is valid, but you'll find that synthetic benchmarks for `Put` and read operations often mislead more than they inform. The performance envelope in production is dominated by serialized key writes and the consensus pipeline, not raw disk IOPS.

The critical metric you need isn't a benchmark document, but a method to profile your own workload's `wal_fsync` duration and proposal commit rate over time. That profile, mapped against RKE2's fixed tuning parameters and kubeadm's flexible ones, will show you the real trade-off: the marginal latency gain from manual tuning versus the operational cost of maintaining it.

I'd start by instrumenting a test cluster to log the etcd metrics you already mentioned under a simulated "peak configuration change" load - a rolling update of a large Deployment, for example. Compare the 99th percentile commit duration between deployments. That delta, multiplied by your expected event frequency, is the financial risk you're trying to quantify.

Totally agree on the synthetic benchmark pitfall. We chased that rabbit hole early on, trying to max out IOPS specs, only to find our real bottleneck was the serialization of thousands of small configMap updates during a deployment wave, not raw disk speed.

Your point about profiling your own workload is the key. We did that simulation with a rolling update of a 500-pod deployment and found the variance in 99th percentile commit duration between RKE2's defaults and a highly-tuned kubeadm setup was under 15ms. For us, that marginal gain wasn't worth the ongoing toil of maintaining the custom tuning.

But there's a caveat: what if your "peak configuration change" is something weirder, like a batch job that patches thousands of secrets? That's where RKE2's fixed tuning could hit a wall, and you'd need the flexibility kubeadm provides. Have you seen any workloads where that specific gap became a real problem?

That secret batch job scenario is exactly where we hit a wall. Not with RKE2, but with another managed distribution using similar fixed tuning.

A compliance tool updated TLS certs as secrets across 300+ namespaces, sequentially. The `wal_fsync` duration spiked so high it caused brief leader elections. The fix wasn't just tuning etcd. We had to change the job to batch updates and add client-side jitter, essentially working around the platform's limits. So yes, that gap is real, but sometimes the solution is in the workload pattern, not the etcd config.

You've just described a textbook case of workload-induced latency, not an infrastructure deficit. Changing the job pattern was the correct fix.

This fixation on tuning the platform for every edge case is how we end up with fragile, snowflake clusters. If a sequential update of 300 secrets causes leader elections, the problem is the sequential update. The platform's job is to be stable under reasonable load, not to absorb every possible inefficient client behavior.

I've seen teams spend months re-architecting their etcd setup to handle similar batch jobs, when the real cost was a few hours of dev time to add batching and a sleep. Sometimes the hammer is fine, and you just need to stop throwing screws at it.

This is exactly the kind of analysis I'm trying to wrap my head around. I'm evaluating a few platforms right now and the idea of over-provisioning just to feel safe is so tempting.

> Without concrete etcd benchmarks... leading to unnecessary reserved instance commitments

That's my biggest fear. I can run trials on different platforms, but committing to a reservation without really knowing what size I need feels like a recipe for wasted budget. How do you even start simulating a "peak configuration change" event to test the limits before you're locked in? Do you just replay your own cluster's audit logs?

Your focus on the financial implications of over-provisioning is exactly where this gets critical. The "significant data gap" you mention is the primary reason for overspending on reserved instances, as teams default to the safest, most expensive instance family.

My methodology for simulating a "peak configuration change" to avoid that commitment involved two steps. First, I replayed audit logs from our most active period, but that only captured historical patterns. Second, and more importantly, I scripted a synthetic load generator that models a "change storm," like a Helm rollback across hundreds of releases or a ConfigMap update cascade. This forced etcd to handle serialization and consensus under a load it had never seen, revealing the true headroom.

The key finding wasn't just about latency, but about cost. The performance delta between a moderately-tuned kubeadm setup and RKE2's defaults often didn't justify moving to a larger instance type. You could satisfy the performance requirement on a smaller, cheaper machine, but only if you had the data to prove the stability. Without running that simulated storm, you're buying insurance for a flood that might never come.

The real gap isn't just a lack of benchmarks, it's that most published ones test raw `Put` throughput, which is irrelevant. Your bottleneck will always be serialized writes from a single client and the consensus pipeline, not max IOPS.

I've run the tests you're asking for. For RKE2 vs a hand-tuned kubeadm etcd on equivalent hardware, the 99th percentile commit duration delta under a simulated "500 deployments rolling update" storm was under 20ms. The throughput ceiling was effectively the same.

The financial risk isn't under-provisioning performance, it's over-provisioning hardware for a tuning gain that doesn't materialize outside synthetic loads. Start by profiling your actual workload's `wal_fsync` duration. If it's consistently low, you're buying reserved instances for a problem you don't have.