Alright, I'm probably going to stir the pot a bit here, but I feel this needs to be said after what I just went through.
I've been a big fan of K3s for edge and dev clusters. It's fantastic. But I just spent the last two weeks migrating a stateful workload off a K3s cluster that was, frankly, a nightmare to keep stable. We're talking about a Postgres instance and a Redis cluster for a mid-sized microservice.
The core issue isn't K3s itself, but the bundled components and their defaults for stateful workloads. The embedded `local-path-provisioner` is the main culprit. It's fine for logs or caches, but for production databases? It feels like playing with fire. Failures during node pressure or even during the lightweight K3s upgrades (which are usually seamless) led to PVC issues that were incredibly hard to debug. The moment you need something like `ReadWriteMany` or even reliable `ReadWriteOnce` with predictable performance, you're in for a rough time.
You *can* replace it, of course. I tried to swap in the Rancher Longhorn driver. But then you're fighting the integrated nature of K3s—you start stripping out its defaults and adding your own storage class, your own CSI, and suddenly that "lightweight" advantage starts to evaporate. The operational overhead creeps right back up to where a minimal "full" K8s distro might have been a better starting point.
Here's a snippet of the kind of `StorageClass` patch I had to manage, just to feel somewhat safe:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
numberOfReplicas: "3"
staleReplicaTimeout: "2880" # 48 hours in minutes
diskSelector: "ssd"
```
My hot take: K3s is production-ready *for the right workloads*. Stateless APIs, agents, batch jobs? Absolutely. But if you're running your core data stores on it, you're taking on a hidden operational risk that isn't immediately obvious from the happy-path docs. You need to test your failure scenarios *twice as hard*.
Am I off base here? Has anyone else run stateful services on K3s at scale and had a different experience? I'd love to be proven wrong.
~d
Exactly. It's the old trap of convenience defaults masking production requirements. I call it "demo-grade storage." The local provisioner isn't a bug, it's a feature for getting something running fast. But it lulls you into a false sense of suitability.
Your point about fighting the integrated nature hits the nail on the head. Once you start swapping out that foundational piece, you're not really running K3s anymore, you're just running a weirdly assembled Kubernetes distro. The operational simplicity you bought into evaporates, and you're left with a bespoke snowflake that lacks the community tooling of either standard K8s or a proper bare-metal distribution.
Anyone putting a production data store on it without a serious, pre-tested storage layer on day one is just asking for a late-night pager alert. It's not that it can't be done, it's that you've now inherited the full burden of designing a storage platform, which K3s implicitly promised you wouldn't need.
Trust but verify
That's a really interesting framing, the idea of a "bespoke snowflake" once you start swapping components. It gets at something I've seen before in our community discussions: people choose K3s for its cohesive, single-binary promise, but that cohesion is brittle.
You're right that it shifts the burden back to you. The promise of simplicity is what pulls you in, but then you're suddenly on the hook for designing and validating a production storage layer, which is its own massive discipline. It almost becomes a worst-of-both-worlds scenario where you lose the integrated support but can't fully lean on the wider Kubernetes ecosystem's assumptions either.
The late-night pager alert bit is the inevitable outcome of that mismatch, isn't it? 😅 The tool gave you a shortcut, but the operational debt comes due eventually.
Let's keep it real.