Managed Kubernetes services like AKS, EKS and GKE handle upgrades for you. With a self-managed cluster built with kubeadm, the upgrade is your job. This guide walks through a real upgrade of a small three-node cluster running in my home lab, step by step, and tells you exactly which machine to run each command on.
It is written for RPM-based Linux distributions (Rocky Linux, AlmaLinux, RHEL). On Debian or Ubuntu the flow is identical, but you swap dnf for apt.
The home lab behind this guide
I did this upgrade on a cluster I built to practice real cluster operations without paying for cloud infrastructure or buying extra machines. Everything runs on a single laptop:
- Host machine: a laptop with a 12-thread Intel CPU and 32 GB of RAM, running Rocky Linux 10.
- Virtualization: KVM with libvirt, using its default NAT network so the virtual machines can reach the internet.
- Nodes: three virtual machines running Rocky Linux 10 (minimal install), each with 2 vCPUs and 4 GB of RAM. One is the control plane and two are workers. I built one template VM, then cloned it.
- Kubernetes: a cluster created with
kubeadm, using containerd as the container runtime and Cilium as the network plugin. Ak8s-apihostname points at the control plane, so a highly available setup can be added later without rebuilding. - Starting version: Kubernetes 1.35.9.
- Safety net: a libvirt snapshot of each VM, taken while powered off, before the upgrade.
You don't need a setup like this to follow along. Any kubeadm cluster works, whether it runs on virtual machines, bare metal, or cloud instances. A home lab is just a cheap, safe place to practice and break things.
What this guide assumes
A kubeadm cluster with three nodes, named like this throughout the article (the real hostnames in my lab are shorter, so I renamed them here for clarity):
| Node name | Role |
|---|---|
control-plane | Runs the Kubernetes control plane (API server, etcd, scheduler) |
worker-1 | Runs your applications |
worker-2 | Runs your applications |
- All nodes currently run Kubernetes 1.35.x.
- The Kubernetes packages (
kubeadm,kubelet,kubectl) were installed from the officialpkgs.k8s.iorepository. - You have
sudoaccess on every node.
Use your own node names wherever you see these. kubectl get nodes lists them.
A few terms
- Minor and patch versions: in
1.36.2, the minor version is36and the patch is2. kubeadm upgrades one minor version at a time, so you go 1.35 to 1.36, and only then to 1.37. - Drain: safely evict the pods from a node so you can work on it. Kubernetes reschedules them elsewhere.
- Cordon / uncordon: mark a node as unschedulable, then schedulable again. Draining cordons the node automatically, and
uncordonbrings it back.
You can't skip minor versions
Kubernetes upgrades must be incremental. If your cluster runs 1.35 and the newest release is 1.37, you cannot jump straight from 1.35 to 1.37. You have to go through every minor version in between:
1.35 → 1.36 → 1.37 The official kubeadm documentation states that skipping minor versions during an upgrade is unsupported. When I ran kubeadm on my 1.35 cluster, it warned that a much newer version (1.37) existed, but it still only offers the next minor version, 1.36.
In practice, this means:
- Each minor version is a full upgrade cycle. You repeat Steps 1 to 4 below once per minor version. For the second hop, change the package repository from
v1.36tov1.37and run the same procedure again. - Patch versions can be skipped. Going from 1.35.2 straight to 1.35.9, or from 1.36.0 to 1.36.4, is fine, because you are staying within the same minor version.
- Falling far behind gets expensive. A cluster that is four minor versions old needs four separate upgrades, which is why regular upgrades are easier than occasional big ones.
This guide covers the first hop, 1.35 to 1.36. For the second hop, repeat it with the next version number.
The plan
- Prepare: take backups and deploy a test application.
- Switch every node to the 1.36 package repository.
- Upgrade the control plane node.
- Upgrade the worker nodes, one at a time.
- Verify everything.
The order matters. The control plane always goes first, because Kubernetes allows a kubelet to be older than the API server, but never newer.
Step 0: Prepare
Run on: your hypervisor or a backup tool (if your nodes are VMs).
Take a snapshot of every node while it is powered off. A kubeadm upgrade can't be undone, so snapshots are your safety net. Snapshotting a running VM can capture an inconsistent state, so wait until each VM has fully shut down before taking the snapshot.
Run on: control-plane.
Confirm the cluster is healthy and that the node uses cgroup v2. Kubernetes 1.36 makes the kubelet refuse to run on cgroup v1 by default, and modern distributions use v2:
kubectl get nodes -o wide kubectl get pods -A | grep -v Running stat -fc %T /sys/fs/cgroup All nodes should be Ready, the second command should print nothing but the header line, and the last should print cgroup2fs.
Run on: control-plane.
Deploy a small test application, so you can prove it keeps working through the upgrade:
kubectl create deployment web --image=nginx --replicas=3 kubectl expose deployment web --type=NodePort --port=80 kubectl rollout status deployment/web Run on: control-plane.
Optionally, take an etcd backup. Replace etcd-control-plane with etcd- plus your control plane node name:
kubectl -n kube-system exec etcd-control-plane -- etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /var/lib/etcd/pre-upgrade.db Step 1: Switch the package repository to 1.36
The Kubernetes package repositories are split by minor version, so each node must be pointed at the 1.36 one. This only changes which packages are available. Nothing is upgraded yet.
Run on: every node (control-plane, worker-1 and worker-2).
sudo sed -i 's/v1\.35/v1.36/g' /etc/yum.repos.d/kubernetes.repo grep -E 'baseurl|gpgkey' /etc/yum.repos.d/kubernetes.repo Both lines in the output should now contain v1.36.
The repository file protects the Kubernetes packages with an
exclude=line, so a routinednf updatecan't upgrade your cluster by accident. That's why every install command below includes--disableexcludes=kubernetes.
Step 2: Upgrade the control plane
Do all of Step 2 on the control-plane node.
2.1 Find the newest 1.36 patch version and store it in a variable:
sudo dnf clean all VER=$(dnf list --showduplicates kubeadm --disableexcludes=kubernetes 2>/dev/null \ | awk '/^kubeadm/ {print $2}' | grep '^1\.36\.' | sort -V | tail -1 | cut -d- -f1) echo "Target version: $VER" This prints something like 1.36.2. If it prints nothing, the repository change in Step 1 didn't work.
2.2 Upgrade kubeadm:
sudo dnf install -y "kubeadm-${VER}-*" --disableexcludes=kubernetes kubeadm version -o short 2.3 Check the upgrade plan:
sudo kubeadm upgrade plan This lists the versions the cluster components will move to. Read it before continuing.
2.4 Apply the upgrade:
sudo kubeadm upgrade apply "v${VER}" Answer y when asked. This takes a few minutes. kubectl may time out for a short while as the API server restarts, which is normal. Wait for the message saying the control plane was upgraded successfully.
2.5 Upgrade the kubelet and kubectl on this node:
kubectl drain control-plane --ignore-daemonsets --delete-emptydir-data sudo dnf install -y "kubelet-${VER}-*" "kubectl-${VER}-*" --disableexcludes=kubernetes sudo systemctl daemon-reload sudo systemctl restart kubelet kubectl uncordon control-plane --ignore-daemonsets skips pods that run on every node by design, and --delete-emptydir-data allows evicting pods that use temporary storage.
2.6 Check:
kubectl get nodes control-plane should now show 1.36.x and Ready, while both workers still show 1.35.x. That mixed state is supported during an upgrade.
Step 3: Upgrade the workers
Upgrade one worker at a time, and confirm it is healthy before starting the next. The steps below use worker-1. Repeat the whole step for worker-2.
3.1 Run on: control-plane. Drain the worker:
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data 3.2 Run on: worker-1. Upgrade the node. Workers use kubeadm upgrade node, not upgrade apply:
sudo dnf clean all VER=$(dnf list --showduplicates kubeadm --disableexcludes=kubernetes 2>/dev/null \ | awk '/^kubeadm/ {print $2}' | grep '^1\.36\.' | sort -V | tail -1 | cut -d- -f1) echo "Target version: $VER" sudo dnf install -y "kubeadm-${VER}-*" --disableexcludes=kubernetes sudo kubeadm upgrade node sudo dnf install -y "kubelet-${VER}-*" "kubectl-${VER}-*" --disableexcludes=kubernetes sudo systemctl daemon-reload sudo systemctl restart kubelet The Target version printed here must match the one you used on the control plane.
3.3 Run on: control-plane. Bring the worker back:
kubectl uncordon worker-1 kubectl get nodes Wait until worker-1 shows Ready on 1.36, then repeat Step 3 for worker-2.
Step 4: Verify
Run on: control-plane.
kubectl get nodes -o wide kubectl get pods -A sudo kubeadm certs check-expiration You are looking for:
- All three nodes
Readyand on the same 1.36 version. - Every pod
Running. - Renewed certificates (kubeadm renews them during an upgrade).
Then test the sample application. Find its port, then request it through each node. Replace the IP addresses with your own node IPs:
NODEPORT=$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}') curl -s -o /dev/null -w "%{http_code}\n" http://<control-plane-ip>:$NODEPORT curl -s -o /dev/null -w "%{http_code}\n" http://<worker-1-ip>:$NODEPORT curl -s -o /dev/null -w "%{http_code}\n" http://<worker-2-ip>:$NODEPORT Each should print 200. Clean up the test app:
kubectl delete svc web kubectl delete deployment web If something goes wrong
- The version variable is empty, or
dnfcan't find the package: the repository wasn't switched on that node. Re-check/etc/yum.repos.d/kubernetes.repo, then runsudo dnf clean all. -
kubeadm upgrade planshows nothing to upgrade:kubeadmitself wasn't upgraded first. Checkkubeadm version -o short. -
kubectlhangs right afterkubeadm upgrade apply: the API server is restarting. Wait one to two minutes. - A drain never finishes: something is blocking eviction, often a PodDisruptionBudget or a pod not managed by a controller. Look at
kubectl get pods -A -o wideto find it. - A node stays
NotReady: checksudo journalctl -u kubelet -n 40 --no-pageron that node, and the health of your network plugin. - Roll back: shut down all nodes and restore all of them to the snapshots from Step 0 together, so etcd and the kubelets stay consistent.
Takeaways
- Upgrade one minor version at a time, control plane first, then workers one by one.
- Take offline snapshots (or an etcd backup) before you start, because kubeadm upgrades can't be reversed.
- Always drain a node before upgrading it and uncordon it afterwards.
- Run
kubeadm upgrade applyonce, on the first control plane node, andkubeadm upgrade nodeeverywhere else. - Test with a real workload, so you know the cluster works, not just that the versions match.
Once you are comfortable with this procedure, repeat it for the next minor version, or practice restoring from the etcd backup.