Managed Kubernetes services like AKS, EKS and GKE handle upgrades for you. With a self-managed cluster built with kubeadm, the upgrade is your job. This guide walks through a real upgrade of a small three-node cluster running in my home lab, step by step, and tells you exactly which machine to run each command on.

It is written for RPM-based Linux distributions (Rocky Linux, AlmaLinux, RHEL). On Debian or Ubuntu the flow is identical, but you swap dnf for apt.

The home lab behind this guide

I did this upgrade on a cluster I built to practice real cluster operations without paying for cloud infrastructure or buying extra machines. Everything runs on a single laptop:

  • Host machine: a laptop with a 12-thread Intel CPU and 32 GB of RAM, running Rocky Linux 10.
  • Virtualization: KVM with libvirt, using its default NAT network so the virtual machines can reach the internet.
  • Nodes: three virtual machines running Rocky Linux 10 (minimal install), each with 2 vCPUs and 4 GB of RAM. One is the control plane and two are workers. I built one template VM, then cloned it.
  • Kubernetes: a cluster created with kubeadm, using containerd as the container runtime and Cilium as the network plugin. A k8s-api hostname points at the control plane, so a highly available setup can be added later without rebuilding.
  • Starting version: Kubernetes 1.35.9.
  • Safety net: a libvirt snapshot of each VM, taken while powered off, before the upgrade.

You don't need a setup like this to follow along. Any kubeadm cluster works, whether it runs on virtual machines, bare metal, or cloud instances. A home lab is just a cheap, safe place to practice and break things.

What this guide assumes

A kubeadm cluster with three nodes, named like this throughout the article (the real hostnames in my lab are shorter, so I renamed them here for clarity):

Node name Role
control-plane Runs the Kubernetes control plane (API server, etcd, scheduler)
worker-1 Runs your applications
worker-2 Runs your applications
  • All nodes currently run Kubernetes 1.35.x.
  • The Kubernetes packages (kubeadm, kubelet, kubectl) were installed from the official pkgs.k8s.io repository.
  • You have sudo access on every node.

Use your own node names wherever you see these. kubectl get nodes lists them.

A few terms

  • Minor and patch versions: in 1.36.2, the minor version is 36 and the patch is 2. kubeadm upgrades one minor version at a time, so you go 1.35 to 1.36, and only then to 1.37.
  • Drain: safely evict the pods from a node so you can work on it. Kubernetes reschedules them elsewhere.
  • Cordon / uncordon: mark a node as unschedulable, then schedulable again. Draining cordons the node automatically, and uncordon brings it back.

You can't skip minor versions

Kubernetes upgrades must be incremental. If your cluster runs 1.35 and the newest release is 1.37, you cannot jump straight from 1.35 to 1.37. You have to go through every minor version in between:

1.35 → 1.36 → 1.37 

The official kubeadm documentation states that skipping minor versions during an upgrade is unsupported. When I ran kubeadm on my 1.35 cluster, it warned that a much newer version (1.37) existed, but it still only offers the next minor version, 1.36.

In practice, this means:

  • Each minor version is a full upgrade cycle. You repeat Steps 1 to 4 below once per minor version. For the second hop, change the package repository from v1.36 to v1.37 and run the same procedure again.
  • Patch versions can be skipped. Going from 1.35.2 straight to 1.35.9, or from 1.36.0 to 1.36.4, is fine, because you are staying within the same minor version.
  • Falling far behind gets expensive. A cluster that is four minor versions old needs four separate upgrades, which is why regular upgrades are easier than occasional big ones.

This guide covers the first hop, 1.35 to 1.36. For the second hop, repeat it with the next version number.

The plan

  1. Prepare: take backups and deploy a test application.
  2. Switch every node to the 1.36 package repository.
  3. Upgrade the control plane node.
  4. Upgrade the worker nodes, one at a time.
  5. Verify everything.

The order matters. The control plane always goes first, because Kubernetes allows a kubelet to be older than the API server, but never newer.

Step 0: Prepare

Run on: your hypervisor or a backup tool (if your nodes are VMs).
Take a snapshot of every node while it is powered off. A kubeadm upgrade can't be undone, so snapshots are your safety net. Snapshotting a running VM can capture an inconsistent state, so wait until each VM has fully shut down before taking the snapshot.

Run on: control-plane.
Confirm the cluster is healthy and that the node uses cgroup v2. Kubernetes 1.36 makes the kubelet refuse to run on cgroup v1 by default, and modern distributions use v2:

kubectl get nodes -o wide kubectl get pods -A | grep -v Running stat -fc %T /sys/fs/cgroup 

All nodes should be Ready, the second command should print nothing but the header line, and the last should print cgroup2fs.

Run on: control-plane.
Deploy a small test application, so you can prove it keeps working through the upgrade:

kubectl create deployment web --image=nginx --replicas=3 kubectl expose deployment web --type=NodePort --port=80 kubectl rollout status deployment/web 

Run on: control-plane.
Optionally, take an etcd backup. Replace etcd-control-plane with etcd- plus your control plane node name:

kubectl -n kube-system exec etcd-control-plane -- etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /var/lib/etcd/pre-upgrade.db 

Step 1: Switch the package repository to 1.36

The Kubernetes package repositories are split by minor version, so each node must be pointed at the 1.36 one. This only changes which packages are available. Nothing is upgraded yet.

Run on: every node (control-plane, worker-1 and worker-2).

sudo sed -i 's/v1\.35/v1.36/g' /etc/yum.repos.d/kubernetes.repo grep -E 'baseurl|gpgkey' /etc/yum.repos.d/kubernetes.repo 

Both lines in the output should now contain v1.36.

The repository file protects the Kubernetes packages with an exclude= line, so a routine dnf update can't upgrade your cluster by accident. That's why every install command below includes --disableexcludes=kubernetes.

Step 2: Upgrade the control plane

Do all of Step 2 on the control-plane node.

2.1 Find the newest 1.36 patch version and store it in a variable:

sudo dnf clean all VER=$(dnf list --showduplicates kubeadm --disableexcludes=kubernetes 2>/dev/null \ | awk '/^kubeadm/ {print $2}' | grep '^1\.36\.' | sort -V | tail -1 | cut -d- -f1) echo "Target version: $VER" 

This prints something like 1.36.2. If it prints nothing, the repository change in Step 1 didn't work.

2.2 Upgrade kubeadm:

sudo dnf install -y "kubeadm-${VER}-*" --disableexcludes=kubernetes kubeadm version -o short 

2.3 Check the upgrade plan:

sudo kubeadm upgrade plan 

This lists the versions the cluster components will move to. Read it before continuing.

2.4 Apply the upgrade:

sudo kubeadm upgrade apply "v${VER}" 

Answer y when asked. This takes a few minutes. kubectl may time out for a short while as the API server restarts, which is normal. Wait for the message saying the control plane was upgraded successfully.

2.5 Upgrade the kubelet and kubectl on this node:

kubectl drain control-plane --ignore-daemonsets --delete-emptydir-data sudo dnf install -y "kubelet-${VER}-*" "kubectl-${VER}-*" --disableexcludes=kubernetes sudo systemctl daemon-reload sudo systemctl restart kubelet kubectl uncordon control-plane 

--ignore-daemonsets skips pods that run on every node by design, and --delete-emptydir-data allows evicting pods that use temporary storage.

2.6 Check:

kubectl get nodes 

control-plane should now show 1.36.x and Ready, while both workers still show 1.35.x. That mixed state is supported during an upgrade.

Step 3: Upgrade the workers

Upgrade one worker at a time, and confirm it is healthy before starting the next. The steps below use worker-1. Repeat the whole step for worker-2.

3.1 Run on: control-plane. Drain the worker:

kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data 

3.2 Run on: worker-1. Upgrade the node. Workers use kubeadm upgrade node, not upgrade apply:

sudo dnf clean all VER=$(dnf list --showduplicates kubeadm --disableexcludes=kubernetes 2>/dev/null \ | awk '/^kubeadm/ {print $2}' | grep '^1\.36\.' | sort -V | tail -1 | cut -d- -f1) echo "Target version: $VER" sudo dnf install -y "kubeadm-${VER}-*" --disableexcludes=kubernetes sudo kubeadm upgrade node sudo dnf install -y "kubelet-${VER}-*" "kubectl-${VER}-*" --disableexcludes=kubernetes sudo systemctl daemon-reload sudo systemctl restart kubelet 

The Target version printed here must match the one you used on the control plane.

3.3 Run on: control-plane. Bring the worker back:

kubectl uncordon worker-1 kubectl get nodes 

Wait until worker-1 shows Ready on 1.36, then repeat Step 3 for worker-2.

Step 4: Verify

Run on: control-plane.

kubectl get nodes -o wide kubectl get pods -A sudo kubeadm certs check-expiration 

You are looking for:

  • All three nodes Ready and on the same 1.36 version.
  • Every pod Running.
  • Renewed certificates (kubeadm renews them during an upgrade).

Then test the sample application. Find its port, then request it through each node. Replace the IP addresses with your own node IPs:

NODEPORT=$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}') curl -s -o /dev/null -w "%{http_code}\n" http://<control-plane-ip>:$NODEPORT curl -s -o /dev/null -w "%{http_code}\n" http://<worker-1-ip>:$NODEPORT curl -s -o /dev/null -w "%{http_code}\n" http://<worker-2-ip>:$NODEPORT 

Each should print 200. Clean up the test app:

kubectl delete svc web kubectl delete deployment web 

If something goes wrong

  • The version variable is empty, or dnf can't find the package: the repository wasn't switched on that node. Re-check /etc/yum.repos.d/kubernetes.repo, then run sudo dnf clean all.
  • kubeadm upgrade plan shows nothing to upgrade: kubeadm itself wasn't upgraded first. Check kubeadm version -o short.
  • kubectl hangs right after kubeadm upgrade apply: the API server is restarting. Wait one to two minutes.
  • A drain never finishes: something is blocking eviction, often a PodDisruptionBudget or a pod not managed by a controller. Look at kubectl get pods -A -o wide to find it.
  • A node stays NotReady: check sudo journalctl -u kubelet -n 40 --no-pager on that node, and the health of your network plugin.
  • Roll back: shut down all nodes and restore all of them to the snapshots from Step 0 together, so etcd and the kubelets stay consistent.

Takeaways

  • Upgrade one minor version at a time, control plane first, then workers one by one.
  • Take offline snapshots (or an etcd backup) before you start, because kubeadm upgrades can't be reversed.
  • Always drain a node before upgrading it and uncordon it afterwards.
  • Run kubeadm upgrade apply once, on the first control plane node, and kubeadm upgrade node everywhere else.
  • Test with a real workload, so you know the cluster works, not just that the versions match.

Once you are comfortable with this procedure, repeat it for the next minor version, or practice restoring from the etcd backup.