About 2 years ago, I had published a much more comprehensive article series of 17 articles called "Setting Up Kubernetes the Hard Way". As the lab environment, we had set it up in 4 different ways: LXD, KVM, Vagrant-Libvirt and Vagramt-Virtualbox. About 2 years have passed since then. There are many details in it, and it's still possible to benefit from those articles for many topics. In fact, in this article too, I'll refer to those pages in some places.
For those who come here without ever reading the other article series, I'll take the introduction from my old article.
The title originally comes from Kelsey Hightower's article series on Github titled "Kubernetes The Hard Way". If you like, for those who haven't heard of it, let's first talk about what this "Kubernetes The Hard Way" thing is.
Simply put, in his article Kelsey Hightower sets up the Controller Plane and Node (Worker) components of Kubernetes without using any deployment tool (for example Kubeadm, Ansible, RKE, etc.).
At first, the clusters he set up were always on Google Cloud, but in his latest update he has now done the installation on a bare-metal ARM64 system.
This time, I'll stick to Kelsey's installation as much as possible, but we'll do the installation on the x64 architecture. I'll drop plenty of notes along the way. We'll try to do a bit of troubleshooting.
Now let's look at which versions of which components we'll install. And a little reminder
- kubectl 1.30.3: Lets users communicate with the Kubernetes API Server and is used to carry out various management tasks.
- kube-apiserver 1.30.3: Serves the Kubernetes API, and all operations are carried out through this API. Users, management tools and other components carry out operations such as creating, deleting or updating resources by communicating with this server. The API Server also manages authentication and authorization operations.
- kube-controller-manager 1.30.3: It's a component that runs the Kubernetes controllers. Controllers observe the state of Kubernetes objects and compare the desired state with the current state. If there's a difference, the controllers carry out the necessary corrective actions. For example, a Deployment controller makes sure that the specified number of Pods are running.
- kube-scheduler: 1.30.3: Determines on which nodes newly created Pods will run. This decision is made by taking into account various factors such as the nodes' current resource usage, hardware capacities, network locations and specific policies. The Scheduler makes sure Pods are distributed in a balanced and efficient way.
- kube-proxy: 1.30.3: Runs on each node and routes traffic to Kubernetes Service objects. By applying network rules, it organizes the communication between Pods and routes the traffic coming in from the outside world to the related Pods. This is a way of managing dynamic IP addressing and doing load balancing within Kubernetes.
- kubelet: 1.30.3: It's an agent running on each Kubernetes node and manages the lifecycle of the Pods and containers on the node. Kubelet receives instructions from the API Server and interacts with the Container Runtime to start, stop or restart containers. It also monitors the health of Pods and carries out restart operations when needed.
- crictl: 1.30.1: Used to debug the Kubernetes cluster. Without using the Kubernetes API, operations can be done directly on Kubernetes components (pod, container, etc.). We can think of it like the Docker cli. It's a tool that can communicate with CRI-based continer runtimes.
- runc 1.1.13: It's a low-level container runtime; it provides isolation using features like cgroups, namespaces, etc. Actually, higher-level runtimes like Docker, Containerd or CRI-O use runc.
- containerd 1.7.20: It's a CRI-compatible Container Runtime.
- etcd 3.5.15: Stores all the state information in the Kubernetes cluster (e.g. Pod, Service, ConfigMap). This information includes the desired state as well as the current state of the cluster, so the system can recreate the correct state in the event of any disruption.
We'll create a service file, certificate and kubeconfig file for the Kubernetes components in the list and run them. There's actually no magic. To bring up just one application, we need to bring up many services with TLS.
We'll set up our lab environment with Vagrant-Virtulbox. You can find the Vagrantfile below.
If you haven't used Vagrant before, you don't need to get hung up on it. We won't have much to do with Vagrant commands. You just need to install Vagrant from this page. And of course you also need to install Virtualbox.
We'll have a system like this.
| Hostname | DNS | ip | Description | CPU | RAM | Disk |
|---|---|---|---|---|---|---|
| bastion | bastion | 192.168.56.31 | The machine where I'll do the installations and manage our cluster | 2 | 2 | 10 |
| master1 | master1.kubernetes.local | 192.168.56.41 | Kubernetes Server Node | 2 | 2 | 10 |
| worker1 | worker0.kubernetes.local | 192.168.56.51 | Kubernetes Worker Node 1 | 2 | 2 | 10 |
| worker2 | worker1.kubernetes.local | 192.168.56.52 | Kubernetes Worker Node 2 | 2 | 2 | 10 |
Setting Up the Lab Environment
For us to be able to set up a Kubernetes cluster, our system needs to meet some prerequisites.
- The machines should be able to connect to each other over ssh with the root account without a password
- The ports you can check in the official Kubernetes documentation should be open within the cluster. To keep it easy, we'll keep the firewall turned off between the servers.
- Swap should be turned off
- The NTP settings of the servers should be made. In other words, the date and time should be the same on all servers.
- There shouldn't be any network obstacles. We already opened all the servers on the same subnet, and the firewall is off.
- (recommendation) If you have a large data center with many servers and different subnets, it'll make more sense to access the servers via DNS names rather than hostnames. Since there's no DNS in our own system, we'll simulate the situation by writing the DNS names into our hosts file.
- (recommendation) In your production environments, block the subnets of the pod network and the service networks across the data center.
Of course, many more topics need attention for production environments. Since I don't know where you'll run the lab environment we created, I just mentioned the small things that won't cause you problems.
The official Kubernetes documentation has prepared a very detailed page for the production environment. You can read it from this link.
The scripts inside the Vagrantfile take care of the prerequisites needed for the Kubernetes installation. The details are written in the comment lines inside the script. If we hadn't written them into the Vagranfile, we'd run the commands below on the servers. Which commands will be run on which server is also written in the scripts.
We create a file named Vagrantfile in a folder and copy the code below into it. Then, while in the same directory, we run the vagrant up command.
We'll be using the Ubuntu 22.04 (jammy64) version.
# -*- mode: ruby -*- # vi: set ft=ruby : MASTER_COUNT = 1 WORKER_COUNT = 2 MASTER_IP_PREFIX = "192.168.56.4" WORKER_IP_PREFIX = "192.168.56.5" PROVIDER = "virtualbox" BASTION_IP = "192.168.56.31" ROOT_PASS = "vagrant" ENV['VAGRANT_DEFAULT_PROVIDER'] = PROVIDER $allscript = <<-ALLSCRIPT MASTER_COUNT=$1 WORKER_COUNT=$2 MASTER_IP_PREFIX=$3 WORKER_IP_PREFIX=$4 BASTION_IP=$5 ROOT_PASS=$6 echo master count : $MASTER_COUNT echo worker count : $WORKER_COUNT echo master ip prefix : $MASTER_IP_PREFIX echo master worker prefix : $WORKER_IP_PREFIX echo bastion ip : $BASTION_IP echo root pass : $ROOT_PASS sudo su echo "============== the required installations are being done =====" sudo apt install ntpstat socat conntrack ipset -y sudo apt install ntp -y echo "============== root user is being activated ====" echo -e "$ROOT_PASS\n$ROOT_PASS" | (passwd root) echo -e "$ROOT_PASS\n$ROOT_PASS" | (passwd vagrant) echo "export TERM=xterm" | tee -a /etc/bash.bashrc echo "============== ntp is being activated ====" sudo apt install ntp sudo systemctl restart ntp timedatectl status echo "============== swap and firewall are being disabled ====" sed -i "/swap/d" /etc/fstab swapoff -a systemctl disable --now ufw echo "============== sysctl is being updated for the kubernetes network ====" # the required kernel modules are being installed modprobe overlay modprobe br_netfilter cat <<EOF | sudo tee -a /etc/sysctl.d/kubernetes.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl --system echo "============= ssh settings are being made =====" sed -i "s/#PasswordAuthentication yes/PasswordAuthentication yes/" /etc/ssh/sshd_config sed -i "s/#PermitRootLogin prohibit-password/PermitRootLogin yes/" /etc/ssh/sshd_config sed -i "s/#PubkeyAuthentication yes/PubkeyAuthentication yes/" /etc/ssh/sshd_config sudo systemctl restart ssh echo "============= host files are being edited ===" for i in $(seq 1 $MASTER_COUNT); do echo "$MASTER_IP_PREFIX$i master$i" >>/etc/hosts echo "$MASTER_IP_PREFIX$i master$i.kubernetes.local" >>/etc/hosts done for i in $(seq 1 $WORKER_COUNT); do echo "$WORKER_IP_PREFIX$i worker$i" >>/etc/hosts echo "$WORKER_IP_PREFIX$i worker$i.kubernetes.local" >>/etc/hosts done echo "$BASTION_IP bastion" >>/etc/hosts ALLSCRIPT $bastionscript = <<-BASTION MASTER_COUNT=$1 WORKER_COUNT=$2 MASTER_IP_PREFIX=$3 WORKER_IP_PREFIX=$4 BASTION_IP=$5 ROOT_PASS=$6 echo master count : $MASTER_COUNT echo worker count : $WORKER_COUNT echo master ip prefix : $MASTER_IP_PREFIX echo master worker prefix : $WORKER_IP_PREFIX echo bastion ip : $BASTION_IP echo root pass : $ROOT_PASS sudo apt install sshpass -y sudo su ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa for i in $(seq 1 $MASTER_COUNT); do sshpass -p $ROOT_PASS ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no root@$MASTER_IP_PREFIX$i echo "yes \n" | scp ~/.ssh/id_rsa root@$MASTER_IP_PREFIX$i:.ssh/ echo key copied: $MASTER_IP_PREFIX$i done for i in $(seq 1 $WORKER_COUNT); do sshpass -p $ROOT_PASS ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no root@$WORKER_IP_PREFIX$i echo "yes \n" | scp ~/.ssh/id_rsa root@$WORKER_IP_PREFIX$i:.ssh/ echo key copied: $WORKER_IP_PREFIX$i done BASTION Vagrant.configure("2") do |config| # master (1..MASTER_COUNT).each do |i| config.vm.define "master#{i}" do |master| master.vm.box = "gusztavvargadr/ubuntu-server-2404-lts" master.vm.network "private_network", ip: MASTER_IP_PREFIX+"#{i}" master.vm.hostname = "master#{i}" master.ssh.username = "vagrant" master.ssh.password = $ROOT_PASS master.ssh.insert_key = "true" master.vm.provider :PROVIDER do |lv| lv.memory=2048 lv.cpus=2 end end end # worker (1..WORKER_COUNT).each do |i| config.vm.define "worker#{i}" do |worker| worker.vm.box = "gusztavvargadr/ubuntu-server-2404-lts" worker.vm.network "private_network", ip: WORKER_IP_PREFIX+"#{i}" worker.vm.hostname = "worker#{i}" worker.ssh.username = "vagrant" worker.ssh.password = $ROOT_PASS worker.ssh.insert_key = "true" worker.vm.provider :PROVIDER do |lv| lv.memory=2048 lv.cpus=2 lv.storage :file, :size => '10G' end end end # bastion config.vm.define "bastion" do |bastion| bastion.vm.box = "gusztavvargadr/ubuntu-server-2404-lts" bastion.vm.network "private_network", ip: BASTION_IP bastion.vm.hostname = "bastion" bastion.ssh.username = "vagrant" bastion.ssh.password = $ROOT_PASS bastion.ssh.insert_key = "true" bastion.vm.provider :PROVIDER do |lv| lv.memory=2048 lv.cpus=2 end bastion.vm.provision "shell", run: "once", inline: $bastionscript, args: [MASTER_COUNT,WORKER_COUNT, MASTER_IP_PREFIX,WORKER_IP_PREFIX, BASTION_IP, ROOT_PASS] end # globally defined shell provision. It will run on all vms config.vm.provision :shell, run: "once" do |s| s.args = [MASTER_COUNT,WORKER_COUNT, MASTER_IP_PREFIX,WORKER_IP_PREFIX, BASTION_IP, ROOT_PASS] s.inline = $allscript end end After the installation finishes, we do the tests below.
- While again in the directory where the Vagrantfile is located, we log in to the Bastion machine with the
vagrant ssh bastioncommand. As soon as we log in, we switch to the root user and test by pinging all the machines by hostname. - We need to be able to ssh from the Bastion machine to all machines with the root user without a password. Be sure to test this.
- While doing the tests, we also need to do them with DNS names. In other words, for example, if we're going to ssh from bastion to worker1, we should use
worker1.kubernetes.localas the domain name.
If a problem came up in the tests, there may have been a problem in one or several of the steps we did in the Vagranfile. In this case, you can run the vagrant up command again, or we can delete all the machines with the vagrant destroy command and do the installation again with the vagrant up command. Finally, if it still isn't fixed, we can manually run the related command in the Vagrantfile in the places where we got an error.
You can also check it on Kelsey's GitHub page.
Preparing the Bastion Server for the Installation
We'll do the installation and configuration of all the servers through our Bastion server. So, since we'll share through this server both the applications the server needs and the binaries to be installed on the other servers, they need to be downloaded too.
Tools like openssl, git, curl, wget and ssh that are likely to be needed come installed already. For the other installations, we connect to our bastion server with the vagrant ssh bastion command and switch to the root user. We create a directory for our files.
KHOME=$HOME/kubernetes-files mkdir $KHOME cd $KHOME We add the download addresses of all the applications we'll download to the downloads.txt file. You can find all the necessary binaries from the links below.
- https://www.downloadkubernetes.com
- https://github.com/opencontainers/runc/releases
- https://github.com/kubernetes-sigs/cri-tools/releases
- https://github.com/containerd/containerd/releases
- https://github.com/etcd-io/etcd/releases
mkdir -p $KHOME/downloads cat << EOF > $KHOME/downloads/downloads.txt https://dl.k8s.io/v1.30.3/bin/linux/amd64/kubectl https://dl.k8s.io/v1.30.3/bin/linux/amd64/kube-apiserver https://dl.k8s.io/v1.30.3/bin/linux/amd64/kube-controller-manager https://dl.k8s.io/v1.30.3/bin/linux/amd64/kube-scheduler https://dl.k8s.io/v1.30.3/bin/linux/amd64/kube-proxy https://dl.k8s.io/v1.30.3/bin/linux/amd64/kubelet https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-amd64-v1.5.1.tgz https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.30.1/crictl-v1.30.1-linux-amd64.tar.gz https://github.com/opencontainers/runc/releases/download/v1.1.13/runc.amd64 https://github.com/containerd/containerd/releases/download/v1.7.20/containerd-1.7.20-linux-amd64.tar.gz https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz EOF We download the binaries with the command below
KHOME=$HOME/kubernetes-files wget -q --show-progress \ --https-only \ --timestamping \ -P $KHOME/downloads \ -i $KHOME/downloads/downloads.txt When we list the files, we should see a list like the one below.
cni-plugins-linux-amd64-v1.5.1.tgz containerd-1.7.20-linux-amd64.tar.gz crictl-v1.30.1-linux-amd64.tar.gz downloads.txt etcd-v3.5.15-linux-amd64.tar.gz kube-apiserver kube-controller-manager kubectl kubelet kube-proxy kube-scheduler runc.amd64 We can install kubectl on our Bastion server.
KHOME=$HOME/kubernetes-files chmod +x $KHOME/downloads/kubectl cp $KHOME/downloads/kubectl /usr/local/bin/ kubectl version --client # Client Version: v1.30.3 # Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Creating the TLS Certificates
We'll create the certificates with openssl. For this, we create our certificate configuration file with the command below. We'll create the certificates below.
- admin
- worker1
- worker2
- kube-proxy
- kube-scheduler
- kube-controller-manager
- kube-api-server
- service-accounts
You can see the certificate names in the configuration.
KHOME=$HOME/kubernetes-files mkdir $KHOME/certs cat << EOF > $KHOME/certs/ca.conf [req] distinguished_name = req_distinguished_name prompt = no x509_extensions = ca_x509_extensions [ca_x509_extensions] basicConstraints = CA:TRUE keyUsage = cRLSign, keyCertSign [req_distinguished_name] C = US ST = Washington L = Seattle CN = CA [admin] distinguished_name = admin_distinguished_name prompt = no req_extensions = default_req_extensions [admin_distinguished_name] CN = admin O = system:masters # Service Accounts: the certificate needed both for the admin user and for privileged operations across the whole system [service-accounts] distinguished_name = service-accounts_distinguished_name prompt = no req_extensions = default_req_extensions [service-accounts_distinguished_name] CN = service-accounts # Worker Nodes: The certificates here are actually being created for kubelet. # for authorization operations, CN (CommonName) naming in the form system:node:<nodeName> is required. # https://kubernetes.io/docs/concepts/overview/components/#kubelet [worker1] distinguished_name = worker1_distinguished_name prompt = no req_extensions = worker1_req_extensions [worker1_req_extensions] basicConstraints = CA:FALSE extendedKeyUsage = clientAuth, serverAuth keyUsage = critical, digitalSignature, keyEncipherment nsCertType = client nsComment = "Worker 1 Certificate" subjectAltName = DNS:worker1, IP:127.0.0.1 subjectKeyIdentifier = hash [worker1_distinguished_name] CN = system:node:worker1 O = system:nodes C = US ST = Washington L = Seattle [worker2] distinguished_name = worker2_distinguished_name prompt = no req_extensions = worker2_req_extensions [worker2_req_extensions] basicConstraints = CA:FALSE extendedKeyUsage = clientAuth, serverAuth keyUsage = critical, digitalSignature, keyEncipherment nsCertType = client nsComment = "Worker 2 Certificate" subjectAltName = DNS:worker2, IP:127.0.0.1 subjectKeyIdentifier = hash [worker2_distinguished_name] CN = system:node:worker2 O = system:nodes C = US ST = Washington L = Seattle # Kube Proxy Section [kube-proxy] distinguished_name = kube-proxy_distinguished_name prompt = no req_extensions = kube-proxy_req_extensions [kube-proxy_req_extensions] basicConstraints = CA:FALSE extendedKeyUsage = clientAuth, serverAuth keyUsage = critical, digitalSignature, keyEncipherment nsCertType = client nsComment = "Kube Proxy Certificate" subjectAltName = DNS:kube-proxy, IP:127.0.0.1 subjectKeyIdentifier = hash [kube-proxy_distinguished_name] CN = system:kube-proxy O = system:node-proxier C = US ST = Washington L = Seattle # Controller Manager [kube-controller-manager] distinguished_name = kube-controller-manager_distinguished_name prompt = no req_extensions = kube-controller-manager_req_extensions [kube-controller-manager_req_extensions] basicConstraints = CA:FALSE extendedKeyUsage = clientAuth, serverAuth keyUsage = critical, digitalSignature, keyEncipherment nsCertType = client nsComment = "Kube Controller Manager Certificate" subjectAltName = DNS:kube-proxy, IP:127.0.0.1 subjectKeyIdentifier = hash [kube-controller-manager_distinguished_name] CN = system:kube-controller-manager O = system:kube-controller-manager C = US ST = Washington L = Seattle # Scheduler [kube-scheduler] distinguished_name = kube-scheduler_distinguished_name prompt = no req_extensions = kube-scheduler_req_extensions [kube-scheduler_req_extensions] basicConstraints = CA:FALSE extendedKeyUsage = clientAuth, serverAuth keyUsage = critical, digitalSignature, keyEncipherment nsCertType = client nsComment = "Kube Scheduler Certificate" subjectAltName = DNS:kube-scheduler, IP:127.0.0.1 subjectKeyIdentifier = hash [kube-scheduler_distinguished_name] CN = system:kube-scheduler O = system:system:kube-scheduler C = US ST = Washington L = Seattle # API Server [kube-api-server] distinguished_name = kube-api-server_distinguished_name prompt = no req_extensions = kube-api-server_req_extensions [kube-api-server_req_extensions] basicConstraints = CA:FALSE extendedKeyUsage = clientAuth, serverAuth keyUsage = critical, digitalSignature, keyEncipherment nsCertType = client nsComment = "Kube Scheduler Certificate" subjectAltName = @kube-api-server_alt_names subjectKeyIdentifier = hash # Here, if we had a load balancer configuration, we'd need to enter its domain too. [kube-api-server_alt_names] IP.0 = 127.0.0.1 IP.1 = 10.32.0.1 DNS.0 = kubernetes DNS.1 = kubernetes.default DNS.2 = kubernetes.default.svc DNS.3 = kubernetes.default.svc.cluster DNS.4 = kubernetes.svc.cluster.local DNS.5 = server.kubernetes.local DNS.6 = api-server.kubernetes.local [kube-api-server_distinguished_name] CN = kubernetes C = US ST = Washington L = Seattle [default_req_extensions] basicConstraints = CA:FALSE extendedKeyUsage = clientAuth keyUsage = critical, digitalSignature, keyEncipherment nsCertType = client nsComment = "Admin Client Certificate"Source：https://dev.to/muratcabuk/setting-up-kubernetes-the-hard-way-single-article-and-up-to-date-2024-version-8glRelated posts
GitHub Actions pricing changes 2026: price per minute, the self-hosted runner fee, and cheaper runners
Top AI SRE Tools in 2026: Compare Vendors and How to Choose
Running a .NET 8+ App on Something Other Than Windows Part 1
What Is the HashiCorp Configuration Language (HCL)? How Is It Used?
Using LXC, Automation with the Terraform LXD Provider, and Creating a Kubernetes Cluster Using LXC with Terraform
