Three laptops and an itch

I had three laptops that nobody was using. All three still booted, and all three were more than capable of running Linux. That was the problem: too good to throw away, too boring to keep as they were.

At the same time I was teaching myself platform engineering: Kubernetes, Terraform, the AWS side of things. Nearly all my practice had happened on throwaway clusters running inside Docker on one machine. Those are fine for learning commands. They teach you nothing about what happens when a machine disappears.

So the idea was simple. Put Linux on all three laptops, join them into one real cluster, and then break it on purpose. No cloud bill, no free-tier clock ticking. Three physical machines on a shelf.

This is how that went, including the parts that did not work.

Why three, not two

My first plan was two laptops. I had a guide for a K3s cluster with two control-plane nodes, and it came with a catch: two servers cannot form a safe quorum on their own. The guide worked around this by running PostgreSQL in Docker as an external datastore.

Then laptop number three turned up, and the whole design got simpler. Kubernetes keeps its state in etcd, and etcd needs a majority of its members to agree. Three members can lose one and carry on. Two cannot. With three nodes I could drop PostgreSQL entirely and use K3s's embedded etcd. One less service to babysit, and a failure mode I could actually test.

The Mac never became part of the cluster. It stayed my personal machine, and its only job was to be the remote control: kubectl and k9s pointed at the laptops from the outside. Three Linux servers, one client, nothing installed on the Mac beyond command-line tools.

The first surprise: I need cat6 cables

With Ubuntu installed on all three, I ran ip a on the first laptop to find its address. The output told me two things. The subnet was 192.168.1.0/24, and the machine was on Wi-Fi. The Ethernet port, enp0s31f6, said NO-CARRIER: nothing plugged in.

That mattered more than I expected. etcd is sensitive to network jitter, and a Wi-Fi blip can trigger a leader election or even cost the cluster its quorum. A server that roams between access points is a bad foundation for a control plane.

So the laptops went on cables. One Cat6 run from each machine into a switch, and one more from the switch into a LAN port on the router. (The WAN port is a different one. Plugging a second cable from the switch back into the router creates a loop, which is a good way to take your whole house offline.) The Mac stayed on Wi-Fi, because it is only the client.

The next ip -br a showed the wired port up with an address. It also corrected an assumption: the home network was 192.168.0.0/24, not the 192.168.1.x range my first check had shown. Every address I had been planning around was wrong by one octet. Checking before assuming became a theme.

Reaching it from anywhere, and pinning the addresses

A homelab you can only reach from the couch is half a homelab. I had been testing SSH against a private LAN address, and I wanted to understand why that would stop working the moment I left the house. The answer is that 192.168.x.x addresses mean nothing outside your own network.

Tailscale fixed that in an afternoon. Install it, run sudo tailscale up, open the login link, and every device you sign in joins a private network with stable 100.x.x.x addresses that work from anywhere. No port forwarding, no exposed SSH. That also turned out to matter for a second reason: home 5G routers typically sit behind carrier-grade NAT, so opening a port to the outside is usually not even possible. I turned off key expiry on each laptop, so a headless machine could never lock me out.

Then came the addresses. A cluster needs its nodes to stay put, and my router's DHCP pool (.100 to .200) was free to hand any of them a different address after a lease expired. I logged in to the router, a 5G broadband unit, and went looking for a DHCP reservation. The first page I found looked right and was not: it was a Wi-Fi allow and block list. The actual feature was a tab called Static IP, where you bind a MAC address to an address.

I reserved 192.168.0.21, .22 and .23, just outside the pool, then bounced each laptop's wired connection so it would pick up the new address. One small lesson from that step: turning Wi-Fi off over an SSH session that came in over Wi-Fi drops the session. Connect through the interface you are not about to change.

The bug that cost me an evening

With the network settled, the K3s install itself is almost anticlimactic. One command on the first laptop starts the cluster with --cluster-init. One command on each of the other two joins them to it. I ran them, and the second and third laptops promptly crashed.

The first error in the logs was not authorized. The join token I had passed was being rejected. Fixing the token produced a different error: etcd disabled. That one was more interesting, because it meant the first laptop was not running etcd at all. It was running K3s's default single-node database, which cannot accept new servers.

But I had passed --cluster-init. So I opened the service file K3s had generated and looked at how the flags had actually been recorded:

' --cluster-init' \ ' --token=...' \ 

Every flag had a space inside the quotes. K3s was being handed " --cluster-init" instead of "--cluster-init", did not recognise it as a flag, and silently ignored it. That one invisible character explained both errors: the flag that enables etcd was ignored, and so was the flag that set my token.

The most likely culprit was the multi-line command I had copied, whose indentation can carry hidden non-breaking spaces into the shell. The fix was boring and worked immediately: uninstall K3s everywhere, generate a fresh token, and run each install as a single line with the token passed as K3S_TOKEN. I also checked the service file afterwards, instead of trusting that the command had done what I typed.

And since I had pasted the old token into a chat while debugging, I replaced it. Treat join tokens as secrets, even on a home LAN.

Three nodes, all Ready

After the clean reinstall, the first laptop came up as Ready with the roles control-plane,etcd. That role list was the proof I had been after: the flag was finally being read, and etcd was running. I joined the second laptop, waited for it to settle, then the third, one at a time so etcd could add each member cleanly.

Then I ran kubectl get nodes and saw three machines, all Ready, all control-plane,etcd. [Add: your own moment here. What you were doing, what you saw, how it felt.]

The real test comes next. Stop K3s on one laptop and the other two should carry on, with the stopped node showing NotReady after about 40 seconds. Start it again and it rejoins. That is the thing a single-node cluster inside Docker could never show me.

What I would tell someone starting today

The hardware was never the hard part. Almost every problem was a layer underneath Kubernetes: a cable that was not plugged in, a subnet I had guessed wrong, an address the router was free to reassign, and one stray space in a command. If you are building something similar, these are the lessons I would pass on:

  • Use wired Ethernet. etcd wants a steady network, and Wi-Fi is not one.
  • Use three nodes, not two. An odd number gives you quorum without extra moving parts.
  • Pin the addresses early, either with a router reservation or static IPs outside the DHCP pool.
  • Put Tailscale on everything first. It solves remote access and the NAT problem together.
  • Paste commands as single lines, then check what was actually recorded instead of trusting what you typed.
  • Keep the control machine separate. My Mac runs no cluster software, only kubectl and k9s.

The lab is a working cluster now.