Home Lab: Part 1 – Cluster Setup

This entry is part 1 of 6 in the series Home Lab

I recently setup a Kubernetes cluster home lab and wanted to do it the hard-way and share it with you. I setup a home lab so I could run my smart home software and learn more about different Kubernetes networking technologies.

This blog post is broken up into several sections. Feel free to skip directly to the section that applies to you.

When I started I had a few things already:

  • I was already using Rancher as a UI to manage my Kubernetes clusters on my dedicated servers
  • A Windows computer that can run K8s
  • A Ubiquiti EdgeRouter 12 acting as my home network’s router

Requirements

I wanted a fully flat network, that means no packet encapsulation. Packet encapsulation tunnels IP packets inside of other IP packets and creates a separate IP network that runs on-top of my existing network.) I wanted all nodes, pods, and services to be fully routable on my home network. Additionally, I had several Sonos speakers and other smart-home devices that I wanted to be control from my k8s cluster which required pods that ran on the same IP network.

Alternatives

Docker Desktop and WSL2 are both great for development Docker projects where you use the Docker CLI, but when you try to run Kubernetes you’ll quickly run into networking issues. WSL2 and Docker Desktop can’t expose services to the rest of your network very easily because they use NAT’d network adapters. (GitHub microsoft/WSL#4150) This means you can’t expose nodes or pods as devices on the network, they will always be NAT’d to the host’s IP address. This failed my requirement.

Continue reading “Home Lab: Part 1 – Cluster Setup”

Home Lab: Part 2 – Networking Setup

This entry is part 2 of 6 in the series Home Lab

Next up in the series, we’re going to manually configure all of the network settings to get our flat network home lab. Our flat network should not use any packet encapsulation with all pods and services fully routable to and from the existing network.

Detailed in the previous post, I want a so-called flat network because packet encapsulation tunnels IP packets inside of other IP packets and creates a separate IP network that runs on-top of my existing network.) I wanted all nodes, pods, and services to be fully routable on my home network. Additionally, I had several Sonos speakers and other smart-home devices that I wanted to be control from my k8s cluster which required pods that ran on the same subnet as my other software.

Install CNI Plugin

The CNI (Container Network Interface) plugin is responsible configuring the network adapter that each Kubernetes pod has. Since each pod usually gets a separate network namespace isolated from the host’s main network adapter, without it, no pod could make any network calls. For more information, check out cni.dev or the K8s documentation.

Continue reading “Home Lab: Part 2 – Networking Setup”

Home Lab: Part 3 – Networking Revisited

This entry is part 3 of 6 in the series Home Lab

The Problem

In my previous post series, I described how I installed my Kubernetes Home Lab using Calico and MetalLB. This worked great up until I started installing smart home software that expected to be able to do local network discovery. For example, Home Assistant and my Sonos control software both attempted to do subnet local discovery using mDNS or broadcast packets. This did not work because the pods were running on a 192.168.4.0/24 subnet, but all of my physical devices were on 192.168.2.0/24.

This prevented Home Assistant from discovering any devices and had to be fixed.

Continue reading “Home Lab: Part 3 – Networking Revisited”

Home Lab: Part 4 – A DHCP IPAM

This entry is part 4 of 6 in the series Home Lab

In the previous post, we end up abusing subnets and routing to get Calico to exist on the correct subnet, but what if we could get rid of Calico’s duplicate IPAM system and just depend on our existing DHCP server to handle reservations? In this post, we’re going to prototype a cluster that uses DHCP + layer 2 Linux bridging to avoid the complications outlined in Part 3.

The official CNI documentation describes two plugins that could be relevant.

With dhcp plugin the containers can get an IP allocated by a DHCP server already running on your network.

https://www.cni.dev/plugins/current/ipam/dhcp/

This avoids overlapping IPAM problems with the previous solution and means that the DHCP server already running on my network would be responsible for handing out IP addresses directly to the containers.

Continue reading “Home Lab: Part 4 – A DHCP IPAM”

Home Lab: Part 5 – Problems with asymmetrical routing

This entry is part 5 of 6 in the series Home Lab

In the previous post (DHCP IPAM), we successfully got our containers running with macvlan + DHCP. I additionally installed MetalLB and everything seemingly worked, however when I tried to retroactively add this to my existing Kubernetes home lab cluster already running Calico, I was not able to access the Metallb service. All connections were timing out.

A quick Wireshark packet capture of the situation exposed this problem:

The SYN packet from my computer made it to the container (LB IP 1921.168.6.2), but the responding SYN/ACK packet that came back had a source address of 192.168.2.76 (the pod’s network interface.) This wouldn’t work because my computer ignored it because it didn’t belong to an active flow.

Continue reading “Home Lab: Part 5 – Problems with asymmetrical routing”

Home Lab: Part 6 – Replacing MACvlan with a Bridge

This entry is part 6 of 6 in the series Home Lab

In previous posts, I leveraged the MACvlan CNI to provide the networking to forward packets between containers and the rest of my network, however I ran into several issues rooted from the fact that MACvlan traffic bypasses several parts of the host’s IP stack including conntrack and IPTables. This conflicted with how Kubernetes expects to handle routing and meant we had to bypass and modify IPTables chains to get it to work.

While I got it to work, there was simply too much wire bending involved and I wanted to investigate alternatives to see if anything was able to fit my requirements better. Let’s consider the bridge CNI.

Continue reading “Home Lab: Part 6 – Replacing MACvlan with a Bridge”