There were a few services that I ran that I wanted to be able to access from both inside my home network and outside my home network. If I was inside my home network, I wanted to route directly to the service, but if I was outside I needed to be able to route traffic through a proxy that would then route into my home lab. Additionally, I wanted to support SSL on all my services for security using cert-manager
Since my IPv4 addresses differ inside my network vs outside, I need to use split-horizon DNS to respond with the correct DNS query. Split-horizon DNS refers to the DNS on one horizon (inside the network) showing different results than outside the network.
One day, we’ll all be on IPv6 and this won’t be needed anymore because all services will be using globally unique IPv6 addresses, but a las we’re stuck in IPv4 land. With IPv6, instead of having separate views of the IP addresses, all services could just get a single global IPv6 address registered. Then no matter where you are, the same IPv6 address gets routed to the correct Kubernetes service.
My cluster is using MetalLB as it’s L4 load balancer and it or an equivalent is required to be able to forward IP-level packets to the DNS server. NodePorts don’t work because they use a random port and other servers expect to be able to use port 53/udp and 53/tcp.
Prior to this, I purchased a domain name, mydomain.com. I reserved a subdomain, *.home.mydomain.com that all of my home lab software will run on-top of. I usually purchase my domain names from porkbun.com, but any domain name registrar will work.
The Problem with TLS
To provision certificates, Let’s Encrypt needs to confirm that you own or at least have control over the DNS domain name that you’re requesting. Since I’m running home lab inside my private network that’s not accessible to the internet, it won’t be able to verify I own *.home.mydomain.com. To fix this, I need cert-manager to be able to create a DNS record on a publicly resolvable DNS record.
The diagram below shows the problem with split horizon DNS in this case. cert-manager needs to be able to update the external DNS server and be able to verify the DNS record is updated, but if it hits the internal DNS server then it won’t be able to confirm everything is working.
Architectural diagram showing how DNS queries progress through the network. Clients call into Pi-Hole, which then conditionally forwards the lab zone to the CoreDNS instance, everything else goes to the internet. CoreDNS then queries etcd for the lab zone results.
Let’s break down the different components and how to configure each one.
Component diagram showing the software we’re going to deploy
Ingress controllers in Kubernetes automatically update each Ingress resource’s status block with an IP address that can be used to access that Ingress. However, there are multiple IP addresses that can be associated with a given ingress. If you’re using a NodePort service, then that ingress would be pointing to the Node’s IP, but a Layer 3 LB service, would need to use the IP of the ingress-nginx’s LB.
See the below example, this ingress is pointing to the node:
The correct IP as exposed by MetalLB for NGINX is 192.168.6.8. To fix this, we need to tell ingress-nginx to instead use it’s service IP. If you’ve deployed nginx using Helm, then change the values.yaml
After deploying, NGINX should modify the Ingress statuses to point to the correct IP:
External-dns will store the DNS records in etcd and CoreDNS will look up query responses for the zone in in etcd.
External-DNS is a Kubernetes tool that will take ingresses or services defined in Kubernetes and automatically create DNS records in some provider for you. For ingresses, it takes the IP address defined in the status of the Ingress by the ingress controller (ingress-nginx.)
Following the Helm install method, udate the following Helm values. Note that I’m using the K8s namespace ‘dns’. If you deploy your etcd/external-dns in a different namespace, make sure to update ETCD_URLs below. Also note, I’m using the fully qualified names because of an issue I discovered and documented in this post.
Once deployed, check the logs to verify that the records are being created correctly.
CoreDNS will answer the DNS queries by querying etcd for records.
Kubernetes has a limitation where we can’t use TCP and UDP on the same Load Balancer service.
To fix this, you have two options:
- Enable the feature gate
- Disable TCP based DNS queries and hope you don’t exceed the size of a UDP packet (ref.)
Many Kubernetes clusters come built in with a CoreDNS instance already to handle pod DNS queries.
- Create a separate CoreDNS instance to handle split-horizon zone queries.
- Update the existing kube-dns instance (if it’s CoreDNS)
If the downstream DNS server is located outside the cluster, then enable the LB. If you’re running something like Pi-hole in the cluster, then you don’t need this and can route directly here
After that, you should have a Load Balancer created like below. Use this to configure your DNS server to forward queries for your internal domain to the LB IP address.
Cert-manager is responsible for requesting certificates from Let’s Encrypt (or any other compliant ACME certificate provider) and storing them in your cluster. To install, following the standard Helm installation process, but make sure to update following values.
--dns01-recursive-nameservers flag tells cert-manager not to use the internal DNS server to verify propagation of the DNS01 verifier and instead hit Google’s public DNS server. Feel free to use any public resolver.
After that, create a ClusterIssuer or Issuer. In the DNS01 solver, you’ll need to configure a DNS solver for your domain name. See the official docs on how to do that for your case.
After the service is deployed, update the router to forward traffic for your given domain name to the newly created service IP address.
In the Ubiquiti EdgeRouter config, this is defined as:
set service dns forwarding options server=/home.mydomain.com/192.168.6.2
After this is setup, you should be able to access any ingress you’ve got configured under your home DNS zone when you’re inside your home network.
If you want to be able to access your home lab services outside your home network, you can use Wireguard or expose NGINX to the internet and create DNS records in a publicly accessible DNS zone. If you expose any services to the internet, do take care to ensure that you securely restrict access. In a future post, I may document how I do this for my own network.