After I’ve had time to run my home lab for a while, I’ve started switching to a more up to date Linux distribution (instead of RancherOS.) I’m currently testing Ubuntu Server which leverages Systemd. Systemd-networkd is responsible for managing the network interface configuration and it differs in behavior compared to NetworkManager enough that we need to update the Home Lab Bridge CNI to handle it.

Previously the CNI was creating a bridge network adapter when the first container started up, but this causes problems with systemd because resolved (the DNS resolver component) was eventually failing to make DNS queries and networkd was duplicating IP addresses on both eth0 (the actual uplink adapter) and on cni0 because we were copying it over.

Prior to identifying this fix, I was experiencing issues where containers would work for a brief time, but then something would go wrong on the node host and it stopped being able to pull images and network traffic wasn’t routing between containers correctly.

After some digging, it looked like systemd was trying to use eth0 even though it had become enslaved to the cni0 bridge and the cni0 bridge interface effectively replaced it. The resolvectl command showed that cni0 had no configuration and only eth0 was configured.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# resolvectl
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2604:dead:beef:cafe::1
       DNS Servers: 192.168.2.1 2604:dead:beef:cafe::1

Link 3 (cni0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

The networkctl command showed that eth0 and cni0 were both operational.

# networkctl
IDX LINK         TYPE     OPERATIONAL SETUP
  1 lo           loopback carrier     unmanaged
  2 eth0         ether    routable    configured
  3 cni0         bridge   routable    unmanaged
  4 docker0      bridge   no-carrier  unmanaged

# networkctl status eth0
● 2: eth0
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: /run/systemd/network/10-netplan-eth0.network
                          Type: ether
                         State: routable (configured)
                    HW Address: 00:15:5d:02:26:00 (Microsoft Corporation)

The Fix

Instead of trying to create the bridge using my CNI, I instead changed cni0 to be created and managed by systemd. I created the following files.

This file changes eth0 to be bridged to cni0 and disables DHCP on this interface.

Note: This file name needs to be the same name as returned in networkctl status eth0 above, but is located in a different folder. Systemd uses this to know that the new file overrides the previous file.

/run/systemd/network/10-netplan-eth0.network -> /etc/systemd/network/10-netplan-eth0.network

1
2
3
4
5
6

# cat /etc/systemd/network/10-netplan-eth0.network
[Match]
Name=eth0

[Network]
Bridge=cni0

This next file tells networkd that it needs to create the cni0 bridge. The MACAddress is the same MAC address as eth0 and I copied it from the networkctl status eth0 output. Interestingly, bridge interfaces usually take the MAC address of the first interface in the bridge (in this case, it should be eth0). Systemd should be doing the same thing as per this GitHub issue, however this was not working for me, so I explicitly added this option to ensure it does.

1
2
3
4
5

# cat /etc/systemd/network/cni0.netdev
[NetDev]
Name=cni0
Kind=bridge
MACAddress=**00:15:5d:02:26:00**

This final file tells networkd that it should enable DHCP and IPv6 autoconfig. If you need to disable IPv6, remove the IPv6AcceptRA line.

1
2
3
4
5
6
7

# cat /etc/systemd/network/cni0.network
[Match]
Name=cni0

[Network]
DHCP=yes
IPv6AcceptRA=yes # Enable IPv6 autoconfig

Then after rebooting, the host should come back up and should now work correctly. Networkctl should now show eth0 is enslaved to cni0.

1
2
3
4
5
6

$ networkctl
IDX LINK         TYPE     OPERATIONAL SETUP
  1 lo           loopback carrier     unmanaged
  2 eth0         ether    enslaved    configured
  3 cni0         bridge   routable    configured
  4 docker0      bridge   no-carrier  unmanaged

resolvectl should now show that cni0 is being used for DNS queries instead of eth0.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# resolvectl
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (cni0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2604:dead:beef:cafe::1
       DNS Servers: 192.168.2.1 2604:dead:beef:cafe::1

Comments

Comments are currently unavailable while I move to this new blog platform. To give feedback, send an email to adam [at] this website url.