Last week, I upgraded my home server from Ubuntu 24.04 LTS (Long Term Support) to Ubuntu 26.04 LTS. I expected smooth sailing, but you and I should know not to expect that, especially if there’s a blog post coming from it.

As soon as I started my Kubernetes cluster, services were failing to start left and right. Cloudnative-pg cluster pods were stuck crashing and restarting forever. Initially, I thought it was some kind of bug in Longhorn (the distributed storage I use for Kubernetes) because I was getting errors when reading/writing data saying “Remote I/O Error”, but the Postgres error was far more impactful.

It finally happened to me. I dared to log into my LinkedIn account and update my job status because I left my job. I dared to update LinkedIn while I was on vacation in Hawaii.

When I tried to login, I found my account locked for security and they just needed a quick peek at my government ID to make sure I’m legit.

I’ve been modernizing my OVH-hosted dedicated servers running my various services moving from a CPU from 2015 to a CPU from 2019. While that’s still not that new, they’re budget servers and I only pay $30 USD a month for them. I managed to upgrade 2 out of 3 of them without any issues, making some improvements as I go, including enabling encryption at rest for all my data. However, the last server proved to be a problem.

Normally, I pay for the server, let OVH install a base OS (like Ubuntu Server), then use nixos-anywhere to remotely wipe the machine and install NixOS over top without any issues. But this time, after rebooting, the server would hang trying to mount my RAID array (each server has 2x SSD drives that I RAID1 for performance and fault tolerance.) I couldn’t SSH to remotely unlock my drives because the SSH would immediately close the connection after negotiating an encrypted tunnel.

OVH included an emergency rescue OS that I could reboot into and inspect the drive, which gave me the error message:

nvme0n1p2 does not have a valid v1.2 superblock, not importing!

In this post, I walk through all the steps I went through to diagnose and arrive at a nixos-anywhere bug caused by a Linux kernel change.

Nix/NixOS is a declarative language for defining your entire operating system. I use it on my dedicated servers to be able to apply GitOps for the servers. I define my services in a Git repo, everything from what version of packages to use, to what services should be installed, and how they should be installed. Those servers run Kubernetes which is where most of my services live.

Nix is a beast. The language is quite complicated and I wrote about my challenges. While I’ve gotten used to the language, I still don’t consider it intuitive. With that out of the way, my next challenge is that if I run nix flake update, which updates the packages that come from NixPkgs (which is most packages that you install.) Then I don’t really know what’s changing.

In my home lab, I use cdk8s which builds on AWS CDK to define my Kubernetes resources infrastructure as code. A lot of people use Helm which uses YAML programming, which I wrote about why I didn’t like, but cdk8s allows me to write it in TypeScript. I can write reusable classes to reduce duplication, like an Ingress construct that handles all the configuration I need. If you want to read more about the pros and cons of Cdk8s, read the post.

In this post, I’m going to show how I use an open-source system, Renovatebot, to keep my home lab up to date.