I’ve been modernizing my OVH-hosted dedicated servers running my various services moving from a CPU from 2015 to a CPU from 2019. While that’s still not that new, they’re budget servers and I only pay $30 USD a month for them. I managed to upgrade 2 out of 3 of them without any issues, making some improvements as I go, including enabling encryption at rest for all my data. However, the last one proved to be a problem.

Normally, I can pay for the server, let OVH install a base OS (like Ubuntu Server), then with nixos-anywhere I remotely wipe the machine and install NixOS over top without any issues. But this time, after rebooting, the server would hang trying to mount my RAID array (each server has 2x SSD drives that I RAID1 for performance and fault tolerance.) I couldn’t SSH to remotely unlock my drives because the SSH would immediately close the connection after negotiating an encrypted tunnel.

OVH included an emergency rescue OS that I could reboot into and inspect the drive, which gave me the error message:

nvme0n1p2 does not have a valid v1.2 superblock, not importing!

In this post, I walk through all the steps I went through to diagnose and arrive at a Linux kernel bug.

Nix/NixOS is a declarative language for defining your entire operating system. I use it on my dedicated servers to be able to apply GitOps for the servers. I define my services in a Git repo, everything from what version of packages to use, to what services should be installed, and how they should be installed. Those servers run Kubernetes which is where most of my services live.

Nix is a beast. The language is quite complicated and I wrote about my challenges. While I’ve gotten used to the language, I still don’t consider it intuitive. With that out of the way, my next challenge is that if I run nix flake update, which updates the packages that come from NixPkgs (which is most packages that you install.) Then I don’t really know what’s changing.

In my home lab, I use cdk8s which builds on AWS CDK to define my Kubernetes resources infrastructure as code. A lot of people use Helm which uses YAML programming, which I wrote about why I didn’t like, but cdk8s allows me to write it in TypeScript. I can write reusable classes to reduce duplication, like an Ingress construct that handles all the configuration I need. If you want to read more about the pros and cons of Cdk8s, read the post.

In this post, I’m going to show how I use an open-source system, Renovatebot, to keep my home lab up to date.

What’s in a Docker image? Nobody knows! Unless, you use an SBOM. What is an SBOM you say? Source Bill of Materials is a file containing all of the resources and tools that were used to build or are included in a software release artifact. SBOMs can be used to identify vulnerable software or even ensure compliance with software licenses. Imagine being able to block vulnerable software using a Kyverno policy.

I wanted to generate a Docker image that contained .net AOT compiled application and also generate an SBOM. Unfortunately, it wasn’t as straight forward as generating it from the output Docker image because it didn’t contain any build packages or any dotnet packages.

Additionally, the build tools have just as much of a role to play in software security as the libraries that end up in the artifact. For example, there have been several supply chain attacks that injected code into a binary or modified release artifacts using stole tokens. An SBOM should contain the build tools too, which simply running Syft won’t give you.

Kyverno is a handy policy engine for Kubernetes. For example, I use them to improve security by enabling user namespaces, or fix compatibility issues. I frequently use to validate or mutate resources that are created.

However, the YAML format can be a bit tricky for some of my more complicated policies. LLMs do alright creating many policies, but can struggle with my more complicated policies because they have to one-shot the policy and create it in one pass with no feedback or validation.

In this post, I’m going to show how I configure OpenWebUI to help me draft Kyverno policies.