I’ve been modernizing my OVH-hosted dedicated servers running my various services moving from a CPU from 2015 to a CPU from 2019. While that’s still not that new, they’re budget servers and I only pay $30 USD a month for them. I managed to upgrade 2 out of 3 of them without any issues, making some improvements as I go, including enabling encryption at rest for all my data. However, the last one proved to be a problem.

Normally, I can pay for the server, let OVH install a base OS (like Ubuntu Server), then with nixos-anywhere I remotely wipe the machine and install NixOS over top without any issues. But this time, after rebooting, the server would hang trying to mount my RAID array (each server has 2x SSD drives that I RAID1 for performance and fault tolerance.) I couldn’t SSH to remotely unlock my drives because the SSH would immediately close the connection after negotiating an encrypted tunnel.

OVH included an emergency rescue OS that I could reboot into and inspect the drive, which gave me the error message:

nvme0n1p2 does not have a valid v1.2 superblock, not importing!

In this post, I walk through all the steps I went through to diagnose and arrive at a Linux kernel bug.

The Start Configuration

I started with a basic configuration that looked exactly like my other successful hosts, albeit with a change to the disk id. I opted to use /dev/disk/by-id/XYZ instead of /dev/sda which most of the Disko examples use because I wanted to be deterministic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
disko.devices = {
  disk = {
    disk1 = {
      type = "disk";
      device = "/dev/disk/by-id/nvme-SAMSUNG_MZVL2512...";
      content = {
        type = "gpt";
        partitions = {
          ESP = {
            type = "EF00";
            size = "500M";
            content = {
              type = "filesystem";
              format = "vfat";
              mountpoint = "/boot";
              mountOptions = [ "umask=0077" ];
            };
          };
          mdadm = {
            size = "100%";
            content = {
              type = "mdraid";
              name = "raid1";
            };
          };
        };
      };
    };
    disk2 = {
      type = "disk";
      device = "/dev/disk/by-id/nvme-WDC_CL_SN720_SDAQNTW...";
      content = {
        type = "gpt";
        partitions = {
          mdadm = {
            size = "100%";
            content = {
              type = "mdraid";
              name = "raid1";
            };
          };
        };
      };
    };
  };
  mdadm = {
    raid1 = {
      type = "mdadm";
      level = 1;
      extraArgs = [ "--bitmap=internal" ];
      content = {
        type = "gpt";
        partitions = {
          luks = {
            size = "100%";
            label = "luks";
            content = {
              type = "luks";
              name = "cryptroot";
              extraOpenArgs = [
                "--allow-discards"
                "--perf-no_read_workqueue"
                "--perf-no_write_workqueue"
              ];
              content = {
                type = "btrfs";
                mountpoint = "/";
                extraArgs = [ "-f" ];
                mountOptions = [ "noatime" ];
                subvolumes = {
                  "@root" = {
                    mountpoint = "/";
                    mountOptions = [ "compress=zstd" "noatime" "ssd" ];
                  };
                  "@home" = {
                    mountpoint = "/home";
                    mountOptions = [ "compress=zstd" "noatime" "ssd" ];
                  };
                  "@nix" = {
                    mountpoint = "/nix";
                    mountOptions = [ "compress=zstd" "noatime" "ssd" ];
                  };
                  "@persist" = {
                    mountpoint = "/persist";
                    mountOptions = [ "compress=zstd" "noatime" "ssd" ];
                  };
                };
              };
            };
          };
        };
      };
    };
  };
};
fileSystems."/persist".neededForBoot = true;

Initial steps into the hole

I’ll admit, I went a little crazy investigating this one. Initially, the error appeared to suggest some kind of intermittent issue or a configuration mistake on my part. Given the issue manifested as my SSH server closing the connection, I initially assumed it was related to that. I tried changing kernel modules, disabling impermanence (a NixOS+btrfs feature that causes NixOS to delete and rebuild the root filesystem each time to force me to configure everything using Nix), I messed with the initrd settings (Linux’s second stage bootloader that contains a kernel that launches my boot-time SSH server for remote unlocking).

While I had been using systemd-initrd to boot my other hosts, I noticed NixOS 26.05 changed initrd to systemd and tried rolling back to the old version. Switching from systemd-boot to the older grub boot wasn’t successful given my servers were EUFI boot based.

Given Nix Flakes are designed to be reproducible and deterministic, I even tried reverting nix.flake file back to an older Nixpkgs release. That failed too which really surprised me, but it turns out Nix Flakes doesn’t pin everything (foreshadowing.)

When nothing worked including removing the LUKS partition, I shifted my focus towards the higher-level partitions–the GPT and mdadm RAID array.

The descent into madness

A screenshot from the host remote KVE showing the Linux boot process stalling. Systemd is stalled trying to mount the raid1-part1 volume.

Clearly the boot errors were telling me something useful that the RAID array was the problem, not the LUKS encryption. I then started investigating other options. I tried configuring only the A drive or B drive at a time.

To confirm whether it was an issue where initrd couldn’t load the RAID array for booting, but it was valid, I tried configuring the root FS on a single drive and mounting the RAID array on a /test folder, that failed until I marked it as optional to boot, then the OS was able to boot. Once inside the OS, tried to manually mount the mdadm array and faced the same error:

nvme0n1p2 does not have a valid v1.2 superblock, not importing!

It’s taunting me. Even waiting for cat /proc/mdstat to show the RAID array was fully rebuilt.

I wrote a previous post about diffing changes between releases, which came in handy.

All the king’s men, AI, GitHub issues couldn’t figure it out

At this point, my hair is turning grey, my browser was growing slow with many tabs, and I had several LLM chats open with Claude Code, GLM-5.2, and Kimi-2.5. All of them confidently telling me to do something I already tried or go down different paths.

Assuming it was an issue with disk partitioning, I opened a GitHub issue disko#1276. They gave me a few ideas including there were was some old data remaining on the drive, thus I zeroed both drives start to the finish before running the disk partitioning:

1
2
dd if=/dev/zero of=/dev/nvme0n1 bs=10M
dd if=/dev/zero of=/dev/nvme1n1 bs=10M

Nope. I got a new dedicated server to test if it was a hardware issue, but I sadly had to renew my old server for a month while this sat around.

Manual Partitioning

I then turned to manually provisioning my RAID array. After using nixos-anywhere to kexec (kexec live boots from one Linux kernel into a new one), I ran the following commands:

The cryptsetup command, which is required to create a LUKS encrypted volume isn’t in nixos-installer by default, so I had to download it using:

1
nix-shell -p cryptsetup -I nixpkgs=channel:nixos-unstable

Then after partitioning the volumes, I ran

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
mdadm --create /dev/md/raid1 --level=1 --raid-devices=2 --metadata=1.2 /dev/nvme0n1p2 /dev/nvme1n1p1

cryptsetup luksFormat /dev/md0
cryptsetup luksOpen /dev/md0 cryptroot

mount -o subvol=@root,compress=zstd,noatime /dev/mapper/cryptroot /mnt

mount -o subvol=@home,compress=zstd,noatime /dev/mapper/cryptroot /mnt/home
mount -o subvol=@nix,compress=zstd,noatime /dev/mapper/cryptroot /mnt/nix
mount -o subvol=@persist,compress=zstd,noatime /dev/mapper/cryptroot /mnt/persist

# Mount first ESP (for installation)
mount /dev/nvme0n1p1 /mnt/boot

Then letting the RAID array rebuild and cleanly unmounting after installing:

1
2
3
cryptsetup luksClose /dev/mapper/cryptroot

mkfs.btrfs -L root /dev/mapper/cryptroot

Increase the log level

But that didn’t work for me, so I then I started to investigate other ways of getting debugging information out. I tried to dump out debug logs to the boot screen using the following:

1
2
3
4
5
6
boot.kernelParams = [
  "systemd.log_level=debug"
  "systemd.log_target=console"
  "systemd.journald.forward_to_console=1"
  "raid=noautodetect"
];

That worked, but unfortunately too much got dumped out and I couldn’t scroll back in the KVM to read anything.

Going serial

I then found that OVH exposed a serial over LAN feature which I could SSH into. If I could get this work, then I could pipe all the boot logs to a local file.

A screenshot from the OVH control panel showing an option to connect to the host serial port using SSH.

Some poking around the emergency distro gave me the commands I needed:

1
2
cat /proc/cmdline
vmlinuz BOOTMAC=... rdinit=/sbin/init console=tty0 console=ttyS1,115200n8 ramdisk_size=51200 load_ramdisk=1 panic=10 net.ifnames=0 rootfstype=ramfs systemd.gpt_auto=0 DATASOURCE=https://releases.mirrors.ovh.net/...

I was able to confirm that I could print things to my Serial over SSH session:

1
2
3
4
echo 'hello' > /dev/ttyS1

stty -F /dev/ttyS1 -a  
speed 115200 baud; rows 24; columns 80; line = 0;

Now, I add console=ttyS1,115200n8 to the Kernel boot params to get more output:

1
2
3
4
5
6
7
8
boot.kernelParams = [
  "systemd.log_level=debug"
  "systemd.log_target=console"
  "systemd.journald.forward_to_console=1"
  "console=tty0"
  "console=ttyS1,115200"
  "raid=noautodetect"
];

I needed more information. I combined that with a boot time systemd job that I manually printed data out. Figuring out when to get this to run is tricky. Consulting bootup shows

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
boot.initrd.systemd.services.mdadm-assemble = {
    description = "Assemble RAID arrays via mdadm";
    wantedBy = [ "initrd.target" ];
    after = [ "systemd-udevd.service" ];
    before = [
      "systemd-cryptsetup@cryptroot.service"
      "initrd-fs.target"
    ];
    script = ''
      set -x
      exec >/dev/kmsg 2>&1
      echo "MDADM-DEBUG: service started, pid=$$"

      echo "MDADM-DEBUG: enabled block devices:"
      lsblk -f

      echo "MDADM-DEBUG: by-id links:"
      ls -l /dev/disk/by-id/ || true

      echo "MDADM-DEBUG: /proc/mdstat before:"
      cat /proc/mdstat

      echo "MDADM-DEBUG: /etc/mdadm/mdadm.conf is:"
      cat /etc/mdadm/mdadm.conf || true

      echo "MDADM-DEBUG: detail scan before assembly:"
      ${pkgs.mdadm}/bin/mdadm --examine --scan --verbose || true

      echo "MDADM-DEBUG: running mdadm --assemble --scan --verbose"
      ${pkgs.mdadm}/bin/mdadm --assemble --scan --verbose
      rc=$?
      echo "MDADM-DEBUG: mdadm exit code: $rc"

      echo "MDADM-DEBUG: /proc/mdstat after:"
      cat /proc/mdstat

      echo "MDADM-DEBUG: /dev/md/* after:"
      ls -l /dev/md* || true

      echo "MDADM-DEBUG: mdadm --detail --scan after:"
      ${pkgs.mdadm}/bin/mdadm --detail --scan || true

      exit $rc
    '';
    serviceConfig = {
      StandardOutput = "journal+console";
      StandardError = "journal+console";
      Type = "oneshot";
      RemainAfterExit = true;
    };
  };

I put the project on the back-burner and went on an international vacation for a few weeks to recover.

Making some progress

While I was scuba diving, other people started reporting the same issue which made me realize there was something bigger including somebody mentioning there was a Kernel bug

I briefly tested using an older version of nixos-anywhere, but that didn’t work.

1
nix run 'github:nix-community/nixos-anywhere?rev=92f82c5196a5f8588be4967e535c4cfd35e85902 -- --flake .#srv10 --target-host root@<ip address>

Back to the build

I continued to try to print out more debugging information during the boot by getting the mdadm-assemble job running during the start-up.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
...
dev-md-raid1.device: Job 35 dev-md-raid1.device/start finished, result=timeout  

systemd-cryptsetup@cryptroot.service: Job 33 systemd-cryptsetup@cryptroot.service/start finished, result=dependency  
cryptsetup.target: Job 32 cryptsetup.target/start finished, result=dependency  
dev-mapper-cryptroot.device: Job 53 dev-mapper-cryptroot.device/start finished, result=dependency  
initrd-root-device.target: Job 59 initrd-root-device.target/start finished, result=dependency  
initrd.target: Job 1 initrd.target/start finished, result=canceled  
sysinit.target: Job 3 sysinit.target/start finished, result=canceled  
mdadm-assemble.service: Job 43 mdadm-assemble.service/start finished, result=canceled  

But that never worked because it was getting stuck due to dev-md-raid1.device failing and it’s not possible to get stuck, but I found the boot logging over serial trick useful enough to share it in the post.

Matching kernel versions

Another GitHub comment mentioned Linux kernels needing matching, I then explored a bit more about that.

What kernel version is getting installed?

1
2
3
nix eval .#nixosConfigurations.srv10.pkgs.linuxPackages.kernel.version

"6.18.39"

What version is the nixos-anywhere installer using?

1
2
3
uname -a

Linux nixos-installer 7.0.9 #1-NixOS SMP PREEMPT_DYNAMIC Sun May 17 15:16:34 UTC 2026 x86_64 GNU/Linux

The NixOS installer kernel is newer than the one that boots. There could be something to this.

The Fix

I grabbed a matching version of the kexec from GitHub releases.

1
wget https://github.com/nix-community/nixos-images/releases/download/nixos-26.05/nixos-kexec-installer-x86_64-linux.tar.gz

Or compile one freshly:

1
2
3
4
nix run github:nix-community/nixos-anywhere -- \
  --kexec "$(nix build --print-out-paths github:nix-community/nixos-images#packages.x86_64-linux.kexec-installer-nixos-unstable-noninteractive)/nixos-kexec-installer-noninteractive-x86_x64-linux.tar.gz" \
  --flake '#.srv10' \
  root@srv10

Let’s see what happens after an install.

1
nix run 'github:nix-community/nixos-anywhere' -- --flake .#srv10 --target-host root@158.69.225.68  --phases kexec --kexec nixos-kexec-installer-x86_64-linux.tar.gz

Looking at the boot log, I get:

1
systemd-ask-password-console.path: Got triggered by '/run/systemd/ask-password'.

Sounds promising. Is that a working boot I hear?

1
2
ssh root@srv10 -p 2231
🔐 Please enter passphrase for disk cryptroot: 

Yes it is success!

My server boots and I’m able to proceed

What changed in the kernel?

Can we find the actual bug? This sounds like a job for AI. I cloned the repository:

1
git clone https://github.com/torvalds/linux.git

Then opened up OpenCode with the Kimi K2.5 model and asked it to review commits between 6.18.38 and 7.0.3 related to superblock changes. It looked at all the commits between those versions, looked for commits related to the superblock, and reviewed the changes.

<ai> The Problem: Logical Block Size (LBS) Superblock Change Commits Involved:

commit 62ed1b582246 (Nov 2025): “md: allow configuring logical block size”

Added a new logical_block_size field to the mdp_superblock_1 structure
This field was carved out of the previously reserved pad3[64-32] padding area
Changed the superblock layout for 1.x metadata arrays

commit 9c47127a807d (Nov 2025): “md: add check_new_feature module parameter”

Added validation that rejects arrays with non-zero padding in super_1_load()
When an older kernel (6.18.x) sees the new logical_block_size field (which was previously padding), it detects non-zero padding and rejects the array

commit a4166f1c4893 (Dec 2025): “md: Fix forward incompatibility from configurable logical block size”

The fix: Only sets logical_block_size for newly created arrays
For existing arrays from old kernels, the LBS field remains 0 in the superblock

When kernel 7.0.9 creates a new RAID array:

It writes the logical_block_size value to the superblock (in what used to be padding)
This makes the superblock incompatible with older kernels (6.18.x)
When 6.18.38 tries to assemble the array, it sees non-zero padding and fails </ai>

Looking at these commits, they’re not really bugs, but changes that are intended to improve reliability. They are forwards incompatible, meaning the older version of the kernel can’t read them.

Conclusion

This bug was difficult and very frustrating to track down. Reinstalling and testing each attempt took a lot of time. I started with the mysterious error: nvme0n1p2 does not have a valid v1.2 superblock, not importing!

After weeks of different attempts to fix it, I finally discovered there was a change in Linux 6.19.x to the MD array superblock (metadata stored in the RAID array) that added more information that Linux 6.18.x kernels couldn’t decode. This was intended to make RAID arrays more reliable.

Since the nixos-anywhere installer used 7.0.x, but nixpkgs currentlyed used 6.18.x, it would fail during booting. The OVH emergency distro also was too old to decode it.

The fix was to explicitly download a matching kexec installer binary from nixos-images and pass that with --kexec to the kexec phase.

Through this, I also learned how to debug kernel booting using the serial over SSH. While it didn’t provide a lot of insight, it still provides a way to get debug info out.

My server has since been added to my Kubernetes cluster and I’m also entirely on faster hosts with encrypted disks.

References

https://github.com/nix-community/disko/issues/1276#issuecomment-4906524888

AI Disclosure

Some LLMs were used in the research of this problem, however outside of the snippet clearly stated above, LLMs provided no other content.

Comments

To give feedback, send an email to adam [at] this website url.

Donate

If you've found these posts helpful and would like to support this work directly, your contribution would be appreciated and enable me to dedicate more time to creating future posts. Thank you for joining me!