Last week, I upgraded my home server from Ubuntu 24.04 LTS (Long Term Support) to Ubuntu 26.04 LTS. I expected smooth sailing, but you and I should know not to expect that, especially if there’s a blog post coming from it.

As soon as I started my Kubernetes cluster, services were failing to start left and right. Cloudnative-pg cluster pods were stuck crashing and restarting forever. Initially, I thought it was some kind of bug in Longhorn (the distributed storage I use for Kubernetes) because I was getting errors when reading/writing data saying “Remote I/O Error”, but the Postgres error was far more impactful.

The postgres pod, managed by cloudnative-pg, was running, but would give the following unhelpful logs. The logs effectively showed this:

1
2
3
4
Instance is still down, will retry in 1 second
Defaulting for Cluster
Instance is still down, will retry in 1 second
Defaulting for Cluster

These logs were difficult to search the web for a root cause. So I kept investigating other options.

Manual fix

Due to messages elsewhere in the logs suggesting data corruption, I assumed cloudnative-pg was aggressively restarting the pod before it could recover. I manually ran Postgres to let it recover. Using nerdctl, a command-line interface for containerd similar to Docker’s, I ran:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
sudo mount /dev/longhorn/pvc-b9ddb5fa-eb02-45fc-a3de-15958dc96ba0 test  
sudo ~/nerdctl run --net=host -ti -v ~/test/pgdata:/var/lib/postgresql/data/pgdata --rm ghcr.io/cloudnative-pg/postgresql:16.2 /bin/bash

cd ~/data/pgdata

cat custom.conf | grep -v ssl | grep -v log > custom.conf.new
mv custom.conf.new custom.conf

postgres -D .

sudo umount test

Then I started the Kubernetes Pod, and it was able to run and serve traffic for some time. I couldn’t explain why at the time—a mystery that the AppArmor investigation later in this post would help clarify.

Stuck in Postgres Auth

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
  "level": "info",
  "ts": "2026-08-20T05:38:03.093412449Z",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "airtrail-1",
  "record": {
    "log_time": "2026-08-20 05:38:03.093 UTC",
    "user_name": "postgres",
    "database_name": "postgres",
    "process_id": "1602",
    "connection_from": "[local]",
    "session_id": "6a8692bb.642",
    "session_line_num": "1",
    "command_tag": "initializing",
    "session_start_time": "2026-08-20 05:38:03 UTC",
    "transaction_id": "0",
    "error_severity": "FATAL",
    "sql_state_code": "53300",
    "message": "sorry, too many clients already",
    "backend_type": "client backend",
    "query_id": "0"
  }
}

dmesg

Time to look elsewhere. Dmesg was a good tool to check next. It captures the kernel-level logs and gave me a number of AppArmor failures.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
sudo dmesg

kauditd_printk_skb: 12 callbacks suppressed
audit:
  type=1400 audit(1787202456.512:58246):
  apparmor="AUDIT"
  operation="change_onexec"
  class="file"
  info="change_profile unprivileged unconfined converted to stacking"
  profile="unconfined"
  name="cri-containerd.apparmor.d"
  pid=854680
  comm="runc:[2:INIT]"
audit:
  type=1400 audit(1787202459.448:58247):
  apparmor="DENIED"
  operation="signal"
  class="signal"
  profile="cri-containerd.apparmor.d"
  pid=832109
  comm="postgres"
  requested_mask="send"
  denied_mask="send"
  signal=urg
  peer="cri-containerd.apparmor.d//&unconfined"
# repeated multiple times

These logs indicate that AppArmor is blocking PostgreSQL from sending internal inter-process signals (SIGUSR1 and SIGURG) to other PostgreSQL processes running inside the same containerd-managed container. Because PostgreSQL relies heavily on signals for internal communication, this blockage can cause database locks, replication issues, or checkpoints to hang.

What this error message shows is that process ID (PID) 247075, which is a postgres process, tried to signal another postgres process and was denied.

What’s going on with AppArmor

Passing the Z argument to ps with ps afxuZ, gives us each PID’s AppArmor loaded profile:

1
2
3
4
5
6
7
ps afxuZ | grep postgres | grep airtrail  

cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608079 0.0  0.0 68680 5672 ? Ss 20:42   0:00          \_ postgres: airtrail: logger    
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608080 0.0  0.0 223968 6132 ? Ss 20:42   0:00          \_ postgres: airtrail: io worker 0  
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608081 0.0  0.0 223968 6188 ? Ss 20:42   0:00          \_ postgres: airtrail: io worker 1  
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608082 0.0  0.0 223968 6124 ? Ss 20:42   0:00          \_ postgres: airtrail: io worker 2  
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608083 0.0  0.0 223968 618 ...

Each postgres process is running in the profile cri-containerd.apparmor.d//&unconfined. The //& signifies it’s a stacked profile. The process started out as cri-containerd.apparmor.d, then tried to transition to unconfined. The error shows that Postgres is trying to send a signal to another process. Even though it tried to transition to unconfined, AppArmor still enforces the AppArmor policy for signals from cri-containerd.apparmor.d instead of unconfined.

Looking at the original error, the source process should be cri-containerd.apparmor.d//&unconfined and the destination process is also a cri-containerd.apparmor.d//&unconfined process. According to the container policy from v2.2.1 (which is the latest in Ubuntu 24.04) and in v2.2.2 (latest in Ubuntu 26.04), they both should have the ability to send and receive from the same container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # runc may send signals to container processes.
  signal (receive) peer=runc,
  # crun may send signals to container processes.
  signal (receive) peer=crun,
  # Manager may send signals to container processes.
  signal (receive) peer={{.DaemonProfile}},
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer={{.Name}},
{{if .RootlessKit}}
  # https://github.com/containerd/nerdctl/issues/2730
  signal (receive) peer={{.RootlessKit}},
{{end}}

At this point, I couldn’t reconcile two issues: why did this change break after upgrading to Ubuntu 26.04, and why does the source profile show cri-containerd.apparmor.d instead of cri-containerd.apparmor.d//&unconfined?

At this time, I had filed an issue against cloudnative-pg thinking it was limited to Postgres. Their doc bot gave me some suggestions including disabling AppArmor, which I did not want to do globally.

Digging into the Kernel

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
apparmor="DENIED"
operation="signal"
class="signal"
profile="cri-containerd.apparmor.d"
pid=247075
comm="postgres"
requested_mask="send"
denied_mask="send"
signal=urg
peer="cri-containerd.apparmor.d//&unconfined"

First, I want to know why the error message source profile showed cri-containerd.apparmor.d instead of cri-containerd.apparmor.d//&unconfined. This would tell me whether the processes are genuinely in different AppArmor profiles or if this is just an artifact of the error log entry.

GLM-5.2 claimed that I had stumbled upon a “notoriously confusing quirk in how AppArmor logs audit entries”. That seemed suspect, so I checked out the apparmor source code and the Linux kernel and asked it to confirm whether that’s true or not based on the source code.

Which then lead me to the claim being probable:

Subject processes (what your postgres process is):

Use profile->base.hname which is a simple name
No profile stack information logged
Shows: cri-containerd.apparmor.d Target/peer processes:
Use full stacked profile formatting with aa_label_snxprint()
Includes “//&” separator for each stack layer
Shows: cri-containerd.apparmor.d//&unconfined (full stack) Key code paths:

1
2
3
4
5
6
7
audit_pre() in security/apparmor/audit.c:116-134:
audit_log_format(ab, " profile=");
audit_log_untrustedstring(ab, profile->base.hname);  // Simple name only
aa_label_xaudit() in security/apparmor/label.c:1724-1752:
len = aa_label_asxprint(&name, ns, label, flags, gfp);  // Full stack when needed
// Or
str = (char *) label->hname;  // Or just hname if it's enough

Why did this break in Ubuntu 26.04

I had a few theories on what could have changed: containerd, Linux kernel, K3s (Rancher’s lightweight Kubernetes distribution), or cloudnative-pg. I had not upgraded anything in Kubernetes, so that left containerd and the Linux kernel.

Looking at the containerd container policy file diff between 2.2.1 and 2.2.2, I see no material changes to the profile that could affect it. Nothing was added or removed in the policy.

Investigating kernel changes

Could it be yet another Linux kernel change? To answer this question, I tasked GLM-4.7-Flash to look through all relevant-looking commits and try to find a likely change.

It claimed commit 9afdc6abb looked suspect as it was changing from a list of rules to a vector rules. The commit didn’t appear obviously broken at first glance. Changing types could cause problems, but it seems to conflict with Linus Torvalds’ famous rant about not breaking user space if that is the cause.

The LLM suggested that the containerd policy was actually wrong and should include

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # runc may send signals to container processes.
  signal (receive) peer=runc,
  # crun may send signals to container processes.
  signal (receive) peer=crun,
  # Manager may send signals to container processes.
  signal (receive) peer={{.DaemonProfile}},
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer={{.Name}},
+ # Allow signals to containers with stacked profiles.
+ signal (send,receive) peer={{.Name}}//&*,
 {{if .RootlessKit}}
   # https://github.com/containerd/nerdctl/issues/2730
   signal (receive) peer={{.RootlessKit}},
 {{end}}

Then I discovered this containerd issue which aligned with the proposed fix above. I missed this issue in my initial issue investigation.

Workarounds

Disable AppArmor for all containerd

I’m not happy about this suggestion, but for clusters where the impact is wider than just a few containers, it might be better to just disable AppArmor entirely.

In my case, I’m running Kubernetes through Rancher-deployed k3s, which means this document applies. The document suggests extending the config using {{ template "base" . }}, but that resulted in a TOML (Tom’s Obvious, Minimal Language) validation error for me, so I ended up with this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
apiVersion: v1
data:
  containerd-config: |
     version = 3

    root = "/var/lib/rancher/k3s/agent/containerd"

    state = "/run/k3s/containerd"


    [grpc]
      address = "/run/k3s/containerd/containerd.sock"

    [plugins.'io.containerd.internal.v1.opt']
      path = "/var/lib/rancher/k3s/agent/containerd"

    [plugins.'io.containerd.grpc.v1.cri']
      stream_server_address = "127.0.0.1"
      stream_server_port = "10010"

    [plugins.'io.containerd.cri.v1.runtime']
      enable_selinux = false
      enable_unprivileged_ports = true
      enable_unprivileged_icmp = true
      device_ownership_from_security_context = false
      disable_apparmor = true

    [plugins.'io.containerd.cri.v1.images']
      snapshotter = "overlayfs"
      disable_snapshot_annotations = true
      use_local_image_pull = true

    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = "rancher/mirrored-pause:3.6"

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
      SystemdCgroup = true

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
      runtime_type = "io.containerd.runhcs.v1"

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia']
      runtime_type = "io.containerd.runc.v2"

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia'.options]
      BinaryName = "/usr/bin/nvidia-container-runtime"
      SystemdCgroup = true

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"
kind: ConfigMap
metadata:
  name: cluster-config
  namespace: fleet-default

Then my cluster object gets updated to deploy the file to each node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
  name: adamsnet
  namespace: fleet-default
spec:
  rkeConfig:
    machineSelectorFiles:
      - fileSources:
          - configMap:
              items:
                - key: containerd-config
                  path: >-
                    /var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl
              name: cluster-config
            secret:
              name: ''
        machineLabelSelector:
          matchLabels:
            rke.cattle.io/control-plane-role: 'true'

Only for Postgres

Since my impact was predominantly limited to Postgres instances, instead of globally disabling, I updated the cluster resource to disable AppArmor on just those containers

1
2
3
4
5
6
7
8
9
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/initdb: unconfined
    container.apparmor.security.beta.kubernetes.io/postgres: unconfined
spec:
  seccompProfile:
    type: Unconfined

Remote I/O Error

With Postgres sorted out, a separate, unrelated issue from the upgrade remained: some Kubernetes pods were reporting an error Remote I/O Error when reading or writing to the persistent volumes mounted into the application. Only some containers were having this issue, but it was very noticeable when it did happen since the application would break and CPU/Memory usage would climb significantly:

A screenshot from Grafana showing CPU of the Longhorn manager node growing to spikes of 2 seconds of CPU time, consuming 2 cores.

A screenshot from Grafana showing RAM usage of the longhorn share-manager pod growing towards 40GB

This issue appeared after I upgraded Longhorn to 1.11.x in the hopes that it would fix my Postgres issue, but no such luck.

A GitHub issue was already opened.

I was able to workaround the issue by restoring my volumes from backup and switching from ReadWriteMany to ReadWriteOnce. ReadWriteMany was a more complicated data path and apparently had some bugs in it.

Conclusion

I didn’t expect that upgrading from one LTS to an LTS would cause my Postgres databases to fail to start, but Ubuntu 26.04 brought in a new kernel version that surfaced a latent incompatibility between containerd’s AppArmor template and AppArmor’s stacking behavior.

The hardest part was diagnosis: random remote I/O errors from some Kubernetes services, Kubernetes having both seccomp profiles and AppArmor profiles hidden as annotations, and AppArmor errors that were misdirecting.

I didn’t discover any new Linux or Longhorn bugs. They were already filed, but in this post I showed my diagnostic strategy to investigate the issue and confirm the bug was the same.

I truly seem to be cursed with computers to find weird bugs.

Comments

To give feedback, send an email to adam [at] this website url.

Donate

If you've found these posts helpful and would like to support this work directly, your contribution would be appreciated and enable me to dedicate more time to creating future posts. Thank you for joining me!