Last week, I upgraded my home server from Ubuntu 24.04 LTS (Long Term Support) to Ubuntu 26.04 LTS. I expected smooth sailing, but you and I should know not to expect that, especially if there’s a blog post coming from it.
As soon as I started my Kubernetes cluster, services were failing to start left and right. Cloudnative-pg cluster pods were stuck crashing and restarting forever. Initially, I thought it was some kind of bug in Longhorn (the distributed storage I use for Kubernetes) because I was getting errors when reading/writing data saying “Remote I/O Error”, but the Postgres error was far more impactful.
The postgres pod, managed by cloudnative-pg, was running, but would give the following unhelpful logs. The logs effectively showed this:
1
2
3
4
| Instance is still down, will retry in 1 second
Defaulting for Cluster
Instance is still down, will retry in 1 second
Defaulting for Cluster
|
These logs were difficult to search the web for a root cause. So I kept investigating other options.
Manual fix
Due to messages elsewhere in the logs suggesting data corruption, I assumed cloudnative-pg was aggressively restarting the pod before it could recover. I manually ran Postgres to let it recover. Using nerdctl, a command-line interface for containerd similar to Docker’s, I ran:
1
2
3
4
5
6
7
8
9
10
11
| sudo mount /dev/longhorn/pvc-b9ddb5fa-eb02-45fc-a3de-15958dc96ba0 test
sudo ~/nerdctl run --net=host -ti -v ~/test/pgdata:/var/lib/postgresql/data/pgdata --rm ghcr.io/cloudnative-pg/postgresql:16.2 /bin/bash
cd ~/data/pgdata
cat custom.conf | grep -v ssl | grep -v log > custom.conf.new
mv custom.conf.new custom.conf
postgres -D .
sudo umount test
|
Then I started the Kubernetes Pod, and it was able to run and serve traffic for some time. I couldn’t explain why at the time—a mystery that the AppArmor investigation later in this post would help clarify.
Stuck in Postgres Auth
1
| {"level":"info","ts":"2026-08-20T05:38:03.093412449Z","logger":"postgres","msg":"record","logging_pod":"airtrail-1","record":{"log_time":"2026-08-20 05:38:03.093 UTC","user_name":"postgres","database_name":"postgres","process_id":"1602","connection_from":"[local]","session_id":"6a8692bb.642","session_line_num":"1","command_tag":"initializing","session_start_time":"2026-08-20 05:38:03 UTC","transaction_id":"0","error_severity":"FATAL","sql_state_code":"53300","message":"sorry, too many clients already","backend_type":"client backend","query_id":"0"}}
|
dmesg
Time to look elsewhere. Dmesg was a good tool to check next. It captures the kernel-level logs and gave me a number of AppArmor failures.
1
2
3
4
5
6
7
8
9
10
| sudo dmesg
[15417.064189] kauditd_printk_skb: 12 callbacks suppressed
[15417.064192] audit: type=1400 audit(1787202456.512:58246): apparmor="AUDIT" operation="change_onexec" class="file" info="change_profile unprivileged unconfined converted to stacking" profile="unconfined" name="cri-containerd.apparmor.d" pid=854680 comm="runc:[2:INIT]"
[15419.999947] audit: type=1400 audit(1787202459.448:58247): apparmor="DENIED" operation="signal" class="signal" profile="cri-containerd.apparmor.d" pid=832109 comm="postgres" requested_mask="send" denied_mask="send" signal=urg peer="cri-containerd.apparmor.d//&unconfined"
[15422.581671] audit: type=1400 audit(1787202462.030:58248): apparmor="DENIED" operation="signal" class="signal" profile="cri-containerd.apparmor.d" pid=247075 comm="postgres" requested_mask="send" denied_mask="send" signal=urg peer="cri-containerd.apparmor.d//&unconfined"
[15423.015197] audit: type=1400 audit(1787202462.463:58249): apparmor="DENIED" operation="signal" class="signal" profile="cri-containerd.apparmor.d" pid=234403 comm="postgres" requested_mask="send" denied_mask="send" signal=urg peer="cri-containerd.apparmor.d//&unconfined"
[15424.404666] audit: type=1400 audit(1787202463.853:58250): apparmor="DENIED" operation="signal" class="signal" profile="cri-containerd.apparmor.d" pid=253246 comm="postgres" requested_mask="send" denied_mask="send" signal=urg peer="cri-containerd.apparmor.d//&unconfined"
[15425.609599] audit: type=1400 audit(1787202465.058:58251): apparmor="DENIED" operation="signal" class="signal" profile="cri-containerd.apparmor.d" pid=247075 comm="postgres" requested_mask="send" denied_mask="send" signal=urg peer="cri-containerd.apparmor.d//&unconfined"
[15425.866460] audit: type=1400 audit(1787202465.315:58252): apparmor="DENIED" operation="signal" class="signal" profile="cri-containerd.apparmor.d" pid=247075 comm="postgres" requested_mask="send" denied_mask="send" signal=urg peer="cri-containerd.apparmor.d//&unconfined"
|
These logs indicate that AppArmor is blocking PostgreSQL from sending internal inter-process signals (SIGUSR1 and SIGURG) to other PostgreSQL processes running inside the same containerd-managed container. Because PostgreSQL relies heavily on signals for internal communication, this blockage can cause database locks, replication issues, or checkpoints to hang.
What this error message shows is that process ID (PID) 247075, which is a postgres process, tried to signal another postgres process and was denied.
What’s going on with AppArmor
Passing the Z argument to ps with ps afxuZ, gives us each PID’s AppArmor loaded profile:
1
2
3
4
5
6
7
| ps afxuZ | grep postgres | grep airtrail
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608079 0.0 0.0 68680 5672 ? Ss 20:42 0:00 \_ postgres: airtrail: logger
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608080 0.0 0.0 223968 6132 ? Ss 20:42 0:00 \_ postgres: airtrail: io worker 0
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608081 0.0 0.0 223968 6188 ? Ss 20:42 0:00 \_ postgres: airtrail: io worker 1
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608082 0.0 0.0 223968 6124 ? Ss 20:42 0:00 \_ postgres: airtrail: io worker 2
cri-containerd.apparmor.d//&unconfined (enforce) 2769747+ 608083 0.0 0.0 223968 618 ...
|
Each postgres process is running in the profile cri-containerd.apparmor.d//&unconfined. The //& signifies it’s a stacked profile. The process started out as cri-containerd.apparmor.d, then tried to transition to unconfined. The error shows that Postgres is trying to send a signal to another process. Even though it tried to transition to unconfined, AppArmor still enforces the AppArmor policy for signals from cri-containerd.apparmor.d instead of unconfined.
Looking at the original error, the source process should be cri-containerd.apparmor.d//&unconfined and the destination process is also a cri-containerd.apparmor.d//&unconfined process. According to the container policy from v2.2.1 (which is the latest in Ubuntu 24.04) and in v2.2.2 (latest in Ubuntu 26.04), they both should have the ability to send and receive from the same container:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| # Host (privileged) processes may send signals to container processes.
signal (receive) peer=unconfined,
# runc may send signals to container processes.
signal (receive) peer=runc,
# crun may send signals to container processes.
signal (receive) peer=crun,
# Manager may send signals to container processes.
signal (receive) peer={{.DaemonProfile}},
# Container processes may send signals amongst themselves.
signal (send,receive) peer={{.Name}},
{{if .RootlessKit}}
# https://github.com/containerd/nerdctl/issues/2730
signal (receive) peer={{.RootlessKit}},
{{end}}
|
At this point, I couldn’t reconcile two issues: why did this change break after upgrading to Ubuntu 26.04, and why does the source profile show cri-containerd.apparmor.d instead of cri-containerd.apparmor.d//&unconfined?
At this time, I had filed an issue against cloudnative-pg thinking it was limited to Postgres. Their doc bot gave me some suggestions including disabling AppArmor, which I did not want to do globally.
Digging into the Kernel
1
| apparmor="DENIED" operation="signal" class="signal" profile="cri-containerd.apparmor.d" pid=247075 comm="postgres" requested_mask="send" denied_mask="send" signal=urg peer="cri-containerd.apparmor.d//&unconfined"
|
First, I want to know why the error message source profile showed cri-containerd.apparmor.d instead of cri-containerd.apparmor.d//&unconfined. This would tell me whether the processes are genuinely in different AppArmor profiles or if this is just an artifact of the error log entry.
GLM-5.2 claimed that I had stumbled upon a “notoriously confusing quirk in how AppArmor logs audit entries”. That seemed suspect, so I checked out the apparmor source code and the Linux kernel and asked it to confirm whether that’s true or not based on the source code.
Which then lead me to the claim being probable:
Subject processes (what your postgres process is):
- Use
profile->base.hname which is a simple name - No profile stack information logged
- Shows: cri-containerd.apparmor.d
Target/peer processes:
- Use full stacked profile formatting with aa_label_snxprint()
- Includes “//&” separator for each stack layer
- Shows:
cri-containerd.apparmor.d//&unconfined (full stack)
Key code paths:
1
2
3
4
5
6
7
| audit_pre() in security/apparmor/audit.c:116-134:
audit_log_format(ab, " profile=");
audit_log_untrustedstring(ab, profile->base.hname); // Simple name only
aa_label_xaudit() in security/apparmor/label.c:1724-1752:
len = aa_label_asxprint(&name, ns, label, flags, gfp); // Full stack when needed
// Or
str = (char *) label->hname; // Or just hname if it's enough
|
Why did this break in Ubuntu 26.04
I had a few theories on what could have changed: containerd, Linux kernel, K3s (Rancher’s lightweight Kubernetes distribution), or cloudnative-pg. I had not upgraded anything in Kubernetes, so that left containerd and the Linux kernel.
Looking at the containerd container policy file diff between 2.2.1 and 2.2.2, I see no material changes to the profile that could affect it. Nothing was added or removed in the policy.
Investigating kernel changes
Could it be yet another Linux kernel change? To answer this question, I tasked GLM-4.7-Flash to look through all relevant-looking commits and try to find a likely change.
It claimed commit 9afdc6abb looked suspect as it was changing from a list of rules to a vector rules. The commit didn’t appear obviously broken at first glance. Changing types could cause problems, but it seems to conflict with Linus Torvalds’ famous rant about not breaking user space if that is the cause.
The LLM suggested that the containerd policy was actually wrong and should include
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| # Host (privileged) processes may send signals to container processes.
signal (receive) peer=unconfined,
# runc may send signals to container processes.
signal (receive) peer=runc,
# crun may send signals to container processes.
signal (receive) peer=crun,
# Manager may send signals to container processes.
signal (receive) peer={{.DaemonProfile}},
# Container processes may send signals amongst themselves.
signal (send,receive) peer={{.Name}},
+ # Allow signals to containers with stacked profiles.
+ signal (send,receive) peer={{.Name}}//&*,
{{if .RootlessKit}}
# https://github.com/containerd/nerdctl/issues/2730
signal (receive) peer={{.RootlessKit}},
{{end}}
|
Then I discovered this containerd issue which aligned with the proposed fix above. I missed this issue in my initial issue investigation.
Workarounds
Disable AppArmor for all containerd
I’m not happy about this suggestion, but for clusters where the impact is wider than just a few containers, it might be better to just disable AppArmor entirely.
In my case, I’m running Kubernetes through Rancher-deployed k3s, which means this document applies. The document suggests extending the config using {{ template "base" . }}, but that resulted in a TOML (Tom’s Obvious, Minimal Language) validation error for me, so I ended up with this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
| apiVersion: v1
data:
containerd-config: |
version = 3
root = "/var/lib/rancher/k3s/agent/containerd"
state = "/run/k3s/containerd"
[grpc]
address = "/run/k3s/containerd/containerd.sock"
[plugins.'io.containerd.internal.v1.opt']
path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.'io.containerd.grpc.v1.cri']
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
[plugins.'io.containerd.cri.v1.runtime']
enable_selinux = false
enable_unprivileged_ports = true
enable_unprivileged_icmp = true
device_ownership_from_security_context = false
disable_apparmor = true
[plugins.'io.containerd.cri.v1.images']
snapshotter = "overlayfs"
disable_snapshot_annotations = true
use_local_image_pull = true
[plugins.'io.containerd.cri.v1.images'.pinned_images]
sandbox = "rancher/mirrored-pause:3.6"
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runhcs-wcow-process]
runtime_type = "io.containerd.runhcs.v1"
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia']
runtime_type = "io.containerd.runc.v2"
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia'.options]
BinaryName = "/usr/bin/nvidia-container-runtime"
SystemdCgroup = true
[plugins.'io.containerd.cri.v1.images'.registry]
config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"
kind: ConfigMap
metadata:
name: cluster-config
namespace: fleet-default
|
Then my cluster object gets updated to deploy the file to each node:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: adamsnet
namespace: fleet-default
spec:
rkeConfig:
machineSelectorFiles:
- fileSources:
- configMap:
items:
- key: containerd-config
path: >-
/var/lib/rancher/k3s/agent/etc/containerd/config-v3.toml.tmpl
name: cluster-config
secret:
name: ''
machineLabelSelector:
matchLabels:
rke.cattle.io/control-plane-role: 'true'
|
Only for Postgres
Since my impact was predominantly limited to Postgres instances, instead of globally disabling, I updated the cluster resource to disable AppArmor on just those containers
1
2
3
4
5
6
7
8
9
| apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
annotations:
container.apparmor.security.beta.kubernetes.io/initdb: unconfined
container.apparmor.security.beta.kubernetes.io/postgres: unconfined
spec:
seccompProfile:
type: Unconfined
|
Remote I/O Error
With Postgres sorted out, a separate, unrelated issue from the upgrade remained: some Kubernetes pods were reporting an error Remote I/O Error when reading or writing to the persistent volumes mounted into the application. Only some containers were having this issue, but it was very noticeable when it did happen since the application would break and CPU/Memory usage would climb significantly:
This issue appeared after I upgraded Longhorn to 1.11.x in the hopes that it would fix my Postgres issue, but no such luck.
A GitHub issue was already opened.
I was able to workaround the issue by restoring my volumes from backup and switching from ReadWriteMany to ReadWriteOnce. ReadWriteMany was a more complicated data path and apparently had some bugs in it.
Conclusion
I didn’t expect that upgrading from one LTS to an LTS would cause my Postgres databases to fail to start, but Ubuntu 26.04 brought in a new kernel version that surfaced a latent incompatibility between containerd’s AppArmor template and AppArmor’s stacking behavior.
The hardest part was diagnosis: random remote I/O errors from some Kubernetes services, Kubernetes having both seccomp profiles and AppArmor profiles hidden as annotations, and AppArmor errors that were misdirecting.
I didn’t discover any new Linux or Longhorn bugs. They were already filed, but in this post I showed my diagnostic strategy to investigate the issue and confirm the bug was the same.
I truly seem to be cursed with computers to find weird bugs.
