Long ago, I installed Longhorn onto my Kubernetes cluster using Helm 2. Then eventually Helm 3 was released and helm 2to3 was made available. However, I was not able to use helm 2to3 for whatever reason because Rancher didn’t deploy Tiller in the way that this CLI expected. Additionally, Rancher did not provide an upgrade mechanism to handle this. Eventually Rancher 2.6 was released which entirely dropped Helm 2 support and I was stuck with a cluster where Longhorn was deployed, but not managed by a working Helm installation.
This blog post outlines how you can recover Longhorn and upgrade it to Helm 3 without deleting all your volumes. This guide isn’t specific to Rancher 2.6.
Any time I tried to install Longhorn using Helm, I was getting this error in Rancher. This is telling me that Kubernetes resources already exist, but are supposedly being managed by a different Helm install.
Waiting for Kubernetes API to be available
helm upgrade –install=true –namespace=longhorn-system –timeout=10m0s –values=/home/shell/helm/values-longhorn-crd-100.1.0-up1.2.2.yaml –version=100.1.0+up1.2.2 –wait=true longhorn-crd /home/shell/helm/longhorn-crd-100.1.0-up1.2.2.tgz
Release “longhorn-crd” does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition “engines.longhorn.io” in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key “meta.helm.sh/release-name”: must be set to “longhorn-crd”; annotation validation error: missing key “meta.helm.sh/release-namespace”: must be set to “longhorn-system”
To fix this issue, we must update every single resource to include the following annotations and labels:
1
2
3
4
5
6
| spec:
annotations:
meta.helm.sh/release-name=longhorn-crd or longhorn
meta.helm.sh/release-namespace=longhorn-system
labels:
app.kubernetes.io/managed-by=Helm
|
First, configure kubectl to be able to connect back to your cluster, then run the following commands to update all resources:
Note KUBE_CONTEXT=prod. Make sure you change this to match the name of the Kubernetes cluster. In Rancher, by default it’s the name of the cluster.
The first step is to fix the CRDs (make sure to update your KUBE_CONTEXT to point to the correct cluster.)
1
2
3
4
5
| KUBE_CONTEXT=default
kubectl --context=$KUBE_CONTEXT get crds -o name | grep longhorn | xargs -I % kubectl --context=$KUBE_CONTEXT label --overwrite -n longhorn-system % app.kubernetes.io/managed-by=Helm
kubectl --context=$KUBE_CONTEXT get crds -o name | grep longhorn | xargs -I % kubectl --context=$KUBE_CONTEXT annotate -n longhorn-system % meta.helm.sh/release-name=longhorn-crd
kubectl --context=$KUBE_CONTEXT get crds -o name | grep longhorn | xargs -I % kubectl --context=$KUBE_CONTEXT annotate -n longhorn-system % meta.helm.sh/release-namespace=longhorn-system
|
After that you can fix the Longhorn application itself. If you’re upgrading from Longhorn 1.2.0 or earlier, you may already have the Longhorn Helm application deployed. If you do, then you should be able to skip the next step. If not because you’re coming from Helm v2 (like I was) then run the following:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| KUBE_CONTEXT=default
kubectl --context=$KUBE_CONTEXT get psp longhorn-psp -o name | xargs -I % kubectl --context=$KUBE_CONTEXT annotate % meta.helm.sh/release-name=longhorn
kubectl --context=$KUBE_CONTEXT get psp longhorn-psp -o name | xargs -I % kubectl --context=$KUBE_CONTEXT annotate % meta.helm.sh/release-namespace=longhorn-system
kubectl --context=$KUBE_CONTEXT get psp longhorn-psp -o name | xargs -I % kubectl --context=$KUBE_CONTEXT label --overwrite -n longhorn-system % app.kubernetes.io/managed-by=Helm
kubectl --context=$KUBE_CONTEXT -n longhorn-system get configMap,service,ds,deploy,serviceaccount,role,rolebinding -o name | xargs -I % kubectl --context=$KUBE_CONTEXT label --overwrite -n longhorn-system % app.kubernetes.io/managed-by=Helm
kubectl --context=$KUBE_CONTEXT -n longhorn-system get configMap,service,ds,deploy,serviceaccount,role,rolebinding -o name | xargs -I % kubectl --context=$KUBE_CONTEXT -n longhorn-system annotate % meta.helm.sh/release-name=longhorn
kubectl --context=$KUBE_CONTEXT -n longhorn-system get configMap,service,ds,deploy,serviceaccount,role,rolebinding -o name | xargs -I % kubectl --context=$KUBE_CONTEXT -n longhorn-system annotate % meta.helm.sh/release-namespace=longhorn-system
kubectl --context=$KUBE_CONTEXT get clusterrole,clusterrolebinding -o name | grep longhorn | xargs -I % kubectl --context=$KUBE_CONTEXT label --overwrite -n longhorn-system % app.kubernetes.io/managed-by=Helm
kubectl --context=$KUBE_CONTEXT get clusterrole,clusterrolebinding -o name | grep longhorn | xargs -I % kubectl --context=$KUBE_CONTEXT annotate % meta.helm.sh/release-namespace=longhorn-system
kubectl --context=$KUBE_CONTEXT get clusterrole,clusterrolebinding -o name | grep longhorn | xargs -I % kubectl --context=$KUBE_CONTEXT annotate % meta.helm.sh/release-name=longhorn
|
After fixing all of these issues, use the Rancher UI to deploy the Longhorn application. Search for Longhorn in the Rancher repository.
Then click Install
Make any changes to the install project or settings, then click Install.
If you’re lucky, everything should successfully deploy and you should now have Longhorn managed by Helm again.
1
2
3
| ---------------------------------------------------------------------
SUCCESS: helm upgrade --history-max=5 --install=true --namespace=longhorn-system --timeout=10m0s --values=/home/shell/helm/values-longhorn-100.1.0-up1.2.2.yaml --version=100.1.0+up1.2.2 --wait=true longhorn /home/shell/helm/longhorn-100.1.0-up1.2.2.tgz
---------------------------------------------------------------------
|
Comments
Comments are currently unavailable while I move to this new blog platform. To give feedback, send an email to adam [at] this website url.
