csi-s3-archiver on k0s

Two ways round the same path#

k0s runs kubelet with /var/lib/k0s/kubelet as its root directory rather than the /var/lib/kubelet that every CSI driver's manifests assume. You can move the driver or move k0s, and unlike most either-or choices this one has a clear answer if you have more than one CSI driver to install.

# Either point the driver at k0s...
$ helm install csi-s3-archiver deploy/helm/csi-s3-archiver \
    -n csi-s3-archiver --create-namespace \
    --set kubeletDir=/var/lib/k0s/kubelet

# ...or point k0s at the path every CSI driver's manifests already assume, once, per worker:
$ k0s install worker --token-file /var/lib/k0s/join-token \
    --kubelet-root-dir=/var/lib/kubelet

$ kubectl get csinode -o jsonpath='{.items[*].spec.drivers[*].name}'
s3archiver.csi.trion.de

Moving an existing cluster#

Changing a running cluster's kubelet root directory is not a live migration. Kubelet's pod directories move with it, so drain the node first and let the workloads be recreated on the new path. That is another reason to make the decision when the cluster is built rather than after it has volumes in flight.

Other distributions#

The install page has the overview. The rest of the awkward ones:

• vcluster. Install once on the host cluster. Volumes work from inside untouched, but every prefix placeholder resolves to the synced pod's host identity.
• Talos Linux. The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed.
• OpenShift. hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container.
• MicroK8s. Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet.
• GKE Autopilot and EKS Fargate. Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine.