csi-s3-archiver on vcluster

Install it on the host cluster#

A virtual cluster has no nodes of its own. Its pods are synced out to the host cluster and run on the host's kubelet, which is the kubelet this driver talks to. So the driver is installed in the host cluster, once, and every vcluster on it gets archived volumes for free. Nothing is installed into the virtual cluster and nothing needs to be configured there.

Both ways of declaring a volume then work from inside a vcluster with no changes, an inline csi: block and a volumeClaimTemplate against the optional Controller's StorageClass alike. What changes is whose identity the driver is told about, and that changes the shape of your archive rather than whether it works.

Do not install the driver inside the vcluster#

It is the obvious first thing to try and it does not work, for a reason worth knowing. vcluster rewrites a synced pod's hostPath for kubelet's pods directory into a sandbox of its own, so the DaemonSet never starts.

$ kubectl -n csi-s3-archiver get pods            # inside the vcluster
NAME                    READY   STATUS              RESTARTS   AGE
csi-s3-archiver-w47rl   0/3     ContainerCreating   0          98s

Warning  FailedMount  34s (x8 over 97s)  kubelet  MountVolume.SetUp failed for volume
"kubelet-pods-dir" : hostPath type check failed:
/tmp/vcluster/vc-v1/v1/kubelet/pods is not a directory

The prefix placeholders resolve to the host's identity#

Pod metadata reaches the driver from kubelet, and kubelet only knows the pod it is actually running, which is the synced one. For a pod named probe in the virtual namespace default, in a vcluster named v1 living in the host namespace vc-v1:

The consequence is a layout, not a failure. Every tenant of a vcluster archives under the same {namespace} segment, and it is {podName} that carries the tenant apart, mangled but present, so nothing collides and nothing is overwritten:

# Two namespaces inside one vcluster, both running a pod called "dumper", both with
#   prefix: "tenants/{namespace}/{podName}/"
$ aws s3 ls --recursive s3://lab/tenants/
tenants/vc-v1/dumper-x-team-a-x-v1/who.txt
tenants/vc-v1/dumper-x-team-b-x-v1/who.txt
#        ^^^^^ the vcluster's namespace on the host, for every tenant
#              ^^^^^^^^^^^^^^^^^^^^^ pod, virtual namespace and vcluster, mangled together

# The same two pods on a plain cluster:
tenants/team-a/dumper/who.txt
tenants/team-b/dumper/who.txt

So build the prefix on {podName} rather than {namespace} when the cluster is virtual, and treat {serviceAccount} as unusable there, because it is one value for every pod in the vcluster. Better, give each vcluster its own bucket or a fixed prefix of its own in the DaemonSet's environment, and let the placeholders sort out what is inside it.

Let the tenants see the driver#

By default a vcluster shows no StorageClass, no CSIDriver and no CSINode, while volumes using them work perfectly. A volumeClaimTemplate naming a class that kubectl get sc says does not exist still binds, because vcluster syncs the claim out to the host and the host's provisioner fulfils it. That is a confusing place to leave whoever has to use the cluster:

# vcluster.yaml -- volumes already work without this. It only makes the driver visible to
# whoever is working inside the virtual cluster, so `kubectl get sc` answers instead of
# printing "No resources found" while a volumeClaimTemplate naming that very class binds fine.
sync:
  fromHost:
    storageClasses:
      enabled: true
    csiDrivers:
      enabled: true
    csiNodes:
      enabled: true

Other distributions#

The install page has the overview. The rest of the awkward ones:

• Talos Linux. The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed.
• OpenShift. hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container.
• MicroK8s. Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet.
• k0s. Point the driver at the k0s path, or point k0s back at the standard one and never think about it again.
• GKE Autopilot and EKS Fargate. Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine.