csi-s3-archiver
Install

Architecture

How it is put together

One binary, two deployment shapes, and about seventy classes with no framework between them. What runs where, and which class is responsible for what.

One binary, two deployments#

The driver is a single native binary with a few subcommands. Everything below is that binary running in a different mode, or not running at all: The base install has no controller, no operator, no CRDs and no webhook.

DeploymentWhat runsWhen you need it
DaemonSet (the base install)One pod per node, three containers, csi-s3-archiver serveAlways. This is the whole product for inline csi: volumes.
Controller (optional)One Deployment, csi-s3-archiver controller beside the upstream provisionerOnly if your workloads declare volumes with a volumeClaimTemplate rather than inline.

The DaemonSet#

The DaemonSet deployment On every node, a three-container DaemonSet pod runs the driver, the node-driver-registrar and the livenessprobe. kubelet calls NodePublishVolume over a Unix socket; the driver creates a plain directory under the kubelet pods directory and watches it. The workload pod writes files there, and the driver uploads them to the S3 bucket over HTTPS. The only cluster-scoped object is the CSIDriver. Nothing in the driver talks to the Kubernetes API server. kubernetes node workload pod writes /dumps volume dir a plain directory kubelet publish · unpublish NodePublishVolume over /csi/csi.sock csi-s3-archiver DaemonSet pod csi-s3-archiver watch · quiesce · upload · sweep node-driver-registrar registers the plugin livenessprobe /healthz on 9808 hostPath plugins/ · plugins_registry/ · pods/ creates · watches CSIDriver object podInfoOnMount · Ephemeral S3 bucket PutObject · multipart retry · concurrency cap AWS · Ceph · SeaweedFS any S3 API HTTPS No Kubernetes API access anywhere in the driver, because kubelet pushes pod metadata and the volume's Secret into the call. No mount(2), no FUSE, no privileged container. The volume is a directory the node plugin creates and removes.

The base install. Two of the three containers are upstream sidecars; the third is this driver.

What kubelet asks for and what it gets back is the whole contract. A CSI driver must make the volume readable and writable at target_path by the time NodePublishVolume returns, and a directory satisfies that as completely as a mount does. Serving one instead lets the DaemonSet drop privileged: true and bidirectional mount propagation. Everything else on this page follows from that single decision.

ObjectScopeWhat it is for
CSIDriverclusterattachRequired: false (there is nothing to attach), podInfoOnMount: true (which is how the driver learns the namespace, pod name and service account without an API call), volumeLifecycleModes: [Ephemeral], and requiresRepublish: true so a rotated Secret reaches running pods.
DaemonSetnamespaceThe driver plus node-driver-registrar, which tells kubelet where the socket is, and livenessprobe, which turns the CSI Probe RPC into a container probe on port 9808.
ServiceAccountnamespaceWith no Role, no RoleBinding and no ClusterRole. It exists because a pod needs one, not because the driver uses it.
NamespaceclusterSomewhere to put the two above.

Three host paths are mounted, and no others. /var/lib/kubelet/plugins/s3archiver.csi.trion.de holds the socket and the driver's state directory, /var/lib/kubelet/plugins_registry is where kubelet expects plugins to register, and /var/lib/kubelet/pods is where the volume directories themselves live. A distribution that moves kubelet's root needs those three paths changed and nothing else; the distribution pages name the handful that move it.

With the Controller#

The Controller variant, for PVC-declared volumes An optional Deployment in the control plane runs the same binary in controller mode beside the upstream csi-provisioner sidecar. A PVC from a volumeClaimTemplate, and the StorageClass naming the driver, reach the provisioner, which calls CreateVolume over a Unix socket. Nothing on the node changes, and kubelet still publishes the volume as a directory that the DaemonSet archives to S3. The RBAC belongs to the provisioner sidecar, not to the driver. cluster (control plane) PVC volumeClaimTemplate StorageClass pod-lifetime csi-s3-archiver Deployment csi-provisioner watches PVCs · holds the RBAC over /csi/csi.sock csi-s3-archiver controller CreateVolume · DeleteVolume kubernetes node the DaemonSet, unchanged NodePublishVolume directory · watch · upload final sweep at unpublish S3 bucket same objects as before The driver still holds no RBAC. The ClusterRole and RoleBinding belong to the csi-provisioner sidecar. Pod-lifetime by default. With durable set to true the contents are restored from S3 at publish, so a second pod sees them.

The optional overlay, for teams whose tooling is built around PVCs. The node side is untouched.

An inline csi: volume needs no controller at all, because kubelet has everything it needs in the pod spec. A PVC does, because something has to answer CreateVolume before the scheduler will bind the claim. That something is the same binary in controller mode, beside the upstream csi-provisioner.

Three things to know before you turn it on.

Worth knowingDetail
The volume is still pod-lifetimeUnless you set durable. The directory is created at publish, archived at unpublish and removed. A generic ephemeral volume is exactly that, so the StorageClass is named for what it gives you rather than for the driver.
The RBAC is the sidecar'sThe ClusterRole that reads PVCs and writes PVs belongs to csi-provisioner. The driver container in the same pod still holds nothing, and the node plugin is unchanged.
Turning it on edits an immutable fieldCSIDriver.spec.volumeLifecycleModes gains Persistent, and that field cannot be updated. Delete the CSIDriver object and let the apply recreate it; no running volume notices, because kubelet reads it at publish time only.

With durable: "true" the same shape gains a restore: The volume's contents live in S3 between pods, and are copied back into a fresh directory before the pod starts. That is the one path in the driver that reads from a bucket. The configuration reference lists the constraints, starting with a pod start that blocks on the download.

Inside the process#

The driver's internal structure Main picks a subcommand and Driver wires the object graph by hand. Three groups of classes follow. The grpc package terminates the Unix-socket gRPC server and the CSI services. The config and volume packages resolve a volume's effective configuration and credentials and persist its state and upload manifest. The archive and s3 packages watch the directories, decide what to upload and when, and run the uploads with bounded concurrency against one of three object stores. Logging, metrics and the web UI sit beside all of it. Main serve · controller · bench Driver wires it by hand, no DI logging · metrics · web UI StderrLoggerProvider · Metrics · WebUiServer grpc UnixSocketGrpcServer UDS · one Netty channel IdentityService GetPluginInfo · Probe NodeService publish · unpublish · stats ControllerService CreateVolume, when enabled RequestLoggingInterceptor one log line per RPC config · volume VolumeConfigResolver defaults, env, attributes CredentialResolver four modes, in order PrefixTemplate {namespace} · {podName} VolumeRegistry which volumes exist here VolumeStateStore state + manifest, atomic archive · s3 S3ArchiverEngine 1 s tick · 60 s rescan FileWatcher one WatchService, all volumes VolumeArchiver what to upload, and when UploadPipeline concurrency · retry · order ObjectStore S3 · presigned · POST policy Every RPC and every upload runs on a virtual thread, in plain blocking style. The engine's timing decisions all go through one tick method, so a test can drive them with a fake clock.

Roughly 70 classes, no framework. These are the ones worth knowing first.

Two decisions shape the rest. Everything is wired by hand, because a native image has to be told about every reflective lookup and a container generates them in bulk. Blocking work runs on virtual threads in plain blocking style, so an upload waiting on S3 costs a stack rather than a callback chain.

ClassResponsibilityWhat it actually does
Main, Driverentry point and wiringOne binary, several subcommands (serve, controller, presign-signer, bench, check-upload). Driver constructs the object graph by hand. There is no DI framework. The graph is a dozen singletons, and a container would only add reflection for the native image to configure.
UnixSocketGrpcServertransportgRPC over a Unix domain socket, on Netty's NIO channel rather than the native epoll transport. No per-architecture shared library ships, so the native-image configuration is identical on amd64 and arm64. Removes a stale socket file left by an unclean shutdown, and shuts down gracefully on SIGTERM.
IdentityServicewho this driver isGetPluginInfo, GetPluginCapabilities and Probe. Probe reports ready only after the state directory has been replayed, so kubelet does not publish into a driver that has not yet remembered its volumes.
NodeServicethe volume lifecycleValidates, creates the directory at target_path, registers the volume, and on unpublish runs the final sweep before deleting anything. Its validation messages are the user's error channel, because kubelet turns them into pod events.
ControllerServicePVC-declared volumesServed only when the driver runs as controller, beside the provisioner. It allocates nothing. CreateVolume validates the StorageClass parameters and hands back a volume id, so a bad StorageClass fails on the PVC and never reaches a pod.
VolumeConfigResolvereffective configurationBuilt-in defaults, then S3A_* environment, then the pod's volumeAttributes. Unknown attributes and malformed values are rejected rather than ignored, so a typo fails the pod instead of producing an empty bucket.
CredentialResolverhow a volume authenticatesThe four modes, in order. Presigned, then the per-volume Secret, then driver-global keys, then the AWS default chain. Credentials live in memory only, and their toString() is redacted.
VolumeRegistry, VolumeStateStorewhat survives a restartThe registry is the in-memory truth while the process lives; the store writes each volume's record and upload manifest to the state directory atomically, before the mutation counts as done. After a SIGKILL the files are the truth, and replaying them keeps a restart from re-uploading everything.
S3ArchiverEnginethe clockTwo ticks. A one-second tick evaluates quiescence and append pacing, finer than the shortest window a user can configure; a slow rescan re-reads every volume from disk, which is the safety net for inotify events the kernel dropped.
FileWatchernoticing writesOne WatchService for every volume on the node. inotify instances are a limited per-user resource, 128 by default. One instance with many watches keeps the driver from putting a ceiling on how many archived pods a node can host.
VolumeArchiverwhat to upload, and whenOne per volume. Holds the quiescence window, the append strategy, the segment bookkeeping and the manifest comparison. All of it is decided in a tick method a test can call with a fake clock. None of those tests sleep.
UploadPipelinehow uploads runBounded concurrency across the node, per-file serialisation so two uploads of one file cannot race, and capped exponential backoff with jitter that retries indefinitely. Virtual threads throughout, in blocking style.
ObjectStorewhere the bytes goA deliberately narrow interface. The driver writes, and for segment compaction lists and deletes, but never reads objects back. The presigned implementation has no S3 API at all and still satisfies the same contract as the SDK-backed one.
VolumeRestoredurable volumesThe one path that reads from S3. A volume marked durable is restored into the fresh directory before publish returns, so a pod start can block on a download. restoreTimeoutSeconds bounds the wait.
PresignSignerServerthe reference signerThe same binary in presign-signer mode: A JDK HTTP server in front of the AWS SDK's presigner. The integration and end-to-end tests sign against it. Treat it as a starting point for your own, not as a hardened service.

The packages map straight onto directories under src/main/java/de/trion/csi/s3archiver/, and the class names above are the file names, so the diagram is also a map of the source tree.

What one publish call does#

The order matters more than it looks. Most of it is about failing before anything exists on disk.

  1. Validate the request. Volume id, target path, access mode, and the ephemeral flag kubelet sets. A PV-backed volume is refused here.
  2. Resolve the configuration and the credentials. Both before any directory is created, so a misconfigured volume fails the pod at start rather than becoming a bucket that quietly stays empty. The failure text is written for whoever wrote the pod spec, because that is who reads the event.
  3. Check for a republish. Same volume, same configuration is a cheap no-op that refreshes only the credentials. That is the route a rotated Secret takes to a running pod. Same volume, different configuration is ALREADY_EXISTS.
  4. Create the directory at target_path, mode 0777, and refuse to follow a symlink there.
  5. Register the volume. The state file is written before the call returns, and the watcher and the engine pick the directory up. A durable volume is restored from S3 here, the only step that can take minutes.

Unpublish runs in reverse and blocks: The final sweep uploads everything new or changed, ignoring the quiescence window, and only then is the directory deleted and the state record dropped. The wait is deliberate. sweepTimeoutSeconds is there for workloads that would rather not have it.

Where next#

  • Security: The same architecture argued by subtraction, for an audit.
  • Operations: What the semantics above mean once it is running, and what to size the container for.
  • Why Java: Virtual threads, the native image, and what each of them costs here.
  • The source: Every decision on this page is written up in full there, next to the code it produced.