# csi-s3-archiver — full documentation > A Kubernetes CSI driver giving pods a local scratch volume whose contents are archived to S3 automatically: no sidecar, no S3 SDK, no application change. Generated from https://s3archiver.csi.trion.de. Each section below is one page of that site, separated by a rule; the Source line above each is where it can be cited from. ============================================================================== Source: https://s3archiver.csi.trion.de/ Updated: 2026-08-30 Summary: A Kubernetes CSI driver giving pods a local scratch volume whose contents are archived to S3 automatically: no sidecar, no S3 SDK, no application change. Kubernetes CSI driver # Pod-local volumes that archive themselves to S3 Heapdumps, rotated logs and backup dumps land in object storage without a sidecar, an S3 SDK, or a single line of change in your application. The pod writes ordinary files; the driver on the node ships them. [Install it](https://s3archiver.csi.trion.de/install.html) [Configuration reference](https://s3archiver.csi.trion.de/docs.html) [How it compares](https://s3archiver.csi.trion.de/compare.html) - CSI v1.12.0 - Java 25, GraalVM native - amd64, arm64 - no RBAC - no sidecar pod spec: ```yaml volumes: - name: dumps csi: driver: s3archiver.csi.trion.de volumeAttributes: bucket: prod-dumps prefix: "heapdumps/{namespace}/{podName}/" ``` Anything the pod writes to that volume shows up under `s3://prod-dumps/heapdumps///…` shortly after the writer finishes, and at the latest when the pod terminates. ## The problem Getting a file out of a pod and into object storage is a recurring chore, and every usual answer puts storage concerns inside the workload: - add an S3 SDK and credentials to the application; - bolt on a sidecar sharing an `emptyDir`; - write a `preStop` script that races pod termination and loses. This driver moves the problem to the node, where it belongs. The application keeps writing files to a directory and never learns that S3 exists. **NOTE — It is not an S3 filesystem** This is not s3fs, goofys or mountpoint-s3. Files are ordinary files on the node disk; archiving is asynchronous and one-way. Objects are never read back into the volume, and there is no restore command. That is a design choice rather than a gap. [See how it compares](https://s3archiver.csi.trion.de/compare.html). ## How it works One DaemonSet pod per node, nothing per workload. kubelet creates the volume directory and hands the driver the pod's metadata and secrets over a Unix domain socket; the driver watches the directory and uploads. Figure: How a file reaches S3 — The pod writes a file into its volume directory on the node. The driver's watcher notices the write, waits for the file to go quiet, and uploads it to the S3 bucket. At pod termination a final sweep uploads anything left. 1. **kubelet publishes the volume** — The driver validates every attribute up front and creates a plain directory under the kubelet target path. A bad value fails the pod with a readable message on its events, rather than a bucket that quietly stays empty. 2. **The pod writes files** — Ordinary POSIX writes at local-disk speed. No FUSE layer, no network in the write path, no surprise latency when the object store is slow. 3. **The archiver notices and uploads** — A shared inotify watcher plus a periodic rescan detect changes. Once a file has been unmodified for `quiescenceSeconds` it is uploaded, streaming single-PUT or multipart, with capped-backoff retry and bounded per-node concurrency. `appendOnly` volumes sync while the file still grows. 4. **The final sweep blocks teardown** — When the pod terminates the driver uploads everything new or changed one last time, ignoring the quiescence window, and holds up pod deletion until it is done. Set `sweepTimeoutSeconds` if you would rather have best-effort teardown. ## What you get - **Nothing in the pod** — No sidecar, no init container, no S3 SDK, no credentials in the application. The pod declares a volume; the node does the rest. - **Quiescence-based upload** — A file is uploaded once the writer stops touching it. A shared inotify watcher does the noticing, with a periodic rescan as a safety net. - **Append-only log shipping** — `appendOnly` syncs files that never stop growing, either by re-uploading or, Loki-style, by shipping only the new tail bytes. - **A final sweep that actually waits** — Pod termination blocks until the last file is in the bucket. The heapdump written seconds before the crash is the one you wanted most. - **Credentials that stay off the node** — Per-volume secrets, driver-global keys, the AWS default chain (IRSA works untouched), or presigned mode, where the node holds nothing. - **Predictable keys** — Prefix templates with `{namespace}`, `{podName}`, `{date}` and friends. A typo fails the pod instead of creating a directory called `{namesapce}`. - **Empty RBAC** — The driver never calls the API server. kubelet pushes pod metadata and per-volume secrets into the CSI calls, so there is no Role to audit. - **Any S3, not just AWS** — Endpoint and path-style overrides per volume. Tested against SeaweedFS on every build; the same code path serves Ceph RGW and AWS. - **One static binary** — A GraalVM native image in a `FROM scratch` image: No shell, no package manager, no libc. Runs unprivileged with a read-only root filesystem. ## See it run Optional statistics Web UI. [![Screenshot of the csi-s3-archiver statistics page rendering a three-node cluster: Headline counters for uploads, bytes archived, failures and retries across every node; a bar chart of uploads per hour over the last day; a table of the three nodes with their individual totals; and a table of recent upload failures showing the object key and the S3 error.](https://s3archiver.csi.trion.de/media/webui.png)](https://s3archiver.csi.trion.de/media/webui.png) *One pod rendering the whole cluster: Uploads per hour, all three nodes with their own totals, and recent failures with their cause.* [All screenshots and recordings](https://s3archiver.csi.trion.de/screenshots.html) ## Getting started Installing is one `kustomize` apply. Archiving your first file is a pod spec with six extra lines in it. install: ```sh $ kubectl apply -k deploy/base $ kubectl -n csi-s3-archiver rollout status daemonset/csi-s3-archiver $ kubectl get csinode -o jsonpath='{.items[*].spec.drivers[*].name}' s3archiver.csi.trion.de ``` [Full install guide](https://s3archiver.csi.trion.de/install.html) [Configuration reference](https://s3archiver.csi.trion.de/docs.html) Then a pod that writes a file. Everything under `/dumps` ends up in the bucket, at the latest when the pod terminates. heapdump-example.yaml: ```yaml apiVersion: v1 kind: Pod metadata: name: heapdump-example spec: restartPolicy: Never containers: - name: app image: busybox:1.36 command: ["sh", "-c", "dd if=/dev/urandom of=/dumps/java_pid1.hprof bs=1M count=4; sleep 60"] volumeMounts: - name: dumps mountPath: /dumps volumes: - name: dumps csi: driver: s3archiver.csi.trion.de volumeAttributes: bucket: prod-dumps prefix: "heapdumps/{namespace}/{podName}/" ``` ## Two more shapes ### A log that never goes quiet A continuously written log never becomes quiescent, so by default it would only be archived at termination. `appendOnly` syncs it while it grows, and `segments` keeps the transfer linear instead of quadratic. ```yaml volumeAttributes: bucket: prod-logs prefix: "logs/{namespace}/{podName}/" appendOnly: "true" # sync while the file is still growing appendStrategy: "segments" # upload only the new tail bytes segmentTargetBytes: "1048576" ``` ### Bounded node disk For a CronJob that dumps a database every few minutes, delete the local file once it is safely in the bucket. The node never accumulates. ```yaml volumeAttributes: bucket: prod-backups prefix: "pg/{namespace}/{podName}/{date}/" deleteAfterUpload: "true" # bounded node disk: the local file goes after a good upload ``` ## Project status **Under development.** The driver works end to end today and is covered by unit, integration and conformance suites. It has not yet been through a tagged release, and it comes with no warranty of any kind. - **Working:** CSI Identity and Node services over a Unix domain socket, persisted volume state with recovery across a driver restart *and a node reboot*, the full configuration and credential model, streaming and multipart uploads with retry and bounded concurrency, and the archiver engine: Quiescence, `appendOnly` (`rewrite` and `segments`), `deleteAfterUpload`, glob filters and the blocking final sweep. csi-sanity passes on both the JVM build and the native binary, with and without the optional Controller. - **Also working:** [gzip compression and SSE/SSE-KMS](https://s3archiver.csi.trion.de/configuration.html#volume-attributes), [server-side append](https://s3archiver.csi.trion.de/configuration.html#append-strategy) so a growing file no longer re-uploads whole, [presigned multipart and signer-less POST policies](https://s3archiver.csi.trion.de/credentials.html#credentials), [Prometheus metrics](https://s3archiver.csi.trion.de/configuration.html#environment-driver-container), a statistics [web UI](https://s3archiver.csi.trion.de/screenshots.html#statistics-page), a Helm chart, [PVC-declared volumes](https://s3archiver.csi.trion.de/faq.html#can-i-declare-the-volume-with-a-pvc-instead-of-inline) and [durable volumes](https://s3archiver.csi.trion.de/faq.html#can-a-volume-outlive-its-pod) that outlive their pod. - **In progress:** the end-to-end suite against a real kubelet in CI, cross-architecture images and the first tagged release. - **Not planned:** reading objects back for ephemeral volumes, Windows nodes, capacity enforcement by deletion, and a POSIX filesystem over S3. Everything documented on this site is behaviour that exists in the code today. [The FAQ goes into what is tested](https://s3archiver.csi.trion.de/faq.html#is-this-ready-for-production), and what that does and does not prove. ============================================================================== Source: https://s3archiver.csi.trion.de/docs.html Updated: 2026-08-30 Summary: Documentation for csi-s3-archiver: Configuration reference, credential modes, operational semantics, and how to install it. Reference # Documentation Start with the install guide if you have not run it yet. The reference is split by the question you are asking: What can I set, how does it authenticate, and what happens once it is running. ## Start here Three commands from nothing to an archived file. The [install guide](https://s3archiver.csi.trion.de/install.html) has the full walkthrough, including credentials and how to verify it worked. install: ```bash kubectl apply -k deploy/base kubectl create secret generic s3-creds \ --from-literal=accessKeyId=... --from-literal=secretAccessKey=... kubectl apply -f my-pod.yaml ``` Or with Helm, which installs the same objects. The chart is in the repository rather than on a chart repository, because nothing has been tagged yet: install with Helm: ```bash helm install csi-s3-archiver deploy/helm/csi-s3-archiver \ --namespace csi-s3-archiver --create-namespace \ --set config.bucket=prod-dumps ``` The values, the credential options and what an upgrade needs are in [the Helm section](https://s3archiver.csi.trion.de/install.html#helm) of the install guide. ## Reference - **[Configuration](https://s3archiver.csi.trion.de/configuration.html)** — Every `S3A_*` variable and `volumeAttribute`, the prefix placeholders, and how to choose an append strategy.Environment · volumeAttributes · Prefix placeholders · Append strategies - **[Credentials](https://s3archiver.csi.trion.de/credentials.html)** — The four ways a volume authenticates, resolved in a fixed order: The AWS default chain, per-volume secrets, a signer, or a static POST policy.Resolution order · Presigned mode · POST policies · Rotation - **[Operations](https://s3archiver.csi.trion.de/operations.html)** — What to expect once it is running: What pod deletion waits for, the archiving semantics and their caveats, what to size the container for, and troubleshooting.The final sweep · Semantics and caveats · Resources · Troubleshooting · Versions - **[Architecture](https://s3archiver.csi.trion.de/architecture.html)** — What runs where and which class does what: Deployment diagrams for the DaemonSet and the optional Controller, and the structure inside the process.The DaemonSet · With the Controller · Inside the process · The publish path ## Going further | Page | Answers | | --- | --- | | [Distributions](https://s3archiver.csi.trion.de/install.html#distributions) | What is different on vcluster, Talos, OpenShift, MicroK8s and k0s, and the two managed products where the driver cannot run at all. | | [Comparison](https://s3archiver.csi.trion.de/compare.html) | Should I use this instead of a FUSE mount, a sidecar, a log shipper or Velero? Includes when *not* to. | | [Benchmarks](https://s3archiver.csi.trion.de/benchmarks.html) | What does it cost to run, measured, and why there is no goofys throughput comparison. | | [Security](https://s3archiver.csi.trion.de/security.html) | What it does not have: No API access, no `mount(2)`, no shell, and in two modes no credentials on the node. | | [FAQ](https://s3archiver.csi.trion.de/faq.html) | The questions that come up before adopting it. | | [Why Java](https://s3archiver.csi.trion.de/why-java.html) | What Java 25 brings to a CSI driver, what the native image costs, and where Go would have been easier. | | [Release history](https://s3archiver.csi.trion.de/releases.html) | What shipped when, and the compatibility rules the project holds to. | ## Machine-readable The same documentation, for the things that read it on your behalf. An assistant answering a question about this driver, a validating admission webhook, or editor completion for `volumeAttributes` can start from one of these instead of from rendered HTML. Each is generated from the source the pages above are generated from, so none of them can quietly fall behind. | File | What it is | | --- | --- | | [reference.json](https://s3archiver.csi.trion.de/reference.json) | Every `S3A_*` variable and every `volumeAttribute` as JSON, each with its default, its description, and the environment variable that overrides it. Built from the same table the [configuration page](https://s3archiver.csi.trion.de/configuration.html) renders. | | [llms.txt](https://s3archiver.csi.trion.de/llms.txt) | An index of the site in the [llms.txt](https://llmstxt.org/) format, one line per page saying what that page answers. | | [llms-full.txt](https://s3archiver.csi.trion.de/llms-full.txt) | Every page on this site as one Markdown document, each with the URL it can be cited from. | Nothing here is behind a robots rule and nothing is rate-limited. If you are building something against this driver and the shape of these files is awkward, that is worth telling us about. ============================================================================== Source: https://s3archiver.csi.trion.de/faq.html Updated: 2026-08-30 Summary: What happens when S3 is down, why pod deletion blocks, how credentials resolve, and what csi-s3-archiver deliberately will not do. FAQ # Questions people actually ask Including the awkward ones. Where a behaviour is a trade-off rather than a feature, the trade is stated instead of glossed over. - [What it is](#what-it-is) - [Operating it](#operating-it) - [Configuration](#configuration) - [Security](#security) - [Limits](#limits) - [Design](#design) ## What it is ### Is this ready for production? **Nothing here comes with a warranty.** The software is provided as is, without warranties or conditions of any kind, and without liability for what it does to your data or your cluster. Nobody is on call for your bucket. What does exist is a test suite that runs on every build: - **Unit tests** over the archiver engine, the upload pipeline and retry policy, config and prefix-template validation, credential resolution, volume state and recovery, and the gRPC services. - **Integration tests against a real object store.** SeaweedFS in Testcontainers exercises single-PUT, multipart and presigned uploads, and the archiver engine end to end against it. - **CSI conformance.** The upstream `csi-sanity` suite runs against both the JVM build and the GraalVM native binary. It proves the Identity service and the Node service's capabilities, node id and argument validation. It does not prove a full publish, write and unpublish cycle, because `csi-test` skips those specs for any plugin without a controller service. - **Crash and restart tests.** One suite forks the real jar and `SIGKILL`s it, checking that persisted volume state is recovered rather than re-uploaded; another covers the same ground for interrupted uploads. - **A log-capture test** that fails the build if any secret material reaches a log line, plus a unit test asserting credentials never reach the state files on disk. - **End to end against a real kubelet**, with K3s in Testcontainers. So the behaviour described on this site is tested behaviour rather than aspiration. What is missing before anyone should call it production-ready is a release, and mileage on real clusters that are not ours. Read [the limits below](#limits) before you decide. Several of them are permanent design choices rather than gaps that will close, and [the comparison page](https://s3archiver.csi.trion.de/compare.html#when-not-to) lists the cases where a different tool is simply the right answer. ### Is this open source? Not yet, but that is the plan. The source is not published at the moment. When it is, this answer will be replaced by a link to the repository and the licence. ### Is this an S3 filesystem? **No, and not by accident.** It is not s3fs, goofys or mountpoint-s3. Files the pod writes are ordinary files on the node's disk, written at local-disk speed; the driver copies them to S3 afterwards, asynchronously and one-way. Nothing in the write path talks to the network, so a slow or unavailable object store never blocks the application's `write(2)`. The trade is that the volume is not a view of the bucket: `ls` shows what this pod wrote, not what is in S3. ### Can my application read objects back from the volume? Not while the pod runs. Archiving is one-way, and there is no read-through, so `ls` shows what this pod wrote rather than what is in the bucket. That is a non-goal rather than a missing feature. One exception, at one moment: [A durable volume](#can-a-volume-outlive-its-pod) is restored from S3 *at publish*, before the containers start, so the next pod finds what the last one left. A fresh directory filled once, not a view of the bucket. If you need read-back during the run, [the comparison page](https://s3archiver.csi.trion.de/compare.html) names tools that do it. ### What is it good at, and what should I not use it for? Good at: JVM heapdumps, core dumps, rotated logs, database dump files, diagnostic bundles, CI artefacts. Anything a pod produces as whole files that somebody may want later. Bad at: Anything that needs the object store to be the source of truth during the run, shared read-write storage between pods, data the application must read back, or files that are rewritten in place thousands of times a second. ## Operating it ### What happens when S3 is unreachable? Uploads retry with exponential backoff and jitter, capped, indefinitely. The contract is at-least-once, eventually. During the pod's life this is invisible: The file is on local disk and the retries happen in the background. At pod termination it is very visible, because the final sweep blocks pod deletion until the upload succeeds. **A pod can sit in `Terminating` for as long as S3 is down.** That is the intended default, since losing the last heapdump is worse, but it is bounded per volume with `sweepTimeoutSeconds`. ### What happens if the node or the driver dies mid-upload? Per-volume state records and upload manifests live under `S3A_STATE_DIR` on a host path that outlives the container, so a restarted driver recovers its volumes and resumes rather than re-uploading everything. Point `S3A_STATE_DIR` at `emptyDir` and you lose that property. One caveat worth knowing: Kubelet only delivers a `nodePublishSecretRef` secret on a fresh publish, so a recovered volume that used one falls back to whatever driver-global credentials exist, possibly none. The driver logs a warning naming the volume when that happens. If the node itself is destroyed, the local files go with it. A pod-local scratch disk is exactly as durable as the node, which is why the sweep is synchronous. ### How do I know an upload actually happened? The driver logs one structured line per event on stderr, which is what `kubectl logs` shows: ```log 2026-07-27T09:15:30.402Z INFO [csi-rpc-3] NodeService - rpc=NodePublishVolume volumeId=csi-9f3a bucket=prod-dumps status=OK duration_ms=6 2026-07-27T09:16:01.884Z INFO [archiver-1] Archiver - event=quiescent file=java_pid1.hprof bytes=4194304 2026-07-27T09:16:04.117Z INFO [upload-2] Uploader - event=upload key=heapdumps/default/heapdump-example/java_pid1.hprof bytes=4194304 attempt=1 status=OK duration_ms=2231 2026-07-27T09:17:12.006Z INFO [csi-rpc-5] NodeService - rpc=NodeUnpublishVolume volumeId=csi-9f3a event=sweep uploaded=0 status=OK duration_ms=41 ``` A permanent failure (any 4xx) is logged as `FAILED_PERMANENTLY` and dropped rather than retried forever. If nothing appears at all, raise `S3A_LOG_LEVEL` to `debug` and run `check-upload` from a pod using the driver image and the same environment. It exercises the same upload path and reports exactly what S3 answered. ### How much does the DaemonSet cost me? The driver container requests 10m CPU and 64 MiB of memory, with a 256 MiB memory limit and, on purpose, no CPU limit: Throttling the driver mid-upload just makes the unpublish sweep, which blocks pod deletion, take longer. The two upstream sidecars request 5m and 16 MiB each. Files are streamed from disk with fixed-size buffers, so a multi-gigabyte heapdump does not need multi-gigabyte heap. The image is a GraalVM native binary in a `FROM scratch` image: No shell, no package manager, no libc. ### How many S3 requests will this generate? For a normal volume, one `PutObject` per file, plus one multipart upload (initiate, *n* parts, complete) per file over 64 MiB, with 32 MiB parts. `appendOnly` changes the arithmetic and is the case to think about. `rewrite` costs one full re-upload every `appendSyncSeconds`, so total bytes transferred grow quadratically with the file size. `appendStrategy: segments` uploads only the new tail bytes, which is linear, at the cost of one object per segment plus deletes at compaction. For a log that runs for hours, use `segments`. ### Can I limit which files get archived? Yes. `include` and `exclude` take comma-separated globs matched against paths inside the volume. A common shape is to exclude the writer's temporary files so half-written objects never reach the bucket: `exclude: "*.tmp,*.partial"`. ### Does a growing log file get re-uploaded every time? Not any more. In `appendOnly` mode with the `rewrite` strategy, `appendUpload` sends only the new tail and lets the server keep what it already has, using `UploadPartCopy` or, on an S3 Express directory bucket, a native offset append. Segment compaction at pod termination also assembles server-side rather than re-uploading the file, which matters because that upload is the one blocking pod deletion. It falls back to a full upload whenever the store cannot do it, and the fallback writes the same object, so losing the optimisation costs bandwidth and never correctness. ### How do I see what it is doing across the cluster? Three ways, all off by default. `S3A_METRICS_PORT` serves Prometheus metrics, which is where alerting belongs. `S3A_WEBUI_PORT` serves a read-only page with uploads per hour and per day, bytes, retries and failures with their cause; set `S3A_WEBUI_PEERS` to a headless Service and any pod renders the whole cluster. And `S3A_LOG_FORMAT=json` gives a collector one JSON object per line instead of a format it would have to re-parse. The page has no authentication. Keep it on a port-forward or behind an authenticating ingress. ## Configuration ### Do I need a PVC, a StorageClass or a provisioner? Not for the default shape. A CSI ephemeral inline volume is declared in the pod spec and lives and dies with the pod: No PV, no PVC, no StorageClass, no controller, and nothing to deploy beyond the DaemonSet. That is the whole install, and it is what the [first file](https://s3archiver.csi.trion.de/install.html#first-file) walkthrough uses. A PVC is an option, not a requirement. Deploying the optional Controller adds a StorageClass and lets a workload declare the volume with a `volumeClaimTemplate`, which suits tooling and admission policies built around PVCs. It is also what `durable: "true"` needs. The driver still holds no Kubernetes permissions in either shape, because the RBAC belongs to the provisioner sidecar rather than to the node plugin. See [declaring the volume with a PVC](#can-i-declare-the-volume-with-a-pvc-instead-of-inline) and [volumes that outlive their pod](#can-a-volume-outlive-its-pod). ### Does the driver need RBAC or privileged mode? Neither. The ServiceAccount has **no Roles or bindings at all**, because the driver never talks to the API server. kubelet pushes pod metadata and per-volume secrets into the CSI calls, and registration happens over a local socket. The container runs as root but *not* privileged, with no added capabilities and a read-only root filesystem. Root is needed only to create and remove directories under `/var/lib/kubelet/pods`, which are root-owned. Because the volume is a plain directory rather than a mount, there is no `mount(2)` in this driver, and so no need for `privileged: true` or bidirectional mount propagation. ### My pod runs as a non-root user. Can it write to the volume? Yes. The driver creates the volume directory world-writable and the `CSIDriver` object sets `fsGroupPolicy: File`, which makes kubelet apply the pod's `fsGroup` ownership itself. A non-root container with an `fsGroup` can write into the volume without further configuration. ### Does it work with something other than AWS? Yes. Set `endpoint` and usually `pathStyle: "true"` per volume, or the `S3A_ENDPOINT` and `S3A_PATH_STYLE` defaults driver-wide. The integration suite runs against SeaweedFS on every build; the same code path serves Ceph RGW and AWS. Anything that implements the S3 API for `PutObject` and multipart should work. ### How do I use IRSA, EKS Pod Identity or an instance profile? Do nothing. If no per-volume secret and no `S3A_ACCESS_KEY_ID` are present, credential resolution falls through to the AWS default provider chain, which picks those up. Give the driver's ServiceAccount the role and it works with no driver configuration. ### Why is the secret field called nodePublishSecretRef? Because that is the fixed upstream Kubernetes field name, not this project's naming. It means "the Secret kubelet passes into the `NodePublishVolume` RPC". The Secret lives in the pod's own namespace, and it is read once, at publish time, so rotating it takes effect for new pods only. ### Two pods write files with the same name. What happens? Whatever your `prefix` says should happen. The default prefix is `{namespace}/{podName}/`, so two pods never collide. If you flatten the prefix on purpose, the second upload overwrites the first: Objects are overwritten, never versioned. Add `{date}`, add `{podUid}`, or enable bucket versioning if you need history. ### I rewrote a file and nothing was uploaded. Change detection is size + mtime. A file rewritten to exactly the same size with the same timestamp is not noticed, by the watcher, by the periodic rescan, or by the final sweep. Hashing every candidate would mean reading multi-gigabyte heapdumps twice on every scan, which is the worse trade for this workload. ### Can a volume outlive its pod? Yes, with `durable: "true"` on a PVC-declared volume, and it works differently from how you might expect. The contents are **not kept on the node**: They live in S3, and the driver restores them into a fresh directory when the next pod publishes the volume, then archives what changed and mirrors deletions when it ends. That design avoids everything node-local storage would have cost: No `mount(2)`, so the driver stays unprivileged; no node affinity, so the volume survives losing its node entirely. What it costs is a blocking download at every pod start, ReadWriteOnce with no attach step enforcing it, and no POSIX semantics between sweeps. If you cannot rebuild the data, use real storage. ### Can I declare the volume with a PVC instead of inline? Yes, if you deploy the optional Controller. It exists because plenty of tooling, charts and admission policies are built around PVCs, and that is a presentation choice rather than a storage one. Without `durable`, the volume a `volumeClaimTemplate` gives you is **still pod-lifetime**: Archived and removed when the pod ends. That is exactly right for a generic ephemeral volume, whose claim Kubernetes deletes with the pod, and it is not what a standalone PVC leads people to expect. The driver cannot tell the two apart, so the StorageClass is named for what it gives you. ## Security ### Can a workload make the driver upload files from the node? No. Symlinks are never followed: The driver runs as root, so following a symlink a workload planted would exfiltrate arbitrary node files into that workload's bucket. Symlinks are skipped and logged, and only regular files are archived. Hard links cannot escape the volume's filesystem scope. ### Do S3 credentials end up on the node disk or in logs? No. Credentials are held in memory only: Never written to the state files, and `toString()` on the resolved credentials is redacted. Both properties are unit-tested, and a log-capture test fails the build if secret material appears in any log line. If you want no credentials on the node at all, use presigned mode: Set `presignEndpoint` and the driver asks a signer service for a fresh URL per upload. Combining presigned mode with static keys in the same secret is rejected, since that would ship to the node exactly what the mode exists to avoid. The cost is a 5 GiB per-file cap, because a single presigned PUT cannot be multipart. A signer that also signs the multipart lifecycle lifts the cap; set `presignMultipart: "true"` on the volume once yours does. ### Is the data encrypted? In transit, yes. Uploads go over HTTPS to whatever endpoint you configure. At rest, both options exist. A bucket default applies to everything the driver writes, and `serverSideEncryption` requests it per volume: `AES256` for SSE-S3, or `aws:kms` with `sseKmsKeyId`. Set the bucket default anyway, since it covers a volume whose author forgot to ask. Three of the six stores on the [endpoint matrix](https://s3archiver.csi.trion.de/s3-compatibility.html) answer 500 or 501 to an SSE request rather than rejecting it cleanly, so check yours first. ### Can I run it without S3 credentials on the node at all? Two ways. **Presigned mode** asks a signer service for a URL authorising one object key, valid for minutes, so the node holds only a bearer token. **POST policy mode** puts one signed, prefix-scoped policy in the volume's Secret, so there is no signer to run at all. The trade for the second is that a policy lives at most 7 days and has to be rotated; include an `expiresAt` in the bundle and the driver warns twelve hours ahead rather than letting every upload start failing at once. Both modes are covered on the [security page](https://s3archiver.csi.trion.de/security.html). ## Limits ### How large a file can it archive? Normal mode streams from disk and switches to multipart above 64 MiB with 32 MiB parts, so file size is bounded by S3's own limits and the node's disk, not by the driver's memory. Presigned mode caps a file at **5 GiB** by default, because a single presigned PUT cannot be multipart; larger files fail with a clear error rather than being silently split into objects you did not ask for. With `presignMultipart: "true"` and a signer that signs the multipart operations the driver runs the whole multipart lifecycle instead, which raises the ceiling to about **312 GiB**, being S3's 10 000 parts at the driver's 32 MiB part size. Object keys are capped at S3's 1024 UTF-8 bytes. A file whose key would exceed that is skipped and logged, never truncated, because a truncated key would overwrite an unrelated object. ### Which architectures and platforms? linux/amd64 and linux/arm64, built on native runners per architecture and stitched into a multi-arch manifest, with no QEMU involved. Linux nodes only; Windows is a non-goal. ### Is there a Helm chart, metrics endpoint or web UI? All three. The chart is equivalent to the kustomize base, so `kubectl apply -k deploy/base` and a `helm install` give the same DaemonSet. Both endpoints are **off unless you set their port**, because a node plugin that opens a listener nobody asked for is a surprise. `S3A_METRICS_PORT` turns on Prometheus metrics; `S3A_WEBUI_PORT` turns on the [statistics page](https://s3archiver.csi.trion.de/screenshots.html#statistics-page), which aggregates every node through a headless Service and has no authentication, so keep it behind something. ### What happens if the node crashes? Whatever had already been archived is safe, and whatever was still only on disk is re-archived when the node comes back: Recovery reads the state files, re-registers the volumes, and the manifest stops it re-uploading what already landed. A volume whose pod directory is gone is dropped without affecting the node's other volumes. What is *not* recoverable is a node that never comes back with its disk intact. That is the trade for keeping the write path local, and it is why the contract is at-least-once and eventual rather than synchronous. If you need a write to be durable before the application continues, you need the application to write to S3. ## Design ### Why Java for a CSI driver? Because it is built as a GraalVM native image, the usual objections do not apply: There is no JVM in the runtime image, no warm-up on the publish path, and the container is `FROM scratch`. What Java buys is the AWS SDK, mature gRPC, and Java 25 virtual threads. Every RPC and every upload is plain blocking code on a virtual thread, with no async framework and no reactive types anywhere in the codebase. ### Why does pod deletion wait for the upload? Because the file written seconds before a pod died is the one you most want. A driver that returns immediately from `NodeUnpublishVolume` and uploads in the background would lose exactly that file, silently, on every OOMKill. The cost is stated rather than hidden: If S3 is unreachable, the pod stays in `Terminating`. `sweepTimeoutSeconds` converts that into best-effort teardown for the volumes where a stuck pod is the worse outcome. ### Why quiescence instead of uploading on every write? A heapdump is written over many seconds; uploading it on the first `IN_MODIFY` would upload a truncated file, then do it again, and again. Waiting for the writer to stop touching the file, `quiescenceSeconds` and 30 by default, means one upload of one complete file. Files that never go quiet are what `appendOnly` is for. **NOTE — Not answered here?** Read [the configuration reference](https://s3archiver.csi.trion.de/docs.html) for the exhaustive attribute list, [the install guide](https://s3archiver.csi.trion.de/install.html) to get going, or [the comparison](https://s3archiver.csi.trion.de/compare.html) to work out whether this is the right shape of tool at all. ============================================================================== Source: https://s3archiver.csi.trion.de/compare.html Updated: 2026-08-30 Summary: How csi-s3-archiver differs from S3 FUSE mounts, sidecars, preStop hooks, log shippers and Velero, including when to pick one of those instead. Comparison # Compared to the alternatives Where this driver fits among S3 FUSE mounts, sidecars, log shippers and backup tools, including, at the end, the cases where you should pick one of those instead. ## The short version Most tools in this space answer the question *"how do I make S3 look like a filesystem?"* This one answers a narrower question: *"how do I get files out of a pod and into a bucket without touching the workload?"* Narrower means it does less, and that is why it is cheap to reason about. - yes - partial or conditional - no | Approach | Nothing in the pod | Local write speed | Reads from S3 | Zero app changes | Survives termination | Any file type | Reach for it when | | --- | --- | --- | --- | --- | --- | --- | --- | | **csi-s3-archiver** | | | | | | | Write-once files you may want later | | S3 SDK in the application | | | | | | | The app really does need object storage | | Sidecar + shared `emptyDir` | | | | | | | You already run one for other reasons | | `preStop` hook script | | | | | | | Nothing; see below | | s3fs / goofys | | | | | | | Legacy code that must see a mounted bucket | | Mountpoint for Amazon S3 (CSI) | | | | | | | Read-heavy AWS workloads | | csi-s3 (FUSE-backed CSI) | | | | | | | You want a PVC that is a bucket | | JuiceFS CSI driver | | | | | | | Shared POSIX storage across pods | | Fluent Bit / Vector | | | | | | | Log *records*, parsed and routed | | Velero | | | | | | | Disaster recovery of the whole cluster | *"Survives termination"* means: A file written moments before the pod is killed still reaches the bucket, without you arranging it. That column is the one that decides most real cases. ## The FUSE mounts: S3fs, goofys, Mountpoint, csi-s3, JuiceFS These present a bucket as a mounted filesystem. Reads and writes go over the network, translated into S3 requests by a FUSE layer. That is a different product category, and if you need to *read* objects, one of them is the answer rather than this driver. What that costs when all you wanted was to archive a heapdump: - **Write latency is object-store latency.** The application's `write(2)` is now a network operation. A JVM writing a 4 GiB heapdump onto a FUSE mount is a very different event from one writing to local disk. - **Object stores are not filesystems.** Rename is copy-then-delete; partial overwrites, appends and random writes range from expensive to unsupported. Mountpoint for Amazon S3, the best-engineered of the group, supports sequential writes to *new* objects; appends only work on S3 Express One Zone directory buckets, and overwriting an existing object needs an explicit `--allow-overwrite` and a full sequential rewrite. - **A FUSE mount can fail underneath a running pod.** When the mount goes away, the application sees I/O errors on a path it thought was a disk. - **Mounting needs privilege.** There is a real `mount(2)`, a FUSE device and usually mount propagation. This driver has none of that: The volume is an ordinary directory, so the container runs unprivileged with a read-only root filesystem and an empty RBAC. - **JuiceFS additionally needs a metadata engine**, Redis or a SQL database, that you now operate and back up. In exchange it gives real POSIX semantics and shared access across pods, which none of the others do. **NOTE — The summary** If your workload must read from the bucket, share files between pods, or see a POSIX filesystem, use one of these and accept the operational surface. If it only produces files, a FUSE mount is a large amount of machinery, and a large amount of failure mode, to move bytes in one direction. ## Sidecars and in-app SDKs The two most common hand-rolled answers, and the reason this driver exists. ### An S3 SDK in the application Correct if the application's job really does involve object storage. Wrong as a way to get diagnostics out. It puts an SDK, a credential, a retry policy and a bucket name into every service that might ever produce a dump, in whatever languages those services happen to be written in, and the code runs inside the process that is currently dying of an OutOfMemoryError. ### A sidecar sharing an `emptyDir` Better: The application stays clean and one image does the uploading. What you pay: - a container, its image pulls, its requests and limits, and its CVE surface, on *every* pod, on every node, rather than once per node; - S3 credentials mounted into the workload's own pod, where the application container can often reach them; - termination ordering. Kubernetes has native sidecars now, so a sidecar can outlive the main container, but you still have to configure it, and getting it wrong is silent until the day it matters. The DaemonSet form of the same idea is one pod per node instead of one per workload, with the credentials outside the workload's namespace and the termination ordering guaranteed by the CSI contract rather than by your YAML. **WARN — The preStop hook deserves its own warning** A `preStop` script that `aws s3 cp`s the directory is the most common version of this, and it loses the exact case it was written for: It does not run on an OOMKill or a node failure, it competes with `terminationGracePeriodSeconds`, and its failures are invisible. Nothing about it is fixable by trying harder. ## Log shippers: Fluent Bit, Vector, Promtail These tail files, parse them into records, and route the records somewhere, including S3. If what you have is *logs*, and what you want is searchable structured events, they are the right tool and this driver is not competing with them. They are the wrong tool when the artefact is a file rather than a stream of lines: A 4 GiB heapdump, a `pg_dump` tarball, a core dump, a JFR recording. A log shipper either mangles those into "records" or ignores them. The dividing line is simple: **if you would ever want to open the thing in a viewer rather than grep it, it is a file, not a log stream.** Many clusters want both, and they compose fine: A log shipper for the log stream, this driver for the dumps. ## Velero Velero backs up Kubernetes API objects and persistent volumes for disaster recovery, on a schedule, with restore as the whole point. Different job, different failure model, different retention story. It has no opinion about a file that appeared in an ephemeral scratch directory ninety seconds ago, and this driver has no opinion about restoring your cluster. Run both. ## Ease of setup Throughput and features get compared; the number of moving parts usually does not, and it is the one an operator lives with. Counted as objects to install and things that can be misconfigured, not as lines of YAML. | Approach | To install | Needs cluster permissions? | Per workload | | --- | --- | --- | --- | | **csi-s3-archiver** | One `CSIDriver`, a namespace, a DaemonSet | **None for the driver.** Its ServiceAccount has no Role at all | Six lines in the pod spec | | s3fs / goofys in a sidecar | Nothing cluster-wide | Usually `privileged` or `SYS_ADMIN` for FUSE | A sidecar, a shared volume, a lifecycle ordering problem | | CSI FUSE drivers (csi-s3, JuiceFS) | A CSIDriver, a DaemonSet, a controller, RBAC, a StorageClass, often a metadata service | RBAC for the provisioner; the node plugin is usually privileged | A PVC | | Mountpoint for Amazon S3 CSI | A DaemonSet, a controller, RBAC, IRSA setup | RBAC, plus an AWS IAM role per service account | A PVC | | S3 SDK in the application | Nothing | None | A dependency, credentials, retry logic, and a code change per application | | Fluent Bit / Vector | A DaemonSet, a ConfigMap, RBAC to read pod metadata | RBAC to list pods | Annotations or a parser entry | | Velero | A CRD set, a controller, RBAC, a BackupStorageLocation | Cluster-admin-shaped permissions | A Backup or Schedule resource | ### What "no RBAC" actually means It is unusual enough to be worth showing. This is the whole of the driver's access to the Kubernetes API: deploy/base/serviceaccount.yaml: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: csi-s3-archiver namespace: csi-s3-archiver # and that is the entire file: no Role, no RoleBinding, no ClusterRole ``` It works because kubelet pushes everything the driver needs into the CSI calls: The pod's name, namespace, UID and service account, and the contents of `nodePublishSecretRef`. Nothing has to be looked up, so nothing has to be granted. If you deploy the optional Controller for PVC-declared volumes, the RBAC that adds belongs to the upstream `external-provisioner` sidecar; the driver container in that pod still talks to nobody. ### The first volume, end to end From nothing to an archived file, assuming a bucket and credentials exist: three commands: ```bash kubectl apply -k deploy/base # 1. the driver kubectl create secret generic s3-creds \ # 2. credentials --from-literal=accessKeyId=... \ --from-literal=secretAccessKey=... kubectl apply -f my-pod.yaml # 3. a pod with six extra lines ``` There is no operator to install first, no CRDs to reconcile, no cert-manager dependency and no webhook to fail closed. The failure mode of a misconfigured volume is a pod event naming the attribute that is wrong, at pod start, rather than a bucket that quietly stays empty. ## When not to use this driver Stated plainly, because a comparison page that only lists wins is an advertisement: - **You need to read objects back.** There is no restore, ever. Use a FUSE mount or an SDK. - **You need shared storage between pods.** The volume is pod-local and dies with the pod. Use JuiceFS or a real network filesystem. - **You need the object to be up to date continuously.** Archiving is asynchronous. `appendOnly` narrows the window to `appendSyncSeconds`, but it is still a window. - **You need a PVC.** This is ephemeral inline volumes only: No StorageClass and no dynamic provisioning in v1. - **Your files are rewritten in place constantly.** Change detection is size + mtime, and every change means a full re-upload of that file. - **You need it today, in production, with a support contract.** There has been no tagged release yet. Where it does fit, which is write-once files that somebody may want later produced by applications that should not know about S3, it is a DaemonSet, an empty RBAC, and six lines of YAML in the pod spec. [Install it](https://s3archiver.csi.trion.de/install.html) · [Read the FAQ](https://s3archiver.csi.trion.de/faq.html) · [Configuration reference](https://s3archiver.csi.trion.de/docs.html) ============================================================================== Source: https://s3archiver.csi.trion.de/security.html Updated: 2026-08-30 Summary: What csi-s3-archiver does not have: No API-server access, no mount(2), no credentials on the node, no shell in the image. Written for regulated environments. Security # Security in regulated environments A CSI driver runs as root on every node in the cluster, which makes it worth reviewing carefully. Most of what follows is a list of things this one does not have: No Kubernetes API access, no `mount(2)`, no shell in the image, and in two of its four credential modes, no S3 credentials on the node at all. ## What is not there Most of this driver's security story is subtraction. Each row is a thing an auditor would otherwise have to assess, and the reason it is absent. | Not present | Why it is not needed | | --- | --- | | **Any Kubernetes API access**. The node plugin's ServiceAccount has no Role, no RoleBinding and no ClusterRole. | kubelet pushes pod metadata and per-volume secrets into the CSI calls. The driver never needs to ask the API server anything. | | **`privileged: true`**, added capabilities, and bidirectional mount propagation. | The volume is a directory, not a mount. There is no `mount(2)` anywhere in the driver, so there is nothing to be privileged for. | | **A writable root filesystem.** | `readOnlyRootFilesystem: true`. The driver writes only to the volume directories and its own state directory. | | **A shell, a package manager, or a libc** in the runtime image. | `FROM scratch` with a statically linked binary and a CA bundle. There is nothing to exec even after a compromise. | | **S3 credentials on the node**, in presigned and POST-policy modes. | A signer holds them elsewhere and issues per-object grants, or a policy authorises a prefix for a bounded time. | | **Inbound network listeners**, by default. | The CSI socket is a Unix domain socket. Metrics and the statistics page are off unless a port is set. | The driver runs as **root but not privileged**. Root is needed for exactly one thing: Creating and removing directories under `/var/lib/kubelet/pods`, which are root-owned. That is a smaller grant than it sounds, and it is the same one every CSI node plugin holds. ## Credential handling Four modes, in decreasing order of what the node is trusted with. | Mode | What the node holds | Suits | | --- | --- | --- | | **Default provider chain** | Nothing. IRSA, EKS Pod Identity or an instance profile supplies short-lived credentials. | AWS clusters with an identity story already in place | | **Presigned (signer)** | Nothing but a bearer token for the signer. Each upload gets a URL for *one object key*, valid for minutes. | Regulated environments: Every grant is auditable at the signer, and revocation is immediate | | **Presigned POST policy** | One signed policy, prefix-scoped, valid at most 7 days. No signer service to run. | Environments that want scoped grants without another deployment | | **Static keys** | An access key and secret, per volume or driver-wide. | Non-AWS S3 endpoints; the fallback when nothing else fits | **NOTE — Secrets never touch the disk** The driver's state files record what was uploaded, not how. Credentials are held in memory only, and the type that writes state is physically incapable of writing them because it is never handed any. Both properties are unit-tested, along with the rule that no `toString()`, log line or error message can contain secret material. Per-volume secrets arrive through `nodePublishSecretRef`, which kubelet reads from the pod's own namespace. A namespace can therefore only use credentials it already has, and the driver is not a way to escalate across namespaces. ## Supply chain | Control | Detail | | --- | --- | | **Keyless signatures** | Every published image and every architecture-specific tag is signed with cosign using the workflow's OIDC identity. No private key exists to leak or rotate. | | **Verifiable provenance** | The signature covers the digest, not the tag, so moving a tag afterwards does not carry the signature with it. | | **Reproducible install** | The release attaches the rendered YAML pinned to the released digest, so what you applied is what was signed. | | **Four runtime dependencies** | The AWS SDK, grpc-java, Jackson streaming, and slf4j. No dependency-injection container, no reflective frameworks, no plugin loading. | | **Nothing loaded at runtime** | A native image has no classpath to add to. There is no plugin mechanism, no scripting engine and no deserialisation of untrusted input. | verify before you deploy: ```bash cosign verify \ --certificate-identity-regexp "^https://github.com/.*/csi-s3-archiver/" \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ ghcr.io/trion-development/csi-s3-archiver:1.2.3 ``` ## The data path Archiving is **one-way and append-only** for an ephemeral volume: Files go to S3 and objects are never deleted. That is a property worth stating plainly in a compliance context, because it means the driver cannot be used to destroy evidence it has already archived. Two deliberate exceptions: - **Segment compaction** deletes the `.parts/` fragments it has just assembled into the canonical object. Nothing a user wrote is removed. - **Durable volumes** mirror deletions, because a volume that resurrects files its workload deleted is not durable. This is opt-in per StorageClass, and the durable class defaults to `reclaimPolicy: Retain` so deleting a claim cannot delete the archive. Other boundaries the driver holds: | Boundary | Enforcement | | --- | --- | | Symbolic links are never followed | A symlink inside a volume is skipped and logged. The driver runs as root, so following one would exfiltrate node files into the workload's bucket. | | Target paths are validated | An absolute path of at least two segments, refusing anything close to the filesystem root, and refusing a symlinked target. | | Object keys cannot escape the prefix | A restored key that resolves outside the volume is refused, which matters if the bucket is writable by anyone else. | | Over-long keys are skipped, never truncated | A truncated key would silently overwrite an unrelated object. | ## Reporting a vulnerability Report privately first, and give us a chance to ship a fix before the details are public. That applies to anything that lets a workload reach files outside its own volume, read credentials it was not given, or make the driver act on another namespace's behalf. | Channel | Use it for | | --- | --- | | **[security.txt](https://s3archiver.csi.trion.de/.well-known/security.txt)** | The address to send the report to, machine-readable and per [RFC 9116](https://www.rfc-editor.org/rfc/rfc9116.html). It is kept there rather than printed on this page, because an address in a web page is harvested within days and then the reports arrive buried in spam. | | **Bluesky**, [@triondev](https://bsky.app/profile/triondev.bsky.social) or [@everflux](https://bsky.app/profile/everflux.bsky.social) | Getting our attention, or asking how to reach us, when the address above bounces. Say that you have something to report; do not put the details in a public post or a DM. | | **[The contact form](https://www.trion.de/kontakt.html)** at TRION Development | Everything else, including whether commercial support exists for your cluster. | **What helps.** The driver version or image digest, the Kubernetes distribution and version, the `volumeAttributes` of the volume involved, and the smallest pod spec that shows the behaviour. Driver logs at `S3A_LOG_LEVEL=debug` are usually the fastest way to a diagnosis; they are designed to carry no secret material, and there is a test enforcing that, so they are safe to attach. **What happens next.** You get a human reply confirming whether we can reproduce it. A fix is released as a normal tagged version and named in the [release history](https://s3archiver.csi.trion.de/releases.html), with credit if you want it and without if you do not. There is no bounty programme. ## Your responsibilities The driver covers what is listed above. These remain yours. - **The statistics page has no authentication.** It is read-only and carries no secret material, but it does reveal object keys inside failure messages and per-node volume counts. It is off by default; keep it on a port-forward or behind an authenticating ingress. - **The reference signer is a reference.** One static bearer token, no TLS termination, no rate limiting, no audit log. It exists so presigned mode is testable and so you have a readable starting point, and the code says so where it matters. - **Bucket policy is yours.** The driver needs `PutObject`, and `ListBucket` plus `DeleteObject` only for segment compaction and durable volumes. Grant the narrowest set your configuration actually uses. - **Encryption is a bucket decision.** The driver can request SSE-S3 or SSE-KMS per volume, but a bucket default is what protects you from a volume that forgot to ask. - **No release has been tagged yet.** Until one is, there is no signed image to verify, and everything above describes the pipeline rather than an artefact you can download today. ============================================================================== Source: https://s3archiver.csi.trion.de/benchmarks.html Updated: 2026-08-30 Summary: Measured resource use for csi-s3-archiver, why the goofys benchmark does not apply to it, and how to reproduce every number on this page. Measurements # Benchmarks What csi-s3-archiver costs to run, measured rather than estimated, and an explanation of why the benchmark people usually ask for, against goofys and other FUSE filesystems, does not apply to it. ## Method **NOTE — Scope of these numbers** Every number on this page comes from a script in the repository, run on one machine, and every one of them is reproducible with one command. The head-to-head against s3fs runs both tools on the same host, against the same S3 endpoint, with the same writer. **It reports two columns, and the two tools win different ones**: Quoting only the flattering column is how benchmarks become marketing. The workload is synthetic and the harness drives the *real* engine: The same watcher, quiescence tracking, upload pipeline and object store a published volume gets. The S3 endpoint is SeaweedFS in a container on the same host, so the network is a loopback and the CPU figures are a floor, not a forecast for your endpoint. | Detail | Value | | --- | --- | | Host | 12-core x86-64, Linux | | Build under test | The native binary, statically linked with musl. Not the JVM build, whose memory profile is unrelated. | | S3 endpoint | SeaweedFS 4.40 in a container, loopback network | | Sampling | RSS and CPU from `/proc` every 200 ms; peak taken from `VmHWM`, the kernel's own high-water mark | ## Resource use under load | Workload shape | Peak RSS | CPU | Data | | --- | --- | --- | --- | | Idle (1 volume, 1 MiB) | 69 MiB | 0.02 s | 1 MiB | | Typical (4 volumes x 25 files x 1 MiB) | 128 MiB | 0.67 s | 100 MiB | | Many volumes (32 x 10 x 1 MiB) | 152 MiB | 4.96 s | 320 MiB | | One large file (2 x 512 MiB) | **156 MiB** | 13.7 s | 1 GiB | | With gzip (4 x 25 x 1 MiB) | 93 MiB | 0.58 s | 100 MiB | | Segments strategy (4 x 25 x 4 MiB) | 147 MiB | 7.47 s | 400 MiB | | Upload concurrency 16 | 138 MiB | 1.53 s | 100 MiB | ### Memory is flat in file size A 512 MiB file peaks no higher than thirty-two 1 MiB ones. Uploads stream from disk in fixed buffers and switch to multipart above a threshold, so there is no point at which the driver holds a file in memory. This is the property that lets a 200 GiB heapdump be archived by a container with a 256 MiB limit. What does move memory is **concurrency**, and `S3A_NODE_CONCURRENCY` bounds it. That is the knob to turn if you need a smaller limit, and the one to watch if you raise it. ### Compression is cheaper than it looks The gzip row uses *less* peak memory than the uncompressed one, because the compressed object is smaller and the multipart threshold is reached later. It costs roughly the same CPU. For log-shaped data, `compression: gzip` is close to free. ## Head to head against s3fs Same host, same SeaweedFS, same writer script, three arms. **local** is a plain directory with nothing uploaded and exists as the control: Without it, "the archiver beats s3fs on writes" cannot be falsified, because the archiver writes to local disk and of course it does. | Workload | Arm | Write | Durable | | --- | --- | --- | --- | | **200 x 256 KiB** | local | 133 ms | n/a | | | s3fs | 2027 ms | 2027 ms | | | **archiver** | 136 ms | 1143 ms | | **1 x 256 MiB** | local | 61 ms | n/a | | | s3fs | 723 ms | 723 ms | | | **archiver** | 59 ms | 2895 ms | | **64 MiB appended** | local | 64 ms | n/a | | | s3fs | 2197 ms | 2197 ms | | | **archiver** | 63 ms | 1287 ms | | **64 MiB, then exit** | local | 19 ms | n/a | | | s3fs | 234 ms | 234 ms | | | **archiver** | 22 ms | 1212 ms | ### Write time What the workload itself waits for. *Lower is better. The same numbers are in the table above.* The archiver arm finishes writing within a few milliseconds of the control in every shape, which is the structural claim of the design: The driver is not in the write path. s3fs costs the writer roughly 12 to 34 times the control, most at the append-heavy shape, which is what a log-producing pod looks like. ### Time until the bytes are in the bucket What you have left if the node dies. The control never uploads, so it does not appear. *Lower is better. The same numbers are in the table above.* **s3fs reaches durability sooner for one large file and for a single quick write.** The cause is a floor rather than throughput: This driver waits for a file to go quiescent, then sweeps at pod termination, so roughly a second passes before anything is durable no matter how small the workload. s3fs streams throughout and is finished when the writer is. Lowering `quiescenceSeconds` drops the floor, at the cost of uploading files that are still being written. **NOTE — Which column applies to you** If archiving is off the critical path, which is what this driver is designed for, the write column is the one you feel and the durable column is a background cost. If a write must be durable before the application continues, neither the number nor the tuning helps: You need the application to write to S3, or a mount that does. The [comparison page](https://s3archiver.csi.trion.de/compare.html) covers that case. ## The benchmark script The harness that produced every number above. It starts the S3 endpoint, builds an s3fs image, runs three arms over four workloads and writes the report. It installs nothing on the host and skips with a reason if `/dev/fuse` is unavailable. run it: ```bash ./build-native.sh # measure the shipped binary bench/head-to-head/run.sh --binary target/csi-s3-archiver ``` One writer serves every arm, so the workload is not a variable: writer.sh: ```bash #!/usr/bin/env bash # # The workload. ONE writer, used by every arm of the comparison, so the only thing that differs # between arms is what sits between this script and the bucket. # # writer.sh # # It prints one line to stdout when it is done: # # WRITER wall_ms= bytes= files= # # Nothing here is clever on purpose. dd with a fixed block size, no parallelism, no fsync games: # a benchmark whose writer is doing something subtle measures the writer. set -euo pipefail SHAPE="${1:?shape}" TARGET="${2:?target directory}" FILES="${3:?file count}" FILE_BYTES="${4:?bytes per file}" BLOCK=65536 COUNT=$(( FILE_BYTES / BLOCK )) [[ "$COUNT" -lt 1 ]] && COUNT=1 mkdir -p "$TARGET" start_ns() { date +%s%N; } START="$(start_ns)" WRITTEN=0 case "$SHAPE" in many-small|one-large) for i in $(seq 1 "$FILES"); do dd if=/dev/zero of="$TARGET/payload-$i.bin" bs="$BLOCK" count="$COUNT" \ conv=notrunc status=none WRITTEN=$(( WRITTEN + COUNT * BLOCK )) done ;; append-log) # A growing file, the shape this driver's appendOnly mode exists for. Appended in chunks # rather than one write, so a FUSE mount pays per flush the way a real logger makes it. for i in $(seq 1 "$FILES"); do dd if=/dev/zero bs="$BLOCK" count="$COUNT" status=none >> "$TARGET/app.log" WRITTEN=$(( WRITTEN + COUNT * BLOCK )) done ;; write-then-exit) # One file, written and then the writer is gone immediately. This is the case a sweep-based # design either handles or does not, and the one a preStop hook races with. dd if=/dev/zero of="$TARGET/dump.bin" bs="$BLOCK" count="$COUNT" status=none WRITTEN=$(( COUNT * BLOCK )) ;; *) echo "unknown shape: $SHAPE" >&2 exit 2 ;; esac # sync so the comparison is not measuring who left more in the page cache. For s3fs this is # where the upload actually happens, which is the point: its cost belongs to the writer. sync END="$(start_ns)" echo "WRITER wall_ms=$(( (END - START) / 1000000 )) bytes=$WRITTEN files=$FILES" ``` And the harness itself: run.sh: ```bash #!/usr/bin/env bash # # Head-to-head: csi-s3-archiver against s3fs, on one machine, against one S3 endpoint, with # one writer. # # bench/head-to-head/run.sh --binary target/csi-s3-archiver # # WHAT IS HELD CONSTANT. Every arm runs in a container on the same Docker network, writes with # the same writer.sh, and targets the same SeaweedFS. The local and archiver arms write to the # same bind-mounted host directory, so the disk underneath them is identical. The only variable # is what sits between the writer and the bucket. # # local a plain directory. Nothing is uploaded. The control. # s3fs an s3fs-fuse mount of the bucket. The writer's writes go to S3. # archiver a plain directory that csi-s3-archiver is archiving concurrently. # # WHY THE CONTROL ARM EXISTS. Without it, "the archiver is faster than s3fs" is unfalsifiable: # the archiver writes to local disk, so of course it is. The control says how much of the # difference is the disk and how much is the design. The archiver's write-path number should # land on top of the control's, and if it does not, the driver is in the write path somewhere # it should not be. # # TWO COLUMNS, NOT ONE. "Write time" and "time until every byte is in the bucket" are different # questions and the arms win different ones. Reporting only the first flatters this project; # reporting only the second flatters s3fs. Both are printed for every arm. # set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$REPO_ROOT" BINARY="" SEAWEED_IMAGE="${SEAWEED_IMAGE:-chrislusf/seaweedfs:4.40}" S3FS_IMAGE="csi-s3-archiver-bench-s3fs" S3FS_VERSION="" # read from the built image; see below BUCKET="headtohead" ACCESS_KEY="bench-key" SECRET_KEY="bench-secret" RESULTS="$HERE/results.md" KEEP=0 usage() { cat <<'EOF' Usage: bench/head-to-head/run.sh [options] Compares csi-s3-archiver against s3fs and against a plain local directory, on one machine, against one S3 endpoint, with one writer. Options: --binary PATH the native driver binary (default: target/csi-s3-archiver) --results FILE where to write the report (default: bench/head-to-head/results.md) --keep leave the containers running for inspection -h, --help this help Requires Docker and /dev/fuse. Nothing is installed on the host: s3fs runs in a pinned container image this script builds. EOF } while [[ $# -gt 0 ]]; do case "$1" in --binary) BINARY="$2"; shift 2 ;; --results) RESULTS="$2"; shift 2 ;; --keep) KEEP=1; shift ;; -h|--help) usage; exit 0 ;; *) echo "error: unknown option: $1" >&2; usage >&2; exit 2 ;; esac done BINARY="${BINARY:-$REPO_ROOT/target/csi-s3-archiver}" # --- preconditions ------------------------------------------------------------------------ command -v docker >/dev/null 2>&1 || { echo "error: docker is required" >&2; exit 1; } if [[ ! -x "$BINARY" ]]; then echo "error: $BINARY is not executable. Run ./build-native.sh first." >&2 echo " The comparison measures the binary that ships, not the JVM build." >&2 exit 1 fi # FUSE is the one thing that cannot be worked around from inside a container. Skip with a # reason rather than fail, the same way the e2e suite handles a kernel without vxlan. if [[ ! -c /dev/fuse ]]; then echo ">> SKIPPING: /dev/fuse is not available on this host, so s3fs cannot mount." >&2 echo ">> The archiver and local arms would still run, but a one-armed comparison is not" >&2 echo ">> a comparison. Run this on a host with the fuse module loaded." >&2 exit 0 fi NETWORK="h2h-net-$$" SEAWEED="h2h-seaweed-$$" WORK_DIR="$(mktemp -d /tmp/h2h-XXXXXX)" chmod 777 "$WORK_DIR" cleanup() { local status=$? if [[ "$KEEP" != 1 ]]; then docker rm -f "$SEAWEED" >/dev/null 2>&1 || true docker network rm "$NETWORK" >/dev/null 2>&1 || true rm -rf "$WORK_DIR" else echo ">> --keep: network $NETWORK, seaweed $SEAWEED, work dir $WORK_DIR" fi return $status } trap cleanup EXIT # --- the S3 endpoint ---------------------------------------------------------------------- # SeaweedFS answers any SigV4-signed request with 400 unless an identity is configured, and # both the AWS SDK and s3fs sign. Same fixture as the rest of this project's harnesses. cat >"$WORK_DIR/s3.json" <> building the s3fs image" docker build -q -f "$HERE/Dockerfile.s3fs" -t "$S3FS_IMAGE" "$HERE" >/dev/null # Reported rather than asserted: apt installs whatever the base image ships, so the honest # thing is to read the version back out of the image that will actually run. S3FS_VERSION="$(docker run --rm "$S3FS_IMAGE" 's3fs --version 2>&1 | head -1' \ | sed -n 's/.*V\([0-9.]*\).*/\1/p')" echo ">> s3fs-fuse $S3FS_VERSION" echo ">> starting SeaweedFS ($SEAWEED_IMAGE)" docker network create "$NETWORK" >/dev/null docker run -d --name "$SEAWEED" --network "$NETWORK" -P \ -v "$WORK_DIR/s3.json":/etc/s3.json:ro \ "$SEAWEED_IMAGE" server -s3 -dir=/data -ip=0.0.0.0 -s3.config=/etc/s3.json >/dev/null S3_PORT="$(docker port "$SEAWEED" 8333/tcp | head -1 | sed 's/.*://')" S3_HOST_URL="http://127.0.0.1:$S3_PORT" S3_NET_URL="http://$SEAWEED:8333" for _ in $(seq 1 120); do curl -s -o /dev/null --max-time 2 "$S3_HOST_URL/" 2>/dev/null && break sleep 1 done echo ">> endpoint: $S3_HOST_URL (in-network: $S3_NET_URL)" # s3fs refuses to mount a bucket that does not exist, and SeaweedFS creates one on the first # PUT. The driver's own upload check is the shortest way to make that PUT, and it doubles as a # proof that the endpoint and credentials work before any arm is timed. echo ">> priming the bucket" S3A_ENDPOINT="$S3_HOST_URL" S3A_BUCKET="$BUCKET" S3A_PATH_STYLE=true \ S3A_PREFIX="prime/" \ S3A_ACCESS_KEY_ID="$ACCESS_KEY" S3A_SECRET_ACCESS_KEY="$SECRET_KEY" \ "$BINARY" check-upload >"$WORK_DIR/prime.log" 2>&1 || { echo "error: could not reach the S3 endpoint with the driver:" >&2 sed 's/^/ | /' "$WORK_DIR/prime.log" >&2 exit 1 } # --- helpers ------------------------------------------------------------------------------ now_ms() { echo $(( $(date +%s%N) / 1000000 )); } # Every result row lands here as: shape arm write_ms durable_ms bytes peak_rss_kb : >"$WORK_DIR/rows.txt" record() { echo "$1 $2 $3 $4 $5 $6" >>"$WORK_DIR/rows.txt" printf ' %-16s %-9s write %7s ms durable %8s ms %6s MiB\n' \ "$1" "$2" "$3" "$4" "$(( $5 / 1048576 ))" } # Peak RSS of a pid, sampled until it exits. VmHWM is the kernel's own high-water mark, so # the peak is exact even between samples; the loop exists only to read it before the process # is gone, which matters because the sweep is the expensive part. sample_peak_rss() { local pid="$1" out="$2" echo 0 >"$out" while kill -0 "$pid" 2>/dev/null; do local hwm hwm="$(awk '/^VmHWM:/ {print $2}' "/proc/$pid/status" 2>/dev/null || true)" [[ -n "$hwm" ]] && echo "$hwm" >"$out" sleep 0.1 done } # --- the arms ----------------------------------------------------------------------------- # local: a plain directory. No upload, so "durable" is not applicable and is reported as -1. run_local() { local shape="$1" files="$2" bytes="$3" local dir="$WORK_DIR/local-$shape" mkdir -p "$dir" local out out="$(docker run --rm --network "$NETWORK" -v "$dir":/data "$S3FS_IMAGE" \ "/usr/local/bin/writer.sh $shape /data $files $bytes")" local write_ms bytes_written write_ms="$(sed -n 's/.*wall_ms=\([0-9]*\).*/\1/p' <<<"$out")" bytes_written="$(sed -n 's/.*bytes=\([0-9]*\).*/\1/p' <<<"$out")" record "$shape" "local" "$write_ms" "-1" "$bytes_written" "0" } # s3fs: the writer writes straight into the mount, so the upload is inside its wall time. run_s3fs() { local shape="$1" files="$2" bytes="$3" local prefix="s3fs-$shape" # -o use_path_request_style for a container-hosted endpoint with no per-bucket DNS, and # nocopyapi/nomultipart off: the defaults are what a user would get. local script=" echo '$ACCESS_KEY:$SECRET_KEY' > /etc/passwd-s3fs && chmod 600 /etc/passwd-s3fs mkdir -p /mnt/s3 s3fs $BUCKET /mnt/s3 \ -o url=$S3_NET_URL \ -o use_path_request_style \ -o passwd_file=/etc/passwd-s3fs \ -o dbglevel=crit \ -o allow_other mkdir -p /mnt/s3/$prefix /usr/local/bin/writer.sh $shape /mnt/s3/$prefix $files $bytes umount /mnt/s3 " local start out start="$(now_ms)" out="$(docker run --rm --network "$NETWORK" \ --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined \ "$S3FS_IMAGE" "$script")" local write_ms bytes_written write_ms="$(sed -n 's/.*wall_ms=\([0-9]*\).*/\1/p' <<<"$out")" bytes_written="$(sed -n 's/.*bytes=\([0-9]*\).*/\1/p' <<<"$out")" # For s3fs the bytes are in the bucket when the writer returns (the unmount above forces # any remainder), so durability costs nothing beyond the write. record "$shape" "s3fs" "$write_ms" "$write_ms" "$bytes_written" "0" } # archiver: the writer writes to a plain directory that the driver is archiving CONCURRENTLY, # which is what happens in a pod. The sweep at the end is pod termination. run_archiver() { local shape="$1" files="$2" bytes="$3" local dir="$WORK_DIR/arch-$shape" local sentinel="$WORK_DIR/done-$shape" local prefix="archiver-$shape/" mkdir -p "$dir" rm -f "$sentinel" local append_only="false" strategy="rewrite" if [[ "$shape" == "append-log" ]]; then append_only="true" strategy="segments" fi local start start="$(now_ms)" # The driver first, so archiving overlaps the writing exactly as it would in a pod. S3A_ENDPOINT="$S3_HOST_URL" \ S3A_BUCKET="$BUCKET" \ S3A_PATH_STYLE=true \ S3A_PREFIX="$prefix" \ S3A_ACCESS_KEY_ID="$ACCESS_KEY" \ S3A_SECRET_ACCESS_KEY="$SECRET_KEY" \ S3A_QUIESCENCE_SECONDS=1 \ S3A_BENCH_SOURCE_DIR="$dir" \ S3A_BENCH_UNTIL_FILE="$sentinel" \ S3A_BENCH_APPEND_ONLY="$append_only" \ S3A_BENCH_APPEND_STRATEGY="$strategy" \ "$BINARY" bench >"$WORK_DIR/arch-$shape.log" 2>&1 & local driver_pid=$! sample_peak_rss "$driver_pid" "$WORK_DIR/rss-$shape" & local sampler_pid=$! sleep 0.5 local out out="$(docker run --rm --network "$NETWORK" -v "$dir":/data "$S3FS_IMAGE" \ "/usr/local/bin/writer.sh $shape /data $files $bytes")" local write_ms bytes_written write_ms="$(sed -n 's/.*wall_ms=\([0-9]*\).*/\1/p' <<<"$out")" bytes_written="$(sed -n 's/.*bytes=\([0-9]*\).*/\1/p' <<<"$out")" # The writer is gone: this is the pod terminating, and the sweep is what unpublish does. touch "$sentinel" wait "$driver_pid" || { echo "error: the archiver arm failed for $shape:" >&2 sed 's/^/ | /' "$WORK_DIR/arch-$shape.log" >&2 exit 1 } local durable_ms=$(( $(now_ms) - start )) wait "$sampler_pid" 2>/dev/null || true local rss rss="$(cat "$WORK_DIR/rss-$shape" 2>/dev/null || echo 0)" record "$shape" "archiver" "$write_ms" "$durable_ms" "$bytes_written" "$rss" } # --- run ---------------------------------------------------------------------------------- # Shapes chosen to cover what the two designs actually differ on: per-file overhead, one big # transfer, an append pattern, and a writer that exits immediately. # shape files bytes-each SHAPES=( "many-small 200 262144" "one-large 1 268435456" "append-log 64 1048576" "write-then-exit 1 67108864" ) echo echo ">> running (one writer, three arms, same endpoint)" echo for spec in "${SHAPES[@]}"; do read -r shape files bytes <<<"$spec" echo " --- $shape ($files x $(( bytes / 1024 )) KiB) ---" run_local "$shape" "$files" "$bytes" run_s3fs "$shape" "$files" "$bytes" run_archiver "$shape" "$files" "$bytes" echo done # --- report ------------------------------------------------------------------------------- # The raw rows are kept beside the report: regenerating the prose should never require # re-measuring, and a reader can check the arithmetic. cp "$WORK_DIR/rows.txt" "$HERE/results.tsv" "$HERE/report.sh" "$HERE/results.tsv" "$S3FS_VERSION" "$SEAWEED_IMAGE" >"$RESULTS" echo ">> wrote $RESULTS (raw rows in $HERE/results.tsv)" ``` ## Why the goofys benchmark does not transfer The [goofys benchmark](https://github.com/kahing/goofys#benchmark) is the one people ask about, and it measures a FUSE filesystem: `ls` of a large directory, sequential read throughput, random read latency, file creation rate. Those are the right measurements for goofys, s3fs and Mountpoint for Amazon S3, and most of them measure something csi-s3-archiver **does not do**. | That benchmark measures | What this driver does | | --- | --- | | Read throughput from S3 | Nothing. An ephemeral volume never reads from S3. A durable one reads once, at pod start. | | Random-read latency | Nothing. There is no read path to be random about. | | `ls` on a large directory | A local directory listing: A syscall, not a request. | | File creation rate | The node's filesystem. The driver is not in the write path, as the table above shows. | | Sequential write to the mount | The one that does transfer, and it is the write column above. | goofys is also archived upstream, which is why the head-to-head above uses s3fs: It is maintained, packaged, and the mount people actually encounter. Worth knowing when reading the write column, though: **s3fs deliberately trades speed for POSIX completeness**, doing consistency work per operation that goofys skips. Part of that 12-to-34x is a win against that choice rather than against FUSE as such. ## What reaches the bucket Since throughput is not the differentiator, the honest axis is *reliability of delivery*: Given a pod that writes a file and then terminates, does the file arrive? | Approach | Survives a pod that exits immediately | Survives a node crash | Costs the write path | | --- | --- | --- | --- | | **csi-s3-archiver** | Yes: Unpublish blocks on a final sweep | Partly: Whatever was archived, plus anything recovered at the next boot | Nothing | | `preStop` hook | Racy: Bounded by the termination grace period | No | Nothing | | Sidecar + `emptyDir` | Racy: The sidecar may be killed first | No | Nothing | | S3 SDK in the application | Yes, if the application handles it | No | Yes, and the application owns the retries | | FUSE mount | Yes: The write already went to S3 | Yes | A round trip per write | csi-s3-archiver's row is deliberately not all-green. A node that loses power between the last upload and the sweep loses whatever had not been archived yet; the driver re-archives what is on disk when the node comes back, but a destroyed node destroys unarchived data. That is the trade for keeping the write path local, and it is why the [semantics section](https://s3archiver.csi.trion.de/operations.html#semantics) states the contract as at-least-once and eventual rather than synchronous. ## Reproduce it The script is in the repository and takes one argument that matters. reproduce: ```bash ./build-native.sh # measure what ships, not the JVM build hack/measure-resources.sh \ --binary target/csi-s3-archiver \ --sweep # all seven shapes; omit for one # or a single shape of your own hack/measure-resources.sh --binary target/csi-s3-archiver \ --volumes 8 --files 50 --file-bytes 4194304 --concurrency 8 ``` It starts a SeaweedFS container, runs the driver's `bench` subcommand against it, samples the process while it works, and prints a suggested `resources` block derived from the measured peak. Re-run it against your own workload shape before trusting the defaults: The numbers above describe one host and one endpoint, and your S3 latency moves the CPU figures more than the code does. ============================================================================== Source: https://s3archiver.csi.trion.de/screenshots.html Updated: 2026-08-30 Summary: The csi-s3-archiver statistics page, and short recordings of installing the driver, archiving a file, shipping a growing log and a rejected volume attribute. Screenshots # csi-s3-archiver in action What the driver looks like once it is running: The optional statistics page, and the command sequence from installing it to seeing a file appear in a bucket. ## The statistics page Set `S3A_WEBUI_PORT` and the driver serves a read-only page: Uploads per hour and per day, bytes archived, retries, permanent failures with the S3 error that caused them, and per-node queue depth and volume counts. It is off unless the port is set. [![Screenshot of the csi-s3-archiver statistics page rendering a three-node cluster: Headline counters for uploads, bytes archived, failures and retries across every node; a bar chart of uploads per hour over the last day; a table of the three nodes with their individual totals; and a table of recent upload failures showing the object key and the S3 error.](https://s3archiver.csi.trion.de/media/webui.png)](https://s3archiver.csi.trion.de/media/webui.png) *One pod rendering the whole cluster: Uploads per hour, all three nodes with their own totals, and recent failures with their cause.* Point `S3A_WEBUI_PEERS` at a headless Service and any pod renders the whole cluster: Each one fetches its siblings and merges by node id. Discovery is plain DNS rather than the Kubernetes API, which is what keeps the driver's ServiceAccount empty. The capture above is three drivers on one network, aggregated by the one serving the page, which is also what the end-to-end suite asserts against a two-node cluster. [![Screenshot of the csi-s3-archiver statistics page rendering a three-node cluster: Headline counters for uploads, bytes archived, failures and retries across every node; a bar chart of uploads per hour over the last day; a table of the three nodes with their individual totals; and a table of recent upload failures showing the object key and the S3 error.](https://s3archiver.csi.trion.de/media/webui-dark.png)](https://s3archiver.csi.trion.de/media/webui-dark.png) *The same page in dark mode. It follows the browser colour scheme.* **WARN — No authentication** The page is read-only and carries no secret material, but it does show object keys inside failure messages and per-node volume counts. Keep it on a `kubectl port-forward` or behind an authenticating ingress rather than exposing it. Alerting belongs on the Prometheus endpoint, which is built for it. ## Recordings Five short recordings of the command sequence: Installing the DaemonSet with kustomize, the same thing with Helm, archiving a file, shipping a growing log with `appendStrategy: segments`, and what a rejected attribute looks like. ![Terminal recording: kubectl apply -k deploy/base creates the CSIDriver, namespace, service account and DaemonSet; the rollout finishes across three nodes; kubectl get csinode shows s3archiver.csi.trion.de registered on every node.](https://s3archiver.csi.trion.de/media/install.gif) *Install: One kustomize apply, one DaemonSet, driver registered on every node. 13.7s loop, [still image](https://s3archiver.csi.trion.de/media/install.png)* ![Terminal recording: git clone fetches the repository because the chart is not published yet; kubectl creates a namespace and a credentials Secret from environment variables; helm install deploys the chart from deploy/helm/csi-s3-archiver with the bucket and the existing Secret set; the DaemonSet rolls out; helm list shows the release deployed.](https://s3archiver.csi.trion.de/media/helm.gif) *Helm: Clone, one Secret, one helm install, driver rolled out. 17.6s loop, [still image](https://s3archiver.csi.trion.de/media/helm.png)* ![Terminal recording: a pod writes a 4 MiB heapdump into its CSI volume; the driver log shows the file going quiescent and being uploaded; aws s3 ls lists the finished object under heapdumps/default/heapdump-example/ in the bucket.](https://s3archiver.csi.trion.de/media/archive.gif) *Archive: Write a file, watch it appear in the bucket. 14.4s loop, [still image](https://s3archiver.csi.trion.de/media/archive.png)* ![Terminal recording: an appendOnly volume using appendStrategy segments uploads tail segments under a .parts prefix while the log keeps growing, then compacts them into a single object at the canonical key when the pod terminates.](https://s3archiver.csi.trion.de/media/appendonly.gif) *Append-only: Tail segments during the run, one clean object at the end. 16.3s loop, [still image](https://s3archiver.csi.trion.de/media/appendonly.png)* ![Terminal recording: a pod with a misspelled prefix placeholder stays in ContainerCreating, and kubectl describe pod shows the driver returning INVALID_ARGUMENT with a message naming the bad placeholder and listing the accepted ones.](https://s3archiver.csi.trion.de/media/error.gif) *Misconfiguration: The driver's own message, on the pod's own events. 10.6s loop, [still image](https://s3archiver.csi.trion.de/media/error.png)* These are scripted reproductions rendered from a checked-in script, not captures of a live cluster: The commands, the log format and the error text all come from the code, but no cluster was recorded. They will be replaced with real captures once the end-to-end suite runs a cluster on every build. ============================================================================== Source: https://s3archiver.csi.trion.de/architecture.html Updated: 2026-08-30 Summary: How csi-s3-archiver is deployed and how it is built: The DaemonSet, the optional Controller, and the classes inside the process with their responsibilities. Architecture # How it is put together One binary, two deployment shapes, and about seventy classes with no framework between them. What runs where, and which class is responsible for what. ## One binary, two deployments The driver is a single native binary with a few subcommands. Everything below is that binary running in a different mode, or not running at all: The base install has no controller, no operator, no CRDs and no webhook. | Deployment | What runs | When you need it | | --- | --- | --- | | [**DaemonSet**](#daemonset) (the base install) | One pod per node, three containers, `csi-s3-archiver serve` | Always. This is the whole product for inline `csi:` volumes. | | [**Controller**](#controller) (optional) | One Deployment, `csi-s3-archiver controller` beside the upstream provisioner | Only if your workloads declare volumes with a `volumeClaimTemplate` rather than inline. | ## The DaemonSet Figure: The DaemonSet deployment — On every node, a three-container DaemonSet pod runs the driver, the node-driver-registrar and the livenessprobe. kubelet calls NodePublishVolume over a Unix socket; the driver creates a plain directory under the kubelet pods directory and watches it. The workload pod writes files there, and the driver uploads them to the S3 bucket over HTTPS. The only cluster-scoped object is the CSIDriver. Nothing in the driver talks to the Kubernetes API server. The base install. Two of the three containers are upstream sidecars; the third is this driver. What kubelet asks for and what it gets back is the whole contract. A CSI driver must make the volume readable and writable at `target_path` by the time `NodePublishVolume` returns, and a directory satisfies that as completely as a mount does. Serving one instead lets the DaemonSet drop `privileged: true` and bidirectional mount propagation. Everything else on this page follows from that single decision. | Object | Scope | What it is for | | --- | --- | --- | | `CSIDriver` | cluster | `attachRequired: false` (there is nothing to attach), `podInfoOnMount: true` (which is how the driver learns the namespace, pod name and service account without an API call), `volumeLifecycleModes: [Ephemeral]`, and `requiresRepublish: true` so a rotated Secret reaches running pods. | | `DaemonSet` | namespace | The driver plus `node-driver-registrar`, which tells kubelet where the socket is, and `livenessprobe`, which turns the CSI `Probe` RPC into a container probe on port 9808. | | `ServiceAccount` | namespace | With no Role, no RoleBinding and no ClusterRole. It exists because a pod needs one, not because the driver uses it. | | `Namespace` | cluster | Somewhere to put the two above. | Three host paths are mounted, and no others. `/var/lib/kubelet/plugins/s3archiver.csi.trion.de` holds the socket and the driver's state directory, `/var/lib/kubelet/plugins_registry` is where kubelet expects plugins to register, and `/var/lib/kubelet/pods` is where the volume directories themselves live. A distribution that moves kubelet's root needs those three paths changed and nothing else; the [distribution pages](https://s3archiver.csi.trion.de/install.html#distributions) name the handful that move it. **NOTE — Why the state directory is on the host** It has to outlive the driver container. After a restart, or a node reboot, the driver replays it to work out which volumes it is responsible for and what it has already uploaded. Without it, a restart would either re-upload everything or forget the volumes entirely. It holds no credentials, by construction. ## With the Controller Figure: The Controller variant, for PVC-declared volumes — An optional Deployment in the control plane runs the same binary in controller mode beside the upstream csi-provisioner sidecar. A PVC from a volumeClaimTemplate, and the StorageClass naming the driver, reach the provisioner, which calls CreateVolume over a Unix socket. Nothing on the node changes, and kubelet still publishes the volume as a directory that the DaemonSet archives to S3. The RBAC belongs to the provisioner sidecar, not to the driver. The optional overlay, for teams whose tooling is built around PVCs. The node side is untouched. An inline `csi:` volume needs no controller at all, because kubelet has everything it needs in the pod spec. A PVC does, because something has to answer `CreateVolume` before the scheduler will bind the claim. That something is the same binary in `controller` mode, beside the upstream `csi-provisioner`. Three things to know before you turn it on. | Worth knowing | Detail | | --- | --- | | **The volume is still pod-lifetime** | Unless you set `durable`. The directory is created at publish, archived at unpublish and removed. A generic ephemeral volume is exactly that, so the StorageClass is named for what it gives you rather than for the driver. | | **The RBAC is the sidecar's** | The ClusterRole that reads PVCs and writes PVs belongs to `csi-provisioner`. The driver container in the same pod still holds nothing, and the node plugin is unchanged. | | **Turning it on edits an immutable field** | `CSIDriver.spec.volumeLifecycleModes` gains `Persistent`, and that field cannot be updated. Delete the `CSIDriver` object and let the apply recreate it; no running volume notices, because kubelet reads it at publish time only. | With `durable: "true"` the same shape gains a restore: The volume's contents live in S3 between pods, and are copied back into a fresh directory before the pod starts. That is the one path in the driver that reads from a bucket. The [configuration reference](https://s3archiver.csi.trion.de/configuration.html) lists the constraints, starting with a pod start that blocks on the download. ## Inside the process Figure: The driver's internal structure — Main picks a subcommand and Driver wires the object graph by hand. Three groups of classes follow. The grpc package terminates the Unix-socket gRPC server and the CSI services. The config and volume packages resolve a volume's effective configuration and credentials and persist its state and upload manifest. The archive and s3 packages watch the directories, decide what to upload and when, and run the uploads with bounded concurrency against one of three object stores. Logging, metrics and the web UI sit beside all of it. Roughly 70 classes, no framework. These are the ones worth knowing first. Two decisions shape the rest. Everything is wired by hand, because a native image has to be told about every reflective lookup and a container generates them in bulk. Blocking work runs on virtual threads in plain blocking style, so an upload waiting on S3 costs a stack rather than a callback chain. | Class | Responsibility | What it actually does | | --- | --- | --- | | `Main`, `Driver` | entry point and wiring | One binary, several subcommands (`serve`, `controller`, `presign-signer`, `bench`, `check-upload`). `Driver` constructs the object graph by hand. There is no DI framework. The graph is a dozen singletons, and a container would only add reflection for the native image to configure. | | `UnixSocketGrpcServer` | transport | gRPC over a Unix domain socket, on Netty's NIO channel rather than the native epoll transport. No per-architecture shared library ships, so the native-image configuration is identical on amd64 and arm64. Removes a stale socket file left by an unclean shutdown, and shuts down gracefully on SIGTERM. | | `IdentityService` | who this driver is | `GetPluginInfo`, `GetPluginCapabilities` and `Probe`. Probe reports ready only after the state directory has been replayed, so kubelet does not publish into a driver that has not yet remembered its volumes. | | `NodeService` | the volume lifecycle | Validates, creates the directory at `target_path`, registers the volume, and on unpublish runs the final sweep before deleting anything. Its validation messages are the user's error channel, because kubelet turns them into pod events. | | `ControllerService` | PVC-declared volumes | Served only when the driver runs as `controller`, beside the provisioner. It allocates nothing. `CreateVolume` validates the StorageClass parameters and hands back a volume id, so a bad StorageClass fails on the PVC and never reaches a pod. | | `VolumeConfigResolver` | effective configuration | Built-in defaults, then `S3A_*` environment, then the pod's `volumeAttributes`. Unknown attributes and malformed values are rejected rather than ignored, so a typo fails the pod instead of producing an empty bucket. | | `CredentialResolver` | how a volume authenticates | The four modes, in order. Presigned, then the per-volume Secret, then driver-global keys, then the AWS default chain. Credentials live in memory only, and their `toString()` is redacted. | | `VolumeRegistry`, `VolumeStateStore` | what survives a restart | The registry is the in-memory truth while the process lives; the store writes each volume's record and upload manifest to the state directory atomically, before the mutation counts as done. After a SIGKILL the files are the truth, and replaying them keeps a restart from re-uploading everything. | | `S3ArchiverEngine` | the clock | Two ticks. A one-second tick evaluates quiescence and append pacing, finer than the shortest window a user can configure; a slow rescan re-reads every volume from disk, which is the safety net for inotify events the kernel dropped. | | `FileWatcher` | noticing writes | One `WatchService` for every volume on the node. inotify *instances* are a limited per-user resource, 128 by default. One instance with many watches keeps the driver from putting a ceiling on how many archived pods a node can host. | | `VolumeArchiver` | what to upload, and when | One per volume. Holds the quiescence window, the append strategy, the segment bookkeeping and the manifest comparison. All of it is decided in a tick method a test can call with a fake clock. None of those tests sleep. | | `UploadPipeline` | how uploads run | Bounded concurrency across the node, per-file serialisation so two uploads of one file cannot race, and capped exponential backoff with jitter that retries indefinitely. Virtual threads throughout, in blocking style. | | `ObjectStore` | where the bytes go | A deliberately narrow interface. The driver writes, and for segment compaction lists and deletes, but never reads objects back. The presigned implementation has no S3 API at all and still satisfies the same contract as the SDK-backed one. | | `VolumeRestore` | durable volumes | The one path that reads from S3. A volume marked `durable` is restored into the fresh directory before publish returns, so a pod start can block on a download. `restoreTimeoutSeconds` bounds the wait. | | `PresignSignerServer` | the reference signer | The same binary in `presign-signer` mode: A JDK HTTP server in front of the AWS SDK's presigner. The integration and end-to-end tests sign against it. Treat it as a starting point for your own, not as a hardened service. | The packages map straight onto directories under `src/main/java/de/trion/csi/s3archiver/`, and the class names above are the file names, so the diagram is also a map of the source tree. ## What one publish call does The order matters more than it looks. Most of it is about failing before anything exists on disk. 1. **Validate the request.** Volume id, target path, access mode, and the ephemeral flag kubelet sets. A PV-backed volume is refused here. 2. **Resolve the configuration and the credentials.** Both before any directory is created, so a misconfigured volume fails the pod at start rather than becoming a bucket that quietly stays empty. The failure text is written for whoever wrote the pod spec, because that is who reads the event. 3. **Check for a republish.** Same volume, same configuration is a cheap no-op that refreshes only the credentials. That is the route a rotated Secret takes to a running pod. Same volume, different configuration is `ALREADY_EXISTS`. 4. **Create the directory** at `target_path`, mode 0777, and refuse to follow a symlink there. 5. **Register the volume.** The state file is written before the call returns, and the watcher and the engine pick the directory up. A durable volume is restored from S3 here, the only step that can take minutes. Unpublish runs in reverse and blocks: The final sweep uploads everything new or changed, ignoring the quiescence window, and only then is the directory deleted and the state record dropped. The wait is deliberate. [`sweepTimeoutSeconds`](https://s3archiver.csi.trion.de/operations.html#final-sweep) is there for workloads that would rather not have it. ## Where next - [Security](https://s3archiver.csi.trion.de/security.html): The same architecture argued by subtraction, for an audit. - [Operations](https://s3archiver.csi.trion.de/operations.html): What the semantics above mean once it is running, and what to size the container for. - [Why Java](https://s3archiver.csi.trion.de/why-java.html): Virtual threads, the native image, and what each of them costs here. - [The source](https://github.com/trion-development/csi-s3-archiver): Every decision on this page is written up in full there, next to the code it produced. ============================================================================== Source: https://s3archiver.csi.trion.de/configuration.html Updated: 2026-08-30 Summary: Every environment variable and volumeAttribute csi-s3-archiver accepts, the prefix placeholders, and how to choose an append strategy. Reference # Configuration Everything the driver reads: The `S3A_*` variables on its container, the `volumeAttributes` on a volume, and the rules for combining them. ## Environment (driver container) | Variable | Default | Meaning | | --- | --- | --- | | `CSI_ENDPOINT` | `unix:///csi/csi.sock` | Socket the driver listens on. Unix domain sockets only; a `tcp://` value is rejected at startup. | | `S3A_STATE_DIR` | `/var/lib/kubelet/plugins/…/state` | Where per-volume state records and upload manifests live. Must be on a host path that outlives the driver container, or a restart re-uploads everything. | | `NODE_NAME` | hostname | The node id reported by `NodeGetInfo`. Set it from the downward API; the hostname fallback exists for local runs and logs a warning. | | `S3A_LOG_LEVEL` | `info` | Global log threshold: `trace`, `debug`, `info`, `warn`, `error`, `off`. | | `S3A_LOG_LEVELS` | unset | Per-logger overrides, comma-separated `prefix=level` (for example `io.grpc=warn,de.trion.csi=debug`). Longest matching prefix wins. | | `S3A_NODE_CONCURRENCY` | `4` | Concurrent uploads across all volumes on the node. A file is never uploaded concurrently with itself. | | `S3A_RESCAN_SECONDS` | `60` | Interval of the full rescan that backs up the inotify watcher. | | `S3A_LOG_FORMAT` | `text` | `text` for one human-readable line per record, `json` for one JSON object per line. JSON carries the full logger name and renders a throwable as fields, because a record that spans lines is not a JSON record. | | `S3A_METRICS_PORT` | unset (off) | Serve Prometheus metrics at `/metrics`. Off by default: A node plugin running as root should not open a listening port because someone upgraded it. Not 9808, which the `livenessprobe` sidecar already holds. | | `S3A_WEBUI_PORT` | unset (off) | Serve the read-only statistics page. Setting the port is what enables it; there is no separate switch. | | `S3A_WEBUI_RETENTION_DAYS` | `7` | How much hourly history each node keeps for the statistics page. 1 to 90. | | `S3A_WEBUI_PEERS` | unset (this node only) | Bare DNS name of a headless Service resolving to every driver pod, so any pod can render the whole cluster. Plain DNS rather than the Kubernetes API, which is what keeps the driver without permissions. | | `S3A_CONTROLLER` | `false` | Also serve the Controller service on the node socket. For `csi-sanity`; in a cluster the controller runs in its own pod. | ## volumeAttributes Set on the pod's inline CSI volume. Every attribute is validated at publish time: An unknown name, a malformed value or a contradictory combination fails the pod with a message on the pod's events, rather than producing a bucket that quietly stays empty. | Attribute | Default (env override) | Meaning | | --- | --- | --- | | `bucket` | required (`S3A_BUCKET`) | Target bucket, from either place. | | `prefix` | `{namespace}/{podName}/` (`S3A_PREFIX`) | Key prefix template. See placeholders below. | | `endpoint` | AWS (`S3A_ENDPOINT`) | S3 endpoint override for Ceph, SeaweedFS and the like. | | `region` | `us-east-1` (`S3A_REGION`) | Region. | | `pathStyle` | `false` (`S3A_PATH_STYLE`) | Force path-style addressing; most non-AWS stores need it. | | `appendOnly` | `false` (`S3A_APPEND_ONLY`) | Periodically sync growing files instead of waiting for them to go quiet. | | `deleteAfterUpload` | `false` (`S3A_DELETE_AFTER_UPLOAD`) | Delete the local file after a successful upload. Mutually exclusive with `appendOnly`. | | `quiescenceSeconds` | `30` (`S3A_QUIESCENCE_SECONDS`) | How long a file must be unmodified before it is uploaded. | | `appendSyncSeconds` | `60` (`S3A_APPEND_SYNC_SECONDS`) | Sync interval in `appendOnly` mode. | | `appendStrategy` | `rewrite` (`S3A_APPEND_STRATEGY`) | `rewrite` re-uploads the whole file per sync; `segments` uploads only new tail bytes and compacts at pod end. | | `segmentTargetBytes` | `8388608` (`S3A_SEGMENT_TARGET_BYTES`) | `segments`: Flush the pending tail at this size. | | `include` / `exclude` | all / none (`S3A_INCLUDE`, `S3A_EXCLUDE`) | Comma-separated glob filters on paths inside the volume. | | `storageClass` | bucket default (`S3A_STORAGE_CLASS`) | For example `STANDARD_IA`. | | `sweepTimeoutSeconds` | unlimited (`S3A_SWEEP_TIMEOUT_SECONDS`) | Bound for the final sweep at pod termination. | | `presignEndpoint` | unset (`S3A_PRESIGN_ENDPOINT`) | Signer service URL; switches the volume to presigned-upload mode. | | `presignMultipart` | `false` (`S3A_PRESIGN_MULTIPART`) | Presigned mode: Upload large files in parts, lifting the 5 GiB single-PUT cap. Needs a signer that signs the multipart operations, and is opt-in because a `putObject`-only signer answers them plausibly and wrongly. | | `compression` | `none` (`S3A_COMPRESSION`) | `gzip` compresses each object and appends `.gz` to the key. `zstd` is refused with a reason: It needs a per-architecture native library, which a static binary does not have. | | `serverSideEncryption` | `none` (`S3A_SERVER_SIDE_ENCRYPTION`) | `AES256` or `aws:kms`, with `sseKmsKeyId`. A KMS key without KMS mode is a hard error rather than a silently ignored setting. | | `sseKmsKeyId` | none (`S3A_SSE_KMS_KEY_ID`) | The KMS key id or ARN to encrypt with. Only meaningful alongside `serverSideEncryption: aws:kms`; setting it without that mode is a hard error rather than a silently ignored value. | | `sizeLimitBytes` | unlimited (`S3A_SIZE_LIMIT_BYTES`) | Reported as the volume capacity by `NodeGetVolumeStats`, and an overrun is logged. **Not enforced by deletion**: The driver will not delete a workload's files to stay under a limit. | | `appendUpload` | `auto` (`S3A_APPEND_UPLOAD`) | How a growing object is extended: `copy` uses `UploadPartCopy`, `offset` uses `x-amz-write-offset-bytes` and needs an S3 Express directory bucket, `off` re-uploads the whole file. Only in `appendOnly` mode. | | `durable` | `false` (`S3A_DURABLE`) | The volume outlives its pod: Restored from the archive at publish, archived and mirrored at unpublish. PVC-declared volumes only, and credential mode only. | | `restoreTimeoutSeconds` | unlimited (`S3A_RESTORE_TIMEOUT_SECONDS`) | Bounds how long a pod start may block on a durable restore. Past it the pod gets what arrived and the driver logs that the volume is incomplete. | `appendSyncSeconds`, `appendStrategy` and `segmentTargetBytes` only mean anything with `appendOnly: "true"`, so setting one on a volume without it is rejected. As a driver-global `S3A_*` default they are fine: They simply apply to the volumes that do use `appendOnly`. ## Prefix placeholders `prefix` may contain `{namespace}`, `{podName}`, `{podUid}`, `{serviceAccount}`, `{nodeName}`, `{volumeId}` and `{date}` (UTC `yyyy-MM-dd` at upload time). Anything else is an error rather than a literal: a typo, caught at publish time: ```sh $ kubectl describe pod heapdump-example Events: Type Reason Message ---- ------ ------- Warning FailedMount MountVolume.SetUp failed for volume "dumps" : rpc error: code = InvalidArgument desc = volumeAttributes: unknown placeholder "{namesapce}" in prefix; known placeholders are {namespace}, {podName}, {podUid}, {serviceAccount}, {nodeName}, {volumeId}, {date} ``` Keys are capped at S3's 1024 UTF-8 bytes. A file whose key would exceed that is skipped and logged, never silently truncated, because a truncated key would overwrite an unrelated object. ## Choosing an append strategy `rewrite` re-uploads the whole file every sync: The object always equals the file and only `PutObject` is needed, but transfer grows quadratically with file size. `segments` uploads only new tail bytes as immutable objects under `{key}.parts/`: Transfer is linear and data reaches S3 promptly, at the cost of a mid-flight reader needing to concatenate segments in sequence order. Both end up as one clean object at the canonical key after the pod terminates, because the sweep compacts. Rule of thumb: **`rewrite` for files that stay small, `segments` for a log that runs for hours.** | | `rewrite` | `segments` | | --- | --- | --- | | Transfer for an *n*-byte file | O(*n*²) across syncs | O(*n*) | | S3 operations | `PutObject` only | `PutObject` plus `DeleteObject` at compaction | | Object during the run | always a complete file | segments under `{key}.parts/` | | Object after termination | complete file | complete file (compacted) | | Reading mid-flight | just read the key | concatenate a generation's segments in `seq` order | Before compaction the data lives at `{key}.parts/{generation}/{seq}-{startOffset}`. A new generation appears when the file is truncated or rotated. ============================================================================== Source: https://s3archiver.csi.trion.de/credentials.html Updated: 2026-08-30 Summary: How csi-s3-archiver resolves credentials per volume: The default provider chain, per-volume secrets, presigned mode and static POST policies. Reference # Credentials Four ways a volume can authenticate, resolved in a fixed order, and the rules that stop a half-configured secret from failing quietly. ## Credentials Resolved per volume, in this order. The first that applies wins. 1. **Presigned mode**, when `presignEndpoint` is set. The node holds no S3 credentials at all; the driver asks a signer service for a fresh URL per upload. Combining it with static credential keys in the same secret is rejected, since that would ship to the node exactly what the mode exists to avoid. 2. **Per-volume secret** via `nodePublishSecretRef`, with keys `accessKeyId`, `secretAccessKey`, optional `sessionToken`, or `presignToken` for presigned mode. 3. **Driver-global** `S3A_ACCESS_KEY_ID`, `S3A_SECRET_ACCESS_KEY` and `S3A_SESSION_TOKEN`. 4. **AWS default provider chain**, which is what makes IRSA, EKS Pod Identity and node instance profiles work with no driver configuration at all. per-volume credentials: ```yaml apiVersion: v1 kind: Secret metadata: name: s3-creds namespace: team-a stringData: accessKeyId: AKIAEXAMPLE secretAccessKey: "…" --- # in the pod spec csi: driver: s3archiver.csi.trion.de volumeAttributes: bucket: prod-dumps nodePublishSecretRef: name: s3-creds ``` The repository ships a **reference signer** for presigned mode, run as `csi-s3-archiver presign-signer`. It reads `S3A_PRESIGN_PORT`, `S3A_PRESIGN_TOKEN` (the bearer token the driver must present) and `S3A_PRESIGN_EXPIRY_SECONDS`, and it holds the S3 credentials the nodes do not. It is a starting point rather than a product: One static token, no TLS termination, no rate limiting and no audit log. [The security page](https://s3archiver.csi.trion.de/security.html#your-work) says what you would need to add. **NOTE — Why the field is called nodePublishSecretRef** That is the fixed upstream Kubernetes field name, not this project's naming. It means "the Secret kubelet passes into the `NodePublishVolume` RPC". The Secret lives in the pod's own namespace. **GOOD — Secrets never reach disk** Credentials are held in memory only. They are never written to the state files on the node, and `toString()` on the resolved credentials is redacted. Both properties are unit-tested, and a log-capture test fails the build if secret material appears in any log line. ============================================================================== Source: https://s3archiver.csi.trion.de/install.html Updated: 2026-08-30 Summary: Install csi-s3-archiver on a cluster: One kustomize apply, credentials, a first pod that archives a file, and how to verify it worked. Install # Install csi-s3-archiver One DaemonSet, no RBAC, no operator. From an empty cluster to a file in a bucket in four steps. ## Before you start You need a Linux Kubernetes cluster you can install a DaemonSet into, and `kubectl`. That is the whole list. There is no operator to install first, no CRDs, no cert-manager, and no API-server access for the driver to negotiate. | Requirement | Detail | | --- | --- | | Nodes | Linux, amd64 or arm64. Windows is a non-goal. | | Permissions | Enough to create a namespace, a `CSIDriver` object and a DaemonSet. The driver itself needs no RBAC. | | Kubelet root | The default `/var/lib/kubelet`. On a cluster that moved it, patch the host paths in the DaemonSet. | | An S3 bucket | AWS, Ceph RGW, SeaweedFS: Anything speaking the S3 API. Credentials can also come from IRSA or an instance profile. | **WARN — Pre-release software** There is no tagged release yet, so there is no published image to pull. Build one from the sources you were given and push it somewhere your cluster can reach. Everything on this page works today; see [project status](https://s3archiver.csi.trion.de/index.html#status) for what does not, and [the FAQ](https://s3archiver.csi.trion.de/faq.html#is-this-ready-for-production) for what is tested. ## 1. Install the driver install: ```sh $ kubectl apply -k deploy/base $ kubectl -n csi-s3-archiver rollout status daemonset/csi-s3-archiver $ kubectl get csinode -o jsonpath='{.items[*].spec.drivers[*].name}' s3archiver.csi.trion.de ``` ![Terminal recording: kubectl apply -k deploy/base creates the CSIDriver, namespace, service account and DaemonSet; the rollout finishes across three nodes; kubectl get csinode shows s3archiver.csi.trion.de registered on every node.](https://s3archiver.csi.trion.de/media/install.gif) *Install: One kustomize apply, one DaemonSet, driver registered on every node. 13.7s loop, [still image](https://s3archiver.csi.trion.de/media/install.png)* That applies one `CSIDriver` object, a namespace with an **RBAC-less** ServiceAccount, and a DaemonSet running the driver alongside the upstream `node-driver-registrar` and `livenessprobe` sidecars ([diagram](https://s3archiver.csi.trion.de/architecture.html#daemonset)). There are no Roles or bindings anywhere: The driver never talks to the API server, because kubelet pushes pod metadata and per-volume secrets into the CSI calls. The driver container runs as **root but not privileged**, with no added capabilities and a read-only root filesystem. Root is needed only to create and remove directories under `/var/lib/kubelet/pods`, which are root-owned. Because the volume is a plain directory rather than a mount, there is no `mount(2)` in this driver, and therefore no `privileged: true` and no bidirectional mount propagation. ## 1, with Helm instead The chart under `deploy/helm/csi-s3-archiver` installs exactly what the kustomize base installs: The `CSIDriver`, the ServiceAccount with no RBAC, the DaemonSet and its two sidecars. Nothing on the rest of this page changes if you use it. Pick one of the two, not both, or the second install fights the first over the cluster-scoped `CSIDriver` object. install with Helm: ```sh $ git clone https://github.com/trion-development/csi-s3-archiver.git $ helm install csi-s3-archiver csi-s3-archiver/deploy/helm/csi-s3-archiver \ --namespace csi-s3-archiver --create-namespace \ --set config.bucket=prod-dumps NAME: csi-s3-archiver NAMESPACE: csi-s3-archiver STATUS: deployed REVISION: 1 $ kubectl -n csi-s3-archiver rollout status daemonset/csi-s3-archiver daemon set "csi-s3-archiver" successfully rolled out ``` ![Terminal recording: git clone fetches the repository because the chart is not published yet; kubectl creates a namespace and a credentials Secret from environment variables; helm install deploys the chart from deploy/helm/csi-s3-archiver with the bucket and the existing Secret set; the DaemonSet rolls out; helm list shows the release deployed.](https://s3archiver.csi.trion.de/media/helm.gif) *Helm: Clone, one Secret, one helm install, driver rolled out. 17.6s loop, [still image](https://s3archiver.csi.trion.de/media/helm.png)* The clone is there because the chart is not on a chart repository yet, for the same reason there is no published image: Nothing has been tagged. Once it is, those two commands become a `helm repo add` and a `helm install`, and every value below keeps its name. The values worth knowing on a first install. `helm show values deploy/helm/csi-s3-archiver` prints all of them with the comments that explain the rest. | Value | Default | What it does | | --- | --- | --- | | `config.bucket` | *empty* | The driver-global bucket. A volume can override it, so this is a default rather than a requirement, but an install with neither archives nothing. | | `config.endpoint`, `config.region`, `config.pathStyle` | AWS, `us-east-1`, `false` | Point it at Ceph, SeaweedFS or any other S3 API. Most non-AWS stores want `pathStyle: true`. | | `config.prefix` | *empty* | Key prefix, with the [placeholders](https://s3archiver.csi.trion.de/configuration.html#prefix-placeholders) the reference lists. | | `credentials.existingSecret` | *empty* | A Secret you manage, with `S3A_ACCESS_KEY_ID` and friends as keys. Leave it empty on EKS and the AWS default chain takes over, which is how IRSA works with no configuration. | | `credentials.create` | `false` | Have the chart create that Secret from values instead. Convenient, and it puts the key into the release. The callout below is about that trade. | | `image.repository`, `image.tag` | GHCR, chart `appVersion` | Point at your own registry while there is no published image. | | `kubeletDir` | `/var/lib/kubelet` | Only [MicroK8s and k0s](#distributions) move it. k3s and RKE2 do not, despite the folklore. | | `metrics.enabled`, `metrics.serviceMonitor.enabled` | `false` | A `/metrics` endpoint on port 9809, and a ServiceMonitor for the Prometheus Operator. | | `webui.enabled` | `false` | The read-only [statistics page](https://s3archiver.csi.trion.de/operations.html). It has no authentication, so reach it with a port-forward or put an authenticating ingress in front. | | `controller.enabled` | `false` | The optional Controller, for volumes declared with a `volumeClaimTemplate` rather than an inline `csi:` block. | | `resources`, `tolerations`, `priorityClassName` | measured; tolerate everything; `system-node-critical` | The defaults come from [measurement](https://s3archiver.csi.trion.de/benchmarks.html). The blanket toleration is deliberate: A node the driver cannot schedule onto silently stops archiving for every pod there. | **WARN — Do not pass a secret key with --set** Helm stores the values of a release in a Secret in the cluster, so a key passed with `--set` lives there for as long as the release does, and in your shell history besides. Create the Secret yourself and name it with `credentials.existingSecret`. credentials without --set: ```sh # From the environment, so no key lands in shell history or in the release secret. $ kubectl create namespace csi-s3-archiver $ kubectl -n csi-s3-archiver create secret generic s3-credentials \ --from-literal=S3A_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" \ --from-literal=S3A_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" $ helm upgrade --install csi-s3-archiver deploy/helm/csi-s3-archiver \ --namespace csi-s3-archiver \ --set config.bucket=prod-dumps \ --set credentials.existingSecret=s3-credentials ``` Past two or three flags, a values file is easier to review and easier to put in Git: values.yaml: ```yaml # values.yaml, applied with: helm upgrade --install ... -f values.yaml image: repository: ghcr.io/trion-development/csi-s3-archiver tag: "" # defaults to the chart's appVersion config: bucket: prod-dumps endpoint: "" # set for Ceph, SeaweedFS or any non-AWS endpoint region: eu-central-1 pathStyle: false # most non-AWS stores need true prefix: "{namespace}/{podName}/" logFormat: text # or json, for a collector credentials: existingSecret: s3-credentials # or leave empty for IRSA / instance profiles metrics: enabled: true port: 9809 # NOT 9808: the livenessprobe sidecar already listens there serviceMonitor: enabled: true # needs the Prometheus Operator webui: enabled: false # read-only statistics page, no authentication kubeletDir: /var/lib/kubelet # MicroK8s and k0s move this ``` Upgrades are `helm upgrade --install ... -f values.yaml`, and the DaemonSet rolls one node at a time. One upgrade needs a manual step: `CSIDriver.spec.volumeLifecycleModes` is immutable and `controller.enabled=true` adds `Persistent` to it, so turning the Controller on over an existing install fails with `field is immutable`. Delete the object and let the upgrade recreate it. No running volume is disturbed, because kubelet reads it at publish time only. Removing it is `helm uninstall`, which takes the `CSIDriver` object with it because the chart created it. A namespace made by `--create-namespace` is not part of the release and stays. The warning in [uninstalling](#uninstalling) applies here too: Delete the pods that use the driver first. uninstall: ```bash helm uninstall csi-s3-archiver -n csi-s3-archiver kubectl delete namespace csi-s3-archiver # not part of the release ``` ## 2. Give it credentials Four sources, resolved per volume, first match wins. Pick whichever fits your cluster; the [reference](https://s3archiver.csi.trion.de/credentials.html#credentials) has the full rules. 1. **Nothing at all**, if the nodes already have them. With no per-volume secret and no `S3A_*` keys, resolution falls through to the AWS default provider chain, which is what makes IRSA, EKS Pod Identity and instance profiles work untouched. 2. **Cluster-global**, for one bucket shared by everything. Create the secret and reference it from the DaemonSet's environment in a kustomize overlay: ```sh $ kubectl -n csi-s3-archiver create secret generic s3-credentials \ --from-literal=accessKeyId=AKIAEXAMPLE \ --from-literal=secretAccessKey=... ``` 3. **Per volume**, when different namespaces use different buckets: A `nodePublishSecretRef` in the pod spec, read by kubelet from the pod's own namespace. per-volume credentials: ```yaml apiVersion: v1 kind: Secret metadata: name: s3-creds namespace: team-a stringData: accessKeyId: AKIAEXAMPLE secretAccessKey: "…" --- # in the pod spec csi: driver: s3archiver.csi.trion.de volumeAttributes: bucket: prod-dumps nodePublishSecretRef: name: s3-creds ``` 4. **No credentials on the node at all**. Set `presignEndpoint` and a signer service hands out a fresh URL per upload. **GOOD — Secrets stay in memory** Credentials are never written to the state files on the node, and `toString()` on the resolved credentials is redacted. Both are unit-tested, and a log-capture test fails the build if secret material appears in any log line. ## 3. Archive your first file A pod that writes four megabytes into a CSI volume and exits. Everything under `/dumps` ends up in the bucket. heapdump-example.yaml: ```yaml apiVersion: v1 kind: Pod metadata: name: heapdump-example spec: restartPolicy: Never containers: - name: app image: busybox:1.36 command: ["sh", "-c", "dd if=/dev/urandom of=/dumps/java_pid1.hprof bs=1M count=4; sleep 60"] volumeMounts: - name: dumps mountPath: /dumps volumes: - name: dumps csi: driver: s3archiver.csi.trion.de volumeAttributes: bucket: prod-dumps prefix: "heapdumps/{namespace}/{podName}/" ``` verify: ```sh $ kubectl apply -f heapdump-example.yaml $ aws s3 ls s3://prod-dumps/heapdumps/default/heapdump-example/ 2026-07-27 09:16:04 4194304 java_pid1.hprof ``` ![Terminal recording: a pod writes a 4 MiB heapdump into its CSI volume; the driver log shows the file going quiescent and being uploaded; aws s3 ls lists the finished object under heapdumps/default/heapdump-example/ in the bucket.](https://s3archiver.csi.trion.de/media/archive.gif) *Archive: Write a file, watch it appear in the bucket. 14.4s loop, [still image](https://s3archiver.csi.trion.de/media/archive.png)* The driver's own log tells the same story, one structured line per event, on stderr, which is what `kubectl logs` shows: kubectl logs -n csi-s3-archiver ds/csi-s3-archiver: ```log 2026-07-27T09:15:30.402Z INFO [csi-rpc-3] NodeService - rpc=NodePublishVolume volumeId=csi-9f3a bucket=prod-dumps status=OK duration_ms=6 2026-07-27T09:16:01.884Z INFO [archiver-1] Archiver - event=quiescent file=java_pid1.hprof bytes=4194304 2026-07-27T09:16:04.117Z INFO [upload-2] Uploader - event=upload key=heapdumps/default/heapdump-example/java_pid1.hprof bytes=4194304 attempt=1 status=OK duration_ms=2231 2026-07-27T09:17:12.006Z INFO [csi-rpc-5] NodeService - rpc=NodeUnpublishVolume volumeId=csi-9f3a event=sweep uploaded=0 status=OK duration_ms=41 ``` **NOTE — Why the wait?** A file is uploaded once it has been unmodified for `quiescenceSeconds`, 30 by default, so that a heapdump written over ten seconds is uploaded once, complete, rather than three times, truncated. Files that never go quiet are what [`appendOnly`](https://s3archiver.csi.trion.de/configuration.html#append-strategy) is for, and anything still pending is uploaded by the final sweep at pod termination. ## 4. Check it before you rely on it Three things worth confirming on a new install, in the order they tend to go wrong: | Check | Command | Expected | | --- | --- | --- | | Driver registered | `kubectl get csinode -o jsonpath='{.items[*].spec.drivers[*].name}'` | `s3archiver.csi.trion.de`, once per node | | Pods healthy | `kubectl -n csi-s3-archiver get pods` | `3/3 Running` on every node | | Uploads working | `check-upload` in a pod using the driver image and the same environment | It exercises the real upload path and reports exactly what S3 answered | If a pod using the driver hangs in `ContainerCreating`, the cause is almost always a rejected attribute, and the driver says so on the pod's own events: ![Terminal recording: a pod with a misspelled prefix placeholder stays in ContainerCreating, and kubectl describe pod shows the driver returning INVALID_ARGUMENT with a message naming the bad placeholder and listing the accepted ones.](https://s3archiver.csi.trion.de/media/error.gif) *Misconfiguration: The driver's own message, on the pod's own events. 10.6s loop, [still image](https://s3archiver.csi.trion.de/media/error.png)* More symptoms and what to do about them are in the [troubleshooting table](https://s3archiver.csi.trion.de/operations.html#troubleshooting). ## Distributions that need more than one apply The four steps above assume a cluster whose kubelet keeps its state in `/var/lib/kubelet` and whose admission controller lets a DaemonSet mount it. Most clusters are that cluster. The ones that are not fail in one of three ways. They **moved the directory**, they **enforce a policy** that forbids the mount, or they are **not a cluster with nodes you can reach** at all. | Distribution | What is different | What it costs you | | --- | --- | --- | | k3s, RKE2, kind, minikube, Docker Desktop, Rancher Desktop | Nothing. Kubelet's root really is `/var/lib/kubelet` | Install exactly as above | | [vcluster](https://s3archiver.csi.trion.de/install-vcluster.html) | The driver belongs to the *host* cluster, and the pod identity it sees is the host's | Nothing to install, but the archive layout changes | | [Talos Linux](https://s3archiver.csi.trion.de/install-talos.html) | Pod Security Admission enforces `baseline` everywhere but `kube-system`, and `baseline` forbids hostPath | One namespace label | | [OpenShift](https://s3archiver.csi.trion.de/install-openshift.html) | A SecurityContextConstraint has to allow hostPath volumes and uid 0 | One `oc adm policy` | | [MicroK8s](https://s3archiver.csi.trion.de/install-microk8s.html) | Kubelet's root directory is under the snap tree | One `kubeletDir` | | [k0s](https://s3archiver.csi.trion.de/install-k0s.html) | Kubelet's root directory is `/var/lib/k0s/kubelet` | One `kubeletDir`, or one worker flag | | [GKE Autopilot and EKS Fargate](https://s3archiver.csi.trion.de/install-autopilot-fargate.html) | No writable hostPath, and on Fargate no DaemonSets at all | It cannot be installed | Each has a page of its own, because the answer is a paragraph in some cases and a change of plan in others: - [**vcluster**](https://s3archiver.csi.trion.de/install-vcluster.html). Install once on the host cluster. Volumes work from inside untouched, but every prefix placeholder resolves to the synced pod's host identity. - [**Talos Linux**](https://s3archiver.csi.trion.de/install-talos.html). The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed. - [**OpenShift**](https://s3archiver.csi.trion.de/install-openshift.html). hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container. - [**MicroK8s**](https://s3archiver.csi.trion.de/install-microk8s.html). Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet. - [**k0s**](https://s3archiver.csi.trion.de/install-k0s.html). Point the driver at the k0s path, or point k0s back at the standard one and never think about it again. - [**GKE Autopilot and EKS Fargate**](https://s3archiver.csi.trion.de/install-autopilot-fargate.html). Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine. **NOTE — Everything else on this page still applies** Those pages are deltas, not alternative installs. Credentials, the first archived file, the checks and the uninstall are the same everywhere, and each page says which one thing is different. ## Building from source Requires JDK 25 or newer and Maven 3.9 or newer. The native build needs only Docker on the host and produces the *host* architecture, because GraalVM cannot cross-compile; multi-arch images are stitched from per-architecture runners. build: ```sh $ mvn package # unit tests + target/csi-s3-archiver.jar $ mvn verify -Pit # + SeaweedFS integration tests (needs Docker) $ ./build-native.sh # -> target/csi-s3-archiver, statically linked $ ./build-image.sh # -> ghcr.io/trion-development/csi-s3-archiver:dev ``` The native build tries a fully static musl link first, which allows a `FROM scratch` image, and falls back to a mostly-static glibc link automatically. It also re-records the GraalVM reachability metadata by running the csi-sanity suite under the tracing agent; commit the result when it changes. ## Uninstalling ```sh $ kubectl delete -k deploy/base ``` Or `helm uninstall csi-s3-archiver -n csi-s3-archiver` for a [chart install](#helm). Delete the pods that use the driver first, or their final sweep has nowhere to run and they will sit in `Terminating` until the CSI calls time out. Objects already in the bucket are untouched: The driver never deletes what it uploaded. ## Where next - [Configuration reference](https://s3archiver.csi.trion.de/docs.html): Every attribute, the prefix placeholders, and the semantics worth knowing before you rely on them. - [Append-only and spool modes](https://s3archiver.csi.trion.de/index.html#shapes): Logs that never go quiet, and bounded node disk. - [FAQ](https://s3archiver.csi.trion.de/faq.html): What happens when S3 is down, and why pod deletion blocks. - [Comparison](https://s3archiver.csi.trion.de/compare.html): The cases where a different tool is the right answer. ============================================================================== Source: https://s3archiver.csi.trion.de/install-autopilot-fargate.html Updated: 2026-08-30 Summary: Why csi-s3-archiver cannot be installed on GKE Autopilot or EKS Fargate, why a mixed cluster is still fine, and what to do instead when it is not. Install # GKE Autopilot and EKS Fargate Neither can run a CSI node plugin, and that is the product working as intended. A mixed cluster is still fine. ## Neither can run the driver Two managed products rule this driver out by design, and no configuration works around either. Both are cases where you do not own a node, and a CSI node plugin is a thing that runs on a node. | Product | Why | | --- | --- | | GKE Autopilot | Writable hostPath is denied by admission (`autogke-no-write-mode-hostpath`). All three of the driver's host paths are writes, and Autopilot allows only a short list of read paths. | | EKS Fargate | Fargate runs no DaemonSets, so there is no node plugin and no kubelet directory to share with one. | | virtual-kubelet, ACI, and other nodeless backends | The same reason as Fargate. The "node" is an API, not a machine with a kubelet you can mount. | **NOTE — This is not a gap to be closed** It is what the products are for. The whole value of Autopilot and Fargate is that nobody gets to put software on the node, and the whole mechanism of a CSI node plugin is putting software on the node. A future version of this driver will not fix it. ## A mixed cluster is fine The restriction is per node, not per cluster. On a cluster with both Fargate and EC2 nodes, the DaemonSet lands on the EC2 nodes and archives volumes there normally; the pods that want an archived volume have to be scheduled there too, which is an ordinary `nodeSelector` on the workload rather than anything the driver needs to know about. The DaemonSet tolerates everything on purpose, so it will also try Fargate nodes and sit `Pending` on them. If that noise matters, add a `nodeSelector` or an affinity that excludes them, the same way every other node-level agent does on such a cluster. It is only an *all*-Fargate or Autopilot cluster that has nowhere for the driver to run. ## What to do instead If that is your cluster, this is the wrong tool for it. What the driver gives you is a directory that gets archived without the workload knowing, and on a nodeless platform there is no place to stand to do that. The nearest things that do work: - **A sidecar that uploads.** Share an `emptyDir` between your container and a small uploader, and keep the S3 layout conventions from [the configuration reference](https://s3archiver.csi.trion.de/configuration.html#volume-attributes) so the archive looks the same. You give up the pod-termination sweep, which is the part that is genuinely hard to reproduce. - **Write to S3 from the application.** For a heap dump this is a flag on the JVM plus an upload step, and it is honest about what is happening. - **A node pool.** On GKE, a Standard node pool alongside Autopilot; on EKS, a managed node group alongside Fargate. Then the section above applies. **NOTE — How this was checked** From the Google and AWS documentation. ## Other distributions The [install page](https://s3archiver.csi.trion.de/install.html#distributions) has the overview. The rest of the awkward ones: - [**vcluster**](https://s3archiver.csi.trion.de/install-vcluster.html). Install once on the host cluster. Volumes work from inside untouched, but every prefix placeholder resolves to the synced pod's host identity. - [**Talos Linux**](https://s3archiver.csi.trion.de/install-talos.html). The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed. - [**OpenShift**](https://s3archiver.csi.trion.de/install-openshift.html). hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container. - [**MicroK8s**](https://s3archiver.csi.trion.de/install-microk8s.html). Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet. - [**k0s**](https://s3archiver.csi.trion.de/install-k0s.html). Point the driver at the k0s path, or point k0s back at the standard one and never think about it again. ============================================================================== Source: https://s3archiver.csi.trion.de/install-k0s.html Updated: 2026-08-30 Summary: Installing the driver on k0s: kubelet keeps its state in /var/lib/k0s/kubelet, so either set kubeletDir or start the workers with --kubelet-root-dir. Install # csi-s3-archiver on k0s Kubelet's root is `/var/lib/k0s/kubelet`. Point the driver at k0s, or point k0s back at the standard path and stop paying for it per driver. ## Two ways round the same path k0s runs kubelet with `/var/lib/k0s/kubelet` as its root directory rather than the `/var/lib/kubelet` that every CSI driver's manifests assume. You can move the driver or move k0s, and unlike most either-or choices this one has a clear answer if you have more than one CSI driver to install. install on k0s: ```sh # Either point the driver at k0s... $ helm install csi-s3-archiver deploy/helm/csi-s3-archiver \ -n csi-s3-archiver --create-namespace \ --set kubeletDir=/var/lib/k0s/kubelet # ...or point k0s at the path every CSI driver's manifests already assume, once, per worker: $ k0s install worker --token-file /var/lib/k0s/join-token \ --kubelet-root-dir=/var/lib/kubelet $ kubectl get csinode -o jsonpath='{.items[*].spec.drivers[*].name}' s3archiver.csi.trion.de ``` | Approach | What it costs | When it is right | | --- | --- | --- | | `--set kubeletDir=/var/lib/k0s/kubelet` | One value, per driver, forever | This is the only CSI driver on the cluster | | `--kubelet-root-dir=/var/lib/kubelet` on each worker | A worker reinstall, once | There are other CSI drivers, or there will be | **NOTE — Start from the ordinary install** This page is only the part that is different on k0s. The four steps every cluster needs, credentials, a first archived file and how to verify it, are on the [install page](https://s3archiver.csi.trion.de/install.html), and everything there applies here too. ## Moving an existing cluster Changing a running cluster's kubelet root directory is not a live migration. Kubelet's pod directories move with it, so drain the node first and let the workloads be recreated on the new path. That is another reason to make the decision when the cluster is built rather than after it has volumes in flight. **WARN — Do not move it under a running driver** The driver's state files live under the plugin directory, and its volume directories live under kubelet's `pods`. Moving the root out from under a running driver leaves both behind. The archiver loses the volumes it was tracking, and the pods that owned them get their final sweep from nobody. Uninstall the driver, move the path, reinstall. **NOTE — How this was checked** The path and the worker flag are from the k0s documentation. ## Other distributions The [install page](https://s3archiver.csi.trion.de/install.html#distributions) has the overview. The rest of the awkward ones: - [**vcluster**](https://s3archiver.csi.trion.de/install-vcluster.html). Install once on the host cluster. Volumes work from inside untouched, but every prefix placeholder resolves to the synced pod's host identity. - [**Talos Linux**](https://s3archiver.csi.trion.de/install-talos.html). The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed. - [**OpenShift**](https://s3archiver.csi.trion.de/install-openshift.html). hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container. - [**MicroK8s**](https://s3archiver.csi.trion.de/install-microk8s.html). Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet. - [**GKE Autopilot and EKS Fargate**](https://s3archiver.csi.trion.de/install-autopilot-fargate.html). Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine. ============================================================================== Source: https://s3archiver.csi.trion.de/install-microk8s.html Updated: 2026-08-30 Summary: Installing the driver on MicroK8s: kubelet keeps its plugin and pod directories under the snap tree, so one kubeletDir value moves all three host paths. Install # csi-s3-archiver on MicroK8s Kubelet's root directory lives under the snap tree. One `kubeletDir` value moves everything that depends on it. ## One value, four places MicroK8s runs kubelet with its root directory inside the snap's data tree rather than at `/var/lib/kubelet`. A CSI node plugin has to agree with kubelet about that path in four places, and the chart derives all four from one value: | What | On MicroK8s | | --- | --- | | The plugin socket and driver state | `/plugins/s3archiver.csi.trion.de` | | Where kubelet asks plugins to register | `/plugins_registry` | | The volume directories themselves | `/pods` | | `--kubelet-registration-path` on the registrar sidecar | the socket path *as kubelet sees it* | where `` is `/var/snap/microk8s/common/var/lib/kubelet`. install on MicroK8s: ```sh # Helm: one value moves all three host paths and the registrar's argument together. $ helm install csi-s3-archiver deploy/helm/csi-s3-archiver \ -n csi-s3-archiver --create-namespace \ --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet # kustomize: the base has the default path in four places, so rewrite the rendered output. $ kubectl kustomize deploy/base \ | sed 's#/var/lib/kubelet#/var/snap/microk8s/common/var/lib/kubelet#g' \ | microk8s kubectl apply -f - # Check that kubelet agrees the driver is there. $ microk8s kubectl get csinode \ -o jsonpath='{.items[*].spec.drivers[*].name}' s3archiver.csi.trion.de ``` **NOTE — Start from the ordinary install** This page is only the part that is different on MicroK8s. The four steps every cluster needs, credentials, a first archived file and how to verify it, are on the [install page](https://s3archiver.csi.trion.de/install.html), and everything there applies here too. ## If it registers but no pod can use a volume The registrar sidecar's `--kubelet-registration-path` is the one path that is not a mount. It is a *string* the driver hands to kubelet, saying where the socket lives on the host. Get the mounts right and that argument wrong and the driver appears in `kubectl get csinode` while every publish fails, because kubelet is dialling a socket that is not there. **NOTE — Some MicroK8s versions symlink the default path** MicroK8s has created a `/var/lib/kubelet` symlink into the snap tree since 1.2, so a driver that hard-codes the default sometimes works by accident. Do not rely on it. Set `kubeletDir` explicitly, and the install stops depending on which MicroK8s version and confinement mode you happen to be on. **NOTE — How this was checked** The path is from the MicroK8s documentation; the knob that sets it is the same one the lab exercised. ## Other distributions The [install page](https://s3archiver.csi.trion.de/install.html#distributions) has the overview. The rest of the awkward ones: - [**vcluster**](https://s3archiver.csi.trion.de/install-vcluster.html). Install once on the host cluster. Volumes work from inside untouched, but every prefix placeholder resolves to the synced pod's host identity. - [**Talos Linux**](https://s3archiver.csi.trion.de/install-talos.html). The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed. - [**OpenShift**](https://s3archiver.csi.trion.de/install-openshift.html). hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container. - [**k0s**](https://s3archiver.csi.trion.de/install-k0s.html). Point the driver at the k0s path, or point k0s back at the standard one and never think about it again. - [**GKE Autopilot and EKS Fargate**](https://s3archiver.csi.trion.de/install-autopilot-fargate.html). Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine. ============================================================================== Source: https://s3archiver.csi.trion.de/install-openshift.html Updated: 2026-08-30 Summary: Installing the driver on OpenShift: the SecurityContextConstraint the node plugin needs, why hostmount-anyuid is enough, and why the privileged SCC is not required. Install # csi-s3-archiver on OpenShift hostPath volumes and uid 0, granted by hostmount-anyuid. The driver is not a privileged container and does not ask to become one. ## One SecurityContextConstraint OpenShift admits pods through SecurityContextConstraints as well as Pod Security. The driver's ServiceAccount needs one that allows hostPath volumes and uid 0; the default `restricted-v2` allows neither, so without this the DaemonSet's pods are refused at admission. grant the node plugin its two exceptions: ```sh # hostmount-anyuid, not privileged. It grants exactly the two things the DaemonSet needs, # hostPath volumes and running as uid 0, and nothing else. The driver is not a privileged # container and does not want to become one. $ oc adm policy add-scc-to-user hostmount-anyuid \ -z csi-s3-archiver -n csi-s3-archiver # With the optional Controller, its ServiceAccount needs nothing extra: it mounts no host path. $ oc -n csi-s3-archiver get pods ``` **NOTE — Start from the ordinary install** This page is only the part that is different on OpenShift. The four steps every cluster needs, credentials, a first archived file and how to verify it, are on the [install page](https://s3archiver.csi.trion.de/install.html), and everything there applies here too. ## Why not the privileged SCC It is worth being precise about what is being granted, because "CSI driver" and "privileged" usually arrive together and here they do not. | What the DaemonSet asks for | Which SCC covers it | | --- | --- | | Three hostPath volumes under `/var/lib/kubelet` | `allowHostDirVolumePlugin`, in `hostmount-anyuid` | | `runAsUser: 0` | `RunAsAny`, in `hostmount-anyuid` | | `privileged: true` | Not asked for | | Added capabilities | None. The container drops `ALL` | | Host network, host PID, host IPC | None of them | | Bidirectional mount propagation | Not asked for; the driver makes no `mount(2)` call | A CSI volume here is a plain directory the node plugin creates, not a mount, which is what lets the whole thing stay this narrow. `hostmount-anyuid` grants exactly the two rows that are needed, and the `privileged` SCC grants a great deal more than the job requires. **NOTE — The Controller needs nothing** If you enable the optional Controller for `volumeClaimTemplate` volumes, its ServiceAccount stays on the default SCC. It mounts no host path and runs no privileged container; it validates StorageClass parameters and hands back a volume id. Only the node plugin touches the host. **NOTE — How this was checked** From the OpenShift documentation and the DaemonSet's own securityContext; not reproduced on a cluster. ## Other distributions The [install page](https://s3archiver.csi.trion.de/install.html#distributions) has the overview. The rest of the awkward ones: - [**vcluster**](https://s3archiver.csi.trion.de/install-vcluster.html). Install once on the host cluster. Volumes work from inside untouched, but every prefix placeholder resolves to the synced pod's host identity. - [**Talos Linux**](https://s3archiver.csi.trion.de/install-talos.html). The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed. - [**MicroK8s**](https://s3archiver.csi.trion.de/install-microk8s.html). Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet. - [**k0s**](https://s3archiver.csi.trion.de/install-k0s.html). Point the driver at the k0s path, or point k0s back at the standard one and never think about it again. - [**GKE Autopilot and EKS Fargate**](https://s3archiver.csi.trion.de/install-autopilot-fargate.html). Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine. ============================================================================== Source: https://s3archiver.csi.trion.de/install-talos.html Updated: 2026-08-30 Summary: The Pod Security label the Talos DaemonSet needs, why the kustomize base already has it and Helm does not, and why no kubelet extra mounts apply. Install # csi-s3-archiver on Talos Linux Talos enforces the baseline Pod Security profile everywhere, and baseline forbids hostPath. One namespace label is the entire difference. ## One label, and no machine configuration Talos turns Pod Security Admission on by default and enforces the `baseline` profile in every namespace except `kube-system`. `baseline` forbids hostPath volumes, and this driver mounts three of them, so the DaemonSet is rejected. That is the whole of the Talos difference. The driver itself needs nothing special from the operating system. **NOTE — Start from the ordinary install** This page is only the part that is different on Talos Linux. The four steps every cluster needs, credentials, a first archived file and how to verify it, are on the [install page](https://s3archiver.csi.trion.de/install.html), and everything there applies here too. ## What the rejection looks like It is rejected by the *DaemonSet controller*, so no failing pod appears for you to describe, and `kubectl get pods` in the namespace shows nothing at all. The message is on the DaemonSet: kubectl -n csi-s3-archiver describe ds csi-s3-archiver: ```log Warning FailedCreate 16s daemonset-controller Error creating: pods "csi-s3-archiver-g9dh8" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volumes "plugin-dir", "registration-dir", "kubelet-pods-dir") ``` The same thing happens on any cluster that enforces `baseline` or `restricted` by default, whether through Talos's admission configuration, a cluster-wide default in the API server, or a policy engine applying the standards. Talos is simply the distribution where it is the default rather than a decision someone made. ## The fix `kubectl apply -k deploy/base` is unaffected: The namespace in the kustomize base already carries the `privileged` labels, and the driver is in a namespace of its own precisely so that they apply to nothing else. **Helm is affected**: The chart does not template the namespace, and `--create-namespace` makes an unlabelled one. Create it yourself first: a namespace the DaemonSet can live in: ```sh # kubectl apply -k deploy/base already carries these labels. Helm does not create the # namespace itself, and --create-namespace makes a bare one, so make it yourself first. $ kubectl create namespace csi-s3-archiver $ kubectl label namespace csi-s3-archiver \ pod-security.kubernetes.io/enforce=privileged \ pod-security.kubernetes.io/audit=privileged \ pod-security.kubernetes.io/warn=privileged $ helm install csi-s3-archiver deploy/helm/csi-s3-archiver -n csi-s3-archiver ``` **GOOD — No Talos extraMounts needed** A CSI driver on Talos usually needs `machine.kubelet.extraMounts` because it wants a directory *outside* kubelet's tree, such as `/var/lib/longhorn`, an iSCSI database or a host binary. This one never leaves `/var/lib/kubelet`, which Talos's kubelet already has, so there is no machine configuration to change, no node to reboot, and nothing to keep in step with a Talos upgrade. **NOTE — Why the driver cannot satisfy baseline** Not stubbornness. A CSI node plugin exists to put files where kubelet expects them, and reaching `/var/lib/kubelet/pods` is the job. It does drop everything else. The container is **not** privileged, adds no capabilities, drops `ALL`, has a read-only root filesystem and makes no `mount(2)` call. Root is needed only because the directories under `/var/lib/kubelet/pods` are root-owned. **NOTE — How this was checked** The rejection and the label that fixes it were reproduced on k3s; that Talos enforces baseline by default is from the Talos documentation. ## Other distributions The [install page](https://s3archiver.csi.trion.de/install.html#distributions) has the overview. The rest of the awkward ones: - [**vcluster**](https://s3archiver.csi.trion.de/install-vcluster.html). Install once on the host cluster. Volumes work from inside untouched, but every prefix placeholder resolves to the synced pod's host identity. - [**OpenShift**](https://s3archiver.csi.trion.de/install-openshift.html). hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container. - [**MicroK8s**](https://s3archiver.csi.trion.de/install-microk8s.html). Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet. - [**k0s**](https://s3archiver.csi.trion.de/install-k0s.html). Point the driver at the k0s path, or point k0s back at the standard one and never think about it again. - [**GKE Autopilot and EKS Fargate**](https://s3archiver.csi.trion.de/install-autopilot-fargate.html). Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine. ============================================================================== Source: https://s3archiver.csi.trion.de/install-vcluster.html Updated: 2026-08-30 Summary: Archived CSI volumes inside a virtual cluster. Install on the host, why a DaemonSet inside the vcluster cannot work, and what the prefix placeholders resolve to. Install # csi-s3-archiver on vcluster One install on the host cluster serves every virtual cluster on it. The volumes work untouched; it is the archive layout that needs a decision. ## Install it on the host cluster A virtual cluster has no nodes of its own. Its pods are synced out to the host cluster and run on the host's kubelet, which is the kubelet this driver talks to. So the driver is installed **in the host cluster**, once, and every vcluster on it gets archived volumes for free. Nothing is installed into the virtual cluster and nothing needs to be configured there. Both ways of declaring a volume then work from inside a vcluster with no changes, an inline `csi:` block and a `volumeClaimTemplate` against the optional Controller's StorageClass alike. What changes is *whose identity* the driver is told about, and that changes the shape of your archive rather than whether it works. **NOTE — Start from the ordinary install** This page is only the part that is different on vcluster. The four steps every cluster needs, credentials, a first archived file and how to verify it, are on the [install page](https://s3archiver.csi.trion.de/install.html), and everything there applies here too. ## Do not install the driver inside the vcluster It is the obvious first thing to try and it does not work, for a reason worth knowing. vcluster rewrites a synced pod's hostPath for kubelet's `pods` directory into a sandbox of its own, so the DaemonSet never starts. the driver, installed one layer too high: ```log $ kubectl -n csi-s3-archiver get pods # inside the vcluster NAME READY STATUS RESTARTS AGE csi-s3-archiver-w47rl 0/3 ContainerCreating 0 98s Warning FailedMount 34s (x8 over 97s) kubelet MountVolume.SetUp failed for volume "kubelet-pods-dir" : hostPath type check failed: /tmp/vcluster/vc-v1/v1/kubelet/pods is not a directory ``` **WARN — The failure is the lucky outcome** Only `/var/lib/kubelet/pods` is rewritten. `plugins/` and `plugins_registry/` are passed through untouched, so a driver that *did* start would register itself with the real host kubelet under the real driver name, competing with the host's own installation, while looking for volume directories in a sandbox the kubelet has never heard of. The `type: Directory` check on that hostPath is what stops it, and it stops it before anything is registered. ## The prefix placeholders resolve to the host's identity Pod metadata reaches the driver from kubelet, and kubelet only knows the pod it is actually running, which is the synced one. For a pod named `probe` in the virtual namespace `default`, in a vcluster named `v1` living in the host namespace `vc-v1`: | Placeholder | Inside the vcluster | What the driver is told | | --- | --- | --- | | `{namespace}` | `default` | `vc-v1`, the vcluster's own host namespace | | `{podName}` | `probe` | `probe-x-default-x-v1` | | `{podUid}` | the virtual UID | the host pod's UID, a different value | | `{serviceAccount}` | `default` | `vc-workload-v1`, one value for the whole vcluster | | `{nodeName}` | the real node | the same real node | The consequence is a layout, not a failure. Every tenant of a vcluster archives under the same `{namespace}` segment, and it is `{podName}` that carries the tenant apart, mangled but present, so nothing collides and nothing is overwritten: two tenants, one vcluster: ```sh # Two namespaces inside one vcluster, both running a pod called "dumper", both with # prefix: "tenants/{namespace}/{podName}/" $ aws s3 ls --recursive s3://lab/tenants/ tenants/vc-v1/dumper-x-team-a-x-v1/who.txt tenants/vc-v1/dumper-x-team-b-x-v1/who.txt # ^^^^^ the vcluster's namespace on the host, for every tenant # ^^^^^^^^^^^^^^^^^^^^^ pod, virtual namespace and vcluster, mangled together # The same two pods on a plain cluster: tenants/team-a/dumper/who.txt tenants/team-b/dumper/who.txt ``` So build the prefix on `{podName}` rather than `{namespace}` when the cluster is virtual, and treat `{serviceAccount}` as unusable there, because it is one value for every pod in the vcluster. Better, give each vcluster its own [`bucket`](https://s3archiver.csi.trion.de/configuration.html#volume-attributes) or a fixed prefix of its own in the DaemonSet's environment, and let the placeholders sort out what is inside it. ## Let the tenants see the driver By default a vcluster shows no `StorageClass`, no `CSIDriver` and no `CSINode`, while volumes using them work perfectly. A `volumeClaimTemplate` naming a class that `kubectl get sc` says does not exist still binds, because vcluster syncs the claim out to the host and the host's provisioner fulfils it. That is a confusing place to leave whoever has to use the cluster: vcluster.yaml: ```yaml # vcluster.yaml -- volumes already work without this. It only makes the driver visible to # whoever is working inside the virtual cluster, so `kubectl get sc` answers instead of # printing "No resources found" while a volumeClaimTemplate naming that very class binds fine. sync: fromHost: storageClasses: enabled: true csiDrivers: enabled: true csiNodes: enabled: true ``` **NOTE — Keeping the tenant's own names** vcluster's namespace syncing creates a real host namespace per virtual namespace and stops rewriting resource names, which would make every placeholder in the table above resolve the way it does on a plain cluster. It is not in the plain CLI, and enabling it asks you to log into vCluster Platform first (the free tier is enough). It also cannot be turned on after the vcluster has been created, so it is a decision to make before the first one. **NOTE — How this was checked** Reproduced on vcluster 0.29.0 over k3s v1.36.2, both inline and PVC-declared volumes. ## Other distributions The [install page](https://s3archiver.csi.trion.de/install.html#distributions) has the overview. The rest of the awkward ones: - [**Talos Linux**](https://s3archiver.csi.trion.de/install-talos.html). The kustomize base already labels its namespace; Helm does not. No kubelet extra mounts are needed. - [**OpenShift**](https://s3archiver.csi.trion.de/install-openshift.html). hostmount-anyuid rather than privileged. The driver mounts host directories and runs as root, but it is not a privileged container. - [**MicroK8s**](https://s3archiver.csi.trion.de/install-microk8s.html). Everything works once the three host paths point at /var/snap/microk8s/common/var/lib/kubelet. - [**k0s**](https://s3archiver.csi.trion.de/install-k0s.html). Point the driver at the k0s path, or point k0s back at the standard one and never think about it again. - [**GKE Autopilot and EKS Fargate**](https://s3archiver.csi.trion.de/install-autopilot-fargate.html). Both rule the driver out by design. What to do instead, and why a mixed cluster is still fine. ============================================================================== Source: https://s3archiver.csi.trion.de/operations.html Updated: 2026-08-30 Summary: Running csi-s3-archiver: What pod deletion waits for, the archiving semantics and their caveats, resource requests and limits, and troubleshooting. Reference # Operations What to expect once it is running: The guarantees, the caveats worth knowing before you rely on them, what to size the container for, and what to do when something looks wrong. ## Pod deletion waits for the final sweep When the pod terminates, the driver uploads everything new or changed one last time, ignoring the quiescence window, and **blocks pod deletion until it finishes**. A heapdump written seconds before the pod died is exactly the file you most want archived, and returning early would lose it. **WARN — If S3 is unreachable, the pod stays in Terminating** Uploads retry with capped backoff indefinitely, and the sweep waits for them. Set `sweepTimeoutSeconds` on a volume to bound it and accept best-effort teardown instead: `sweepTimeoutSeconds: "300"` gives up after five minutes and lets the pod go. For a workload where losing the last file is worse than a stuck pod, leave it unset. For one where the reverse is true, a CronJob that runs every few minutes for instance, set it. ## Semantics and caveats Things that will bite you if you assume otherwise. Each is a design choice rather than an oversight. - **Objects are overwritten, not versioned.** A file uploaded twice lands on the same key the second time. If you need history, enable bucket versioning or put `{date}` in the prefix. The same applies in spool mode: After `deleteAfterUpload` removes a file, a later file with the same name overwrites the object. - **Change detection is size + mtime.** A file rewritten with exactly the same size and timestamp is not noticed. Hashing every candidate would mean reading multi-gigabyte heapdumps twice on every scan. The periodic rescan and the final sweep use the same comparison, so they do not rescue this case either. - **Per-volume secrets are read once, at publish.** Rotating the Secret takes effect for new pods only; existing pods keep the credentials they were published with. Presigned mode does not have this limitation, because every upload fetches a fresh URL. - **A driver restart re-resolves credentials from the driver's own environment.** kubelet only delivers a `nodePublishSecretRef` secret on a fresh publish, so a recovered volume that used one falls back to whatever driver-global credentials exist, possibly none. The driver logs a warning naming the volume when this happens. - **Presigned mode caps files at 5 GiB, unless the signer does multipart.** A single presigned PUT cannot be multipart, so larger files fail with a clear error rather than being split into objects you did not ask for. Setting `presignMultipart: "true"` has the driver ask the signer for the multipart operations instead, which raises the ceiling to about 312 GiB. It is opt-in because a `putObject`-only signer answers those requests plausibly and wrongly. - **Symlinks are never followed.** The driver runs as root on the node, so following a symlink a workload planted would upload arbitrary node files into that workload's bucket. Symlinks are skipped and logged; only regular files are archived. - **Object keys are capped at 1024 UTF-8 bytes.** A file whose key would exceed S3's limit is skipped and logged, never truncated, because a truncated key would overwrite an unrelated object. - **No restore.** This driver only writes. Objects are never read back into a volume, and there is no command to reverse an archive. ## Resource requests and limits The values shipped in `deploy/` are **a starting point, not a recommendation for your cluster**. They come from `hack/measure-resources.sh`, which archives a synthetic workload through the real engine and samples the native binary's RSS and CPU while it works. | Shape | Peak RSS | CPU | | --- | --- | --- | | idle (1 volume, 1 MiB) | 69 MiB | 0.02 s | | typical (4 volumes x 25 x 1 MiB) | 128 MiB | 0.67 s | | many volumes (32 x 10 x 1 MiB) | 152 MiB | 4.96 s | | one large file (2 x 512 MiB) | **156 MiB** | 13.7 s | | gzip (4 x 25 x 1 MiB) | 93 MiB | 0.58 s | | segments (4 x 25 x 4 MiB) | 147 MiB | 7.47 s | | concurrency 16 | 138 MiB | 1.53 s | **Memory is flat in file size.** A 512 MiB file peaks no higher than thirty-two small ones, because uploads stream from disk in fixed buffers and switch to multipart above a threshold: There is no point at which the driver holds a file in memory. What moves the number is concurrency, and `S3A_NODE_CONCURRENCY` bounds that. deploy/base/daemonset.yaml: ```yaml resources: requests: cpu: 10m memory: 128Mi limits: memory: 256Mi # a starting point -- measure your own workload ``` **WARN — Re-measure before you trust these** The request is the number that is firmly justified: Idle RSS alone is 69 MiB, so anything below ~96Mi under-provisions every node. The **limit is deliberately modest** at ~1.6x the measured peak, so the driver is a good neighbour out of the box. If you archive large files at high concurrency, raise it. The failure is asymmetric: An OOM kill mid-sweep loses the archive the driver exists to produce, while an unused reservation costs only scheduling headroom. **No CPU limit**, on purpose. Archiving is bursty and idle most of the time, and throttling the driver mid-upload only makes the unpublish sweep -- which blocks pod deletion -- take longer. If a noisy-neighbour policy forces one, give it at least two cores: The sweep runs `S3A_NODE_CONCURRENCY` uploads at once and measured 2.4 cores at concurrency 16. ## Troubleshooting | Symptom | What to do | | --- | --- | | Pod stuck in `ContainerCreating` | Run `kubectl describe pod`. A configuration error surfaces as an event carrying the driver's own message, naming the attribute and what is accepted. | | Pod stuck in `Terminating` | The final sweep is waiting on an upload. Check the driver log for retries; set `sweepTimeoutSeconds` if best-effort teardown is acceptable for that volume. | | Nothing appears in the bucket | Raise `S3A_LOG_LEVEL` to `debug`. Then run `check-upload` in a pod using the driver image and the same environment: It exercises the same upload path and reports exactly what S3 answered. | | Driver missing from `kubectl get csinode` | Run `kubectl logs -c node-driver-registrar` in the driver pod. | | Repeated `status=RETRY` lines | The log line carries the S3 error and the attempt count. A 4xx is reported as `FAILED_PERMANENTLY` and dropped rather than retried. | | Leftover `.parts/` objects | Compaction could not delete them (missing `DeleteObject`, or presigned mode). The canonical object is still correct; the leftover keys are named in a warning. | ## Versions | Component | Version | | --- | --- | | CSI spec | **v1.12.0**, vendored in the build | | Java | 25 | | Driver name | `s3archiver.csi.trion.de` | | Architectures | linux/amd64, linux/arm64 | The vendored `csi.proto` carries one documented local patch: The Controller-service map field `mutable_parameters` is renamed to `mutable_params`, because protoc's Java generator cannot emit a message that has both it and a `parameters` map. Field numbers, and therefore the wire format, are unchanged. ============================================================================== Source: https://s3archiver.csi.trion.de/s3-cloudserver.html Updated: 2026-08-30 Summary: Running Zenko CloudServer with docker compose and pointing csi-s3-archiver at it: The endpoint configuration, what it supports, and what to watch out for. S3 endpoints # Zenko CloudServer Scality's S3 server, the storage engine behind Zenko. ## What it is CloudServer is a mature implementation with the widest S3 surface of the six, and the one whose behaviour is closest to AWS on the features this driver uses. The in-memory backend below is for trying it out; use the file backend or a real Scality RING for anything you intend to keep. | | | | --- | --- | | **Project** | [https://github.com/scality/cloudserver](https://github.com/scality/cloudserver) | | **Licence** | Apache-2.0 | | **Image tested** | `zenko/cloudserver:latest-7.70.10` | | **S3 port** | 8000 | ## Run it A single-node setup, enough to archive into and to try the driver against. It is not a production topology for any of these products; each project's own documentation covers that. docker-compose.yml: ```yaml services: cloudserver: image: zenko/cloudserver:latest-7.70.10 environment: # "mem" keeps nothing across a restart. Use S3BACKEND=file with a volume to persist. S3BACKEND: mem SCALITY_ACCESS_KEY_ID: archiver-key SCALITY_SECRET_ACCESS_KEY: archiver-secret REMOTE_MANAGEMENT_DISABLE: "1" ports: - "8000:8000" ``` ## Point the driver at it Endpoint and credentials go on the volume; nothing about the driver's installation changes. `pathStyle` is on because a container reached by address has no per-bucket DNS, which is the usual shape outside AWS. a volume archiving into Zenko CloudServer: ```yaml apiVersion: v1 kind: Secret metadata: name: cloudserver-credentials namespace: default stringData: accessKeyId: archiver-key secretAccessKey: archiver-secret --- apiVersion: v1 kind: Pod metadata: name: writer spec: containers: - name: app image: busybox:1.36 command: ["sh", "-c", "echo hello > /dumps/first.txt; sleep 3600"] volumeMounts: - { name: dumps, mountPath: /dumps } volumes: - name: dumps csi: driver: s3archiver.csi.trion.de nodePublishSecretRef: name: cloudserver-credentials volumeAttributes: bucket: archives prefix: "{namespace}/{podName}/" endpoint: http://cloudserver.storage.svc.cluster.local:8000 pathStyle: "true" region: us-east-1 ``` To make it the default for every volume instead, set `S3A_ENDPOINT`, `S3A_PATH_STYLE` and `S3A_REGION` on the DaemonSet and leave them off the volumes. The [configuration reference](https://s3archiver.csi.trion.de/configuration.html#environment-driver-container) lists both halves. ## What works Measured, not claimed. An [opt-in test suite](https://s3archiver.csi.trion.de/s3-compatibility.html#how-it-is-tested) runs every one of these against Zenko CloudServer through the driver's own code paths. | Capability | | What it gives you | | --- | --- | --- | | Single PutObject | ●yes | Archiving anything at all. | | Multipart upload | ●yes | Files over the 64 MiB threshold. A heapdump is almost always over it. | | ListObjectsV2 | ●yes | Segment compaction and durable volumes. Ephemeral archiving never lists. | | DeleteObject | ●yes | Compaction removes fragments it has assembled; durable volumes mirror deletions. | | GetObject | ●yes | Restoring a durable volume at pod start. Ephemeral volumes never read back. | | UploadPartCopy | ●yes | Server-side append and segment assembly. Without it a growing file is re-uploaded whole. | | Offset append [1](https://s3archiver.csi.trion.de/s3-compatibility.html#offset-append) | ○no | The cheapest append, one request carrying only the new bytes. An S3 Express feature. | | SSE-S3 | ●yes | Requesting AES256 encryption per volume. A bucket default covers you regardless. | | SSE-KMS | ●yes | Per-volume encryption with a customer-managed key. | | Presigned PUT | ●yes | Presigned credential mode, where the node holds no S3 keys. | | Presigned POST policy | ○no | Signer-less mode, with one prefix-scoped policy in the volume Secret. | 9 of 11 supported. Missing: Offset append, Presigned POST policy. The driver degrades rather than failing for all of these except where noted below. ## Worth knowing **NOTE — POST policies are not implemented** A browser-style POST upload returns `501 NotImplemented`, so signer-less presigned mode does not work. The signer-based presigned mode does. **NOTE — `S3BACKEND=mem` is a demo** It is the fastest way to see the driver work end to end and it loses everything on restart. Switch to `file` with a mounted volume before anything matters. ============================================================================== Source: https://s3archiver.csi.trion.de/s3-compatibility.html Updated: 2026-08-30 Summary: csi-s3-archiver against SeaweedFS, Garage, S3Proxy, CloudServer, Versity and RustFS. A measured capability matrix, and a docker-compose setup for each. S3 endpoints # What works on which S3 The driver talks to anything with an S3 API, but "S3-compatible" covers a wide range. Here is what six open-source implementations actually do when the driver asks, measured rather than claimed, with a working setup for each. - [The matrix](#the-matrix) - [How to read it](#how-to-read-it) - [Choosing one](#choosing) - [How it is tested](#how-it-is-tested) - [Note 1: Offset append, and why the column is empty](#offset-append) ## The matrix Every feature the driver asks an object store for, against every implementation it is tested with. These are **test results**. An opt-in suite starts each product and runs each probe through the driver's own code, and the numbers below are what it observed. | Capability | [SeaweedFS](https://s3archiver.csi.trion.de/s3-seaweedfs.html) | [Garage](https://s3archiver.csi.trion.de/s3-garage.html) | [S3Proxy](https://s3archiver.csi.trion.de/s3-s3proxy.html) | [Zenko CloudServer](https://s3archiver.csi.trion.de/s3-cloudserver.html) | [Versity S3 Gateway](https://s3archiver.csi.trion.de/s3-versitygw.html) | [RustFS](https://s3archiver.csi.trion.de/s3-rustfs.html) | | --- | --- | --- | --- | --- | --- | --- | | Single PutObject | ●yes | ●yes | ●yes | ●yes | ●yes | ●yes | | Multipart upload | ●yes | ●yes | ●yes | ●yes | ●yes | ●yes | | ListObjectsV2 | ●yes | ●yes | ●yes | ●yes | ●yes | ●yes | | DeleteObject | ●yes | ●yes | ●yes | ●yes | ●yes | ●yes | | GetObject | ●yes | ●yes | ●yes | ●yes | ●yes | ●yes | | UploadPartCopy | ●yes | ●yes | ●yes | ●yes | ●yes | ●yes | | Offset append [1](#offset-append) | ○no | ○no | ○no | ○no | ○no | ○no | | SSE-S3 | ○no | ●yes | ○no | ●yes | ●yes | ○no | | SSE-KMS | ○no | ●yes | ○no | ●yes | ●yes | ○no | | Presigned PUT | ●yes | ●yes | ●yes | ●yes | ●yes | ●yes | | Presigned POST policy | ●yes | ○no | ●yes | ○no | ○no | ●yes | Each product has its own page with a `docker-compose.yml`, the volume configuration to point the driver at it, and the things that caught us out: - [**SeaweedFS**](https://s3archiver.csi.trion.de/s3-seaweedfs.html). A distributed file and object store; the S3 endpoint this project tests against by default. - [**Garage**](https://s3archiver.csi.trion.de/s3-garage.html). A self-contained store aimed at small, self-hosted, geo-distributed clusters. - [**S3Proxy**](https://s3archiver.csi.trion.de/s3-s3proxy.html). An S3 API in front of something else. A local filesystem, Azure, Google Cloud Storage. - [**Zenko CloudServer**](https://s3archiver.csi.trion.de/s3-cloudserver.html). Scality's S3 server, the storage engine behind Zenko. - [**Versity S3 Gateway**](https://s3archiver.csi.trion.de/s3-versitygw.html). An S3 protocol translator over POSIX filesystems, aimed at existing HPC storage. - [**RustFS**](https://s3archiver.csi.trion.de/s3-rustfs.html). A newer, Apache-2.0 object store written in Rust. **NOTE — AWS S3 itself is not in the table** It supports every row. The table is about the alternatives, because that is where the answer is not obvious. S3 Express One Zone directory buckets are the only place offset append works at all, which is why every column below says no. ## How to read it A missing capability is usually a slower driver, not a broken one. What each one costs: | Capability | Missing it means | | --- | --- | | Single PutObject | Archiving anything at all. | | Multipart upload | Files over the 64 MiB threshold. A heapdump is almost always over it. | | ListObjectsV2 | Segment compaction and durable volumes. Ephemeral archiving never lists. | | DeleteObject | Compaction removes fragments it has assembled; durable volumes mirror deletions. | | GetObject | Restoring a durable volume at pod start. Ephemeral volumes never read back. | | UploadPartCopy | Server-side append and segment assembly. Without it a growing file is re-uploaded whole. | | Offset append | The cheapest append, one request carrying only the new bytes. An S3 Express feature. | | SSE-S3 | Requesting AES256 encryption per volume. A bucket default covers you regardless. | | SSE-KMS | Per-volume encryption with a customer-managed key. | | Presigned PUT | Presigned credential mode, where the node holds no S3 keys. | | Presigned POST policy | Signer-less mode, with one prefix-scoped policy in the volume Secret. | Two are not negotiable. Without **single PutObject** and **multipart upload** the driver cannot archive at all, and the test suite treats their absence differently from the rest. Every product here has both. The rest degrade. Without `UploadPartCopy` a growing log is re-uploaded whole instead of appended, which still archives correctly and costs bandwidth that grows with the square of the file size, so use `appendStrategy: segments` there. Without `ListObjectsV2` or `DeleteObject` you lose segment compaction and durable volumes, but ephemeral archiving never calls either. ## Choosing one | If you want | Start with | | --- | --- | | The least setup | [SeaweedFS](https://s3archiver.csi.trion.de/s3-seaweedfs.html). One process, one config file, and it is what this project's own tests run against. | | The most complete S3 surface | [Zenko CloudServer](https://s3archiver.csi.trion.de/s3-cloudserver.html) or [Garage](https://s3archiver.csi.trion.de/s3-garage.html). Both do server-side encryption; CloudServer is the closest to AWS behaviour, Garage the easiest to run across several sites. | | An S3 API over storage you already have | [Versity gateway](https://s3archiver.csi.trion.de/s3-versitygw.html) for a POSIX filesystem, keeping objects as ordinary files, or [S3Proxy](https://s3archiver.csi.trion.de/s3-s3proxy.html) to front a directory or another cloud. | | Signer-less presigned mode | [SeaweedFS](https://s3archiver.csi.trion.de/s3-seaweedfs.html), [S3Proxy](https://s3archiver.csi.trion.de/s3-s3proxy.html) or [RustFS](https://s3archiver.csi.trion.de/s3-rustfs.html). They are the three that accept a POST policy; on the others use the [signer](https://s3archiver.csi.trion.de/credentials.html#credentials). | ## How it is tested The suite starts each product in a container, creates a bucket and runs eleven probes through the driver's `ObjectStore`, the same code an archiving volume uses. A pass means the driver's path works against that store, not that the store implements some verb. ```bash mvn verify -Pcompat # every product mvn verify -Pcompat -Dcompat.only=GARAGE # one of them ``` It is **not part of the normal build**. It starts a different object store per product and pushes real multi-megabyte objects through each, so it costs minutes and gigabytes of image pulls. **NOTE — The matrix cannot drift** Each product declares what it is expected to support, and the suite fails if a capability starts working *or* stops. Failing on a gained feature sounds strange until you notice the alternative, which is a page saying a store cannot do something it now does, which is the direction that makes someone choose the wrong store. Every value in the table was corrected by its first run rather than written from the products' documentation. **WARN — Versions matter** The table describes the pinned image on each product's page, not `:latest`. These projects move, and two of the six answered differently from what their documentation implied. ## Note 1: Offset append, and why the column is empty Every product in the table answers no to this row, which looks like six projects missing the same feature. They are not. **Offset append is not part of the S3 API.** ### It belongs to one storage class, at one provider A `PutObject` carrying `x-amz-write-offset-bytes` appends to an object in place. AWS added it in 2024 for **S3 Express One Zone directory buckets** only. It does not work on AWS's own general-purpose buckets, so a store cannot implement "S3" and get it along the way. It would have to implement a directory-bucket API that has no specification outside AWS's documentation and no other implementation to interoperate with. That is why the column is empty, and why it is likely to stay empty. It is not a gap these projects are behind on. ### Ceph has an append, and it is a different one Ceph's RADOS Gateway is the closest thing to an exception. It supports appending through `PUT /bucket/key?append&position=N`, an extension it inherited from the Aliyun OSS API rather than from S3. The idea is the same and the wire format is not. It uses a query parameter instead of a header, with a different request shape and a different response. So a driver that speaks AWS's version does not speak Ceph's. Supporting it would be a third append mode next to the two below, not a matter of turning something on. Ceph is not in the table because it is not in the test suite yet; if it is added, this is the row that would need a second footnote rather than a yes. ### What the driver does instead The driver *does* implement offset append, and uses it when it is really there. `appendUpload` defaults to `auto`, which resolves per bucket: | Mode | When it is used | What it costs | | --- | --- | --- | | `offset` | S3 Express directory buckets, which the driver recognises by bucket name | One request carrying only the new bytes. Nothing cheaper exists. | | `copy` | Everywhere else, including all six products above | A multipart upload whose first part is the existing object, copied server-side. No bytes for the existing part cross the network. | | `off` | When you set it, or when a store has refused an append | The whole file is re-uploaded on each sync. Correct, and the bandwidth grows with the square of the file size. | So on every store in this table the answer is `copy`, which all six support. Nothing is lost by the empty column except one request per append. ### Why not always use copy? It is the obvious question, since `copy` works nearly everywhere and `offset` works almost nowhere. Three reasons, in increasing order of how much they cost you. **An append by copy is four requests, not one.** It opens a multipart upload, copies the existing object into part 1, uploads the new tail as part 2, and completes. `offset` is a single `PutObject`. For a log synced every few seconds that is a four-fold difference in request count, and requests are what most stores bill for. **Copy has a floor at 5 MiB.** S3 requires every part but the last to be at least 5 MiB, so an object smaller than that cannot be part 1. The driver detects this and declines rather than failing, which means a file re-uploads whole until it passes 5 MiB. `offset` has no minimum. That matters most for exactly the files people turn `appendOnly` on for, which start small and grow slowly. **The store still copies the whole object every time.** This is the one worth understanding, because it is invisible from the outside. `UploadPartCopy` moves no bytes over the network, which is what makes it attractive, but the store internally rewrites the entire existing object on every append. Append *n* times to a file that reaches *S* bytes and the store has done roughly *n × S* bytes of internal copying, even though only *S* bytes crossed the network. | Strategy | Over the network | Work inside the store | | --- | --- | --- | | `appendUpload: off` (full re-upload) | **n × S**, which is why this is the mode to avoid on a long-lived log | n × S | | `copy` | S, only the new tail each time | **n × S**, invisible to you and not always free | | `offset` | S | S, appended in place | | `appendStrategy: segments` | S | S, plus one server-side assembly at the end | So `copy` fixes the bandwidth problem and leaves the storage-side one. On a managed service you may never notice; on a store you run yourself, on your own disks, it is your I/O. **For a file that grows for hours, use [`appendStrategy: segments`](https://s3archiver.csi.trion.de/configuration.html#append-strategy)**, which uploads each new chunk as its own object and assembles them once, rather than either append mode. The last reason is smaller but real. A multipart upload is a resource that exists between requests. The driver aborts it on failure, but a process killed between opening and aborting one leaves it behind, and abandoned uploads are billed until a lifecycle rule reaps them. `offset` has no such lifecycle, because there is nothing to leave open. **WARN — The failure mode this row exists to warn about** S3 says a server ignores `x-amz-*` headers it does not recognise. A store without offset-append support therefore treats the request as an ordinary `PutObject` and **replaces the object with just the appended tail**, answering 200. Nothing in the response says so. SeaweedFS does exactly this. The driver checks the resulting object's size against what it should be, refuses the result as a permanent error, disables the optimisation for that volume and re-uploads the whole file. That check is the only reason `appendUpload: offset` is safe to leave on `auto` against a store you have not tested. ============================================================================== Source: https://s3archiver.csi.trion.de/s3-garage.html Updated: 2026-08-30 Summary: Running Garage with docker compose and pointing csi-s3-archiver at it: The endpoint configuration, what it supports, and what to watch out for. S3 endpoints # Garage A self-contained store aimed at small, self-hosted, geo-distributed clusters. ## What it is Garage is a single static binary with no external database, designed to run on modest hardware across sites. It has the most complete feature coverage of the six here, and the most setup. A node has to be given a storage layout before it will serve. | | | | --- | --- | | **Project** | [https://garagehq.deuxfleurs.fr](https://garagehq.deuxfleurs.fr) | | **Licence** | AGPL-3.0 | | **Image tested** | `dxflrs/garage:v1.0.1` | | **S3 port** | 3900 | ## Run it A single-node setup, enough to archive into and to try the driver against. It is not a production topology for any of these products; each project's own documentation covers that. docker-compose.yml: ```yaml services: garage: image: dxflrs/garage:v1.0.1 ports: - "3900:3900" # S3 API - "3903:3903" # admin API volumes: - ./garage.toml:/etc/garage.toml:ro - garage-meta:/var/lib/garage/meta - garage-data:/var/lib/garage/data volumes: garage-meta: garage-data: ``` garage.toml — replication_factor 1 is a single-node setup; raise it for a real cluster: ```toml metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" db_engine = "sqlite" replication_factor = 1 rpc_bind_addr = "[::]:3901" rpc_public_addr = "127.0.0.1:3901" # openssl rand -hex 32 rpc_secret = "<64 hex characters>" [s3_api] s3_region = "garage" api_bind_addr = "[::]:3900" root_domain = ".s3.garage" [admin] api_bind_addr = "[::]:3903" admin_token = "" ``` This one needs setting up before it will serve: after the first start: ```bash # Garage will not serve until a storage layout exists. docker compose exec garage /garage layout assign -z dc1 -c 1G "$( docker compose exec -T garage /garage node id -q | cut -d@ -f1)" docker compose exec garage /garage layout apply --version 1 # Garage issues its own credentials; it will not accept one you pick. docker compose exec garage /garage key create csi-archiver docker compose exec garage /garage key allow --create-bucket csi-archiver docker compose exec garage /garage bucket create archives docker compose exec garage /garage bucket allow --read --write archives --key csi-archiver ``` ## Point the driver at it Endpoint and credentials go on the volume; nothing about the driver's installation changes. `pathStyle` is on because a container reached by address has no per-bucket DNS, which is the usual shape outside AWS. a volume archiving into Garage: ```yaml apiVersion: v1 kind: Secret metadata: name: garage-credentials namespace: default stringData: accessKeyId: archiver-key secretAccessKey: archiver-secret --- apiVersion: v1 kind: Pod metadata: name: writer spec: containers: - name: app image: busybox:1.36 command: ["sh", "-c", "echo hello > /dumps/first.txt; sleep 3600"] volumeMounts: - { name: dumps, mountPath: /dumps } volumes: - name: dumps csi: driver: s3archiver.csi.trion.de nodePublishSecretRef: name: garage-credentials volumeAttributes: bucket: archives prefix: "{namespace}/{podName}/" endpoint: http://garage.storage.svc.cluster.local:3900 pathStyle: "true" region: garage ``` To make it the default for every volume instead, set `S3A_ENDPOINT`, `S3A_PATH_STYLE` and `S3A_REGION` on the DaemonSet and leave them off the volumes. The [configuration reference](https://s3archiver.csi.trion.de/configuration.html#environment-driver-container) lists both halves. ## What works Measured, not claimed. An [opt-in test suite](https://s3archiver.csi.trion.de/s3-compatibility.html#how-it-is-tested) runs every one of these against Garage through the driver's own code paths. | Capability | | What it gives you | | --- | --- | --- | | Single PutObject | ●yes | Archiving anything at all. | | Multipart upload | ●yes | Files over the 64 MiB threshold. A heapdump is almost always over it. | | ListObjectsV2 | ●yes | Segment compaction and durable volumes. Ephemeral archiving never lists. | | DeleteObject | ●yes | Compaction removes fragments it has assembled; durable volumes mirror deletions. | | GetObject | ●yes | Restoring a durable volume at pod start. Ephemeral volumes never read back. | | UploadPartCopy | ●yes | Server-side append and segment assembly. Without it a growing file is re-uploaded whole. | | Offset append [1](https://s3archiver.csi.trion.de/s3-compatibility.html#offset-append) | ○no | The cheapest append, one request carrying only the new bytes. An S3 Express feature. | | SSE-S3 | ●yes | Requesting AES256 encryption per volume. A bucket default covers you regardless. | | SSE-KMS | ●yes | Per-volume encryption with a customer-managed key. | | Presigned PUT | ●yes | Presigned credential mode, where the node holds no S3 keys. | | Presigned POST policy | ○no | Signer-less mode, with one prefix-scoped policy in the volume Secret. | 9 of 11 supported. Missing: Offset append, Presigned POST policy. The driver degrades rather than failing for all of these except where noted below. ## Worth knowing **NOTE — The region is `garage`, not `us-east-1`** Set `region: garage` on the volume or `S3A_REGION` on the driver, or every request fails its signature check. **NOTE — You cannot choose the access key** Garage generates key ids of the form `GK…` and rejects anything else, so the credentials come out of `garage key create` rather than out of your configuration management. **NOTE — A new key may not create buckets** Either run `garage key allow --create-bucket` or create the bucket with the CLI and grant the key access to it. **NOTE — POST policies are rejected** Garage requires a `bucket` form field that S3's POST contract puts in the URL, so a conformant policy comes back as `InvalidRequest`. Use the signer-based presigned mode instead. **NOTE — The image has no shell** Handy to know when scripting: `docker compose exec garage sh` does not work. Invoke `/garage` directly. ============================================================================== Source: https://s3archiver.csi.trion.de/s3-rustfs.html Updated: 2026-08-30 Summary: Running RustFS with docker compose and pointing csi-s3-archiver at it: The endpoint configuration, what it supports, and what to watch out for. S3 endpoints # RustFS A newer, Apache-2.0 object store written in Rust. ## What it is RustFS is the youngest project here and covers everything the driver needs for ephemeral and durable volumes, including POST policies. It is also the least battle-tested of the six, so weigh it accordingly. | | | | --- | --- | | **Project** | [https://github.com/rustfs/rustfs](https://github.com/rustfs/rustfs) | | **Licence** | Apache-2.0 | | **Image tested** | `rustfs/rustfs:1.0.0-alpha.60` | | **S3 port** | 9000 | ## Run it A single-node setup, enough to archive into and to try the driver against. It is not a production topology for any of these products; each project's own documentation covers that. docker-compose.yml: ```yaml services: rustfs: image: rustfs/rustfs:1.0.0-alpha.60 environment: RUSTFS_ACCESS_KEY: archiver-key RUSTFS_SECRET_KEY: archiver-secret RUSTFS_ADDRESS: 0.0.0.0:9000 RUSTFS_VOLUMES: /data RUSTFS_CONSOLE_ENABLE: "false" ports: - "9000:9000" volumes: - rustfs-data:/data volumes: rustfs-data: ``` ## Point the driver at it Endpoint and credentials go on the volume; nothing about the driver's installation changes. `pathStyle` is on because a container reached by address has no per-bucket DNS, which is the usual shape outside AWS. a volume archiving into RustFS: ```yaml apiVersion: v1 kind: Secret metadata: name: rustfs-credentials namespace: default stringData: accessKeyId: archiver-key secretAccessKey: archiver-secret --- apiVersion: v1 kind: Pod metadata: name: writer spec: containers: - name: app image: busybox:1.36 command: ["sh", "-c", "echo hello > /dumps/first.txt; sleep 3600"] volumeMounts: - { name: dumps, mountPath: /dumps } volumes: - name: dumps csi: driver: s3archiver.csi.trion.de nodePublishSecretRef: name: rustfs-credentials volumeAttributes: bucket: archives prefix: "{namespace}/{podName}/" endpoint: http://rustfs.storage.svc.cluster.local:9000 pathStyle: "true" region: us-east-1 ``` To make it the default for every volume instead, set `S3A_ENDPOINT`, `S3A_PATH_STYLE` and `S3A_REGION` on the DaemonSet and leave them off the volumes. The [configuration reference](https://s3archiver.csi.trion.de/configuration.html#environment-driver-container) lists both halves. ## What works Measured, not claimed. An [opt-in test suite](https://s3archiver.csi.trion.de/s3-compatibility.html#how-it-is-tested) runs every one of these against RustFS through the driver's own code paths. | Capability | | What it gives you | | --- | --- | --- | | Single PutObject | ●yes | Archiving anything at all. | | Multipart upload | ●yes | Files over the 64 MiB threshold. A heapdump is almost always over it. | | ListObjectsV2 | ●yes | Segment compaction and durable volumes. Ephemeral archiving never lists. | | DeleteObject | ●yes | Compaction removes fragments it has assembled; durable volumes mirror deletions. | | GetObject | ●yes | Restoring a durable volume at pod start. Ephemeral volumes never read back. | | UploadPartCopy | ●yes | Server-side append and segment assembly. Without it a growing file is re-uploaded whole. | | Offset append [1](https://s3archiver.csi.trion.de/s3-compatibility.html#offset-append) | ○no | The cheapest append, one request carrying only the new bytes. An S3 Express feature. | | SSE-S3 | ○no | Requesting AES256 encryption per volume. A bucket default covers you regardless. | | SSE-KMS | ○no | Per-volume encryption with a customer-managed key. | | Presigned PUT | ●yes | Presigned credential mode, where the node holds no S3 keys. | | Presigned POST policy | ●yes | Signer-less mode, with one prefix-scoped policy in the volume Secret. | 8 of 11 supported. Missing: Offset append, SSE-S3, SSE-KMS. The driver degrades rather than failing for all of these except where noted below. ## Worth knowing **NOTE — Encryption requests fail with 500** SSE-S3 returns an internal error rather than a clean rejection, so leave encryption to the bucket or the disk. **NOTE — Young project, moving fast** Pin the image tag. The compatibility suite pins one, and the matrix on this page describes that tag rather than `:latest`. ============================================================================== Source: https://s3archiver.csi.trion.de/s3-s3proxy.html Updated: 2026-08-30 Summary: Running S3Proxy with docker compose and pointing csi-s3-archiver at it: The endpoint configuration, what it supports, and what to watch out for. S3 endpoints # S3Proxy An S3 API in front of something else. A local filesystem, Azure, Google Cloud Storage. ## What it is S3Proxy translates S3 to a jclouds backend. Pointed at the filesystem it is the shortest path from "I have a directory" to "I have an S3 endpoint", which makes it a good fit for development clusters and for archiving onto an existing NFS mount. | | | | --- | --- | | **Project** | [https://github.com/gaul/s3proxy](https://github.com/gaul/s3proxy) | | **Licence** | Apache-2.0 | | **Image tested** | `andrewgaul/s3proxy:sha-54ef861` | | **S3 port** | 80 | ## Run it A single-node setup, enough to archive into and to try the driver against. It is not a production topology for any of these products; each project's own documentation covers that. docker-compose.yml: ```yaml services: s3proxy: image: andrewgaul/s3proxy:sha-54ef861 environment: S3PROXY_AUTHORIZATION: aws-v4 S3PROXY_IDENTITY: archiver-key S3PROXY_CREDENTIAL: archiver-secret S3PROXY_ENDPOINT: http://0.0.0.0:80 JCLOUDS_PROVIDER: filesystem JCLOUDS_FILESYSTEM_BASEDIR: /data ports: - "8080:80" volumes: - s3proxy-data:/data volumes: s3proxy-data: ``` ## Point the driver at it Endpoint and credentials go on the volume; nothing about the driver's installation changes. `pathStyle` is on because a container reached by address has no per-bucket DNS, which is the usual shape outside AWS. a volume archiving into S3Proxy: ```yaml apiVersion: v1 kind: Secret metadata: name: s3proxy-credentials namespace: default stringData: accessKeyId: archiver-key secretAccessKey: archiver-secret --- apiVersion: v1 kind: Pod metadata: name: writer spec: containers: - name: app image: busybox:1.36 command: ["sh", "-c", "echo hello > /dumps/first.txt; sleep 3600"] volumeMounts: - { name: dumps, mountPath: /dumps } volumes: - name: dumps csi: driver: s3archiver.csi.trion.de nodePublishSecretRef: name: s3proxy-credentials volumeAttributes: bucket: archives prefix: "{namespace}/{podName}/" endpoint: http://s3proxy.storage.svc.cluster.local:80 pathStyle: "true" region: us-east-1 ``` To make it the default for every volume instead, set `S3A_ENDPOINT`, `S3A_PATH_STYLE` and `S3A_REGION` on the DaemonSet and leave them off the volumes. The [configuration reference](https://s3archiver.csi.trion.de/configuration.html#environment-driver-container) lists both halves. ## What works Measured, not claimed. An [opt-in test suite](https://s3archiver.csi.trion.de/s3-compatibility.html#how-it-is-tested) runs every one of these against S3Proxy through the driver's own code paths. | Capability | | What it gives you | | --- | --- | --- | | Single PutObject | ●yes | Archiving anything at all. | | Multipart upload | ●yes | Files over the 64 MiB threshold. A heapdump is almost always over it. | | ListObjectsV2 | ●yes | Segment compaction and durable volumes. Ephemeral archiving never lists. | | DeleteObject | ●yes | Compaction removes fragments it has assembled; durable volumes mirror deletions. | | GetObject | ●yes | Restoring a durable volume at pod start. Ephemeral volumes never read back. | | UploadPartCopy | ●yes | Server-side append and segment assembly. Without it a growing file is re-uploaded whole. | | Offset append [1](https://s3archiver.csi.trion.de/s3-compatibility.html#offset-append) | ○no | The cheapest append, one request carrying only the new bytes. An S3 Express feature. | | SSE-S3 | ○no | Requesting AES256 encryption per volume. A bucket default covers you regardless. | | SSE-KMS | ○no | Per-volume encryption with a customer-managed key. | | Presigned PUT | ●yes | Presigned credential mode, where the node holds no S3 keys. | | Presigned POST policy | ●yes | Signer-less mode, with one prefix-scoped policy in the volume Secret. | 8 of 11 supported. Missing: Offset append, SSE-S3, SSE-KMS. The driver degrades rather than failing for all of these except where noted below. ## Worth knowing **NOTE — Encryption is the backend's business** SSE requests come back as `501 NotImplemented`. With the filesystem backend there is nothing to encrypt with; use disk encryption underneath. **NOTE — Durability is whatever the backend gives you** The filesystem provider is exactly as durable as the volume behind it. That is fine for a scratch archive and not fine as the only copy of a heapdump you care about. ============================================================================== Source: https://s3archiver.csi.trion.de/s3-seaweedfs.html Updated: 2026-08-30 Summary: Running SeaweedFS with docker compose and pointing csi-s3-archiver at it: The endpoint configuration, what it supports, and what to watch out for. S3 endpoints # SeaweedFS A distributed file and object store; the S3 endpoint this project tests against by default. ## What it is SeaweedFS runs master, volume server, filer and S3 gateway in one process, which makes it the easiest of these to stand up and the reason every integration test in this repository uses it. | | | | --- | --- | | **Project** | [https://github.com/seaweedfs/seaweedfs](https://github.com/seaweedfs/seaweedfs) | | **Licence** | Apache-2.0 | | **Image tested** | `chrislusf/seaweedfs:4.40` | | **S3 port** | 8333 | ## Run it A single-node setup, enough to archive into and to try the driver against. It is not a production topology for any of these products; each project's own documentation covers that. docker-compose.yml: ```yaml services: seaweedfs: image: chrislusf/seaweedfs:4.40 command: ["server", "-s3", "-dir=/data", "-ip=0.0.0.0", "-s3.config=/etc/seaweedfs/s3.json"] ports: - "8333:8333" volumes: - ./s3.json:/etc/seaweedfs/s3.json:ro - seaweed-data:/data volumes: seaweed-data: ``` s3.json — the identity SeaweedFS needs before it will accept a signed request: ```json { "identities": [ { "name": "csi-archiver", "credentials": [ { "accessKey": "archiver-key", "secretKey": "archiver-secret" } ], "actions": ["Admin", "Read", "Write", "List", "Tagging"] } ] } ``` ## Point the driver at it Endpoint and credentials go on the volume; nothing about the driver's installation changes. `pathStyle` is on because a container reached by address has no per-bucket DNS, which is the usual shape outside AWS. a volume archiving into SeaweedFS: ```yaml apiVersion: v1 kind: Secret metadata: name: seaweedfs-credentials namespace: default stringData: accessKeyId: archiver-key secretAccessKey: archiver-secret --- apiVersion: v1 kind: Pod metadata: name: writer spec: containers: - name: app image: busybox:1.36 command: ["sh", "-c", "echo hello > /dumps/first.txt; sleep 3600"] volumeMounts: - { name: dumps, mountPath: /dumps } volumes: - name: dumps csi: driver: s3archiver.csi.trion.de nodePublishSecretRef: name: seaweedfs-credentials volumeAttributes: bucket: archives prefix: "{namespace}/{podName}/" endpoint: http://seaweedfs.storage.svc.cluster.local:8333 pathStyle: "true" region: us-east-1 ``` To make it the default for every volume instead, set `S3A_ENDPOINT`, `S3A_PATH_STYLE` and `S3A_REGION` on the DaemonSet and leave them off the volumes. The [configuration reference](https://s3archiver.csi.trion.de/configuration.html#environment-driver-container) lists both halves. ## What works Measured, not claimed. An [opt-in test suite](https://s3archiver.csi.trion.de/s3-compatibility.html#how-it-is-tested) runs every one of these against SeaweedFS through the driver's own code paths. | Capability | | What it gives you | | --- | --- | --- | | Single PutObject | ●yes | Archiving anything at all. | | Multipart upload | ●yes | Files over the 64 MiB threshold. A heapdump is almost always over it. | | ListObjectsV2 | ●yes | Segment compaction and durable volumes. Ephemeral archiving never lists. | | DeleteObject | ●yes | Compaction removes fragments it has assembled; durable volumes mirror deletions. | | GetObject | ●yes | Restoring a durable volume at pod start. Ephemeral volumes never read back. | | UploadPartCopy | ●yes | Server-side append and segment assembly. Without it a growing file is re-uploaded whole. | | Offset append [1](https://s3archiver.csi.trion.de/s3-compatibility.html#offset-append) | ○no | The cheapest append, one request carrying only the new bytes. An S3 Express feature. | | SSE-S3 | ○no | Requesting AES256 encryption per volume. A bucket default covers you regardless. | | SSE-KMS | ○no | Per-volume encryption with a customer-managed key. | | Presigned PUT | ●yes | Presigned credential mode, where the node holds no S3 keys. | | Presigned POST policy | ●yes | Signer-less mode, with one prefix-scoped policy in the volume Secret. | 8 of 11 supported. Missing: Offset append, SSE-S3, SSE-KMS. The driver degrades rather than failing for all of these except where noted below. ## Worth knowing **NOTE — An identity file is not optional** SeaweedFS serves *unsigned* requests anonymously but answers any SigV4-signed request with `400 "Signed request requires setting up SeaweedFS S3 authentication"` unless an identity exists. The AWS SDK always signs, so without `s3.json` nothing works. **NOTE — Offset append is accepted and ignored** SeaweedFS returns 200 to a PutObject carrying `x-amz-write-offset-bytes` and replaces the object with just the tail. S3 semantics say unknown `x-amz-*` headers are ignored, so this is legal and silent. The driver verifies the resulting object size, refuses the result and falls back to full re-uploads, so use `appendUpload: copy` or leave it on `auto`. **NOTE — Encryption requests fail with 500** Asking for SSE-S3 returns an internal error rather than a clean rejection. Set encryption on the bucket instead of per volume. ============================================================================== Source: https://s3archiver.csi.trion.de/s3-versitygw.html Updated: 2026-08-30 Summary: Running Versity S3 Gateway with docker compose and pointing csi-s3-archiver at it: The endpoint configuration, what it supports, and what to watch out for. S3 endpoints # Versity S3 Gateway An S3 protocol translator over POSIX filesystems, aimed at existing HPC storage. ## What it is VersityGW puts an S3 API over a POSIX filesystem while keeping the objects as ordinary files, so what the driver archives stays readable with `ls` and `cat`. That suits sites with existing parallel filesystems. | | | | --- | --- | | **Project** | [https://github.com/versity/versitygw](https://github.com/versity/versitygw) | | **Licence** | Apache-2.0 | | **Image tested** | `versity/versitygw:v1.0.9` | | **S3 port** | 7070 | ## Run it A single-node setup, enough to archive into and to try the driver against. It is not a production topology for any of these products; each project's own documentation covers that. docker-compose.yml: ```yaml services: versitygw: image: versity/versitygw:v1.0.9 command: ["--port", "0.0.0.0:7070", "posix", "/data"] environment: ROOT_ACCESS_KEY_ID: archiver-key ROOT_SECRET_ACCESS_KEY: archiver-secret ports: - "7070:7070" volumes: # Must exist: the posix backend chdirs into it and refuses to start otherwise. - versity-data:/data volumes: versity-data: ``` ## Point the driver at it Endpoint and credentials go on the volume; nothing about the driver's installation changes. `pathStyle` is on because a container reached by address has no per-bucket DNS, which is the usual shape outside AWS. a volume archiving into Versity S3 Gateway: ```yaml apiVersion: v1 kind: Secret metadata: name: versitygw-credentials namespace: default stringData: accessKeyId: archiver-key secretAccessKey: archiver-secret --- apiVersion: v1 kind: Pod metadata: name: writer spec: containers: - name: app image: busybox:1.36 command: ["sh", "-c", "echo hello > /dumps/first.txt; sleep 3600"] volumeMounts: - { name: dumps, mountPath: /dumps } volumes: - name: dumps csi: driver: s3archiver.csi.trion.de nodePublishSecretRef: name: versitygw-credentials volumeAttributes: bucket: archives prefix: "{namespace}/{podName}/" endpoint: http://versitygw.storage.svc.cluster.local:7070 pathStyle: "true" region: us-east-1 ``` To make it the default for every volume instead, set `S3A_ENDPOINT`, `S3A_PATH_STYLE` and `S3A_REGION` on the DaemonSet and leave them off the volumes. The [configuration reference](https://s3archiver.csi.trion.de/configuration.html#environment-driver-container) lists both halves. ## What works Measured, not claimed. An [opt-in test suite](https://s3archiver.csi.trion.de/s3-compatibility.html#how-it-is-tested) runs every one of these against Versity S3 Gateway through the driver's own code paths. | Capability | | What it gives you | | --- | --- | --- | | Single PutObject | ●yes | Archiving anything at all. | | Multipart upload | ●yes | Files over the 64 MiB threshold. A heapdump is almost always over it. | | ListObjectsV2 | ●yes | Segment compaction and durable volumes. Ephemeral archiving never lists. | | DeleteObject | ●yes | Compaction removes fragments it has assembled; durable volumes mirror deletions. | | GetObject | ●yes | Restoring a durable volume at pod start. Ephemeral volumes never read back. | | UploadPartCopy | ●yes | Server-side append and segment assembly. Without it a growing file is re-uploaded whole. | | Offset append [1](https://s3archiver.csi.trion.de/s3-compatibility.html#offset-append) | ○no | The cheapest append, one request carrying only the new bytes. An S3 Express feature. | | SSE-S3 | ●yes | Requesting AES256 encryption per volume. A bucket default covers you regardless. | | SSE-KMS | ●yes | Per-volume encryption with a customer-managed key. | | Presigned PUT | ●yes | Presigned credential mode, where the node holds no S3 keys. | | Presigned POST policy | ○no | Signer-less mode, with one prefix-scoped policy in the volume Secret. | 9 of 11 supported. Missing: Offset append, Presigned POST policy. The driver degrades rather than failing for all of these except where noted below. ## Worth knowing **NOTE — The backend directory must already exist** The posix backend does `chdir` into its root at startup and exits with `no such file or directory` if it is not there. A named volume or a bind mount both work; an unmounted path does not. **NOTE — POST policies are rejected** A POST-policy upload comes back as `400` complaining about the Authorization header, which such an upload does not carry. Use the signer-based presigned mode. **NOTE — Objects are files** An archived object is a plain file on the filesystem, which is the point of this gateway and worth knowing when you go looking for what the driver wrote. ============================================================================== Source: https://s3archiver.csi.trion.de/releases.html Updated: 2026-08-30 Summary: What shipped in each version of csi-s3-archiver, what changed for users, and the versioning and support policy. History # Release history What shipped when, what it means for a cluster running the previous version, and the rules the project holds itself to about compatibility. ## Status **WARN — No release has been tagged yet** There is no published image and no version number to pin. Everything listed under *Unreleased* is on the main branch and passes its tests, but the first tag has not been cut. Until then, build from source: The [install page](https://s3archiver.csi.trion.de/install.html) covers it. ## Unreleased in progress Everything below is on the main branch and has not been tagged yet. | Change | What it means for you | | --- | --- | | Durable volumes | A volume can now outlive its pod. Its contents live in S3 and are restored into a fresh directory at publish, archived and mirrored at unpublish. No `mount(2)`, no privileged container and no node affinity, so the volume survives losing its node. | | PVC-declared volumes | An optional Controller service lets a workload use a `volumeClaimTemplate` instead of an inline `csi:` block. The driver still holds no Kubernetes permissions; the RBAC belongs to the provisioner sidecar. | | Server-side append | A growing file no longer re-uploads whole. `appendUpload` selects `UploadPartCopy`, or native offset-append on S3 Express directory buckets, and segment compaction now assembles server-side. | | Presigned multipart, and signer-less POST policies | A signer that signs the multipart operations lifts presigned mode's 5 GiB cap. A static POST policy in the volume's Secret removes the signer service entirely. | | Statistics web UI | An optional read-only page with uploads per hour and per day, bytes, retries and failures, aggregated across every node through a headless Service. Off by default. | | Prometheus metrics, volume stats, Helm chart | A `/metrics` endpoint, `NodeGetVolumeStats` reporting usage to kubelet, and a chart equivalent to the kustomize base. | | Compression and server-side encryption | `compression: gzip` and SSE-S3 or SSE-KMS per volume. | | Secret rotation without pod restarts | `requiresRepublish`, so a rotated `nodePublishSecretRef` reaches running pods. | | JSON logging | `S3A_LOG_FORMAT=json` for collectors that would otherwise re-parse the text format. | | Measured resource limits | The shipped requests and limits come from `hack/measure-resources.sh` rather than from guesswork, and the measurement is repeatable. | ## Breaking changes None yet. This section exists so that when there is one, it is somewhere obvious rather than buried in a table. Two changes illustrate what counts as one: - The volume state file's schema is at v3, having gained segment bookkeeping and the object length that server-side append needs. Both additions are **additive**: An older file still reads, with the new fields defaulting to zero, so upgrading a driver never discards a node's manifests and re-uploads everything. That is the property to preserve. - The memory request is 128Mi, set from measurement rather than estimate. A value below the driver's idle usage under-provisions every node, which is the kind of change that warrants a note here. ## Versioning policy | Component | Policy | | --- | --- | | **The driver** | Semantic versioning once 1.0 is tagged. Before that, minor versions may change behaviour and the release notes say so. | | **Volume attributes** | Additive. An attribute is never repurposed; an unknown one fails the publish loudly rather than being ignored, so a typo is never silent. | | **State file schema** | Explicitly versioned, and at version 1: Nothing has been released, so no other version exists anywhere. Additions that default harmlessly keep the number and stay readable; one that does not raises it, and a file the driver cannot read is dropped with an error rather than guessed at, because a wrong manifest causes silent data loss. | | **The signer contract** | Versioned explicitly, and at v1. The multipart operations are opt-in per volume rather than probed, because a `putObject`-only signer answers them plausibly and wrongly. | | **Object layout in the bucket** | Stable. Keys are `{prefix}{relative path}`, and the `.parts/` layout for the segments strategy is documented. | | **The CSI spec** | Vendored at v1.12.0. A spec upgrade is a release note. | ## How releases are built Worth knowing if you verify what you deploy. - A `v*` tag triggers the release workflow. Each architecture builds its own native binary on its own runner, because GraalVM cannot cross-compile. - Every architecture image is verified before it is pushed: The CSI conformance suite, an upload round trip in both credential modes, and the statistics UI, all against the binary that ships. - Images are signed with **keyless cosign**. The signature covers the digest, so moving a tag afterwards does not carry it along. - The multi-architecture tag is stitched from the per-architecture ones, and the stitch step refuses to sign a manifest that does not carry every architecture. - The release attaches the rendered install YAML pinned to that release, so applying it later installs the same thing. verify a release: ```bash cosign verify \ --certificate-identity-regexp "^https://github.com/.*/csi-s3-archiver/" \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ ghcr.io/trion-development/csi-s3-archiver:1.2.3 ``` ============================================================================== Source: https://s3archiver.csi.trion.de/why-java.html Updated: 2026-08-30 Summary: A CSI driver in Java 25: What the language gives this problem, what the native image costs and saves, and where Go would still have been the easier choice. Background # Why Java A CSI driver is systems software: It runs as root on every node, it starts before the workloads do, and its memory is charged to the cluster rather than the application. Here is what Java 25 brings to that, what it costs, and where another language would have been the easier choice. ## Summary Kubernetes infrastructure is written in Go by default, and that default is a reasonable one: The CSI spec ships Go bindings, every sidecar is Go, and the ecosystem's examples are Go. This driver is Java. **Modern Java answers the three usual objections**, and the reasons in its favour apply to this problem. | The objection | The answer | | --- | --- | | "A JVM on every node costs hundreds of MiB" | GraalVM native image. The binary idles at 69 MiB RSS and peaks at 156 MiB under load, measured. There is no JVM on the node. | | "Startup takes seconds" | The native binary is serving on its socket in milliseconds. A CSI driver that starts slowly delays every pod on the node. | | "Concurrency means thread pools and tuning" | Virtual threads. Every upload, every RPC and every peer fetch is a blocking call on its own thread, and the code reads like it. | What is left is a language with a mature S3 SDK, exhaustive pattern matching that the compiler enforces, and records that make the domain model unambiguous. The rest of this page is the detail. ## Virtual threads An archiver is almost entirely blocking I/O: Stat a file, read a range, PUT it, wait. The alternatives are a thread pool sized by guesswork, or an async framework and the colour-of-your-function problem that comes with it. Virtual threads make the obvious code correct. ```java // One virtual thread per upload. Blocking, sequential, readable, and // cheap enough that the only bound is a semaphore expressing a policy // rather than a resource. Executors.newThreadPerTaskExecutor( Thread.ofVirtual().name("csi-upload-", 0).factory()); ``` The consequence shows up in the code, not the benchmarks. The upload pipeline retries with backoff by calling `Thread.sleep`. The final sweep waits for the queue to drain by joining futures. The cluster view fetches every peer concurrently with `invokeAll` and a timeout. None of that needs a reactive library, and all of it can be read top to bottom. **NOTE — Where it does not help** Virtual threads park cheaply, they do not make S3 faster. Node upload concurrency is still a bounded semaphore, because the limit that matters is the endpoint's willingness to be hammered, not the JVM's ability to make threads. ## The native image The shipped artefact is a statically linked binary in a `FROM scratch` image: No JVM, no shell, no package manager, no libc. That is a genuine operational benefit for a DaemonSet running as root on every node, and it is not free. | What you get | What it costs | | --- | --- | | 69 MiB idle RSS, milliseconds to start | A build that needs Docker and takes minutes | | A container image with no shell and no CVE surface from a base OS | Reachability metadata to keep current, regenerated by a tracing agent | | No JIT warm-up, so the first upload is as fast as the thousandth | Reflection has to be declared. In practice: The AWS SDK, and nothing this project wrote | | One file to ship per architecture | GraalVM cannot cross-compile, so each architecture builds on its own runner | The failure mode to plan for is **divergence**: Code that works on the JVM and not in the image. A missing `java.net.URL` protocol handler, for instance, breaks every custom S3 endpoint while `mvn verify` stays green. The answer is to make the difference testable rather than to distrust the image: The binary self-checks its upload path, its statistics UI and its CSI conformance in CI, so a divergence fails a build rather than a cluster. ## Language features used ### Sealed interfaces make the compiler check the modes Credentials are one of four things: Static keys, the AWS default chain, a signer endpoint, or a static POST policy. As a sealed interface, *every* switch over them must handle all four, and adding a mode is a compile error in each place that has to change. ```java return switch (credentials) { case Credentials.Static s -> StaticCredentialsProvider.create(...); case Credentials.DefaultChain d -> DefaultCredentialsProvider.create(); case Credentials.Presigned p -> throw new IllegalArgumentException(...); case Credentials.PresignedPost p -> throw new IllegalArgumentException(...); // no default: adding a fifth mode breaks the build here, on purpose }; ``` That is the feature earning its place: The compiler, not a reviewer, finds every site that a new mode affects. ### Records make the domain model unambiguous A volume's configuration is a record with two dozen components. It is immutable, its `equals` is structural (which is what makes "is this republish the same configuration?" a one-liner), and its components are named in one place. Credentials are records too, with hand-written `toString` that redacts, because these objects reach log statements by accident far more often than by design. ### Text blocks, and knowing when not to use them The statistics page is a real HTML file that Maven inlines into a text block at build time. Text blocks suit content that has no editor of its own; beyond that they get in the way, because they strip trailing whitespace and interpret backslashes. The build therefore refuses HTML containing either, and a test asserts the served page is byte-identical to the file. ### The foreign-function API is *not* used Worth saying because it is the obvious place to reach for it. The driver performs no `mount(2)`, which is why it runs unprivileged. That is a design decision rather than a limitation of the language. ## Libraries used | Dependency | Why | What it costs | | --- | --- | --- | | AWS SDK v2 | SigV4, multipart, retries, endpoint handling, and the long tail of S3-compatible store quirks. Reimplementing it would be the project. | Most of the image size, and most of the reachability metadata | | grpc-java + Netty | The CSI transport. Uses the NIO Unix-domain-socket channel, so no native `.so` ships. | A dependency tree, and one silenced log line about a socket option | | Jackson *streaming only* | State files and the statistics API. Streaming, not databind, so no reflective mapping. | Hand-written codecs, which is the point: Every field is read deliberately | | slf4j + an in-project binding | Structured single-line logs. The binding is ~200 lines. | Written rather than chosen, for a reason worth reading | **NOTE — Why the logging backend is hand-written** slf4j-simple reads its configuration in a static initializer. Native image runs static initializers at *build* time for classes it initializes early, and Netty obtains loggers during the build. The log level would have been frozen into the binary and `S3A_LOG_LEVEL` would have silently done nothing. Two hundred lines of binding was cheaper than that surprise. Four dependencies, one of which is the S3 SDK. There is no dependency-injection container: The object graph is a handful of singletons wired by hand in one file, which reads better than annotations and adds no reflection for the image to configure. ## Where Go fits better This page would be less useful without the other column. - **The CSI ecosystem is Go.** The spec's reference bindings, every sidecar, csi-sanity, and every example a maintainer will search for. The Java side of that is a vendored `.proto` and one local patch to it, because protoc's Java generator cannot emit a message with both a `parameters` and a `mutable_parameters` map. - **Static binaries are the default**, not a build mode with a toolchain image, a tracing agent and metadata to keep current. - **Cross-compilation is one environment variable.** GraalVM cannot cross-compile at all, so multi-architecture images are stitched from per-architecture runners. - **A smaller binary.** An equivalent Go driver would be a fraction of csi-s3-archiver's 78 MiB image, which is dominated by the AWS SDK. The trade the project made: A heavier build and a larger image, for a language its authors write faster and more safely, with a mature S3 client and a compiler that enforces the domain's invariants. If the deciding factor for you is image size or fitting the Go ecosystem, those are good reasons, and this page is not an argument against them.