Questions people actually ask

Nothing here comes with a warranty. The software is provided as is, without warranties or conditions of any kind, and without liability for what it does to your data or your cluster. Nobody is on call for your bucket.

What does exist is a test suite that runs on every build:

• Unit tests over the archiver engine, the upload pipeline and retry policy, config and prefix-template validation, credential resolution, volume state and recovery, and the gRPC services.
• Integration tests against a real object store. SeaweedFS in Testcontainers exercises single-PUT, multipart and presigned uploads, and the archiver engine end to end against it.
• CSI conformance. The upstream csi-sanity suite runs against both the JVM build and the GraalVM native binary. It proves the Identity service and the Node service's capabilities, node id and argument validation. It does not prove a full publish, write and unpublish cycle, because csi-test skips those specs for any plugin without a controller service.
• Crash and restart tests. One suite forks the real jar and SIGKILLs it, checking that persisted volume state is recovered rather than re-uploaded; another covers the same ground for interrupted uploads.
• A log-capture test that fails the build if any secret material reaches a log line, plus a unit test asserting credentials never reach the state files on disk.
• End to end against a real kubelet, with K3s in Testcontainers.

So the behaviour described on this site is tested behaviour rather than aspiration. What is missing before anyone should call it production-ready is a release, and mileage on real clusters that are not ours. Read the limits below before you decide. Several of them are permanent design choices rather than gaps that will close, and the comparison page lists the cases where a different tool is simply the right answer.

Not yet, but that is the plan. The source is not published at the moment. When it is, this answer will be replaced by a link to the repository and the licence.

No, and not by accident. It is not s3fs, goofys or mountpoint-s3. Files the pod writes are ordinary files on the node's disk, written at local-disk speed; the driver copies them to S3 afterwards, asynchronously and one-way. Nothing in the write path talks to the network, so a slow or unavailable object store never blocks the application's write(2).

The trade is that the volume is not a view of the bucket: ls shows what this pod wrote, not what is in S3.

Not while the pod runs. Archiving is one-way, and there is no read-through, so ls shows what this pod wrote rather than what is in the bucket. That is a non-goal rather than a missing feature.

One exception, at one moment: A durable volume is restored from S3 at publish, before the containers start, so the next pod finds what the last one left. A fresh directory filled once, not a view of the bucket. If you need read-back during the run, the comparison page names tools that do it.

Good at: JVM heapdumps, core dumps, rotated logs, database dump files, diagnostic bundles, CI artefacts. Anything a pod produces as whole files that somebody may want later.

Bad at: Anything that needs the object store to be the source of truth during the run, shared read-write storage between pods, data the application must read back, or files that are rewritten in place thousands of times a second.

Uploads retry with exponential backoff and jitter, capped, indefinitely. The contract is at-least-once, eventually. During the pod's life this is invisible: The file is on local disk and the retries happen in the background.

At pod termination it is very visible, because the final sweep blocks pod deletion until the upload succeeds. A pod can sit in Terminating for as long as S3 is down. That is the intended default, since losing the last heapdump is worse, but it is bounded per volume with sweepTimeoutSeconds.

Per-volume state records and upload manifests live under S3A_STATE_DIR on a host path that outlives the container, so a restarted driver recovers its volumes and resumes rather than re-uploading everything. Point S3A_STATE_DIR at emptyDir and you lose that property.

One caveat worth knowing: Kubelet only delivers a nodePublishSecretRef secret on a fresh publish, so a recovered volume that used one falls back to whatever driver-global credentials exist, possibly none. The driver logs a warning naming the volume when that happens.

If the node itself is destroyed, the local files go with it. A pod-local scratch disk is exactly as durable as the node, which is why the sweep is synchronous.

The driver logs one structured line per event on stderr, which is what kubectl logs shows:

2026-07-27T09:15:30.402Z INFO  [csi-rpc-3] NodeService - rpc=NodePublishVolume volumeId=csi-9f3a bucket=prod-dumps status=OK duration_ms=6
2026-07-27T09:16:01.884Z INFO  [archiver-1] Archiver - event=quiescent file=java_pid1.hprof bytes=4194304
2026-07-27T09:16:04.117Z INFO  [upload-2] Uploader - event=upload key=heapdumps/default/heapdump-example/java_pid1.hprof bytes=4194304 attempt=1 status=OK duration_ms=2231
2026-07-27T09:17:12.006Z INFO  [csi-rpc-5] NodeService - rpc=NodeUnpublishVolume volumeId=csi-9f3a event=sweep uploaded=0 status=OK duration_ms=41

A permanent failure (any 4xx) is logged as FAILED_PERMANENTLY and dropped rather than retried forever. If nothing appears at all, raise S3A_LOG_LEVEL to debug and run check-upload from a pod using the driver image and the same environment. It exercises the same upload path and reports exactly what S3 answered.

The driver container requests 10m CPU and 64 MiB of memory, with a 256 MiB memory limit and, on purpose, no CPU limit: Throttling the driver mid-upload just makes the unpublish sweep, which blocks pod deletion, take longer. The two upstream sidecars request 5m and 16 MiB each. Files are streamed from disk with fixed-size buffers, so a multi-gigabyte heapdump does not need multi-gigabyte heap.

The image is a GraalVM native binary in a FROM scratch image: No shell, no package manager, no libc.

For a normal volume, one PutObject per file, plus one multipart upload (initiate, n parts, complete) per file over 64 MiB, with 32 MiB parts.

appendOnly changes the arithmetic and is the case to think about. rewrite costs one full re-upload every appendSyncSeconds, so total bytes transferred grow quadratically with the file size. appendStrategy: segments uploads only the new tail bytes, which is linear, at the cost of one object per segment plus deletes at compaction. For a log that runs for hours, use segments.

Yes. include and exclude take comma-separated globs matched against paths inside the volume. A common shape is to exclude the writer's temporary files so half-written objects never reach the bucket: exclude: "*.tmp,*.partial".

Not any more. In appendOnly mode with the rewrite strategy, appendUpload sends only the new tail and lets the server keep what it already has, using UploadPartCopy or, on an S3 Express directory bucket, a native offset append. Segment compaction at pod termination also assembles server-side rather than re-uploading the file, which matters because that upload is the one blocking pod deletion.

It falls back to a full upload whenever the store cannot do it, and the fallback writes the same object, so losing the optimisation costs bandwidth and never correctness.

Three ways, all off by default. S3A_METRICS_PORT serves Prometheus metrics, which is where alerting belongs. S3A_WEBUI_PORT serves a read-only page with uploads per hour and per day, bytes, retries and failures with their cause; set S3A_WEBUI_PEERS to a headless Service and any pod renders the whole cluster. And S3A_LOG_FORMAT=json gives a collector one JSON object per line instead of a format it would have to re-parse.

The page has no authentication. Keep it on a port-forward or behind an authenticating ingress.

Not for the default shape. A CSI ephemeral inline volume is declared in the pod spec and lives and dies with the pod: No PV, no PVC, no StorageClass, no controller, and nothing to deploy beyond the DaemonSet. That is the whole install, and it is what the first file walkthrough uses.

A PVC is an option, not a requirement. Deploying the optional Controller adds a StorageClass and lets a workload declare the volume with a volumeClaimTemplate, which suits tooling and admission policies built around PVCs. It is also what durable: "true" needs. The driver still holds no Kubernetes permissions in either shape, because the RBAC belongs to the provisioner sidecar rather than to the node plugin. See declaring the volume with a PVC and volumes that outlive their pod.

Neither. The ServiceAccount has no Roles or bindings at all, because the driver never talks to the API server. kubelet pushes pod metadata and per-volume secrets into the CSI calls, and registration happens over a local socket.

The container runs as root but not privileged, with no added capabilities and a read-only root filesystem. Root is needed only to create and remove directories under /var/lib/kubelet/pods, which are root-owned. Because the volume is a plain directory rather than a mount, there is no mount(2) in this driver, and so no need for privileged: true or bidirectional mount propagation.

Yes. The driver creates the volume directory world-writable and the CSIDriver object sets fsGroupPolicy: File, which makes kubelet apply the pod's fsGroup ownership itself. A non-root container with an fsGroup can write into the volume without further configuration.

Yes. Set endpoint and usually pathStyle: "true" per volume, or the S3A_ENDPOINT and S3A_PATH_STYLE defaults driver-wide. The integration suite runs against SeaweedFS on every build; the same code path serves Ceph RGW and AWS. Anything that implements the S3 API for PutObject and multipart should work.

Do nothing. If no per-volume secret and no S3A_ACCESS_KEY_ID are present, credential resolution falls through to the AWS default provider chain, which picks those up. Give the driver's ServiceAccount the role and it works with no driver configuration.

Because that is the fixed upstream Kubernetes field name, not this project's naming. It means "the Secret kubelet passes into the NodePublishVolume RPC". The Secret lives in the pod's own namespace, and it is read once, at publish time, so rotating it takes effect for new pods only.

Whatever your prefix says should happen. The default prefix is {namespace}/{podName}/, so two pods never collide. If you flatten the prefix on purpose, the second upload overwrites the first: Objects are overwritten, never versioned. Add {date}, add {podUid}, or enable bucket versioning if you need history.

Change detection is size + mtime. A file rewritten to exactly the same size with the same timestamp is not noticed, by the watcher, by the periodic rescan, or by the final sweep. Hashing every candidate would mean reading multi-gigabyte heapdumps twice on every scan, which is the worse trade for this workload.

Yes, with durable: "true" on a PVC-declared volume, and it works differently from how you might expect. The contents are not kept on the node: They live in S3, and the driver restores them into a fresh directory when the next pod publishes the volume, then archives what changed and mirrors deletions when it ends.

That design avoids everything node-local storage would have cost: No mount(2), so the driver stays unprivileged; no node affinity, so the volume survives losing its node entirely. What it costs is a blocking download at every pod start, ReadWriteOnce with no attach step enforcing it, and no POSIX semantics between sweeps. If you cannot rebuild the data, use real storage.

Yes, if you deploy the optional Controller. It exists because plenty of tooling, charts and admission policies are built around PVCs, and that is a presentation choice rather than a storage one.

Without durable, the volume a volumeClaimTemplate gives you is still pod-lifetime: Archived and removed when the pod ends. That is exactly right for a generic ephemeral volume, whose claim Kubernetes deletes with the pod, and it is not what a standalone PVC leads people to expect. The driver cannot tell the two apart, so the StorageClass is named for what it gives you.

No. Symlinks are never followed: The driver runs as root, so following a symlink a workload planted would exfiltrate arbitrary node files into that workload's bucket. Symlinks are skipped and logged, and only regular files are archived. Hard links cannot escape the volume's filesystem scope.

No. Credentials are held in memory only: Never written to the state files, and toString() on the resolved credentials is redacted. Both properties are unit-tested, and a log-capture test fails the build if secret material appears in any log line.

If you want no credentials on the node at all, use presigned mode: Set presignEndpoint and the driver asks a signer service for a fresh URL per upload. Combining presigned mode with static keys in the same secret is rejected, since that would ship to the node exactly what the mode exists to avoid. The cost is a 5 GiB per-file cap, because a single presigned PUT cannot be multipart. A signer that also signs the multipart lifecycle lifts the cap; set presignMultipart: "true" on the volume once yours does.

In transit, yes. Uploads go over HTTPS to whatever endpoint you configure.

At rest, both options exist. A bucket default applies to everything the driver writes, and serverSideEncryption requests it per volume: AES256 for SSE-S3, or aws:kms with sseKmsKeyId. Set the bucket default anyway, since it covers a volume whose author forgot to ask. Three of the six stores on the endpoint matrix answer 500 or 501 to an SSE request rather than rejecting it cleanly, so check yours first.

Two ways. Presigned mode asks a signer service for a URL authorising one object key, valid for minutes, so the node holds only a bearer token. POST policy mode puts one signed, prefix-scoped policy in the volume's Secret, so there is no signer to run at all.

The trade for the second is that a policy lives at most 7 days and has to be rotated; include an expiresAt in the bundle and the driver warns twelve hours ahead rather than letting every upload start failing at once. Both modes are covered on the security page.

Normal mode streams from disk and switches to multipart above 64 MiB with 32 MiB parts, so file size is bounded by S3's own limits and the node's disk, not by the driver's memory.

Presigned mode caps a file at 5 GiB by default, because a single presigned PUT cannot be multipart; larger files fail with a clear error rather than being silently split into objects you did not ask for. With presignMultipart: "true" and a signer that signs the multipart operations the driver runs the whole multipart lifecycle instead, which raises the ceiling to about 312 GiB, being S3's 10 000 parts at the driver's 32 MiB part size.

Object keys are capped at S3's 1024 UTF-8 bytes. A file whose key would exceed that is skipped and logged, never truncated, because a truncated key would overwrite an unrelated object.

linux/amd64 and linux/arm64, built on native runners per architecture and stitched into a multi-arch manifest, with no QEMU involved. Linux nodes only; Windows is a non-goal.

All three. The chart is equivalent to the kustomize base, so kubectl apply -k deploy/base and a helm install give the same DaemonSet.

Both endpoints are off unless you set their port, because a node plugin that opens a listener nobody asked for is a surprise. S3A_METRICS_PORT turns on Prometheus metrics; S3A_WEBUI_PORT turns on the statistics page, which aggregates every node through a headless Service and has no authentication, so keep it behind something.

Whatever had already been archived is safe, and whatever was still only on disk is re-archived when the node comes back: Recovery reads the state files, re-registers the volumes, and the manifest stops it re-uploading what already landed. A volume whose pod directory is gone is dropped without affecting the node's other volumes.

What is not recoverable is a node that never comes back with its disk intact. That is the trade for keeping the write path local, and it is why the contract is at-least-once and eventual rather than synchronous. If you need a write to be durable before the application continues, you need the application to write to S3.

Because it is built as a GraalVM native image, the usual objections do not apply: There is no JVM in the runtime image, no warm-up on the publish path, and the container is FROM scratch. What Java buys is the AWS SDK, mature gRPC, and Java 25 virtual threads. Every RPC and every upload is plain blocking code on a virtual thread, with no async framework and no reactive types anywhere in the codebase.

Because the file written seconds before a pod died is the one you most want. A driver that returns immediately from NodeUnpublishVolume and uploads in the background would lose exactly that file, silently, on every OOMKill.

The cost is stated rather than hidden: If S3 is unreachable, the pod stays in Terminating. sweepTimeoutSeconds converts that into best-effort teardown for the volumes where a stuck pod is the worse outcome.

A heapdump is written over many seconds; uploading it on the first IN_MODIFY would upload a truncated file, then do it again, and again. Waiting for the writer to stop touching the file, quiescenceSeconds and 30 by default, means one upload of one complete file. Files that never go quiet are what appendOnly is for.