Operations

Pod deletion waits for the final sweep#

When the pod terminates, the driver uploads everything new or changed one last time, ignoring the quiescence window, and blocks pod deletion until it finishes. A heapdump written seconds before the pod died is exactly the file you most want archived, and returning early would lose it.

Semantics and caveats#

Things that will bite you if you assume otherwise. Each is a design choice rather than an oversight.

Objects are overwritten, not versioned.

A file uploaded twice lands on the same key the second time. If you need history, enable bucket versioning or put {date} in the prefix. The same applies in spool mode: After deleteAfterUpload removes a file, a later file with the same name overwrites the object.

Change detection is size + mtime.

A file rewritten with exactly the same size and timestamp is not noticed. Hashing every candidate would mean reading multi-gigabyte heapdumps twice on every scan. The periodic rescan and the final sweep use the same comparison, so they do not rescue this case either.

Per-volume secrets are read once, at publish.

Rotating the Secret takes effect for new pods only; existing pods keep the credentials they were published with. Presigned mode does not have this limitation, because every upload fetches a fresh URL.

A driver restart re-resolves credentials from the driver's own environment.

kubelet only delivers a nodePublishSecretRef secret on a fresh publish, so a recovered volume that used one falls back to whatever driver-global credentials exist, possibly none. The driver logs a warning naming the volume when this happens.

Presigned mode caps files at 5 GiB, unless the signer does multipart.

A single presigned PUT cannot be multipart, so larger files fail with a clear error rather than being split into objects you did not ask for. Setting presignMultipart: "true" has the driver ask the signer for the multipart operations instead, which raises the ceiling to about 312 GiB. It is opt-in because a putObject-only signer answers those requests plausibly and wrongly.

Symlinks are never followed.

The driver runs as root on the node, so following a symlink a workload planted would upload arbitrary node files into that workload's bucket. Symlinks are skipped and logged; only regular files are archived.

Object keys are capped at 1024 UTF-8 bytes.

A file whose key would exceed S3's limit is skipped and logged, never truncated, because a truncated key would overwrite an unrelated object.

No restore.

This driver only writes. Objects are never read back into a volume, and there is no command to reverse an archive.

Resource requests and limits#

The values shipped in deploy/ are a starting point, not a recommendation for your cluster. They come from hack/measure-resources.sh, which archives a synthetic workload through the real engine and samples the native binary's RSS and CPU while it works.

Memory is flat in file size. A 512 MiB file peaks no higher than thirty-two small ones, because uploads stream from disk in fixed buffers and switch to multipart above a threshold: There is no point at which the driver holds a file in memory. What moves the number is concurrency, and S3A_NODE_CONCURRENCY bounds that.

resources:
  requests:
    cpu: 10m
    memory: 128Mi
  limits:
    memory: 256Mi   # a starting point -- measure your own workload

No CPU limit, on purpose. Archiving is bursty and idle most of the time, and throttling the driver mid-upload only makes the unpublish sweep -- which blocks pod deletion -- take longer. If a noisy-neighbour policy forces one, give it at least two cores: The sweep runs S3A_NODE_CONCURRENCY uploads at once and measured 2.4 cores at concurrency 16.

Troubleshooting#

Versions#

The vendored csi.proto carries one documented local patch: The Controller-service map field mutable_parameters is renamed to mutable_params, because protoc's Java generator cannot emit a message that has both it and a parameters map. Field numbers, and therefore the wire format, are unchanged.