What works on which S3

The matrix#

Every feature the driver asks an object store for, against every implementation it is tested with. These are test results. An opt-in suite starts each product and runs each probe through the driver's own code, and the numbers below are what it observed.

Each product has its own page with a docker-compose.yml, the volume configuration to point the driver at it, and the things that caught us out:

• SeaweedFS. A distributed file and object store; the S3 endpoint this project tests against by default.
• Garage. A self-contained store aimed at small, self-hosted, geo-distributed clusters.
• S3Proxy. An S3 API in front of something else. A local filesystem, Azure, Google Cloud Storage.
• Zenko CloudServer. Scality's S3 server, the storage engine behind Zenko.
• Versity S3 Gateway. An S3 protocol translator over POSIX filesystems, aimed at existing HPC storage.
• RustFS. A newer, Apache-2.0 object store written in Rust.

How to read it#

A missing capability is usually a slower driver, not a broken one. What each one costs:

Two are not negotiable. Without single PutObject and multipart upload the driver cannot archive at all, and the test suite treats their absence differently from the rest. Every product here has both.

The rest degrade. Without UploadPartCopy a growing log is re-uploaded whole instead of appended, which still archives correctly and costs bandwidth that grows with the square of the file size, so use appendStrategy: segments there. Without ListObjectsV2 or DeleteObject you lose segment compaction and durable volumes, but ephemeral archiving never calls either.

Choosing one#

How it is tested#

The suite starts each product in a container, creates a bucket and runs eleven probes through the driver's ObjectStore, the same code an archiving volume uses. A pass means the driver's path works against that store, not that the store implements some verb.

mvn verify -Pcompat                        # every product
mvn verify -Pcompat -Dcompat.only=GARAGE   # one of them

It is not part of the normal build. It starts a different object store per product and pushes real multi-megabyte objects through each, so it costs minutes and gigabytes of image pulls.

Note 1: Offset append, and why the column is empty#

Every product in the table answers no to this row, which looks like six projects missing the same feature. They are not. Offset append is not part of the S3 API.

It belongs to one storage class, at one provider

A PutObject carrying x-amz-write-offset-bytes appends to an object in place. AWS added it in 2024 for S3 Express One Zone directory buckets only. It does not work on AWS's own general-purpose buckets, so a store cannot implement "S3" and get it along the way. It would have to implement a directory-bucket API that has no specification outside AWS's documentation and no other implementation to interoperate with.

That is why the column is empty, and why it is likely to stay empty. It is not a gap these projects are behind on.

Ceph has an append, and it is a different one

Ceph's RADOS Gateway is the closest thing to an exception. It supports appending through PUT /bucket/key?append&position=N, an extension it inherited from the Aliyun OSS API rather than from S3. The idea is the same and the wire format is not. It uses a query parameter instead of a header, with a different request shape and a different response.

So a driver that speaks AWS's version does not speak Ceph's. Supporting it would be a third append mode next to the two below, not a matter of turning something on. Ceph is not in the table because it is not in the test suite yet; if it is added, this is the row that would need a second footnote rather than a yes.

What the driver does instead

The driver does implement offset append, and uses it when it is really there. appendUpload defaults to auto, which resolves per bucket:

So on every store in this table the answer is copy, which all six support. Nothing is lost by the empty column except one request per append.

Why not always use copy?

It is the obvious question, since copy works nearly everywhere and offset works almost nowhere. Three reasons, in increasing order of how much they cost you.

An append by copy is four requests, not one. It opens a multipart upload, copies the existing object into part 1, uploads the new tail as part 2, and completes. offset is a single PutObject. For a log synced every few seconds that is a four-fold difference in request count, and requests are what most stores bill for.

Copy has a floor at 5 MiB. S3 requires every part but the last to be at least 5 MiB, so an object smaller than that cannot be part 1. The driver detects this and declines rather than failing, which means a file re-uploads whole until it passes 5 MiB. offset has no minimum. That matters most for exactly the files people turn appendOnly on for, which start small and grow slowly.

The store still copies the whole object every time. This is the one worth understanding, because it is invisible from the outside. UploadPartCopy moves no bytes over the network, which is what makes it attractive, but the store internally rewrites the entire existing object on every append. Append n times to a file that reaches S bytes and the store has done roughly n × S bytes of internal copying, even though only S bytes crossed the network.

So copy fixes the bandwidth problem and leaves the storage-side one. On a managed service you may never notice; on a store you run yourself, on your own disks, it is your I/O. For a file that grows for hours, use appendStrategy: segments, which uploads each new chunk as its own object and assembles them once, rather than either append mode.

The last reason is smaller but real. A multipart upload is a resource that exists between requests. The driver aborts it on failure, but a process killed between opening and aborting one leaves it behind, and abandoned uploads are billed until a lifecycle rule reaps them. offset has no such lifecycle, because there is nothing to leave open.