[gavin-thompson-postman]

Stacked on #1207, which is stacked on #1139. This branch contains both PRs' commits. Until they merge, review only the last three commits, 8db4b07d (the suspended state), 711d7526 (restricted-credential fixes) and bd5c8c59 (no Copy Blob fallback when versioned): diff against #1207. I'll rebase once they're in.

Rebased onto #1207's fix for refused version listings (d4b1917b). That fix adds S3Exceptions.accessDenied, and 711d7526 now reuses it instead of adding its own. Because of it, an account SAS with srt=o now gets 403 AccessDenied instead of 500 on a GET or HEAD of a missing key and on a copy from a missing source.

This is phase 3 of the plan in #1139: suspended-state semantics (the null version and suspended delete markers). Until now PutBucketVersioning Suspended answered 501. Everything here applies only when s3proxy.azureblob.versioning=true. An unversioned store is unchanged.

Why it needs emulating. Azure turns versioning on for the whole storage account and can't stop versioning one container. S3 keeps the state per bucket. A suspended bucket keeps its existing versions, but each new write becomes the single null version and replaces the one before it.

What S3Proxy does now:

• PutBucketVersioning / GetBucketVersioning: Suspended is stored as container metadata, s3proxy_versioning=Suspended. Enabled is stored as the entry being absent. Azure replaces container metadata as a whole, so the entry is merged into what the container already has. A container nobody has suspended still reports Enabled, never "unversioned" (same gap as before).
• PUT, copy and CompleteMultipartUpload while suspended: the blob is written with s3proxy_null_version=true, and the response has no x-amz-version-id. The transient store also leaves the header out. After the write lands, the null version it replaced is deleted for good:
  • an older tagged version, by its id
  • a base blob written before the account was versioned: Azure gives it an id of its own when it's overwritten, so it's found by its ETag, the way Version objects on azureblob #1139's deleteNullVersion finds it
• DELETE with no version id while suspended: writes a delete marker that is itself the null version. It replaces the previous null version and reports x-amz-delete-marker: true, x-amz-version-id: null. DeleteObjects reports DeleteMarkerVersionId null.
• versionId=null: GET, HEAD, copy-source and DELETE resolve it to the null version. That's the base blob while it has no version id, or else the newest tagged version. GET, HEAD and ListObjectVersions report that version as null, and the null delete marker as a DeleteMarker named null. DELETE versionId=null removes every tagged version. When the null version was current, the newest remaining version is promoted, as in Emulate delete markers on a versioned azureblob store #1207.
• Re-enabling: new writes get real ids, and the null version stays listed as null among them. It's no longer the newest version, so a null version-id marker can't be compared as a timestamp. Paging now resolves it to the version it names.
• Reserved metadata: s3proxy_null_version is reserved like the marker key. It's dropped from caller metadata, and from the metadata GET and HEAD report on a versioned store. A copy of the null version made while Enabled isn't tagged.

One fix to #1139's copy along the way. On a versioned store, copy now sends metadata in the Put Blob From URL request itself, as x-ms-meta-* headers through the SDK's AddHeadersFromContextPolicy. The SDK's options have no metadata field. It also passes x-ms-copy-source-blob-properties: false and sets the source's content headers explicitly, with REPLACE overriding each one. Without that, Azure copies the source's metadata, null tag included, whenever the request names none.

The old REPLACE path ran setMetadata after the copy. On a real account that call creates a version of its own. So the copy left two versions, and the response named the wrong one. Measured through S3Proxy on the test account:

• Emulate delete markers on a versioned azureblob store #1207: copy VersionId ≠ HEAD VersionId, 2 versions listed
• This branch: same id, 1 version

Content-type and content-language behave as before: the source's are kept unless REPLACE names them.

No Copy Blob fallback on a versioned store (bd5c8c59). When Put Blob From URL answers 501 (only Azurite does), copy falls back to the asynchronous Copy Blob API. That fallback can't copy correctly onto a versioned store: Copy Blob copies the source's metadata when the request names none, so an Enabled copy of the null version would keep its tag, and REPLACE's content headers are set by a second, versioned write. Azurite keeps no versions, so a versioned store never needs the fallback; it now refuses the copy as not supported (501). An unversioned store falls back as before.

Costs (versioning on):

• Every write and plain delete reads the container's properties: one extra request. It isn't cached, so every proxy instance in front of the container sees a suspend immediately.
• While suspended, a write also reads the blob's properties first (to spot a pre-versioning base blob). It then lists the key's versions and deletes the one it replaced.
• versionId=null costs a properties call. If the null version isn't current, it also costs a version listing.

Races and failure ordering:

• Write first, then delete the old null version. If the delete fails, or the process dies in between, the key has an extra tagged version rather than none. The newer one is reported as the null version. The older one is hidden from listings and removed by the next suspended write or by DELETE versionId=null. A failed delete doesn't fail a write that already landed, since a client retry would only add another version. How it's logged depends on the cause (next section).
• Concurrent suspended writes. A write only deletes tagged versions older than itself, so the later write survives. 5 concurrent PUTs on the real account left exactly one Azure version.
• No conditional container metadata write. Set Container Metadata can't be conditioned on an ETag. A concurrent write of the container's metadata by something else between the read and the write would be lost. S3Proxy writes no other container metadata.
• Changing the state mid-write. A write that started before a status change can land under the old state. AWS also documents that a versioning change can take a while to apply.

Restricted credentials:

• Can't read the container's properties (a container service SAS, or an account SAS without resource type c): with versioning on, every write is refused 403 AccessDenied, and nothing is written. That covers PUT, copy, CompleteMultipartUpload and a plain delete. The error message names what's missing. Such a credential can't tell whether a broader one has suspended the container. Before this, it wrote as Enabled, so in a Suspended bucket every write silently landed as a real-id version.
• Can't delete versions (no deleteBlobVersion, for example Storage Blob Data Contributor or a SAS without x): the suspended write still succeeds. Azure refuses the cleanup with 403 OperationNotAllowedOnAutomaticSnapshot, and every suspended overwrite will be refused the same way, so hidden versions pile up. This is now logged at ERROR, naming the permission, at most once per container every 5 minutes. Before, it was a WARN on every write. Other cleanup failures are still logged at WARN.

Permissions (versioning on). These are in the README too:

• Every versioned write reads the container's properties. That needs RBAC Microsoft.Storage/storageAccounts/blobServices/containers/read, or an account SAS with resource type c.
• Deletes by version and suspended overwrites need Microsoft.Storage/storageAccounts/blobServices/containers/blobs/deleteBlobVersion/action (SAS permission x). Storage Blob Data Contributor lacks it; Storage Blob Data Owner or a custom role has it.
• Changing the versioning status needs container write.
• A container service SAS can't read container properties at all, so it can't be used with versioning on.

Not covered / open questions:

• The PUT response header. Whether S3 sends x-amz-version-id: null on a suspended PUT, or no header, I couldn't confirm: my AWS session had expired, so I couldn't probe it. This follows the transient store, which leaves it out, and VersionedBlobStoreTest.testSuspendedWritesReplaceTheNullVersion.
• Deleting a null marker by id: DELETE versionId=null on a null delete marker doesn't report x-amz-delete-marker: true. Deleting any marker by id has the same gap in Emulate delete markers on a versioned azureblob store #1207.
• Legacy base blob, end to end: I didn't exercise the case of a base blob written before the account was versioned. It would mean turning versioning off on a shared test account. The legacy path keeps Version objects on azureblob #1139's deleteNullVersion logic, and the unit tests cover it.

Testing:

• Unit tests (AzureBlobStoreTest, row-level test doubles in the existing style), 15 new and 2 updated:

  • container status read, merge and removal
  • the null tag, reserved-key stripping and reported ids
  • which version is the null version (newest tagged, base blob first)
  • listing a tagged null version among newer Enabled versions, and hiding a replaced one
  • paging past a null marker
  • the null delete marker
  • which versions a suspended write removes, including a racing newer write
  • copy metadata (COPY, REPLACE, suspended) and the x-ms-meta-* headers
  • against a local fake Azure endpoint: a write and a plain delete are refused AccessDenied when Get Container Properties answers 403, with nothing written; a suspended overwrite whose version delete answers 403 OperationNotAllowedOnAutomaticSnapshot still succeeds and takes the permission path; a 500 on cleanup still succeeds and takes the WARN path
  • the per-container rate limit on the ERROR

  After the rebase, the full mvn test passes (676 tests, 0 failures; 677 with bd5c8c59's test), as does mvn verify -DskipTests -Pjdeps (checkstyle, Error Prone/NullAway) on JDK 25.

• Against the real account (blob versioning on), with the Azure SDK first:

  • Put Blob From URL with x-ms-meta-* headers replaces the source's metadata and keeps its content headers
  • with no metadata headers, it copies the source's metadata
  • copy-source-blob-properties: false copies neither
  • Set Blob Metadata creates a new version
  • container metadata round-trips

• End to end through a local S3Proxy (azureblob-sdk, s3proxy.azureblob.versioning=true), all passing. Raw Azure state was checked with az storage blob list --include vm.

  • fresh container reports Enabled; suspend, then status reads Suspended
  • two suspended PUTs: no VersionId, one null version, Azure holds exactly 2 versions (the enabled one and the null one)
  • GET and HEAD with versionId=null report null; the tag doesn't leak into metadata; the enabled version is still readable
  • suspended DELETE: null marker, still 2 Azure versions; GET answers 404, GET versionId=null answers 405
  • DELETE versionId=null removes the marker and promotes the enabled version; repeating it is a no-op
  • suspended copy (COPY and REPLACE) and suspended multipart: no VersionId, a single null version, metadata and content-type correct
  • DeleteObjects while suspended: DeleteMarkerVersionId null
  • 5 concurrent suspended PUTs: one null version
  • re-enable: PUT gets a real id and the null version stays; paging with MaxKeys=1 yields E4, E3, null
  • copy of versionId=null while Enabled is untagged
  • DELETE versionId=null while Enabled removes only the null version; a plain delete writes a real-id marker
  • caller-supplied x-amz-meta-s3proxy_null_version is dropped

• The external S3 versioning conformance suite used for Version objects on azureblob #1139: 40 of 41 pass (Emulate delete markers on a versioned azureblob store #1207: 37 of 41). The 4 Suspended-state tests now pass. The one remaining failure is "fresh bucket versioning unset", which is by design (Azure versioning is account-wide).

• Restricted credentials, end to end on the real account (one local S3Proxy per credential, raw state from az storage blob list --include vm). Before is 5e1d100a; after is 1aab9fe2 (now rebased as 8db4b07d and 711d7526):

  Credential	Before	After
  account SAS srt=o, racwdlx	writes to a Suspended bucket 200 with real version ids; 2 PUTs left 2 untagged Azure versions	PUT, copy and plain DELETE 403 AccessDenied naming containers/read / resource type c; 0 Azure versions written
  container service SAS racwdlx	same as above: 2 untagged versions	PUT 403 AccessDenied (Azure code AuthorizationFailure); 0 versions written
  account SAS srt=co, racwdl (no x)	PUTs 200, one WARN per write, 4 PUTs left 4 Azure versions	PUTs 200 with no version id, listed as one null version; 4 PUTs + PUT + DELETE left 6 Azure versions (documented); exactly one ERROR naming deleteBlobVersion for the container; DELETE versionId=null 403
  account SAS srt=co, racdlx (no w)	suspend 403, overwrite 403	unchanged
  account key (control)	correct	unchanged: 4 suspended PUTs leave 1 Azure version; DELETE versionId=null leaves 0; PUT + DELETE leaves only the null marker

  The account-key regression of the main flow also passed: suspend, 2 PUTs leave one null version beside the enabled one, a plain DELETE writes the null marker, DELETE versionId=null promotes the enabled version, and re-enabling gives real ids again. After the rebase it passed again on 711d7526: suspend, 2 PUTs leave 1 null version, a plain DELETE writes the null marker, DELETE versionId=null leaves none, and re-enabling gives a real id. With the account key, a missing key answers 404 NoSuchKey, and promotion after a delete by id still serves the previous version to GET, HEAD and copy.

• With versioning unset: the same script against Emulate delete markers on a versioned azureblob store #1207's build and this branch's gave identical output. It covered PutBucketVersioning Enabled and Suspended (501), GetBucketVersioning (no status), put, get, head, copy, REPLACE copy, multipart, delete, versionId=null and ListObjectVersions (501), and it confirmed no container metadata is written. A rerun with 1aab9fe2 gave the same results.

Generated with Claude Code

[gavin-thompson-postman]

@gaul I believe the versioning work is complete (at least from a known gap standpoint) with this PR (This one is 3 of 3).

I did see your note on #1139 so will watch for movement on the Azurite side.

We are testing internally with a fork of the s3 proxy (pinned to this PRs hash) which hopefully results in this new code becoming a little more battle hardened.