-`backlog-2102` rc.2/rc.3 empty scanner usage floor recovery: old DeleteBucket cleanup could synthesize an empty incomplete v2 usage primary/backup before leadership added an epoch, while newer scanners require a durable authoritative baseline identity. New scanners recognize only that exact serialized empty-fence shape, preserve its epoch through a CAS-protected recovery marker, and rebuild namespace coverage without treating zero usage as authoritative. Remove this recovery path and marker after rc.2 and rc.3 are no longer supported direct-upgrade sources.
-`s3gate-metadata-xml` persisted bucket XML migration: mixed-version site-replication peers, retained `.metadata.bin` objects, and backup archives can all carry XML written by the s3s codec, so the gateway migration must keep the legacy codec available until every stored form has crossed a verified rewrite boundary. Remove the legacy s3s parser and serializer only after the minimum supported direct-upgrade release reads and writes every persisted XML configuration family through the gateway codec, the four-way D1-D5 gate has remained clean for one full support window, every supported mixed-version site-replication topology has completed its writer upgrade, and migration tooling has verified or rewritten every retained bucket metadata object and restorable backup archive.
-`rustfs-6339` legacy bucket policy ID casing: earlier RustFS releases persisted the top-level policy identifier as "ID", while current writes use the S3-compatible "Id" spelling. Readers accept both spellings so retained bucket metadata remains usable after upgrade. Remove the legacy alias after migration tooling has rewritten every retained bucket policy using "ID".
-`table-publication-fence-v1` table publication fencing: nodes that predate table and table-bucket publication fences can mutate live files while a new node is publishing a catalog pointer. New nodes retain exact object guards until the operator confirms that every serving node uses the new fences. Fleet confirmation also requires non-overlapping active warehouse prefixes and lifecycle workers that exclude table buckets. Remove the exact live-file fallback and the fleet-confirmation gate after the minimum supported RustFS release acquires table fences for registered-table mutations and table-bucket fences for unresolved-prefix mutations.
-`table-catalog-strong-snapshot-v1` durable strong catalog snapshot compatibility: version 1 writes continue during mixed-version rollout until operators confirm that every serving node reads version 2, and version 1 table/view identifier collisions remain available only for cleanup. Remove version 1 writes and collision cleanup after the minimum supported RustFS release reads version 2 and every retained durable strong snapshot is collision-free and has been upgraded to version 2.
| Catalog config | Supported | `GET /v1/config` advertises RustFS catalog defaults and only the supported OpenAPI REST paths in `endpoints`. RustFS administration, maintenance, migration, diagnostics, refs, and metadata-location extensions remain available but are not presented as standard Iceberg REST endpoints. |
| Table bucket discovery | Supported | `PUT` and `GET /v1/buckets/{warehouse}` enable and inspect table bucket state. |
| Namespaces | Supported | Create, list, load, existence check, and drop namespace routes are registered on both catalog prefixes. List responses support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Namespace identifiers are limited to 512 ASCII characters so persisted paths and stateless continuation tokens remain bounded. |
| Tables | Supported | Create, register, list, load, existence check, commit, metadata-location get/update, and drop table routes are registered on both catalog prefixes. Table and view listings support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Commit identifiers must match the URL resource; unknown requirements, updates, and snapshot operations fail as bad requests; staged create, register overwrite, purge-on-drop, and v3-only encryption-key updates return an explicit unsupported-operation response. Standard statistics, partition statistics, and schema/spec cleanup updates are accepted. |
| Tables | Supported | Create, register, list, load, existence check, rename, commit, metadata-location get/update, and drop table routes are registered on both catalog prefixes. Object-backed rename uses a bucket-scoped persistent fence, recoverable intent, and conditional publication of the destination, source tombstone, and warehouse index; the source identifier is reusable only through an ETag-conditional tombstone replacement. Table and view listings support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Commit identifiers must match the URL resource; unknown requirements, updates, and snapshot operations fail as bad requests; staged create, register overwrite, purge-on-drop, and v3-only encryption-key updates return an explicit unsupported-operation response. Standard statistics, partition statistics, and schema/spec cleanup updates are accepted. |
| Commit CAS | Supported | Single-table commits validate base metadata, expected version token, referenced object existence, warehouse scope, and Iceberg commit requirements before advancing the current metadata pointer. Externally supplied metadata transitions preserve monotonic column, partition, and sequence assignment watermarks and immutable definitions for retained schemas, partition specs, sort orders, and snapshots. Standard commits preserve the normal commit-token file name and use an immutable-table-scoped fallback when rename followed by source-name reuse would otherwise collide at the same generation and commit ID. The catalog does not advertise `idempotency-key-lifetime`; clients must treat standard mutation-wide `Idempotency-Key` semantics as unsupported. |
| Commit recovery | Supported | Commit log, idempotency lookup, diagnostics, and recovery routes expose staged/finalization gaps and repair safe idempotency gaps without moving the table pointer. |
| Snapshot refs | Supported | Refs can be listed, created or replaced, and deleted through catalog commits. `main` is protected and refs with explicit retention require forced delete. |
- Every consumer in the table below **binds** this epoch. None defines its own.
The concrete ordering semantics are not settled, however. The original #1326
proposal requires a total-ordered, monotonic lock-grant epoch. Current main does
not implement that proposal. PR #6077 instead implements an opaque transaction
identity:
### Monotonicity persistence semantics
-`assign_object_transaction_epoch` mints a random non-nil UUID for PUT and
CompleteMultipartUpload when the object-transaction gate is active.
- The UUID is written through `FileInfo::set_object_transaction_epoch` into the
dual internal metadata map.
- The coordinator reads the current UUID (or `Absent`) and revalidates exact
equality immediately before `rename_data`.
- Old-data cleanup receipts carry the committed UUID and reconciliation deletes
only when the receipt UUID still equals the current object UUID.
The epoch must be **monotonic across lock-plane restart and failover**
(#1312 B4). Today the distributed lock entry is in-memory only
(`crates/lock/src/distributed_lock.rs` has no persistence path), so a lock-service
restart resets the counter to zero: a new writer draws epoch 1 while disks have
already observed epoch 100, producing either a permanent write rejection or a
fence *inversion*. To prevent this, the epoch must be one of:
This is a useful **equality-CAS fence and cleanup identity**. It is not a
monotonic epoch, is not minted by the distributed lock grant, and is not
compared atomically at each disk's `xl.meta` commit point. Until the decision
below is made, documents and issue checklists must call it the *object
transaction UUID* rather than use it as proof that the target generation
authority exists.
1.**Quorum-persisted** before it is handed to a writer, or
2.**Derived from a durable monotonic source** — a `(term, counter)` pair where
`term` advances on every lock-service leadership change and is itself durable,
so the composite never regresses even when `counter` resets.
### Ordering decision required
The comparison at the disk commit point is on the full composite; a lower
`(term, counter)` is always rejected.
Before #1313, #1314, or a unified quota binding can consume the authority, one
of these contracts must be selected and tested:
1.**Total-ordered fencing epoch.** A lock grant returns a durable per-object
`(term, counter)` (or another specified total-order type). Every disk rejects
a lower epoch at the atomic metadata commit point. The value never regresses
across lock-plane restart, failover, or minority recovery.
2.**Opaque commit-generation identity.** Consumers compare only exact identity;
no `<` / `>` semantics are permitted. The authoritative commit must perform
an atomic expected-generation CAS, and all lease, cleanup, prepared-read, and
quota contracts must be rewritten in terms of “references this exact
generation,” not “lower/newer generation.”
The current UUID implementation proves neither a durable total order nor a
per-disk atomic expected-generation CAS, so it does not by itself decide between
these options.
### Persistence semantics if total order is selected
A total-ordered epoch must be **monotonic across lock-plane restart and
failover**. The distributed lock entry remains in-memory; deriving a counter
from that entry alone would reset it after restart. The chosen source therefore
must be either quorum-persisted before grant or derived from a durable term whose
full `(term, counter)` comparison cannot regress. This requirement does not
apply to an opaque UUID as an ordering rule; the opaque alternative instead
requires atomic expected-identity comparison and durable crash recovery.
## Consumer binding contracts
### Current implementation snapshot (2026-08-31, main@9ee7b1221)
This table separates code that exists on current main from the target contract.
Closing an implementation issue does not imply that its token is already the
unified authority.
| Surface | Current main | Gap against this contract |
|---|---|---|
| PUT / CompleteMultipartUpload (#1312, PR #6077) | Owned commit tasks retain the relevant guards; an opt-in gate persists a random object transaction UUID and performs a quorum metadata equality recheck before rename | no lock-grant monotonic source; no per-disk atomic epoch/CAS comparison; the live proof is the reused remote-version-state fleet proof, not a dedicated generation capability |
| Old-data cleanup (#1323, PR #6077) | JSON receipt carries transaction UUID, old dir, and committed dir; reconciliation is gated and requires UUID equality | no generation-bound read lease is consulted, so this is crash cleanup fencing rather than the full #1313/#1323 lease lifetime contract |
| Read lease (#1313) | short-term streaming/multipart path holds the namespace read lock through EOF/drop; deterministic part-boundary coverage is tracked by PR #6887 | no cross-node generation-bound lease registry, TTL reclamation, or crash recovery |
| Prepared pool read (#1314) | PR #6889 tracks a pool-local prepared identity and fails closed/refetches when pool state changes | not merged on this snapshot; pool-local identity is not a cross-pool generation authority; black-box mixed-version/rebalance coverage remains open |
| Quota reservation (#1318) | durable per-bucket ledger plus independent snapshot-lease mutation-fence tokens; issue closed after PR #6058 | reservation and settle are not bound to the object transaction UUID; the independent fence must be reconciled with the selected authority or explicitly proven to be a separate, non-generation arbitration domain |
| Internode integrity (#1327, #1541, #1542) | v2/v3 HMAC binds audience, exact method, timestamp, nonce, canonical body digest, and receiver boot epoch; body-bound RPC policy has exact-set coverage | signature/body/replay strict switches remain default-off rollout gates; generation enforcement cannot treat an unrelated fleet-version proof as proof that these strict contracts converged |
| Consumer | How it binds generation | Key invariant |
|---|---|---|
| #1312 commit fence | epoch compared at three disk-write points —`rename`, rollback `delete`, and `commit_rename_data_dir` cleanup | stale epoch rejected on **all** disks; an already-ACK'd write is never rolled back |
| #1313 read lease | lease binds the generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms no lease referencing a lower generation still points at it | `old_dir != committed_dir`; a still-referenced directory is never deleted |
| #1312 commit fence | selected generation is checked at`rename`, rollback restore/delete, and cleanup mutation points using the chosen ordered or exact-CAS rule | a stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back |
| #1313 read lease | lease binds the exact generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms that no lease for the generation owning that directory remains | `old_dir != committed_dir`; a still-referenced directory is never deleted |
| #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout |
| #1318 quota reservation | reservation / settle token binds the object generation | a late commit holding an old-generation token cannot settle a newer generation |
| #1318 quota reservation | reservation / settle record binds the exact object generation (and an ordered epoch too, if that option is selected) | a late commit cannot settle quota for a different committed generation |
### Fence coverage is three disk-write points, not one (#1312 B2)
Comparing the epoch at the `rename` commit point alone is insufficient. The
Checking the generation only before the `rename` fanout is insufficient. The
**Requirement.** The RPC body digest carrying a generation/epoch/token must be
folded into the RPC HMAC, binding `method + object key + generation`, and the
request must carry a nonce / one-shot identifier inside the 300s replay window.
The nonce is only meaningful if the **receiver enforces it**: each disk keeps a
bounded seen-nonce cache covering the 300s freshness window and rejects any
request whose nonce was already observed. A nonce that is merely transmitted but
not checked provides no replay protection.
**Requirement.** The canonical body carrying a generation or derived token must
be folded into the internode HMAC. The authenticated scope binds the target
audience, exact service/method, timestamp, nonce, canonical body digest, and
receiver replay epoch. The receiver must consume the nonce in a bounded replay
cache; transmitting a nonce without receiver-side consumption is not replay
protection.
This generalizes the existing `walk_dir` pattern: `walk_dir` computes a
`Sha256` of the request body and places it in the signed URL query as
`walk_dir_body_sha256`
(`crates/ecstore/src/cluster/rpc/internode_data_transport.rs:187`), so the body
digest is transitively covered by the URL signature. New generation-bearing RPCs
adopt the same `*_body_sha256` mechanism.
**Current substrate (verified on main).** The original legacy-only description
is obsolete:
**Current gap (verified).** The internode HMAC covers only
`{path_and_query}|{method}|{timestamp}`
(`signature_payload`, `crates/ecstore/src/cluster/rpc/http_auth.rs:75-83`). It
binds neither the request body nor a nonce, and the 300s freshness window has no
one-shot guard. Without the binding above:
- RPC v2 binds target audience, exact method, POST, timestamp, nonce, and body
digest.
- Body-bound policy covers mutating disk RPCs including `RenameData`; its
versioned canonical body includes every `RenameDataRequest` field, so the
`FileInfo` metadata map carrying the transaction UUID is authenticated.
- PR #5425 extended canonical-body enforcement to implemented non-disk mutating
unary RPCs and added an exact policy/handler coverage partition.
- PR #5455 added the receiver boot epoch and rotating replay scope so signatures
captured before a receiver restart are rejected after capability convergence.
- An on-path or replaying attacker can inject a high epoch (e.g. `u32::MAX`) and
**permanently fence out** a key's legitimate writes — monotonicity only
rejects *low/old* epochs, never a forged-high one.
- A captured lease/reservation token can be replayed within 300s to block
old-dir GC (storage-exhaustion DoS) or to double-reserve / prematurely settle
quota.
The rollout switches
`RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`,
`RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, and
`RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` remain default-off for rolling
compatibility. The compatibility register and fallback/overflow metrics govern
their fleet convergence. Therefore a generation capability may claim strong
transport binding only when the relevant strict modes have converged; the
object-transaction gate's current remote-version-state fleet proof is not, by
itself, proof of RPC signature/body/replay strictness.
Acceptance for each consumer must include: "a replayed old signature to a
different method, and a forged-high-epoch request, are both rejected."
Acceptance for each generation consumer includes method substitution, canonical
body tamper, nonce replay, receiver restart, and stripped-strict-metadata
negative tests. Generation rollout must also record which strict-mode evidence
authorized enforcement.
### Encoding contract (#1312 B1)
@@ -167,11 +218,15 @@ The on-disk persistence of generation must not perturb the file format:
`xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a
total read failure, not a graceful downgrade.
- **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged.
- **Where it may live.** Only inside a version's internal metadata **map**
(MinIO skips unknown internal keys and the map encoding is extensible) or in a
per-disk sidecar outside `xl.meta`. If it goes in the metadata map, it must
obey the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`, see
AGENTS.md "Cross-Cutting Domain Invariants").
- **Where it lives today.** The object transaction UUID uses the version's
internal metadata map under the dual-key contract
(`x-rustfs-internal-*` / `x-minio-internal-*`) via
`set_object_transaction_epoch`. Missing, malformed, nil, or conflicting dual
values fail closed when fencing is active.
- **Sidecars are not an equivalent alternative.** A future sidecar is admissible
only if it commits atomically with `xl.meta` and has a specified crash-recovery
protocol. No such protocol is implemented, so a sidecar cannot be selected by
an implementation issue merely because this document mentions one.
- **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop
regression (the fixture family around `crates/filemeta/src/filemeta.rs`):
objects written by a new node must still be readable by old RustFS nodes and
@@ -179,82 +234,151 @@ The on-disk persistence of generation must not perturb the file format:
The internode RPC layer is mid-migration from JSON to msgpack binary, and generation-bearing fields must respect that migration window — this is not optional context, it changes how epoch is transported.
The internode RPC layer retains a JSON/msgpack rolling-compatibility window, and
generation-bearing fields must respect it.
- **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info`#4 alongside `file_info_bin`#7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`).
- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the `record_msgpack_json_fallback` metric reads zero fleet-wide and the convergence runbook is followed. If only the request flag is set, RustFS keeps dual-writing JSON compatibility fields. **Reuse this exact capability + metric-reads-zero model as the mixed-version gate for generation** rather than inventing a parallel handshake; the section above ("Capability negotiation") is layered on top of it, not instead of it.
- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the JSON-fallback metric reads zero fleet-wide and the convergence runbook is followed. Generation follows the same default-off, fleet-confirmed, metric-reads-zero rollout discipline, but a msgpack proof is not itself a generation capability proof.
- **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch.
- **Signature must bind a canonical form.** Because a field is transmitted as both JSON and msgpack and a peer may consume either, the body-digest binding in "RPC signature binding" above must be computed over a single canonical representation (the msgpack `_bin` bytes) — not over whichever copy happened to be decoded. Once the `generation` capability is negotiated for a request, a fenced / generation-bearing request must **reject the JSON fallback path** so a downgrade to the unsigned/loosely-bound JSON copy cannot bypass the epoch check.
- **Signature binds a canonical form.** `RenameDataRequest` now has a versioned,
injective canonical-body encoder that covers both compatibility fields and is
authenticated independently of whichever JSON/msgpack decoder branch a peer
consumes. A generation-capable strict request must reject missing or
mismatched canonical-body metadata; it must not silently downgrade to an
unauthenticated JSON twin.
### Proto evolution
New generation/epoch proto fields use **proto3 `optional`** (explicit presence).
A non-optional field is forbidden: an old coordinator talking to a new disk
decodes an absent field as `0`, which is indistinguishable from a real
`epoch == 0` and silently breaks the "stale epoch rejected" invariant during
upgrade.
No top-level proto field is required by the current metadata-map UUID. If a
future ordered epoch or explicit expected-generation is added to proto, it uses
**proto3 `optional`** (explicit presence). A non-optional scalar is forbidden:
an old coordinator talking to a new disk decodes absence as a plausible zero.
### Mixed-version gate — one direction
When the cluster-level generation capability is **not** negotiated on every
target disk, the behavior **falls back to current semantics** (existing lock +
`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at
`rustfs/src/app/object/get.rs`; full fanout for #1314). Fail-closed is
**only** an explicit administrator strict mode. Defaulting to fail-closed is
forbidden — it makes writes unavailable for the whole rolling-upgrade window.
When generation enforcement is not explicitly requested, or fleet confirmation
is absent, behavior falls back to current semantics. Fail-closed is reserved for
an explicit administrator-confirmed strict rollout.
Current object transaction fencing follows that direction:
-`RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` and
`RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` both default false.
- With either flag absent, PUT/MPU does not persist or consume the transaction
UUID.
- With both flags enabled, failure to obtain or retain the live fleet proof
rejects the commit before rename.
This is an opt-in strict gate, not a negotiated generation capability. The
proof is currently borrowed from the remote-version-state writer rollout. It
proves current membership/process-epoch convergence for that feature, but does
not prove an epoch type, per-disk generation CAS support, or RPC strict-mode
convergence. Treating it as the final handshake is forbidden without an
explicit proof mapping for those properties.
## Capability negotiation
Generation enforcement is a **cluster-level handshake**, not a per-request
probe:
Generation enforcement requires one **live fleet proof**, not independent
boolean guesses in each consumer. The proof contract contains at least:
- A node advertises a `generation` capability once it can (a) mint quorum-durable
epochs, (b) compare epochs at all three disk-write points, and (c) verify the
body-digest-bound RPC signature.
- The authoritative writer enables hard enforcement for an object only when
**all** target disks in the set advertise the capability. Any missing
advertisement pins that commit to the mixed-version fallback above.
-The capability is surfaced through the existing runtime capability contract
surface (see [runtime-capability-contracts.md](runtime-capability-contracts.md)),
so consumers read one negotiated flag rather than each re-deriving support.
- Enforcement tracks the current membership rather than latching: it turns on
for a set only while every disk in that set advertises `generation`, and a
single old node rejoining drops the affected sets back to the mixed-version
fallback rather than failing closed. It never regresses the on-disk epoch —
falling back stops *comparing* new epochs, it does not lower any epoch already
persisted.
1. the selected authority version and comparison mode (ordered or exact-CAS),
2. the current membership/topology fingerprint and process epochs,
3. support for every required disk mutation point,
4. RPC signature/body/replay strict convergence, and
5. the on-disk encoding version (the current metadata-map UUID is version 1).
The authoritative writer enables enforcement only while every target disk in
the set is covered by a current proof. Membership change or an old node rejoin
revokes that proof. Revocation before commit fails an explicitly strict request;
when strict generation was never requested, the request remains on the legacy
path. Revocation never rewrites or lowers an already-persisted generation.
The existing fleet-proof machinery in `notification_sys` may be reused if its
authenticated statements are extended to cover the properties above. The
runtime capability contract may instead expose the proof. This document does
not choose the storage mechanism; it requires one token whose acquisition and
revalidation semantics are shared by all consumers.
## Implementation order
1.**#1312 first.** It defines the epoch, its persistence, the three fence
points, the RPC signature binding, and the encoding location. Everything
downstream depends on its epoch existing.
2.**#1313** (read lease) reuses the #1312 epoch as the lease generation and
must land before or alongside #1323.
3.**#1323** (old-dir GC) depends on #1313 leases being present and
cross-node-visible; its "no lease references old_dir" check has nothing to
query otherwise.
4.**#1318** (quota reservation) and **#1314** (prepared pool read) bind the
epoch independently; both gate on the same capability handshake.
Some original prerequisites have landed, but not in the originally proposed
form. Remaining work follows this order:
## Open design decisions (pin before implementation)
1.**Resolve the authority mode in #1326.** Select total order or opaque
exact-CAS, specify its atomic commit point, and audit PR #6077 against it.
Do not retrofit ordering semantics onto the existing random UUID.
2.**Define the generation fleet proof.** Map generation enablement to the RPC
signature/body/replay strict proofs delivered by #1327/#1541/#1542 and to
the selected per-disk comparison capability. Keep all strict defaults off
until fallback metrics converge.
3.**Implement #1313 generation-bound read leases.** The lease registry,
cross-node visibility, TTL, and crash recovery must exist before old-dir GC
can claim the full snapshot-lifetime guarantee. #1325 supplies the required
multi-node failure tests.
4.**Bind #1314 prepared reads.** A bundle binds the exact selected generation
within its source pool. Cross-pool ordering is forbidden until a common
authority is demonstrated. Validate rebalance and mixed-version fallback in
the #1325 multi-pool harness.
5.**Reconcile #1318 quota fencing.** Either bind reserve/settle/reconcile to
the selected object generation or document and prove that its independent
snapshot-lease fence is a separate arbitration domain that cannot settle a
different generation.
6.**Re-audit #1323 cleanup.** The existing UUID receipt remains valid crash
cleanup, but full closure against active readers requires the #1313 lease
check and the selected generation semantics.
This document fixes the transport, encoding, proto, and gate constraints, but it is not yet a complete implementable algorithm. The following must be decided and written down before any of the five consumers is coded (per the #1307 maintainer re-review, issuecomment-4992956256):
## Open design decisions (pin before contract closure)
- **Epoch type and total order.** The concrete token type and its total-order rule — a term+counter tuple, its persistence, and overflow behavior. Whether monotonicity is global or strictly per-object.
- **Never-regress on lock-service restart / minority recovery.** The epoch source must survive a lock-service restart or minority-quorum recovery without ever handing out an epoch lower than one already persisted on disk (an in-memory counter reset to zero is a fencing inversion). This is the same requirement as "Monotonicity persistence semantics" above, elevated to a hard, tested acceptance.
- **Complete xl.meta-writer coverage.** Every code path that writes xl.meta (commit rename, rollback delete/metadata restore, old-dir cleanup, heal, transition) must be enumerated and shown to compare or carry the epoch. A single unfenced writer voids the guarantee.
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure rollback at `io_primitives.rs:2646-2691` restores a metadata backup, not just a per-writer tmp delete, so a late rollback by writer A can overwrite writer B's committed xl.meta. Rollback must execute only when `stored_epoch == failed_writer_epoch`; a higher stored epoch must abort the rollback. Task panic / cancel / timeout at `io_primitives.rs:2602-2605` must be reaped into the coordinator's state machine, never bubble out via `?` and skip convergence.
The following decisions remain blockers for calling the contract implemented:
- **Authority mode.** Choose total order or opaque exact-CAS. If total order is
selected, define the type, per-object scope, persistence, overflow, and
never-regress restart/minority-recovery tests. If opaque identity is selected,
define the atomic expected-generation CAS and remove all ordered wording.
restore/delete, cleanup, heal, transition, restore, replication, and data
movement. Each path must compare/carry the selected generation or be proved
incapable of replacing the authoritative object identity.
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure
rollback in `rename_data` can restore backup metadata, not just remove a
writer-private temporary file. It must execute only when the stored generation
still matches the failed writer's expected generation. Panic, cancel, and
timeout outcomes must be reaped into coordinator convergence rather than skip
rollback through an early return.
- **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe.
- **Read-lease and GC crash recovery.** Lease registry location (local vs cross-node), TTL reclamation, and crash recovery for both the lease holder and the GC executor.
- **Quota reserve → commit → settle idempotency.** The cross-stage reconcile / idempotency story for #1318, including owner-crash reconciliation, so a reservation is neither lost nor double-counted.
- **Generation capability proof.** Decide whether to extend the current
authenticated fleet proof or the runtime capability contract. It must prove
authority version, mutation coverage, topology/process epoch, and RPC strict
convergence in one revalidatable token.
- **Read-lease and GC crash recovery.** Select the cross-node registry, TTL
reclamation, lease-holder crash behavior, and GC-executor recovery. The
current cleanup receipt equality check does not answer these questions.
exists, but its independent mutation tokens must be related to the selected
object generation with a concrete late-settle rejection test.
- **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited.
- **Hot-path cost is a blocking metric.** If per-PUT fencing grant, quota reserve, or cleanup journal adds a consensus write / fsync / centralized serialization point, it must be measured under 4KiB and high-concurrency hot-key / hot-bucket A/B as a blocking gate, not accepted by default.
- **Hot-path cost is a blocking metric.** Measure any additional consensus
write, fsync, fleet-proof lookup, lease operation, or centralized serialization
under 4 KiB and high-concurrency hot-key/hot-bucket A/B.
- **Test infrastructure.** #1325 still lacks the complete 4-node × 4-drive,
2-pool, directed network-fault, and large-object budget needed for restart,
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
marshawcoco