From c2d84887285bbec4d1d3b01559f2bd3ed6ab04cf Mon Sep 17 00:00:00 2001 From: houseme Date: Mon, 31 Aug 2026 00:20:39 +0800 Subject: [PATCH] docs(architecture): reconcile generation contract (#6901) Co-authored-by: heihutu --- .../architecture/unified-object-generation.md | 364 ++++++++++++------ 1 file changed, 244 insertions(+), 120 deletions(-) diff --git a/docs/architecture/unified-object-generation.md b/docs/architecture/unified-object-generation.md index 40656c429..a5bf7b4e5 100644 --- a/docs/architecture/unified-object-generation.md +++ b/docs/architecture/unified-object-generation.md @@ -30,73 +30,120 @@ lifetime**. Left independent, they diverge and punch through one another: different monotonic sources, cannot be compared — a late commit fenced on one plane can still settle quota on the other. -The fix is a single authority with one monotonic source, one persistence +The fix is a single authority with one selected comparison rule, one persistence semantics, and one transport binding, that every consumer references rather than re-derives. -## The authority (single source) +## Target authority and the current bounded token -**The per-object fencing epoch defined by #1312 is the sole generation -authority.** No other monotonic counter, timestamp, or random token may stand in -for generation. +The target contract still requires **one per-object commit identity** consumed +by commit fencing, read leases, cleanup, prepared reads, and quota settlement. +No consumer may mint a second value and call it the same generation. -- The distributed lock grant returns a monotonic `epoch` for the object key. - Acquiring the object write-lock is the only way to mint a new generation. -- The epoch travels down the authoritative commit path (with - `RenameDataRequest` / the local `DiskAPI` call) and is compared at each disk's - atomic `xl.meta` commit point, rejecting stale epochs. It adds no extra - network round trip (#1312 implementation clause 2). -- Every consumer in the table below **binds** this epoch. None defines its own. +The concrete ordering semantics are not settled, however. The original #1326 +proposal requires a total-ordered, monotonic lock-grant epoch. Current main does +not implement that proposal. PR #6077 instead implements an opaque transaction +identity: -### Monotonicity persistence semantics +- `assign_object_transaction_epoch` mints a random non-nil UUID for PUT and + CompleteMultipartUpload when the object-transaction gate is active. +- The UUID is written through `FileInfo::set_object_transaction_epoch` into the + dual internal metadata map. +- The coordinator reads the current UUID (or `Absent`) and revalidates exact + equality immediately before `rename_data`. +- Old-data cleanup receipts carry the committed UUID and reconciliation deletes + only when the receipt UUID still equals the current object UUID. -The epoch must be **monotonic across lock-plane restart and failover** -(#1312 B4). Today the distributed lock entry is in-memory only -(`crates/lock/src/distributed_lock.rs` has no persistence path), so a lock-service -restart resets the counter to zero: a new writer draws epoch 1 while disks have -already observed epoch 100, producing either a permanent write rejection or a -fence *inversion*. To prevent this, the epoch must be one of: +This is a useful **equality-CAS fence and cleanup identity**. It is not a +monotonic epoch, is not minted by the distributed lock grant, and is not +compared atomically at each disk's `xl.meta` commit point. Until the decision +below is made, documents and issue checklists must call it the *object +transaction UUID* rather than use it as proof that the target generation +authority exists. -1. **Quorum-persisted** before it is handed to a writer, or -2. **Derived from a durable monotonic source** — a `(term, counter)` pair where - `term` advances on every lock-service leadership change and is itself durable, - so the composite never regresses even when `counter` resets. +### Ordering decision required -The comparison at the disk commit point is on the full composite; a lower -`(term, counter)` is always rejected. +Before #1313, #1314, or a unified quota binding can consume the authority, one +of these contracts must be selected and tested: + +1. **Total-ordered fencing epoch.** A lock grant returns a durable per-object + `(term, counter)` (or another specified total-order type). Every disk rejects + a lower epoch at the atomic metadata commit point. The value never regresses + across lock-plane restart, failover, or minority recovery. +2. **Opaque commit-generation identity.** Consumers compare only exact identity; + no `<` / `>` semantics are permitted. The authoritative commit must perform + an atomic expected-generation CAS, and all lease, cleanup, prepared-read, and + quota contracts must be rewritten in terms of “references this exact + generation,” not “lower/newer generation.” + +The current UUID implementation proves neither a durable total order nor a +per-disk atomic expected-generation CAS, so it does not by itself decide between +these options. + +### Persistence semantics if total order is selected + +A total-ordered epoch must be **monotonic across lock-plane restart and +failover**. The distributed lock entry remains in-memory; deriving a counter +from that entry alone would reset it after restart. The chosen source therefore +must be either quorum-persisted before grant or derived from a durable term whose +full `(term, counter)` comparison cannot regress. This requirement does not +apply to an opaque UUID as an ordering rule; the opaque alternative instead +requires atomic expected-identity comparison and durable crash recovery. ## Consumer binding contracts +### Current implementation snapshot (2026-08-31, main@9ee7b1221) + +This table separates code that exists on current main from the target contract. +Closing an implementation issue does not imply that its token is already the +unified authority. + +| Surface | Current main | Gap against this contract | +|---|---|---| +| PUT / CompleteMultipartUpload (#1312, PR #6077) | Owned commit tasks retain the relevant guards; an opt-in gate persists a random object transaction UUID and performs a quorum metadata equality recheck before rename | no lock-grant monotonic source; no per-disk atomic epoch/CAS comparison; the live proof is the reused remote-version-state fleet proof, not a dedicated generation capability | +| Old-data cleanup (#1323, PR #6077) | JSON receipt carries transaction UUID, old dir, and committed dir; reconciliation is gated and requires UUID equality | no generation-bound read lease is consulted, so this is crash cleanup fencing rather than the full #1313/#1323 lease lifetime contract | +| Read lease (#1313) | short-term streaming/multipart path holds the namespace read lock through EOF/drop; deterministic part-boundary coverage is tracked by PR #6887 | no cross-node generation-bound lease registry, TTL reclamation, or crash recovery | +| Prepared pool read (#1314) | PR #6889 tracks a pool-local prepared identity and fails closed/refetches when pool state changes | not merged on this snapshot; pool-local identity is not a cross-pool generation authority; black-box mixed-version/rebalance coverage remains open | +| Quota reservation (#1318) | durable per-bucket ledger plus independent snapshot-lease mutation-fence tokens; issue closed after PR #6058 | reservation and settle are not bound to the object transaction UUID; the independent fence must be reconciled with the selected authority or explicitly proven to be a separate, non-generation arbitration domain | +| Internode integrity (#1327, #1541, #1542) | v2/v3 HMAC binds audience, exact method, timestamp, nonce, canonical body digest, and receiver boot epoch; body-bound RPC policy has exact-set coverage | signature/body/replay strict switches remain default-off rollout gates; generation enforcement cannot treat an unrelated fleet-version proof as proof that these strict contracts converged | + | Consumer | How it binds generation | Key invariant | |---|---|---| -| #1312 commit fence | epoch compared at three disk-write points — `rename`, rollback `delete`, and `commit_rename_data_dir` cleanup | stale epoch rejected on **all** disks; an already-ACK'd write is never rolled back | -| #1313 read lease | lease binds the generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL | -| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms no lease referencing a lower generation still points at it | `old_dir != committed_dir`; a still-referenced directory is never deleted | +| #1312 commit fence | selected generation is checked at `rename`, rollback restore/delete, and cleanup mutation points using the chosen ordered or exact-CAS rule | a stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back | +| #1313 read lease | lease binds the exact generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL | +| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms that no lease for the generation owning that directory remains | `old_dir != committed_dir`; a still-referenced directory is never deleted | | #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout | -| #1318 quota reservation | reservation / settle token binds the object generation | a late commit holding an old-generation token cannot settle a newer generation | +| #1318 quota reservation | reservation / settle record binds the exact object generation (and an ordered epoch too, if that option is selected) | a late commit cannot settle quota for a different committed generation | ### Fence coverage is three disk-write points, not one (#1312 B2) -Comparing the epoch at the `rename` commit point alone is insufficient. The +Checking the generation only before the `rename` fanout is insufficient. The authoritative commit sequence is `tmp sync → data-dir rename → xl.meta commit → directory sync` in `crates/ecstore/src/disk/local.rs`, and there are two further detachable disk-write points in `crates/ecstore/src/set_disk/core/io_primitives.rs`: -- **Rollback delete** — on quorum failure each disk runs - `delete_version(undo_write=true)`. A fenced old writer's rollback must also - compare epoch, otherwise it deletes the winner's already-committed version. +- **Rollback restore/delete** — on quorum failure each disk can restore backup + metadata or delete the failed version. A stale writer's rollback must compare + the expected generation, otherwise it can overwrite or delete the winner's + already-committed metadata. - **`commit_rename_data_dir`** — a cancel-then-detach disk-write point; the coordinator's "reap all child tasks" must explicitly include it so a cancelled writer cannot bypass fence/lease and keep deleting directories. -If the epoch is validated only at the `xl.meta` commit point, a fenced writer +If generation is validated only after data-dir rename, a fenced writer may already have renamed its data-dir into the object path, leaving a staged orphan. Either move the fence ahead of the data-dir rename, or declare that orphan an acceptable residue accounted for by GC metrics — the white-box acceptance "no background disk write after release" must be rewritten accordingly. +Current PR #6077 performs a quorum metadata equality recheck before rename and +reaps owned commit work. That closes important cancellation windows, but it is +not evidence that every disk mutation above performs the selected generation +comparison atomically. The writer inventory and per-point CAS/ordering proof +remain acceptance work for #1326 even though #1312 is closed. + ### Post-commit convergence is orthogonal to the fence (#1321) The same `SetDisks::rename_data` path already returns a post-commit @@ -125,36 +172,40 @@ internode RPC bodies. Every such flow must be signature-bound. ### RPC signature binding (#1312 B3, #1313, #1318) -**Requirement.** The RPC body digest carrying a generation/epoch/token must be -folded into the RPC HMAC, binding `method + object key + generation`, and the -request must carry a nonce / one-shot identifier inside the 300s replay window. -The nonce is only meaningful if the **receiver enforces it**: each disk keeps a -bounded seen-nonce cache covering the 300s freshness window and rejects any -request whose nonce was already observed. A nonce that is merely transmitted but -not checked provides no replay protection. +**Requirement.** The canonical body carrying a generation or derived token must +be folded into the internode HMAC. The authenticated scope binds the target +audience, exact service/method, timestamp, nonce, canonical body digest, and +receiver replay epoch. The receiver must consume the nonce in a bounded replay +cache; transmitting a nonce without receiver-side consumption is not replay +protection. -This generalizes the existing `walk_dir` pattern: `walk_dir` computes a -`Sha256` of the request body and places it in the signed URL query as -`walk_dir_body_sha256` -(`crates/ecstore/src/cluster/rpc/internode_data_transport.rs:187`), so the body -digest is transitively covered by the URL signature. New generation-bearing RPCs -adopt the same `*_body_sha256` mechanism. +**Current substrate (verified on main).** The original legacy-only description +is obsolete: -**Current gap (verified).** The internode HMAC covers only -`{path_and_query}|{method}|{timestamp}` -(`signature_payload`, `crates/ecstore/src/cluster/rpc/http_auth.rs:75-83`). It -binds neither the request body nor a nonce, and the 300s freshness window has no -one-shot guard. Without the binding above: +- RPC v2 binds target audience, exact method, POST, timestamp, nonce, and body + digest. +- Body-bound policy covers mutating disk RPCs including `RenameData`; its + versioned canonical body includes every `RenameDataRequest` field, so the + `FileInfo` metadata map carrying the transaction UUID is authenticated. +- PR #5425 extended canonical-body enforcement to implemented non-disk mutating + unary RPCs and added an exact policy/handler coverage partition. +- PR #5455 added the receiver boot epoch and rotating replay scope so signatures + captured before a receiver restart are rejected after capability convergence. -- An on-path or replaying attacker can inject a high epoch (e.g. `u32::MAX`) and - **permanently fence out** a key's legitimate writes — monotonicity only - rejects *low/old* epochs, never a forged-high one. -- A captured lease/reservation token can be replayed within 300s to block - old-dir GC (storage-exhaustion DoS) or to double-reserve / prematurely settle - quota. +The rollout switches +`RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`, +`RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, and +`RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` remain default-off for rolling +compatibility. The compatibility register and fallback/overflow metrics govern +their fleet convergence. Therefore a generation capability may claim strong +transport binding only when the relevant strict modes have converged; the +object-transaction gate's current remote-version-state fleet proof is not, by +itself, proof of RPC signature/body/replay strictness. -Acceptance for each consumer must include: "a replayed old signature to a -different method, and a forged-high-epoch request, are both rejected." +Acceptance for each generation consumer includes method substitution, canonical +body tamper, nonce replay, receiver restart, and stripped-strict-metadata +negative tests. Generation rollout must also record which strict-mode evidence +authorized enforcement. ### Encoding contract (#1312 B1) @@ -167,11 +218,15 @@ The on-disk persistence of generation must not perturb the file format: `xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a total read failure, not a graceful downgrade. - **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged. -- **Where it may live.** Only inside a version's internal metadata **map** - (MinIO skips unknown internal keys and the map encoding is extensible) or in a - per-disk sidecar outside `xl.meta`. If it goes in the metadata map, it must - obey the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`, see - AGENTS.md "Cross-Cutting Domain Invariants"). +- **Where it lives today.** The object transaction UUID uses the version's + internal metadata map under the dual-key contract + (`x-rustfs-internal-*` / `x-minio-internal-*`) via + `set_object_transaction_epoch`. Missing, malformed, nil, or conflicting dual + values fail closed when fencing is active. +- **Sidecars are not an equivalent alternative.** A future sidecar is admissible + only if it commits atomically with `xl.meta` and has a specified crash-recovery + protocol. No such protocol is implemented, so a sidecar cannot be selected by + an implementation issue merely because this document mentions one. - **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop regression (the fixture family around `crates/filemeta/src/filemeta.rs`): objects written by a new node must still be readable by old RustFS nodes and @@ -179,82 +234,151 @@ The on-disk persistence of generation must not perturb the file format: ### Wire-encoding migration (JSON → msgpack) interaction -The internode RPC layer is mid-migration from JSON to msgpack binary, and generation-bearing fields must respect that migration window — this is not optional context, it changes how epoch is transported. +The internode RPC layer retains a JSON/msgpack rolling-compatibility window, and +generation-bearing fields must respect it. - **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info` #4 alongside `file_info_bin` #7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`). -- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the `record_msgpack_json_fallback` metric reads zero fleet-wide and the convergence runbook is followed. If only the request flag is set, RustFS keeps dual-writing JSON compatibility fields. **Reuse this exact capability + metric-reads-zero model as the mixed-version gate for generation** rather than inventing a parallel handshake; the section above ("Capability negotiation") is layered on top of it, not instead of it. +- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the JSON-fallback metric reads zero fleet-wide and the convergence runbook is followed. Generation follows the same default-off, fleet-confirmed, metric-reads-zero rollout discipline, but a msgpack proof is not itself a generation capability proof. - **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch. -- **Signature must bind a canonical form.** Because a field is transmitted as both JSON and msgpack and a peer may consume either, the body-digest binding in "RPC signature binding" above must be computed over a single canonical representation (the msgpack `_bin` bytes) — not over whichever copy happened to be decoded. Once the `generation` capability is negotiated for a request, a fenced / generation-bearing request must **reject the JSON fallback path** so a downgrade to the unsigned/loosely-bound JSON copy cannot bypass the epoch check. +- **Signature binds a canonical form.** `RenameDataRequest` now has a versioned, + injective canonical-body encoder that covers both compatibility fields and is + authenticated independently of whichever JSON/msgpack decoder branch a peer + consumes. A generation-capable strict request must reject missing or + mismatched canonical-body metadata; it must not silently downgrade to an + unauthenticated JSON twin. ### Proto evolution -New generation/epoch proto fields use **proto3 `optional`** (explicit presence). -A non-optional field is forbidden: an old coordinator talking to a new disk -decodes an absent field as `0`, which is indistinguishable from a real -`epoch == 0` and silently breaks the "stale epoch rejected" invariant during -upgrade. +No top-level proto field is required by the current metadata-map UUID. If a +future ordered epoch or explicit expected-generation is added to proto, it uses +**proto3 `optional`** (explicit presence). A non-optional scalar is forbidden: +an old coordinator talking to a new disk decodes absence as a plausible zero. ### Mixed-version gate — one direction -When the cluster-level generation capability is **not** negotiated on every -target disk, the behavior **falls back to current semantics** (existing lock + -`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at -`rustfs/src/app/object/get.rs`; full fanout for #1314). Fail-closed is -**only** an explicit administrator strict mode. Defaulting to fail-closed is -forbidden — it makes writes unavailable for the whole rolling-upgrade window. +When generation enforcement is not explicitly requested, or fleet confirmation +is absent, behavior falls back to current semantics. Fail-closed is reserved for +an explicit administrator-confirmed strict rollout. + +Current object transaction fencing follows that direction: + +- `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` and + `RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` both default false. +- With either flag absent, PUT/MPU does not persist or consume the transaction + UUID. +- With both flags enabled, failure to obtain or retain the live fleet proof + rejects the commit before rename. + +This is an opt-in strict gate, not a negotiated generation capability. The +proof is currently borrowed from the remote-version-state writer rollout. It +proves current membership/process-epoch convergence for that feature, but does +not prove an epoch type, per-disk generation CAS support, or RPC strict-mode +convergence. Treating it as the final handshake is forbidden without an +explicit proof mapping for those properties. ## Capability negotiation -Generation enforcement is a **cluster-level handshake**, not a per-request -probe: +Generation enforcement requires one **live fleet proof**, not independent +boolean guesses in each consumer. The proof contract contains at least: -- A node advertises a `generation` capability once it can (a) mint quorum-durable - epochs, (b) compare epochs at all three disk-write points, and (c) verify the - body-digest-bound RPC signature. -- The authoritative writer enables hard enforcement for an object only when - **all** target disks in the set advertise the capability. Any missing - advertisement pins that commit to the mixed-version fallback above. -- The capability is surfaced through the existing runtime capability contract - surface (see [runtime-capability-contracts.md](runtime-capability-contracts.md)), - so consumers read one negotiated flag rather than each re-deriving support. -- Enforcement tracks the current membership rather than latching: it turns on - for a set only while every disk in that set advertises `generation`, and a - single old node rejoining drops the affected sets back to the mixed-version - fallback rather than failing closed. It never regresses the on-disk epoch — - falling back stops *comparing* new epochs, it does not lower any epoch already - persisted. +1. the selected authority version and comparison mode (ordered or exact-CAS), +2. the current membership/topology fingerprint and process epochs, +3. support for every required disk mutation point, +4. RPC signature/body/replay strict convergence, and +5. the on-disk encoding version (the current metadata-map UUID is version 1). + +The authoritative writer enables enforcement only while every target disk in +the set is covered by a current proof. Membership change or an old node rejoin +revokes that proof. Revocation before commit fails an explicitly strict request; +when strict generation was never requested, the request remains on the legacy +path. Revocation never rewrites or lowers an already-persisted generation. + +The existing fleet-proof machinery in `notification_sys` may be reused if its +authenticated statements are extended to cover the properties above. The +runtime capability contract may instead expose the proof. This document does +not choose the storage mechanism; it requires one token whose acquisition and +revalidation semantics are shared by all consumers. ## Implementation order -1. **#1312 first.** It defines the epoch, its persistence, the three fence - points, the RPC signature binding, and the encoding location. Everything - downstream depends on its epoch existing. -2. **#1313** (read lease) reuses the #1312 epoch as the lease generation and - must land before or alongside #1323. -3. **#1323** (old-dir GC) depends on #1313 leases being present and - cross-node-visible; its "no lease references old_dir" check has nothing to - query otherwise. -4. **#1318** (quota reservation) and **#1314** (prepared pool read) bind the - epoch independently; both gate on the same capability handshake. +Some original prerequisites have landed, but not in the originally proposed +form. Remaining work follows this order: -## Open design decisions (pin before implementation) +1. **Resolve the authority mode in #1326.** Select total order or opaque + exact-CAS, specify its atomic commit point, and audit PR #6077 against it. + Do not retrofit ordering semantics onto the existing random UUID. +2. **Define the generation fleet proof.** Map generation enablement to the RPC + signature/body/replay strict proofs delivered by #1327/#1541/#1542 and to + the selected per-disk comparison capability. Keep all strict defaults off + until fallback metrics converge. +3. **Implement #1313 generation-bound read leases.** The lease registry, + cross-node visibility, TTL, and crash recovery must exist before old-dir GC + can claim the full snapshot-lifetime guarantee. #1325 supplies the required + multi-node failure tests. +4. **Bind #1314 prepared reads.** A bundle binds the exact selected generation + within its source pool. Cross-pool ordering is forbidden until a common + authority is demonstrated. Validate rebalance and mixed-version fallback in + the #1325 multi-pool harness. +5. **Reconcile #1318 quota fencing.** Either bind reserve/settle/reconcile to + the selected object generation or document and prove that its independent + snapshot-lease fence is a separate arbitration domain that cannot settle a + different generation. +6. **Re-audit #1323 cleanup.** The existing UUID receipt remains valid crash + cleanup, but full closure against active readers requires the #1313 lease + check and the selected generation semantics. -This document fixes the transport, encoding, proto, and gate constraints, but it is not yet a complete implementable algorithm. The following must be decided and written down before any of the five consumers is coded (per the #1307 maintainer re-review, issuecomment-4992956256): +## Open design decisions (pin before contract closure) -- **Epoch type and total order.** The concrete token type and its total-order rule — a term+counter tuple, its persistence, and overflow behavior. Whether monotonicity is global or strictly per-object. -- **Never-regress on lock-service restart / minority recovery.** The epoch source must survive a lock-service restart or minority-quorum recovery without ever handing out an epoch lower than one already persisted on disk (an in-memory counter reset to zero is a fencing inversion). This is the same requirement as "Monotonicity persistence semantics" above, elevated to a hard, tested acceptance. -- **Complete xl.meta-writer coverage.** Every code path that writes xl.meta (commit rename, rollback delete/metadata restore, old-dir cleanup, heal, transition) must be enumerated and shown to compare or carry the epoch. A single unfenced writer voids the guarantee. -- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure rollback at `io_primitives.rs:2646-2691` restores a metadata backup, not just a per-writer tmp delete, so a late rollback by writer A can overwrite writer B's committed xl.meta. Rollback must execute only when `stored_epoch == failed_writer_epoch`; a higher stored epoch must abort the rollback. Task panic / cancel / timeout at `io_primitives.rs:2602-2605` must be reaped into the coordinator's state machine, never bubble out via `?` and skip convergence. +The following decisions remain blockers for calling the contract implemented: + +- **Authority mode.** Choose total order or opaque exact-CAS. If total order is + selected, define the type, per-object scope, persistence, overflow, and + never-regress restart/minority-recovery tests. If opaque identity is selected, + define the atomic expected-generation CAS and remove all ordered wording. +- **Complete xl.meta-writer coverage.** Enumerate commit rename, rollback + restore/delete, cleanup, heal, transition, restore, replication, and data + movement. Each path must compare/carry the selected generation or be proved + incapable of replacing the authoritative object identity. +- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure + rollback in `rename_data` can restore backup metadata, not just remove a + writer-private temporary file. It must execute only when the stored generation + still matches the failed writer's expected generation. Panic, cancel, and + timeout outcomes must be reaped into coordinator convergence rather than skip + rollback through an early return. - **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe. -- **Read-lease and GC crash recovery.** Lease registry location (local vs cross-node), TTL reclamation, and crash recovery for both the lease holder and the GC executor. -- **Quota reserve → commit → settle idempotency.** The cross-stage reconcile / idempotency story for #1318, including owner-crash reconciliation, so a reservation is neither lost nor double-counted. +- **Generation capability proof.** Decide whether to extend the current + authenticated fleet proof or the runtime capability contract. It must prove + authority version, mutation coverage, topology/process epoch, and RPC strict + convergence in one revalidatable token. +- **Read-lease and GC crash recovery.** Select the cross-node registry, TTL + reclamation, lease-holder crash behavior, and GC-executor recovery. The + current cleanup receipt equality check does not answer these questions. +- **Quota reserve → commit → settle binding.** The durable ledger's idempotency + exists, but its independent mutation tokens must be related to the selected + object generation with a concrete late-settle rejection test. - **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited. -- **Hot-path cost is a blocking metric.** If per-PUT fencing grant, quota reserve, or cleanup journal adds a consensus write / fsync / centralized serialization point, it must be measured under 4KiB and high-concurrency hot-key / hot-bucket A/B as a blocking gate, not accepted by default. +- **Hot-path cost is a blocking metric.** Measure any additional consensus + write, fsync, fleet-proof lookup, lease operation, or centralized serialization + under 4 KiB and high-concurrency hot-key/hot-bucket A/B. +- **Test infrastructure.** #1325 still lacks the complete 4-node × 4-drive, + 2-pool, directed network-fault, and large-object budget needed for restart, + mixed-version, and cross-node lease acceptance. ## Acceptance for this contract -- #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this unified - generation and no longer define their own token. -- The five constraints — transport signature, encoding, proto presence, - mixed-version gate direction, and capability negotiation — are pinned here - once; each implementation sub-issue follows them rather than re-deciding. +- [x] Architecture document exists and is linked from the architecture index. +- [x] Transport signature, encoding, proto presence, mixed-version direction, + and capability-proof requirements are defined once. +- [x] Current implementations are separated from target guarantees; a closed + child issue is not treated as proof of unified generation binding. +- [ ] Authority mode and atomic comparison semantics are selected and tested. +- [ ] #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this document and + use the selected authority terminology. +- [ ] #1313 and #1314 bind the selected generation and pass #1325 multi-node / + multi-pool failure tests. +- [ ] #1318 either binds reserve/settle to the selected generation or provides + an accepted proof that its separate fence cannot cross-settle generations. +- [ ] #1323 reconciliation checks both committed generation and active + generation-bound leases. +- [ ] Generation strict enablement is backed by one live proof that includes RPC + signature/body/replay strict convergence and per-disk comparison support.