docs(architecture): reconcile generation contract (#6901)

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-08-31 00:20:39 +08:00
committed by GitHub
parent 1370434f3a
commit c2d8488728
+244 -120
View File
@@ -30,73 +30,120 @@ lifetime**. Left independent, they diverge and punch through one another:
different monotonic sources, cannot be compared — a late commit fenced on one different monotonic sources, cannot be compared — a late commit fenced on one
plane can still settle quota on the other. plane can still settle quota on the other.
The fix is a single authority with one monotonic source, one persistence The fix is a single authority with one selected comparison rule, one persistence
semantics, and one transport binding, that every consumer references rather than semantics, and one transport binding, that every consumer references rather than
re-derives. re-derives.
## The authority (single source) ## Target authority and the current bounded token
**The per-object fencing epoch defined by #1312 is the sole generation The target contract still requires **one per-object commit identity** consumed
authority.** No other monotonic counter, timestamp, or random token may stand in by commit fencing, read leases, cleanup, prepared reads, and quota settlement.
for generation. No consumer may mint a second value and call it the same generation.
- The distributed lock grant returns a monotonic `epoch` for the object key. The concrete ordering semantics are not settled, however. The original #1326
Acquiring the object write-lock is the only way to mint a new generation. proposal requires a total-ordered, monotonic lock-grant epoch. Current main does
- The epoch travels down the authoritative commit path (with not implement that proposal. PR #6077 instead implements an opaque transaction
`RenameDataRequest` / the local `DiskAPI` call) and is compared at each disk's identity:
atomic `xl.meta` commit point, rejecting stale epochs. It adds no extra
network round trip (#1312 implementation clause 2).
- Every consumer in the table below **binds** this epoch. None defines its own.
### Monotonicity persistence semantics - `assign_object_transaction_epoch` mints a random non-nil UUID for PUT and
CompleteMultipartUpload when the object-transaction gate is active.
- The UUID is written through `FileInfo::set_object_transaction_epoch` into the
dual internal metadata map.
- The coordinator reads the current UUID (or `Absent`) and revalidates exact
equality immediately before `rename_data`.
- Old-data cleanup receipts carry the committed UUID and reconciliation deletes
only when the receipt UUID still equals the current object UUID.
The epoch must be **monotonic across lock-plane restart and failover** This is a useful **equality-CAS fence and cleanup identity**. It is not a
(#1312 B4). Today the distributed lock entry is in-memory only monotonic epoch, is not minted by the distributed lock grant, and is not
(`crates/lock/src/distributed_lock.rs` has no persistence path), so a lock-service compared atomically at each disk's `xl.meta` commit point. Until the decision
restart resets the counter to zero: a new writer draws epoch 1 while disks have below is made, documents and issue checklists must call it the *object
already observed epoch 100, producing either a permanent write rejection or a transaction UUID* rather than use it as proof that the target generation
fence *inversion*. To prevent this, the epoch must be one of: authority exists.
1. **Quorum-persisted** before it is handed to a writer, or ### Ordering decision required
2. **Derived from a durable monotonic source** — a `(term, counter)` pair where
`term` advances on every lock-service leadership change and is itself durable,
so the composite never regresses even when `counter` resets.
The comparison at the disk commit point is on the full composite; a lower Before #1313, #1314, or a unified quota binding can consume the authority, one
`(term, counter)` is always rejected. of these contracts must be selected and tested:
1. **Total-ordered fencing epoch.** A lock grant returns a durable per-object
`(term, counter)` (or another specified total-order type). Every disk rejects
a lower epoch at the atomic metadata commit point. The value never regresses
across lock-plane restart, failover, or minority recovery.
2. **Opaque commit-generation identity.** Consumers compare only exact identity;
no `<` / `>` semantics are permitted. The authoritative commit must perform
an atomic expected-generation CAS, and all lease, cleanup, prepared-read, and
quota contracts must be rewritten in terms of “references this exact
generation,” not “lower/newer generation.”
The current UUID implementation proves neither a durable total order nor a
per-disk atomic expected-generation CAS, so it does not by itself decide between
these options.
### Persistence semantics if total order is selected
A total-ordered epoch must be **monotonic across lock-plane restart and
failover**. The distributed lock entry remains in-memory; deriving a counter
from that entry alone would reset it after restart. The chosen source therefore
must be either quorum-persisted before grant or derived from a durable term whose
full `(term, counter)` comparison cannot regress. This requirement does not
apply to an opaque UUID as an ordering rule; the opaque alternative instead
requires atomic expected-identity comparison and durable crash recovery.
## Consumer binding contracts ## Consumer binding contracts
### Current implementation snapshot (2026-08-31, main@9ee7b1221)
This table separates code that exists on current main from the target contract.
Closing an implementation issue does not imply that its token is already the
unified authority.
| Surface | Current main | Gap against this contract |
|---|---|---|
| PUT / CompleteMultipartUpload (#1312, PR #6077) | Owned commit tasks retain the relevant guards; an opt-in gate persists a random object transaction UUID and performs a quorum metadata equality recheck before rename | no lock-grant monotonic source; no per-disk atomic epoch/CAS comparison; the live proof is the reused remote-version-state fleet proof, not a dedicated generation capability |
| Old-data cleanup (#1323, PR #6077) | JSON receipt carries transaction UUID, old dir, and committed dir; reconciliation is gated and requires UUID equality | no generation-bound read lease is consulted, so this is crash cleanup fencing rather than the full #1313/#1323 lease lifetime contract |
| Read lease (#1313) | short-term streaming/multipart path holds the namespace read lock through EOF/drop; deterministic part-boundary coverage is tracked by PR #6887 | no cross-node generation-bound lease registry, TTL reclamation, or crash recovery |
| Prepared pool read (#1314) | PR #6889 tracks a pool-local prepared identity and fails closed/refetches when pool state changes | not merged on this snapshot; pool-local identity is not a cross-pool generation authority; black-box mixed-version/rebalance coverage remains open |
| Quota reservation (#1318) | durable per-bucket ledger plus independent snapshot-lease mutation-fence tokens; issue closed after PR #6058 | reservation and settle are not bound to the object transaction UUID; the independent fence must be reconciled with the selected authority or explicitly proven to be a separate, non-generation arbitration domain |
| Internode integrity (#1327, #1541, #1542) | v2/v3 HMAC binds audience, exact method, timestamp, nonce, canonical body digest, and receiver boot epoch; body-bound RPC policy has exact-set coverage | signature/body/replay strict switches remain default-off rollout gates; generation enforcement cannot treat an unrelated fleet-version proof as proof that these strict contracts converged |
| Consumer | How it binds generation | Key invariant | | Consumer | How it binds generation | Key invariant |
|---|---|---| |---|---|---|
| #1312 commit fence | epoch compared at three disk-write points — `rename`, rollback `delete`, and `commit_rename_data_dir` cleanup | stale epoch rejected on **all** disks; an already-ACK'd write is never rolled back | | #1312 commit fence | selected generation is checked at `rename`, rollback restore/delete, and cleanup mutation points using the chosen ordered or exact-CAS rule | a stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back |
| #1313 read lease | lease binds the generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL | | #1313 read lease | lease binds the exact generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms no lease referencing a lower generation still points at it | `old_dir != committed_dir`; a still-referenced directory is never deleted | | #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms that no lease for the generation owning that directory remains | `old_dir != committed_dir`; a still-referenced directory is never deleted |
| #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout | | #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout |
| #1318 quota reservation | reservation / settle token binds the object generation | a late commit holding an old-generation token cannot settle a newer generation | | #1318 quota reservation | reservation / settle record binds the exact object generation (and an ordered epoch too, if that option is selected) | a late commit cannot settle quota for a different committed generation |
### Fence coverage is three disk-write points, not one (#1312 B2) ### Fence coverage is three disk-write points, not one (#1312 B2)
Comparing the epoch at the `rename` commit point alone is insufficient. The Checking the generation only before the `rename` fanout is insufficient. The
authoritative commit sequence is `tmp sync → data-dir rename → xl.meta commit → authoritative commit sequence is `tmp sync → data-dir rename → xl.meta commit →
directory sync` in `crates/ecstore/src/disk/local.rs`, and there are two further directory sync` in `crates/ecstore/src/disk/local.rs`, and there are two further
detachable disk-write points in detachable disk-write points in
`crates/ecstore/src/set_disk/core/io_primitives.rs`: `crates/ecstore/src/set_disk/core/io_primitives.rs`:
- **Rollback delete** — on quorum failure each disk runs - **Rollback restore/delete** — on quorum failure each disk can restore backup
`delete_version(undo_write=true)`. A fenced old writer's rollback must also metadata or delete the failed version. A stale writer's rollback must compare
compare epoch, otherwise it deletes the winner's already-committed version. the expected generation, otherwise it can overwrite or delete the winner's
already-committed metadata.
- **`commit_rename_data_dir`** — a cancel-then-detach disk-write point; the - **`commit_rename_data_dir`** — a cancel-then-detach disk-write point; the
coordinator's "reap all child tasks" must explicitly include it so a cancelled coordinator's "reap all child tasks" must explicitly include it so a cancelled
writer cannot bypass fence/lease and keep deleting directories. writer cannot bypass fence/lease and keep deleting directories.
If the epoch is validated only at the `xl.meta` commit point, a fenced writer If generation is validated only after data-dir rename, a fenced writer
may already have renamed its data-dir into the object path, leaving a staged may already have renamed its data-dir into the object path, leaving a staged
orphan. Either move the fence ahead of the data-dir rename, or declare that orphan. Either move the fence ahead of the data-dir rename, or declare that
orphan an acceptable residue accounted for by GC metrics — the white-box orphan an acceptable residue accounted for by GC metrics — the white-box
acceptance "no background disk write after release" must be rewritten acceptance "no background disk write after release" must be rewritten
accordingly. accordingly.
Current PR #6077 performs a quorum metadata equality recheck before rename and
reaps owned commit work. That closes important cancellation windows, but it is
not evidence that every disk mutation above performs the selected generation
comparison atomically. The writer inventory and per-point CAS/ordering proof
remain acceptance work for #1326 even though #1312 is closed.
### Post-commit convergence is orthogonal to the fence (#1321) ### Post-commit convergence is orthogonal to the fence (#1321)
The same `SetDisks::rename_data` path already returns a post-commit The same `SetDisks::rename_data` path already returns a post-commit
@@ -125,36 +172,40 @@ internode RPC bodies. Every such flow must be signature-bound.
### RPC signature binding (#1312 B3, #1313, #1318) ### RPC signature binding (#1312 B3, #1313, #1318)
**Requirement.** The RPC body digest carrying a generation/epoch/token must be **Requirement.** The canonical body carrying a generation or derived token must
folded into the RPC HMAC, binding `method + object key + generation`, and the be folded into the internode HMAC. The authenticated scope binds the target
request must carry a nonce / one-shot identifier inside the 300s replay window. audience, exact service/method, timestamp, nonce, canonical body digest, and
The nonce is only meaningful if the **receiver enforces it**: each disk keeps a receiver replay epoch. The receiver must consume the nonce in a bounded replay
bounded seen-nonce cache covering the 300s freshness window and rejects any cache; transmitting a nonce without receiver-side consumption is not replay
request whose nonce was already observed. A nonce that is merely transmitted but protection.
not checked provides no replay protection.
This generalizes the existing `walk_dir` pattern: `walk_dir` computes a **Current substrate (verified on main).** The original legacy-only description
`Sha256` of the request body and places it in the signed URL query as is obsolete:
`walk_dir_body_sha256`
(`crates/ecstore/src/cluster/rpc/internode_data_transport.rs:187`), so the body
digest is transitively covered by the URL signature. New generation-bearing RPCs
adopt the same `*_body_sha256` mechanism.
**Current gap (verified).** The internode HMAC covers only - RPC v2 binds target audience, exact method, POST, timestamp, nonce, and body
`{path_and_query}|{method}|{timestamp}` digest.
(`signature_payload`, `crates/ecstore/src/cluster/rpc/http_auth.rs:75-83`). It - Body-bound policy covers mutating disk RPCs including `RenameData`; its
binds neither the request body nor a nonce, and the 300s freshness window has no versioned canonical body includes every `RenameDataRequest` field, so the
one-shot guard. Without the binding above: `FileInfo` metadata map carrying the transaction UUID is authenticated.
- PR #5425 extended canonical-body enforcement to implemented non-disk mutating
unary RPCs and added an exact policy/handler coverage partition.
- PR #5455 added the receiver boot epoch and rotating replay scope so signatures
captured before a receiver restart are rejected after capability convergence.
- An on-path or replaying attacker can inject a high epoch (e.g. `u32::MAX`) and The rollout switches
**permanently fence out** a key's legitimate writes — monotonicity only `RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`,
rejects *low/old* epochs, never a forged-high one. `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, and
- A captured lease/reservation token can be replayed within 300s to block `RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` remain default-off for rolling
old-dir GC (storage-exhaustion DoS) or to double-reserve / prematurely settle compatibility. The compatibility register and fallback/overflow metrics govern
quota. their fleet convergence. Therefore a generation capability may claim strong
transport binding only when the relevant strict modes have converged; the
object-transaction gate's current remote-version-state fleet proof is not, by
itself, proof of RPC signature/body/replay strictness.
Acceptance for each consumer must include: "a replayed old signature to a Acceptance for each generation consumer includes method substitution, canonical
different method, and a forged-high-epoch request, are both rejected." body tamper, nonce replay, receiver restart, and stripped-strict-metadata
negative tests. Generation rollout must also record which strict-mode evidence
authorized enforcement.
### Encoding contract (#1312 B1) ### Encoding contract (#1312 B1)
@@ -167,11 +218,15 @@ The on-disk persistence of generation must not perturb the file format:
`xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a `xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a
total read failure, not a graceful downgrade. total read failure, not a graceful downgrade.
- **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged. - **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged.
- **Where it may live.** Only inside a version's internal metadata **map** - **Where it lives today.** The object transaction UUID uses the version's
(MinIO skips unknown internal keys and the map encoding is extensible) or in a internal metadata map under the dual-key contract
per-disk sidecar outside `xl.meta`. If it goes in the metadata map, it must (`x-rustfs-internal-*` / `x-minio-internal-*`) via
obey the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`, see `set_object_transaction_epoch`. Missing, malformed, nil, or conflicting dual
AGENTS.md "Cross-Cutting Domain Invariants"). values fail closed when fencing is active.
- **Sidecars are not an equivalent alternative.** A future sidecar is admissible
only if it commits atomically with `xl.meta` and has a specified crash-recovery
protocol. No such protocol is implemented, so a sidecar cannot be selected by
an implementation issue merely because this document mentions one.
- **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop - **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop
regression (the fixture family around `crates/filemeta/src/filemeta.rs`): regression (the fixture family around `crates/filemeta/src/filemeta.rs`):
objects written by a new node must still be readable by old RustFS nodes and objects written by a new node must still be readable by old RustFS nodes and
@@ -179,82 +234,151 @@ The on-disk persistence of generation must not perturb the file format:
### Wire-encoding migration (JSON → msgpack) interaction ### Wire-encoding migration (JSON → msgpack) interaction
The internode RPC layer is mid-migration from JSON to msgpack binary, and generation-bearing fields must respect that migration window — this is not optional context, it changes how epoch is transported. The internode RPC layer retains a JSON/msgpack rolling-compatibility window, and
generation-bearing fields must respect it.
- **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info` #4 alongside `file_info_bin` #7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`). - **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info` #4 alongside `file_info_bin` #7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`).
- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the `record_msgpack_json_fallback` metric reads zero fleet-wide and the convergence runbook is followed. If only the request flag is set, RustFS keeps dual-writing JSON compatibility fields. **Reuse this exact capability + metric-reads-zero model as the mixed-version gate for generation** rather than inventing a parallel handshake; the section above ("Capability negotiation") is layered on top of it, not instead of it. - **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the JSON-fallback metric reads zero fleet-wide and the convergence runbook is followed. Generation follows the same default-off, fleet-confirmed, metric-reads-zero rollout discipline, but a msgpack proof is not itself a generation capability proof.
- **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch. - **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch.
- **Signature must bind a canonical form.** Because a field is transmitted as both JSON and msgpack and a peer may consume either, the body-digest binding in "RPC signature binding" above must be computed over a single canonical representation (the msgpack `_bin` bytes) — not over whichever copy happened to be decoded. Once the `generation` capability is negotiated for a request, a fenced / generation-bearing request must **reject the JSON fallback path** so a downgrade to the unsigned/loosely-bound JSON copy cannot bypass the epoch check. - **Signature binds a canonical form.** `RenameDataRequest` now has a versioned,
injective canonical-body encoder that covers both compatibility fields and is
authenticated independently of whichever JSON/msgpack decoder branch a peer
consumes. A generation-capable strict request must reject missing or
mismatched canonical-body metadata; it must not silently downgrade to an
unauthenticated JSON twin.
### Proto evolution ### Proto evolution
New generation/epoch proto fields use **proto3 `optional`** (explicit presence). No top-level proto field is required by the current metadata-map UUID. If a
A non-optional field is forbidden: an old coordinator talking to a new disk future ordered epoch or explicit expected-generation is added to proto, it uses
decodes an absent field as `0`, which is indistinguishable from a real **proto3 `optional`** (explicit presence). A non-optional scalar is forbidden:
`epoch == 0` and silently breaks the "stale epoch rejected" invariant during an old coordinator talking to a new disk decodes absence as a plausible zero.
upgrade.
### Mixed-version gate — one direction ### Mixed-version gate — one direction
When the cluster-level generation capability is **not** negotiated on every When generation enforcement is not explicitly requested, or fleet confirmation
target disk, the behavior **falls back to current semantics** (existing lock + is absent, behavior falls back to current semantics. Fail-closed is reserved for
`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at an explicit administrator-confirmed strict rollout.
`rustfs/src/app/object/get.rs`; full fanout for #1314). Fail-closed is
**only** an explicit administrator strict mode. Defaulting to fail-closed is Current object transaction fencing follows that direction:
forbidden — it makes writes unavailable for the whole rolling-upgrade window.
- `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` and
`RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` both default false.
- With either flag absent, PUT/MPU does not persist or consume the transaction
UUID.
- With both flags enabled, failure to obtain or retain the live fleet proof
rejects the commit before rename.
This is an opt-in strict gate, not a negotiated generation capability. The
proof is currently borrowed from the remote-version-state writer rollout. It
proves current membership/process-epoch convergence for that feature, but does
not prove an epoch type, per-disk generation CAS support, or RPC strict-mode
convergence. Treating it as the final handshake is forbidden without an
explicit proof mapping for those properties.
## Capability negotiation ## Capability negotiation
Generation enforcement is a **cluster-level handshake**, not a per-request Generation enforcement requires one **live fleet proof**, not independent
probe: boolean guesses in each consumer. The proof contract contains at least:
- A node advertises a `generation` capability once it can (a) mint quorum-durable 1. the selected authority version and comparison mode (ordered or exact-CAS),
epochs, (b) compare epochs at all three disk-write points, and (c) verify the 2. the current membership/topology fingerprint and process epochs,
body-digest-bound RPC signature. 3. support for every required disk mutation point,
- The authoritative writer enables hard enforcement for an object only when 4. RPC signature/body/replay strict convergence, and
**all** target disks in the set advertise the capability. Any missing 5. the on-disk encoding version (the current metadata-map UUID is version 1).
advertisement pins that commit to the mixed-version fallback above.
- The capability is surfaced through the existing runtime capability contract The authoritative writer enables enforcement only while every target disk in
surface (see [runtime-capability-contracts.md](runtime-capability-contracts.md)), the set is covered by a current proof. Membership change or an old node rejoin
so consumers read one negotiated flag rather than each re-deriving support. revokes that proof. Revocation before commit fails an explicitly strict request;
- Enforcement tracks the current membership rather than latching: it turns on when strict generation was never requested, the request remains on the legacy
for a set only while every disk in that set advertises `generation`, and a path. Revocation never rewrites or lowers an already-persisted generation.
single old node rejoining drops the affected sets back to the mixed-version
fallback rather than failing closed. It never regresses the on-disk epoch — The existing fleet-proof machinery in `notification_sys` may be reused if its
falling back stops *comparing* new epochs, it does not lower any epoch already authenticated statements are extended to cover the properties above. The
persisted. runtime capability contract may instead expose the proof. This document does
not choose the storage mechanism; it requires one token whose acquisition and
revalidation semantics are shared by all consumers.
## Implementation order ## Implementation order
1. **#1312 first.** It defines the epoch, its persistence, the three fence Some original prerequisites have landed, but not in the originally proposed
points, the RPC signature binding, and the encoding location. Everything form. Remaining work follows this order:
downstream depends on its epoch existing.
2. **#1313** (read lease) reuses the #1312 epoch as the lease generation and
must land before or alongside #1323.
3. **#1323** (old-dir GC) depends on #1313 leases being present and
cross-node-visible; its "no lease references old_dir" check has nothing to
query otherwise.
4. **#1318** (quota reservation) and **#1314** (prepared pool read) bind the
epoch independently; both gate on the same capability handshake.
## Open design decisions (pin before implementation) 1. **Resolve the authority mode in #1326.** Select total order or opaque
exact-CAS, specify its atomic commit point, and audit PR #6077 against it.
Do not retrofit ordering semantics onto the existing random UUID.
2. **Define the generation fleet proof.** Map generation enablement to the RPC
signature/body/replay strict proofs delivered by #1327/#1541/#1542 and to
the selected per-disk comparison capability. Keep all strict defaults off
until fallback metrics converge.
3. **Implement #1313 generation-bound read leases.** The lease registry,
cross-node visibility, TTL, and crash recovery must exist before old-dir GC
can claim the full snapshot-lifetime guarantee. #1325 supplies the required
multi-node failure tests.
4. **Bind #1314 prepared reads.** A bundle binds the exact selected generation
within its source pool. Cross-pool ordering is forbidden until a common
authority is demonstrated. Validate rebalance and mixed-version fallback in
the #1325 multi-pool harness.
5. **Reconcile #1318 quota fencing.** Either bind reserve/settle/reconcile to
the selected object generation or document and prove that its independent
snapshot-lease fence is a separate arbitration domain that cannot settle a
different generation.
6. **Re-audit #1323 cleanup.** The existing UUID receipt remains valid crash
cleanup, but full closure against active readers requires the #1313 lease
check and the selected generation semantics.
This document fixes the transport, encoding, proto, and gate constraints, but it is not yet a complete implementable algorithm. The following must be decided and written down before any of the five consumers is coded (per the #1307 maintainer re-review, issuecomment-4992956256): ## Open design decisions (pin before contract closure)
- **Epoch type and total order.** The concrete token type and its total-order rule — a term+counter tuple, its persistence, and overflow behavior. Whether monotonicity is global or strictly per-object. The following decisions remain blockers for calling the contract implemented:
- **Never-regress on lock-service restart / minority recovery.** The epoch source must survive a lock-service restart or minority-quorum recovery without ever handing out an epoch lower than one already persisted on disk (an in-memory counter reset to zero is a fencing inversion). This is the same requirement as "Monotonicity persistence semantics" above, elevated to a hard, tested acceptance.
- **Complete xl.meta-writer coverage.** Every code path that writes xl.meta (commit rename, rollback delete/metadata restore, old-dir cleanup, heal, transition) must be enumerated and shown to compare or carry the epoch. A single unfenced writer voids the guarantee. - **Authority mode.** Choose total order or opaque exact-CAS. If total order is
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure rollback at `io_primitives.rs:2646-2691` restores a metadata backup, not just a per-writer tmp delete, so a late rollback by writer A can overwrite writer B's committed xl.meta. Rollback must execute only when `stored_epoch == failed_writer_epoch`; a higher stored epoch must abort the rollback. Task panic / cancel / timeout at `io_primitives.rs:2602-2605` must be reaped into the coordinator's state machine, never bubble out via `?` and skip convergence. selected, define the type, per-object scope, persistence, overflow, and
never-regress restart/minority-recovery tests. If opaque identity is selected,
define the atomic expected-generation CAS and remove all ordered wording.
- **Complete xl.meta-writer coverage.** Enumerate commit rename, rollback
restore/delete, cleanup, heal, transition, restore, replication, and data
movement. Each path must compare/carry the selected generation or be proved
incapable of replacing the authoritative object identity.
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure
rollback in `rename_data` can restore backup metadata, not just remove a
writer-private temporary file. It must execute only when the stored generation
still matches the failed writer's expected generation. Panic, cancel, and
timeout outcomes must be reaped into coordinator convergence rather than skip
rollback through an early return.
- **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe. - **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe.
- **Read-lease and GC crash recovery.** Lease registry location (local vs cross-node), TTL reclamation, and crash recovery for both the lease holder and the GC executor. - **Generation capability proof.** Decide whether to extend the current
- **Quota reserve → commit → settle idempotency.** The cross-stage reconcile / idempotency story for #1318, including owner-crash reconciliation, so a reservation is neither lost nor double-counted. authenticated fleet proof or the runtime capability contract. It must prove
authority version, mutation coverage, topology/process epoch, and RPC strict
convergence in one revalidatable token.
- **Read-lease and GC crash recovery.** Select the cross-node registry, TTL
reclamation, lease-holder crash behavior, and GC-executor recovery. The
current cleanup receipt equality check does not answer these questions.
- **Quota reserve → commit → settle binding.** The durable ledger's idempotency
exists, but its independent mutation tokens must be related to the selected
object generation with a concrete late-settle rejection test.
- **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited. - **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited.
- **Hot-path cost is a blocking metric.** If per-PUT fencing grant, quota reserve, or cleanup journal adds a consensus write / fsync / centralized serialization point, it must be measured under 4KiB and high-concurrency hot-key / hot-bucket A/B as a blocking gate, not accepted by default. - **Hot-path cost is a blocking metric.** Measure any additional consensus
write, fsync, fleet-proof lookup, lease operation, or centralized serialization
under 4 KiB and high-concurrency hot-key/hot-bucket A/B.
- **Test infrastructure.** #1325 still lacks the complete 4-node × 4-drive,
2-pool, directed network-fault, and large-object budget needed for restart,
mixed-version, and cross-node lease acceptance.
## Acceptance for this contract ## Acceptance for this contract
- #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this unified - [x] Architecture document exists and is linked from the architecture index.
generation and no longer define their own token. - [x] Transport signature, encoding, proto presence, mixed-version direction,
- The five constraints — transport signature, encoding, proto presence, and capability-proof requirements are defined once.
mixed-version gate direction, and capability negotiation — are pinned here - [x] Current implementations are separated from target guarantees; a closed
once; each implementation sub-issue follows them rather than re-deciding. child issue is not treated as proof of unified generation binding.
- [ ] Authority mode and atomic comparison semantics are selected and tested.
- [ ] #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this document and
use the selected authority terminology.
- [ ] #1313 and #1314 bind the selected generation and pass #1325 multi-node /
multi-pool failure tests.
- [ ] #1318 either binds reserve/settle to the selected generation or provides
an accepted proof that its separate fence cannot cross-settle generations.
- [ ] #1323 reconciliation checks both committed generation and active
generation-bound leases.
- [ ] Generation strict enablement is backed by one live proof that includes RPC
signature/body/replay strict convergence and per-disk comparison support.