docs(architecture): reconcile generation contract (#6901)

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-08-31 00:20:39 +08:00
committed by GitHub
parent 1370434f3a
commit c2d8488728
+244 -120
View File
@@ -30,73 +30,120 @@ lifetime**. Left independent, they diverge and punch through one another:
different monotonic sources, cannot be compared — a late commit fenced on one
plane can still settle quota on the other.
The fix is a single authority with one monotonic source, one persistence
The fix is a single authority with one selected comparison rule, one persistence
semantics, and one transport binding, that every consumer references rather than
re-derives.
## The authority (single source)
## Target authority and the current bounded token
**The per-object fencing epoch defined by #1312 is the sole generation
authority.** No other monotonic counter, timestamp, or random token may stand in
for generation.
The target contract still requires **one per-object commit identity** consumed
by commit fencing, read leases, cleanup, prepared reads, and quota settlement.
No consumer may mint a second value and call it the same generation.
- The distributed lock grant returns a monotonic `epoch` for the object key.
Acquiring the object write-lock is the only way to mint a new generation.
- The epoch travels down the authoritative commit path (with
`RenameDataRequest` / the local `DiskAPI` call) and is compared at each disk's
atomic `xl.meta` commit point, rejecting stale epochs. It adds no extra
network round trip (#1312 implementation clause 2).
- Every consumer in the table below **binds** this epoch. None defines its own.
The concrete ordering semantics are not settled, however. The original #1326
proposal requires a total-ordered, monotonic lock-grant epoch. Current main does
not implement that proposal. PR #6077 instead implements an opaque transaction
identity:
### Monotonicity persistence semantics
- `assign_object_transaction_epoch` mints a random non-nil UUID for PUT and
CompleteMultipartUpload when the object-transaction gate is active.
- The UUID is written through `FileInfo::set_object_transaction_epoch` into the
dual internal metadata map.
- The coordinator reads the current UUID (or `Absent`) and revalidates exact
equality immediately before `rename_data`.
- Old-data cleanup receipts carry the committed UUID and reconciliation deletes
only when the receipt UUID still equals the current object UUID.
The epoch must be **monotonic across lock-plane restart and failover**
(#1312 B4). Today the distributed lock entry is in-memory only
(`crates/lock/src/distributed_lock.rs` has no persistence path), so a lock-service
restart resets the counter to zero: a new writer draws epoch 1 while disks have
already observed epoch 100, producing either a permanent write rejection or a
fence *inversion*. To prevent this, the epoch must be one of:
This is a useful **equality-CAS fence and cleanup identity**. It is not a
monotonic epoch, is not minted by the distributed lock grant, and is not
compared atomically at each disk's `xl.meta` commit point. Until the decision
below is made, documents and issue checklists must call it the *object
transaction UUID* rather than use it as proof that the target generation
authority exists.
1. **Quorum-persisted** before it is handed to a writer, or
2. **Derived from a durable monotonic source** — a `(term, counter)` pair where
`term` advances on every lock-service leadership change and is itself durable,
so the composite never regresses even when `counter` resets.
### Ordering decision required
The comparison at the disk commit point is on the full composite; a lower
`(term, counter)` is always rejected.
Before #1313, #1314, or a unified quota binding can consume the authority, one
of these contracts must be selected and tested:
1. **Total-ordered fencing epoch.** A lock grant returns a durable per-object
`(term, counter)` (or another specified total-order type). Every disk rejects
a lower epoch at the atomic metadata commit point. The value never regresses
across lock-plane restart, failover, or minority recovery.
2. **Opaque commit-generation identity.** Consumers compare only exact identity;
no `<` / `>` semantics are permitted. The authoritative commit must perform
an atomic expected-generation CAS, and all lease, cleanup, prepared-read, and
quota contracts must be rewritten in terms of “references this exact
generation,” not “lower/newer generation.”
The current UUID implementation proves neither a durable total order nor a
per-disk atomic expected-generation CAS, so it does not by itself decide between
these options.
### Persistence semantics if total order is selected
A total-ordered epoch must be **monotonic across lock-plane restart and
failover**. The distributed lock entry remains in-memory; deriving a counter
from that entry alone would reset it after restart. The chosen source therefore
must be either quorum-persisted before grant or derived from a durable term whose
full `(term, counter)` comparison cannot regress. This requirement does not
apply to an opaque UUID as an ordering rule; the opaque alternative instead
requires atomic expected-identity comparison and durable crash recovery.
## Consumer binding contracts
### Current implementation snapshot (2026-08-31, main@9ee7b1221)
This table separates code that exists on current main from the target contract.
Closing an implementation issue does not imply that its token is already the
unified authority.
| Surface | Current main | Gap against this contract |
|---|---|---|
| PUT / CompleteMultipartUpload (#1312, PR #6077) | Owned commit tasks retain the relevant guards; an opt-in gate persists a random object transaction UUID and performs a quorum metadata equality recheck before rename | no lock-grant monotonic source; no per-disk atomic epoch/CAS comparison; the live proof is the reused remote-version-state fleet proof, not a dedicated generation capability |
| Old-data cleanup (#1323, PR #6077) | JSON receipt carries transaction UUID, old dir, and committed dir; reconciliation is gated and requires UUID equality | no generation-bound read lease is consulted, so this is crash cleanup fencing rather than the full #1313/#1323 lease lifetime contract |
| Read lease (#1313) | short-term streaming/multipart path holds the namespace read lock through EOF/drop; deterministic part-boundary coverage is tracked by PR #6887 | no cross-node generation-bound lease registry, TTL reclamation, or crash recovery |
| Prepared pool read (#1314) | PR #6889 tracks a pool-local prepared identity and fails closed/refetches when pool state changes | not merged on this snapshot; pool-local identity is not a cross-pool generation authority; black-box mixed-version/rebalance coverage remains open |
| Quota reservation (#1318) | durable per-bucket ledger plus independent snapshot-lease mutation-fence tokens; issue closed after PR #6058 | reservation and settle are not bound to the object transaction UUID; the independent fence must be reconciled with the selected authority or explicitly proven to be a separate, non-generation arbitration domain |
| Internode integrity (#1327, #1541, #1542) | v2/v3 HMAC binds audience, exact method, timestamp, nonce, canonical body digest, and receiver boot epoch; body-bound RPC policy has exact-set coverage | signature/body/replay strict switches remain default-off rollout gates; generation enforcement cannot treat an unrelated fleet-version proof as proof that these strict contracts converged |
| Consumer | How it binds generation | Key invariant |
|---|---|---|
| #1312 commit fence | epoch compared at three disk-write points — `rename`, rollback `delete`, and `commit_rename_data_dir` cleanup | stale epoch rejected on **all** disks; an already-ACK'd write is never rolled back |
| #1313 read lease | lease binds the generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms no lease referencing a lower generation still points at it | `old_dir != committed_dir`; a still-referenced directory is never deleted |
| #1312 commit fence | selected generation is checked at `rename`, rollback restore/delete, and cleanup mutation points using the chosen ordered or exact-CAS rule | a stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back |
| #1313 read lease | lease binds the exact generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms that no lease for the generation owning that directory remains | `old_dir != committed_dir`; a still-referenced directory is never deleted |
| #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout |
| #1318 quota reservation | reservation / settle token binds the object generation | a late commit holding an old-generation token cannot settle a newer generation |
| #1318 quota reservation | reservation / settle record binds the exact object generation (and an ordered epoch too, if that option is selected) | a late commit cannot settle quota for a different committed generation |
### Fence coverage is three disk-write points, not one (#1312 B2)
Comparing the epoch at the `rename` commit point alone is insufficient. The
Checking the generation only before the `rename` fanout is insufficient. The
authoritative commit sequence is `tmp sync → data-dir rename → xl.meta commit →
directory sync` in `crates/ecstore/src/disk/local.rs`, and there are two further
detachable disk-write points in
`crates/ecstore/src/set_disk/core/io_primitives.rs`:
- **Rollback delete** — on quorum failure each disk runs
`delete_version(undo_write=true)`. A fenced old writer's rollback must also
compare epoch, otherwise it deletes the winner's already-committed version.
- **Rollback restore/delete** — on quorum failure each disk can restore backup
metadata or delete the failed version. A stale writer's rollback must compare
the expected generation, otherwise it can overwrite or delete the winner's
already-committed metadata.
- **`commit_rename_data_dir`** — a cancel-then-detach disk-write point; the
coordinator's "reap all child tasks" must explicitly include it so a cancelled
writer cannot bypass fence/lease and keep deleting directories.
If the epoch is validated only at the `xl.meta` commit point, a fenced writer
If generation is validated only after data-dir rename, a fenced writer
may already have renamed its data-dir into the object path, leaving a staged
orphan. Either move the fence ahead of the data-dir rename, or declare that
orphan an acceptable residue accounted for by GC metrics — the white-box
acceptance "no background disk write after release" must be rewritten
accordingly.
Current PR #6077 performs a quorum metadata equality recheck before rename and
reaps owned commit work. That closes important cancellation windows, but it is
not evidence that every disk mutation above performs the selected generation
comparison atomically. The writer inventory and per-point CAS/ordering proof
remain acceptance work for #1326 even though #1312 is closed.
### Post-commit convergence is orthogonal to the fence (#1321)
The same `SetDisks::rename_data` path already returns a post-commit
@@ -125,36 +172,40 @@ internode RPC bodies. Every such flow must be signature-bound.
### RPC signature binding (#1312 B3, #1313, #1318)
**Requirement.** The RPC body digest carrying a generation/epoch/token must be
folded into the RPC HMAC, binding `method + object key + generation`, and the
request must carry a nonce / one-shot identifier inside the 300s replay window.
The nonce is only meaningful if the **receiver enforces it**: each disk keeps a
bounded seen-nonce cache covering the 300s freshness window and rejects any
request whose nonce was already observed. A nonce that is merely transmitted but
not checked provides no replay protection.
**Requirement.** The canonical body carrying a generation or derived token must
be folded into the internode HMAC. The authenticated scope binds the target
audience, exact service/method, timestamp, nonce, canonical body digest, and
receiver replay epoch. The receiver must consume the nonce in a bounded replay
cache; transmitting a nonce without receiver-side consumption is not replay
protection.
This generalizes the existing `walk_dir` pattern: `walk_dir` computes a
`Sha256` of the request body and places it in the signed URL query as
`walk_dir_body_sha256`
(`crates/ecstore/src/cluster/rpc/internode_data_transport.rs:187`), so the body
digest is transitively covered by the URL signature. New generation-bearing RPCs
adopt the same `*_body_sha256` mechanism.
**Current substrate (verified on main).** The original legacy-only description
is obsolete:
**Current gap (verified).** The internode HMAC covers only
`{path_and_query}|{method}|{timestamp}`
(`signature_payload`, `crates/ecstore/src/cluster/rpc/http_auth.rs:75-83`). It
binds neither the request body nor a nonce, and the 300s freshness window has no
one-shot guard. Without the binding above:
- RPC v2 binds target audience, exact method, POST, timestamp, nonce, and body
digest.
- Body-bound policy covers mutating disk RPCs including `RenameData`; its
versioned canonical body includes every `RenameDataRequest` field, so the
`FileInfo` metadata map carrying the transaction UUID is authenticated.
- PR #5425 extended canonical-body enforcement to implemented non-disk mutating
unary RPCs and added an exact policy/handler coverage partition.
- PR #5455 added the receiver boot epoch and rotating replay scope so signatures
captured before a receiver restart are rejected after capability convergence.
- An on-path or replaying attacker can inject a high epoch (e.g. `u32::MAX`) and
**permanently fence out** a key's legitimate writes — monotonicity only
rejects *low/old* epochs, never a forged-high one.
- A captured lease/reservation token can be replayed within 300s to block
old-dir GC (storage-exhaustion DoS) or to double-reserve / prematurely settle
quota.
The rollout switches
`RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`,
`RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, and
`RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` remain default-off for rolling
compatibility. The compatibility register and fallback/overflow metrics govern
their fleet convergence. Therefore a generation capability may claim strong
transport binding only when the relevant strict modes have converged; the
object-transaction gate's current remote-version-state fleet proof is not, by
itself, proof of RPC signature/body/replay strictness.
Acceptance for each consumer must include: "a replayed old signature to a
different method, and a forged-high-epoch request, are both rejected."
Acceptance for each generation consumer includes method substitution, canonical
body tamper, nonce replay, receiver restart, and stripped-strict-metadata
negative tests. Generation rollout must also record which strict-mode evidence
authorized enforcement.
### Encoding contract (#1312 B1)
@@ -167,11 +218,15 @@ The on-disk persistence of generation must not perturb the file format:
`xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a
total read failure, not a graceful downgrade.
- **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged.
- **Where it may live.** Only inside a version's internal metadata **map**
(MinIO skips unknown internal keys and the map encoding is extensible) or in a
per-disk sidecar outside `xl.meta`. If it goes in the metadata map, it must
obey the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`, see
AGENTS.md "Cross-Cutting Domain Invariants").
- **Where it lives today.** The object transaction UUID uses the version's
internal metadata map under the dual-key contract
(`x-rustfs-internal-*` / `x-minio-internal-*`) via
`set_object_transaction_epoch`. Missing, malformed, nil, or conflicting dual
values fail closed when fencing is active.
- **Sidecars are not an equivalent alternative.** A future sidecar is admissible
only if it commits atomically with `xl.meta` and has a specified crash-recovery
protocol. No such protocol is implemented, so a sidecar cannot be selected by
an implementation issue merely because this document mentions one.
- **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop
regression (the fixture family around `crates/filemeta/src/filemeta.rs`):
objects written by a new node must still be readable by old RustFS nodes and
@@ -179,82 +234,151 @@ The on-disk persistence of generation must not perturb the file format:
### Wire-encoding migration (JSON → msgpack) interaction
The internode RPC layer is mid-migration from JSON to msgpack binary, and generation-bearing fields must respect that migration window — this is not optional context, it changes how epoch is transported.
The internode RPC layer retains a JSON/msgpack rolling-compatibility window, and
generation-bearing fields must respect it.
- **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info` #4 alongside `file_info_bin` #7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`).
- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the `record_msgpack_json_fallback` metric reads zero fleet-wide and the convergence runbook is followed. If only the request flag is set, RustFS keeps dual-writing JSON compatibility fields. **Reuse this exact capability + metric-reads-zero model as the mixed-version gate for generation** rather than inventing a parallel handshake; the section above ("Capability negotiation") is layered on top of it, not instead of it.
- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the JSON-fallback metric reads zero fleet-wide and the convergence runbook is followed. Generation follows the same default-off, fleet-confirmed, metric-reads-zero rollout discipline, but a msgpack proof is not itself a generation capability proof.
- **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch.
- **Signature must bind a canonical form.** Because a field is transmitted as both JSON and msgpack and a peer may consume either, the body-digest binding in "RPC signature binding" above must be computed over a single canonical representation (the msgpack `_bin` bytes) — not over whichever copy happened to be decoded. Once the `generation` capability is negotiated for a request, a fenced / generation-bearing request must **reject the JSON fallback path** so a downgrade to the unsigned/loosely-bound JSON copy cannot bypass the epoch check.
- **Signature binds a canonical form.** `RenameDataRequest` now has a versioned,
injective canonical-body encoder that covers both compatibility fields and is
authenticated independently of whichever JSON/msgpack decoder branch a peer
consumes. A generation-capable strict request must reject missing or
mismatched canonical-body metadata; it must not silently downgrade to an
unauthenticated JSON twin.
### Proto evolution
New generation/epoch proto fields use **proto3 `optional`** (explicit presence).
A non-optional field is forbidden: an old coordinator talking to a new disk
decodes an absent field as `0`, which is indistinguishable from a real
`epoch == 0` and silently breaks the "stale epoch rejected" invariant during
upgrade.
No top-level proto field is required by the current metadata-map UUID. If a
future ordered epoch or explicit expected-generation is added to proto, it uses
**proto3 `optional`** (explicit presence). A non-optional scalar is forbidden:
an old coordinator talking to a new disk decodes absence as a plausible zero.
### Mixed-version gate — one direction
When the cluster-level generation capability is **not** negotiated on every
target disk, the behavior **falls back to current semantics** (existing lock +
`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at
`rustfs/src/app/object/get.rs`; full fanout for #1314). Fail-closed is
**only** an explicit administrator strict mode. Defaulting to fail-closed is
forbidden — it makes writes unavailable for the whole rolling-upgrade window.
When generation enforcement is not explicitly requested, or fleet confirmation
is absent, behavior falls back to current semantics. Fail-closed is reserved for
an explicit administrator-confirmed strict rollout.
Current object transaction fencing follows that direction:
- `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` and
`RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` both default false.
- With either flag absent, PUT/MPU does not persist or consume the transaction
UUID.
- With both flags enabled, failure to obtain or retain the live fleet proof
rejects the commit before rename.
This is an opt-in strict gate, not a negotiated generation capability. The
proof is currently borrowed from the remote-version-state writer rollout. It
proves current membership/process-epoch convergence for that feature, but does
not prove an epoch type, per-disk generation CAS support, or RPC strict-mode
convergence. Treating it as the final handshake is forbidden without an
explicit proof mapping for those properties.
## Capability negotiation
Generation enforcement is a **cluster-level handshake**, not a per-request
probe:
Generation enforcement requires one **live fleet proof**, not independent
boolean guesses in each consumer. The proof contract contains at least:
- A node advertises a `generation` capability once it can (a) mint quorum-durable
epochs, (b) compare epochs at all three disk-write points, and (c) verify the
body-digest-bound RPC signature.
- The authoritative writer enables hard enforcement for an object only when
**all** target disks in the set advertise the capability. Any missing
advertisement pins that commit to the mixed-version fallback above.
- The capability is surfaced through the existing runtime capability contract
surface (see [runtime-capability-contracts.md](runtime-capability-contracts.md)),
so consumers read one negotiated flag rather than each re-deriving support.
- Enforcement tracks the current membership rather than latching: it turns on
for a set only while every disk in that set advertises `generation`, and a
single old node rejoining drops the affected sets back to the mixed-version
fallback rather than failing closed. It never regresses the on-disk epoch —
falling back stops *comparing* new epochs, it does not lower any epoch already
persisted.
1. the selected authority version and comparison mode (ordered or exact-CAS),
2. the current membership/topology fingerprint and process epochs,
3. support for every required disk mutation point,
4. RPC signature/body/replay strict convergence, and
5. the on-disk encoding version (the current metadata-map UUID is version 1).
The authoritative writer enables enforcement only while every target disk in
the set is covered by a current proof. Membership change or an old node rejoin
revokes that proof. Revocation before commit fails an explicitly strict request;
when strict generation was never requested, the request remains on the legacy
path. Revocation never rewrites or lowers an already-persisted generation.
The existing fleet-proof machinery in `notification_sys` may be reused if its
authenticated statements are extended to cover the properties above. The
runtime capability contract may instead expose the proof. This document does
not choose the storage mechanism; it requires one token whose acquisition and
revalidation semantics are shared by all consumers.
## Implementation order
1. **#1312 first.** It defines the epoch, its persistence, the three fence
points, the RPC signature binding, and the encoding location. Everything
downstream depends on its epoch existing.
2. **#1313** (read lease) reuses the #1312 epoch as the lease generation and
must land before or alongside #1323.
3. **#1323** (old-dir GC) depends on #1313 leases being present and
cross-node-visible; its "no lease references old_dir" check has nothing to
query otherwise.
4. **#1318** (quota reservation) and **#1314** (prepared pool read) bind the
epoch independently; both gate on the same capability handshake.
Some original prerequisites have landed, but not in the originally proposed
form. Remaining work follows this order:
## Open design decisions (pin before implementation)
1. **Resolve the authority mode in #1326.** Select total order or opaque
exact-CAS, specify its atomic commit point, and audit PR #6077 against it.
Do not retrofit ordering semantics onto the existing random UUID.
2. **Define the generation fleet proof.** Map generation enablement to the RPC
signature/body/replay strict proofs delivered by #1327/#1541/#1542 and to
the selected per-disk comparison capability. Keep all strict defaults off
until fallback metrics converge.
3. **Implement #1313 generation-bound read leases.** The lease registry,
cross-node visibility, TTL, and crash recovery must exist before old-dir GC
can claim the full snapshot-lifetime guarantee. #1325 supplies the required
multi-node failure tests.
4. **Bind #1314 prepared reads.** A bundle binds the exact selected generation
within its source pool. Cross-pool ordering is forbidden until a common
authority is demonstrated. Validate rebalance and mixed-version fallback in
the #1325 multi-pool harness.
5. **Reconcile #1318 quota fencing.** Either bind reserve/settle/reconcile to
the selected object generation or document and prove that its independent
snapshot-lease fence is a separate arbitration domain that cannot settle a
different generation.
6. **Re-audit #1323 cleanup.** The existing UUID receipt remains valid crash
cleanup, but full closure against active readers requires the #1313 lease
check and the selected generation semantics.
This document fixes the transport, encoding, proto, and gate constraints, but it is not yet a complete implementable algorithm. The following must be decided and written down before any of the five consumers is coded (per the #1307 maintainer re-review, issuecomment-4992956256):
## Open design decisions (pin before contract closure)
- **Epoch type and total order.** The concrete token type and its total-order rule — a term+counter tuple, its persistence, and overflow behavior. Whether monotonicity is global or strictly per-object.
- **Never-regress on lock-service restart / minority recovery.** The epoch source must survive a lock-service restart or minority-quorum recovery without ever handing out an epoch lower than one already persisted on disk (an in-memory counter reset to zero is a fencing inversion). This is the same requirement as "Monotonicity persistence semantics" above, elevated to a hard, tested acceptance.
- **Complete xl.meta-writer coverage.** Every code path that writes xl.meta (commit rename, rollback delete/metadata restore, old-dir cleanup, heal, transition) must be enumerated and shown to compare or carry the epoch. A single unfenced writer voids the guarantee.
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure rollback at `io_primitives.rs:2646-2691` restores a metadata backup, not just a per-writer tmp delete, so a late rollback by writer A can overwrite writer B's committed xl.meta. Rollback must execute only when `stored_epoch == failed_writer_epoch`; a higher stored epoch must abort the rollback. Task panic / cancel / timeout at `io_primitives.rs:2602-2605` must be reaped into the coordinator's state machine, never bubble out via `?` and skip convergence.
The following decisions remain blockers for calling the contract implemented:
- **Authority mode.** Choose total order or opaque exact-CAS. If total order is
selected, define the type, per-object scope, persistence, overflow, and
never-regress restart/minority-recovery tests. If opaque identity is selected,
define the atomic expected-generation CAS and remove all ordered wording.
- **Complete xl.meta-writer coverage.** Enumerate commit rename, rollback
restore/delete, cleanup, heal, transition, restore, replication, and data
movement. Each path must compare/carry the selected generation or be proved
incapable of replacing the authoritative object identity.
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure
rollback in `rename_data` can restore backup metadata, not just remove a
writer-private temporary file. It must execute only when the stored generation
still matches the failed writer's expected generation. Panic, cancel, and
timeout outcomes must be reaped into coordinator convergence rather than skip
rollback through an early return.
- **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe.
- **Read-lease and GC crash recovery.** Lease registry location (local vs cross-node), TTL reclamation, and crash recovery for both the lease holder and the GC executor.
- **Quota reserve → commit → settle idempotency.** The cross-stage reconcile / idempotency story for #1318, including owner-crash reconciliation, so a reservation is neither lost nor double-counted.
- **Generation capability proof.** Decide whether to extend the current
authenticated fleet proof or the runtime capability contract. It must prove
authority version, mutation coverage, topology/process epoch, and RPC strict
convergence in one revalidatable token.
- **Read-lease and GC crash recovery.** Select the cross-node registry, TTL
reclamation, lease-holder crash behavior, and GC-executor recovery. The
current cleanup receipt equality check does not answer these questions.
- **Quota reserve → commit → settle binding.** The durable ledger's idempotency
exists, but its independent mutation tokens must be related to the selected
object generation with a concrete late-settle rejection test.
- **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited.
- **Hot-path cost is a blocking metric.** If per-PUT fencing grant, quota reserve, or cleanup journal adds a consensus write / fsync / centralized serialization point, it must be measured under 4KiB and high-concurrency hot-key / hot-bucket A/B as a blocking gate, not accepted by default.
- **Hot-path cost is a blocking metric.** Measure any additional consensus
write, fsync, fleet-proof lookup, lease operation, or centralized serialization
under 4 KiB and high-concurrency hot-key/hot-bucket A/B.
- **Test infrastructure.** #1325 still lacks the complete 4-node × 4-drive,
2-pool, directed network-fault, and large-object budget needed for restart,
mixed-version, and cross-node lease acceptance.
## Acceptance for this contract
- #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this unified
generation and no longer define their own token.
- The five constraints — transport signature, encoding, proto presence,
mixed-version gate direction, and capability negotiation — are pinned here
once; each implementation sub-issue follows them rather than re-deciding.
- [x] Architecture document exists and is linked from the architecture index.
- [x] Transport signature, encoding, proto presence, mixed-version direction,
and capability-proof requirements are defined once.
- [x] Current implementations are separated from target guarantees; a closed
child issue is not treated as proof of unified generation binding.
- [ ] Authority mode and atomic comparison semantics are selected and tested.
- [ ] #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this document and
use the selected authority terminology.
- [ ] #1313 and #1314 bind the selected generation and pass #1325 multi-node /
multi-pool failure tests.
- [ ] #1318 either binds reserve/settle to the selected generation or provides
an accepted proof that its separate fence cannot cross-settle generations.
- [ ] #1323 reconciliation checks both committed generation and active
generation-bound leases.
- [ ] Generation strict enablement is backed by one live proof that includes RPC
signature/body/replay strict convergence and per-disk comparison support.