mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-01 17:58:22 +00:00
docs(architecture): reconcile generation contract (#6901)
Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
@@ -30,73 +30,120 @@ lifetime**. Left independent, they diverge and punch through one another:
|
||||
different monotonic sources, cannot be compared — a late commit fenced on one
|
||||
plane can still settle quota on the other.
|
||||
|
||||
The fix is a single authority with one monotonic source, one persistence
|
||||
The fix is a single authority with one selected comparison rule, one persistence
|
||||
semantics, and one transport binding, that every consumer references rather than
|
||||
re-derives.
|
||||
|
||||
## The authority (single source)
|
||||
## Target authority and the current bounded token
|
||||
|
||||
**The per-object fencing epoch defined by #1312 is the sole generation
|
||||
authority.** No other monotonic counter, timestamp, or random token may stand in
|
||||
for generation.
|
||||
The target contract still requires **one per-object commit identity** consumed
|
||||
by commit fencing, read leases, cleanup, prepared reads, and quota settlement.
|
||||
No consumer may mint a second value and call it the same generation.
|
||||
|
||||
- The distributed lock grant returns a monotonic `epoch` for the object key.
|
||||
Acquiring the object write-lock is the only way to mint a new generation.
|
||||
- The epoch travels down the authoritative commit path (with
|
||||
`RenameDataRequest` / the local `DiskAPI` call) and is compared at each disk's
|
||||
atomic `xl.meta` commit point, rejecting stale epochs. It adds no extra
|
||||
network round trip (#1312 implementation clause 2).
|
||||
- Every consumer in the table below **binds** this epoch. None defines its own.
|
||||
The concrete ordering semantics are not settled, however. The original #1326
|
||||
proposal requires a total-ordered, monotonic lock-grant epoch. Current main does
|
||||
not implement that proposal. PR #6077 instead implements an opaque transaction
|
||||
identity:
|
||||
|
||||
### Monotonicity persistence semantics
|
||||
- `assign_object_transaction_epoch` mints a random non-nil UUID for PUT and
|
||||
CompleteMultipartUpload when the object-transaction gate is active.
|
||||
- The UUID is written through `FileInfo::set_object_transaction_epoch` into the
|
||||
dual internal metadata map.
|
||||
- The coordinator reads the current UUID (or `Absent`) and revalidates exact
|
||||
equality immediately before `rename_data`.
|
||||
- Old-data cleanup receipts carry the committed UUID and reconciliation deletes
|
||||
only when the receipt UUID still equals the current object UUID.
|
||||
|
||||
The epoch must be **monotonic across lock-plane restart and failover**
|
||||
(#1312 B4). Today the distributed lock entry is in-memory only
|
||||
(`crates/lock/src/distributed_lock.rs` has no persistence path), so a lock-service
|
||||
restart resets the counter to zero: a new writer draws epoch 1 while disks have
|
||||
already observed epoch 100, producing either a permanent write rejection or a
|
||||
fence *inversion*. To prevent this, the epoch must be one of:
|
||||
This is a useful **equality-CAS fence and cleanup identity**. It is not a
|
||||
monotonic epoch, is not minted by the distributed lock grant, and is not
|
||||
compared atomically at each disk's `xl.meta` commit point. Until the decision
|
||||
below is made, documents and issue checklists must call it the *object
|
||||
transaction UUID* rather than use it as proof that the target generation
|
||||
authority exists.
|
||||
|
||||
1. **Quorum-persisted** before it is handed to a writer, or
|
||||
2. **Derived from a durable monotonic source** — a `(term, counter)` pair where
|
||||
`term` advances on every lock-service leadership change and is itself durable,
|
||||
so the composite never regresses even when `counter` resets.
|
||||
### Ordering decision required
|
||||
|
||||
The comparison at the disk commit point is on the full composite; a lower
|
||||
`(term, counter)` is always rejected.
|
||||
Before #1313, #1314, or a unified quota binding can consume the authority, one
|
||||
of these contracts must be selected and tested:
|
||||
|
||||
1. **Total-ordered fencing epoch.** A lock grant returns a durable per-object
|
||||
`(term, counter)` (or another specified total-order type). Every disk rejects
|
||||
a lower epoch at the atomic metadata commit point. The value never regresses
|
||||
across lock-plane restart, failover, or minority recovery.
|
||||
2. **Opaque commit-generation identity.** Consumers compare only exact identity;
|
||||
no `<` / `>` semantics are permitted. The authoritative commit must perform
|
||||
an atomic expected-generation CAS, and all lease, cleanup, prepared-read, and
|
||||
quota contracts must be rewritten in terms of “references this exact
|
||||
generation,” not “lower/newer generation.”
|
||||
|
||||
The current UUID implementation proves neither a durable total order nor a
|
||||
per-disk atomic expected-generation CAS, so it does not by itself decide between
|
||||
these options.
|
||||
|
||||
### Persistence semantics if total order is selected
|
||||
|
||||
A total-ordered epoch must be **monotonic across lock-plane restart and
|
||||
failover**. The distributed lock entry remains in-memory; deriving a counter
|
||||
from that entry alone would reset it after restart. The chosen source therefore
|
||||
must be either quorum-persisted before grant or derived from a durable term whose
|
||||
full `(term, counter)` comparison cannot regress. This requirement does not
|
||||
apply to an opaque UUID as an ordering rule; the opaque alternative instead
|
||||
requires atomic expected-identity comparison and durable crash recovery.
|
||||
|
||||
## Consumer binding contracts
|
||||
|
||||
### Current implementation snapshot (2026-08-31, main@9ee7b1221)
|
||||
|
||||
This table separates code that exists on current main from the target contract.
|
||||
Closing an implementation issue does not imply that its token is already the
|
||||
unified authority.
|
||||
|
||||
| Surface | Current main | Gap against this contract |
|
||||
|---|---|---|
|
||||
| PUT / CompleteMultipartUpload (#1312, PR #6077) | Owned commit tasks retain the relevant guards; an opt-in gate persists a random object transaction UUID and performs a quorum metadata equality recheck before rename | no lock-grant monotonic source; no per-disk atomic epoch/CAS comparison; the live proof is the reused remote-version-state fleet proof, not a dedicated generation capability |
|
||||
| Old-data cleanup (#1323, PR #6077) | JSON receipt carries transaction UUID, old dir, and committed dir; reconciliation is gated and requires UUID equality | no generation-bound read lease is consulted, so this is crash cleanup fencing rather than the full #1313/#1323 lease lifetime contract |
|
||||
| Read lease (#1313) | short-term streaming/multipart path holds the namespace read lock through EOF/drop; deterministic part-boundary coverage is tracked by PR #6887 | no cross-node generation-bound lease registry, TTL reclamation, or crash recovery |
|
||||
| Prepared pool read (#1314) | PR #6889 tracks a pool-local prepared identity and fails closed/refetches when pool state changes | not merged on this snapshot; pool-local identity is not a cross-pool generation authority; black-box mixed-version/rebalance coverage remains open |
|
||||
| Quota reservation (#1318) | durable per-bucket ledger plus independent snapshot-lease mutation-fence tokens; issue closed after PR #6058 | reservation and settle are not bound to the object transaction UUID; the independent fence must be reconciled with the selected authority or explicitly proven to be a separate, non-generation arbitration domain |
|
||||
| Internode integrity (#1327, #1541, #1542) | v2/v3 HMAC binds audience, exact method, timestamp, nonce, canonical body digest, and receiver boot epoch; body-bound RPC policy has exact-set coverage | signature/body/replay strict switches remain default-off rollout gates; generation enforcement cannot treat an unrelated fleet-version proof as proof that these strict contracts converged |
|
||||
|
||||
| Consumer | How it binds generation | Key invariant |
|
||||
|---|---|---|
|
||||
| #1312 commit fence | epoch compared at three disk-write points — `rename`, rollback `delete`, and `commit_rename_data_dir` cleanup | stale epoch rejected on **all** disks; an already-ACK'd write is never rolled back |
|
||||
| #1313 read lease | lease binds the generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
|
||||
| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms no lease referencing a lower generation still points at it | `old_dir != committed_dir`; a still-referenced directory is never deleted |
|
||||
| #1312 commit fence | selected generation is checked at `rename`, rollback restore/delete, and cleanup mutation points using the chosen ordered or exact-CAS rule | a stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back |
|
||||
| #1313 read lease | lease binds the exact generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
|
||||
| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms that no lease for the generation owning that directory remains | `old_dir != committed_dir`; a still-referenced directory is never deleted |
|
||||
| #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout |
|
||||
| #1318 quota reservation | reservation / settle token binds the object generation | a late commit holding an old-generation token cannot settle a newer generation |
|
||||
| #1318 quota reservation | reservation / settle record binds the exact object generation (and an ordered epoch too, if that option is selected) | a late commit cannot settle quota for a different committed generation |
|
||||
|
||||
### Fence coverage is three disk-write points, not one (#1312 B2)
|
||||
|
||||
Comparing the epoch at the `rename` commit point alone is insufficient. The
|
||||
Checking the generation only before the `rename` fanout is insufficient. The
|
||||
authoritative commit sequence is `tmp sync → data-dir rename → xl.meta commit →
|
||||
directory sync` in `crates/ecstore/src/disk/local.rs`, and there are two further
|
||||
detachable disk-write points in
|
||||
`crates/ecstore/src/set_disk/core/io_primitives.rs`:
|
||||
|
||||
- **Rollback delete** — on quorum failure each disk runs
|
||||
`delete_version(undo_write=true)`. A fenced old writer's rollback must also
|
||||
compare epoch, otherwise it deletes the winner's already-committed version.
|
||||
- **Rollback restore/delete** — on quorum failure each disk can restore backup
|
||||
metadata or delete the failed version. A stale writer's rollback must compare
|
||||
the expected generation, otherwise it can overwrite or delete the winner's
|
||||
already-committed metadata.
|
||||
- **`commit_rename_data_dir`** — a cancel-then-detach disk-write point; the
|
||||
coordinator's "reap all child tasks" must explicitly include it so a cancelled
|
||||
writer cannot bypass fence/lease and keep deleting directories.
|
||||
|
||||
If the epoch is validated only at the `xl.meta` commit point, a fenced writer
|
||||
If generation is validated only after data-dir rename, a fenced writer
|
||||
may already have renamed its data-dir into the object path, leaving a staged
|
||||
orphan. Either move the fence ahead of the data-dir rename, or declare that
|
||||
orphan an acceptable residue accounted for by GC metrics — the white-box
|
||||
acceptance "no background disk write after release" must be rewritten
|
||||
accordingly.
|
||||
|
||||
Current PR #6077 performs a quorum metadata equality recheck before rename and
|
||||
reaps owned commit work. That closes important cancellation windows, but it is
|
||||
not evidence that every disk mutation above performs the selected generation
|
||||
comparison atomically. The writer inventory and per-point CAS/ordering proof
|
||||
remain acceptance work for #1326 even though #1312 is closed.
|
||||
|
||||
### Post-commit convergence is orthogonal to the fence (#1321)
|
||||
|
||||
The same `SetDisks::rename_data` path already returns a post-commit
|
||||
@@ -125,36 +172,40 @@ internode RPC bodies. Every such flow must be signature-bound.
|
||||
|
||||
### RPC signature binding (#1312 B3, #1313, #1318)
|
||||
|
||||
**Requirement.** The RPC body digest carrying a generation/epoch/token must be
|
||||
folded into the RPC HMAC, binding `method + object key + generation`, and the
|
||||
request must carry a nonce / one-shot identifier inside the 300s replay window.
|
||||
The nonce is only meaningful if the **receiver enforces it**: each disk keeps a
|
||||
bounded seen-nonce cache covering the 300s freshness window and rejects any
|
||||
request whose nonce was already observed. A nonce that is merely transmitted but
|
||||
not checked provides no replay protection.
|
||||
**Requirement.** The canonical body carrying a generation or derived token must
|
||||
be folded into the internode HMAC. The authenticated scope binds the target
|
||||
audience, exact service/method, timestamp, nonce, canonical body digest, and
|
||||
receiver replay epoch. The receiver must consume the nonce in a bounded replay
|
||||
cache; transmitting a nonce without receiver-side consumption is not replay
|
||||
protection.
|
||||
|
||||
This generalizes the existing `walk_dir` pattern: `walk_dir` computes a
|
||||
`Sha256` of the request body and places it in the signed URL query as
|
||||
`walk_dir_body_sha256`
|
||||
(`crates/ecstore/src/cluster/rpc/internode_data_transport.rs:187`), so the body
|
||||
digest is transitively covered by the URL signature. New generation-bearing RPCs
|
||||
adopt the same `*_body_sha256` mechanism.
|
||||
**Current substrate (verified on main).** The original legacy-only description
|
||||
is obsolete:
|
||||
|
||||
**Current gap (verified).** The internode HMAC covers only
|
||||
`{path_and_query}|{method}|{timestamp}`
|
||||
(`signature_payload`, `crates/ecstore/src/cluster/rpc/http_auth.rs:75-83`). It
|
||||
binds neither the request body nor a nonce, and the 300s freshness window has no
|
||||
one-shot guard. Without the binding above:
|
||||
- RPC v2 binds target audience, exact method, POST, timestamp, nonce, and body
|
||||
digest.
|
||||
- Body-bound policy covers mutating disk RPCs including `RenameData`; its
|
||||
versioned canonical body includes every `RenameDataRequest` field, so the
|
||||
`FileInfo` metadata map carrying the transaction UUID is authenticated.
|
||||
- PR #5425 extended canonical-body enforcement to implemented non-disk mutating
|
||||
unary RPCs and added an exact policy/handler coverage partition.
|
||||
- PR #5455 added the receiver boot epoch and rotating replay scope so signatures
|
||||
captured before a receiver restart are rejected after capability convergence.
|
||||
|
||||
- An on-path or replaying attacker can inject a high epoch (e.g. `u32::MAX`) and
|
||||
**permanently fence out** a key's legitimate writes — monotonicity only
|
||||
rejects *low/old* epochs, never a forged-high one.
|
||||
- A captured lease/reservation token can be replayed within 300s to block
|
||||
old-dir GC (storage-exhaustion DoS) or to double-reserve / prematurely settle
|
||||
quota.
|
||||
The rollout switches
|
||||
`RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`,
|
||||
`RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, and
|
||||
`RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` remain default-off for rolling
|
||||
compatibility. The compatibility register and fallback/overflow metrics govern
|
||||
their fleet convergence. Therefore a generation capability may claim strong
|
||||
transport binding only when the relevant strict modes have converged; the
|
||||
object-transaction gate's current remote-version-state fleet proof is not, by
|
||||
itself, proof of RPC signature/body/replay strictness.
|
||||
|
||||
Acceptance for each consumer must include: "a replayed old signature to a
|
||||
different method, and a forged-high-epoch request, are both rejected."
|
||||
Acceptance for each generation consumer includes method substitution, canonical
|
||||
body tamper, nonce replay, receiver restart, and stripped-strict-metadata
|
||||
negative tests. Generation rollout must also record which strict-mode evidence
|
||||
authorized enforcement.
|
||||
|
||||
### Encoding contract (#1312 B1)
|
||||
|
||||
@@ -167,11 +218,15 @@ The on-disk persistence of generation must not perturb the file format:
|
||||
`xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a
|
||||
total read failure, not a graceful downgrade.
|
||||
- **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged.
|
||||
- **Where it may live.** Only inside a version's internal metadata **map**
|
||||
(MinIO skips unknown internal keys and the map encoding is extensible) or in a
|
||||
per-disk sidecar outside `xl.meta`. If it goes in the metadata map, it must
|
||||
obey the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`, see
|
||||
AGENTS.md "Cross-Cutting Domain Invariants").
|
||||
- **Where it lives today.** The object transaction UUID uses the version's
|
||||
internal metadata map under the dual-key contract
|
||||
(`x-rustfs-internal-*` / `x-minio-internal-*`) via
|
||||
`set_object_transaction_epoch`. Missing, malformed, nil, or conflicting dual
|
||||
values fail closed when fencing is active.
|
||||
- **Sidecars are not an equivalent alternative.** A future sidecar is admissible
|
||||
only if it commits atomically with `xl.meta` and has a specified crash-recovery
|
||||
protocol. No such protocol is implemented, so a sidecar cannot be selected by
|
||||
an implementation issue merely because this document mentions one.
|
||||
- **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop
|
||||
regression (the fixture family around `crates/filemeta/src/filemeta.rs`):
|
||||
objects written by a new node must still be readable by old RustFS nodes and
|
||||
@@ -179,82 +234,151 @@ The on-disk persistence of generation must not perturb the file format:
|
||||
|
||||
### Wire-encoding migration (JSON → msgpack) interaction
|
||||
|
||||
The internode RPC layer is mid-migration from JSON to msgpack binary, and generation-bearing fields must respect that migration window — this is not optional context, it changes how epoch is transported.
|
||||
The internode RPC layer retains a JSON/msgpack rolling-compatibility window, and
|
||||
generation-bearing fields must respect it.
|
||||
|
||||
- **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info` #4 alongside `file_info_bin` #7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`).
|
||||
- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the `record_msgpack_json_fallback` metric reads zero fleet-wide and the convergence runbook is followed. If only the request flag is set, RustFS keeps dual-writing JSON compatibility fields. **Reuse this exact capability + metric-reads-zero model as the mixed-version gate for generation** rather than inventing a parallel handshake; the section above ("Capability negotiation") is layered on top of it, not instead of it.
|
||||
- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the JSON-fallback metric reads zero fleet-wide and the convergence runbook is followed. Generation follows the same default-off, fleet-confirmed, metric-reads-zero rollout discipline, but a msgpack proof is not itself a generation capability proof.
|
||||
- **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch.
|
||||
- **Signature must bind a canonical form.** Because a field is transmitted as both JSON and msgpack and a peer may consume either, the body-digest binding in "RPC signature binding" above must be computed over a single canonical representation (the msgpack `_bin` bytes) — not over whichever copy happened to be decoded. Once the `generation` capability is negotiated for a request, a fenced / generation-bearing request must **reject the JSON fallback path** so a downgrade to the unsigned/loosely-bound JSON copy cannot bypass the epoch check.
|
||||
- **Signature binds a canonical form.** `RenameDataRequest` now has a versioned,
|
||||
injective canonical-body encoder that covers both compatibility fields and is
|
||||
authenticated independently of whichever JSON/msgpack decoder branch a peer
|
||||
consumes. A generation-capable strict request must reject missing or
|
||||
mismatched canonical-body metadata; it must not silently downgrade to an
|
||||
unauthenticated JSON twin.
|
||||
|
||||
### Proto evolution
|
||||
|
||||
New generation/epoch proto fields use **proto3 `optional`** (explicit presence).
|
||||
A non-optional field is forbidden: an old coordinator talking to a new disk
|
||||
decodes an absent field as `0`, which is indistinguishable from a real
|
||||
`epoch == 0` and silently breaks the "stale epoch rejected" invariant during
|
||||
upgrade.
|
||||
No top-level proto field is required by the current metadata-map UUID. If a
|
||||
future ordered epoch or explicit expected-generation is added to proto, it uses
|
||||
**proto3 `optional`** (explicit presence). A non-optional scalar is forbidden:
|
||||
an old coordinator talking to a new disk decodes absence as a plausible zero.
|
||||
|
||||
### Mixed-version gate — one direction
|
||||
|
||||
When the cluster-level generation capability is **not** negotiated on every
|
||||
target disk, the behavior **falls back to current semantics** (existing lock +
|
||||
`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at
|
||||
`rustfs/src/app/object/get.rs`; full fanout for #1314). Fail-closed is
|
||||
**only** an explicit administrator strict mode. Defaulting to fail-closed is
|
||||
forbidden — it makes writes unavailable for the whole rolling-upgrade window.
|
||||
When generation enforcement is not explicitly requested, or fleet confirmation
|
||||
is absent, behavior falls back to current semantics. Fail-closed is reserved for
|
||||
an explicit administrator-confirmed strict rollout.
|
||||
|
||||
Current object transaction fencing follows that direction:
|
||||
|
||||
- `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` and
|
||||
`RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` both default false.
|
||||
- With either flag absent, PUT/MPU does not persist or consume the transaction
|
||||
UUID.
|
||||
- With both flags enabled, failure to obtain or retain the live fleet proof
|
||||
rejects the commit before rename.
|
||||
|
||||
This is an opt-in strict gate, not a negotiated generation capability. The
|
||||
proof is currently borrowed from the remote-version-state writer rollout. It
|
||||
proves current membership/process-epoch convergence for that feature, but does
|
||||
not prove an epoch type, per-disk generation CAS support, or RPC strict-mode
|
||||
convergence. Treating it as the final handshake is forbidden without an
|
||||
explicit proof mapping for those properties.
|
||||
|
||||
## Capability negotiation
|
||||
|
||||
Generation enforcement is a **cluster-level handshake**, not a per-request
|
||||
probe:
|
||||
Generation enforcement requires one **live fleet proof**, not independent
|
||||
boolean guesses in each consumer. The proof contract contains at least:
|
||||
|
||||
- A node advertises a `generation` capability once it can (a) mint quorum-durable
|
||||
epochs, (b) compare epochs at all three disk-write points, and (c) verify the
|
||||
body-digest-bound RPC signature.
|
||||
- The authoritative writer enables hard enforcement for an object only when
|
||||
**all** target disks in the set advertise the capability. Any missing
|
||||
advertisement pins that commit to the mixed-version fallback above.
|
||||
- The capability is surfaced through the existing runtime capability contract
|
||||
surface (see [runtime-capability-contracts.md](runtime-capability-contracts.md)),
|
||||
so consumers read one negotiated flag rather than each re-deriving support.
|
||||
- Enforcement tracks the current membership rather than latching: it turns on
|
||||
for a set only while every disk in that set advertises `generation`, and a
|
||||
single old node rejoining drops the affected sets back to the mixed-version
|
||||
fallback rather than failing closed. It never regresses the on-disk epoch —
|
||||
falling back stops *comparing* new epochs, it does not lower any epoch already
|
||||
persisted.
|
||||
1. the selected authority version and comparison mode (ordered or exact-CAS),
|
||||
2. the current membership/topology fingerprint and process epochs,
|
||||
3. support for every required disk mutation point,
|
||||
4. RPC signature/body/replay strict convergence, and
|
||||
5. the on-disk encoding version (the current metadata-map UUID is version 1).
|
||||
|
||||
The authoritative writer enables enforcement only while every target disk in
|
||||
the set is covered by a current proof. Membership change or an old node rejoin
|
||||
revokes that proof. Revocation before commit fails an explicitly strict request;
|
||||
when strict generation was never requested, the request remains on the legacy
|
||||
path. Revocation never rewrites or lowers an already-persisted generation.
|
||||
|
||||
The existing fleet-proof machinery in `notification_sys` may be reused if its
|
||||
authenticated statements are extended to cover the properties above. The
|
||||
runtime capability contract may instead expose the proof. This document does
|
||||
not choose the storage mechanism; it requires one token whose acquisition and
|
||||
revalidation semantics are shared by all consumers.
|
||||
|
||||
## Implementation order
|
||||
|
||||
1. **#1312 first.** It defines the epoch, its persistence, the three fence
|
||||
points, the RPC signature binding, and the encoding location. Everything
|
||||
downstream depends on its epoch existing.
|
||||
2. **#1313** (read lease) reuses the #1312 epoch as the lease generation and
|
||||
must land before or alongside #1323.
|
||||
3. **#1323** (old-dir GC) depends on #1313 leases being present and
|
||||
cross-node-visible; its "no lease references old_dir" check has nothing to
|
||||
query otherwise.
|
||||
4. **#1318** (quota reservation) and **#1314** (prepared pool read) bind the
|
||||
epoch independently; both gate on the same capability handshake.
|
||||
Some original prerequisites have landed, but not in the originally proposed
|
||||
form. Remaining work follows this order:
|
||||
|
||||
## Open design decisions (pin before implementation)
|
||||
1. **Resolve the authority mode in #1326.** Select total order or opaque
|
||||
exact-CAS, specify its atomic commit point, and audit PR #6077 against it.
|
||||
Do not retrofit ordering semantics onto the existing random UUID.
|
||||
2. **Define the generation fleet proof.** Map generation enablement to the RPC
|
||||
signature/body/replay strict proofs delivered by #1327/#1541/#1542 and to
|
||||
the selected per-disk comparison capability. Keep all strict defaults off
|
||||
until fallback metrics converge.
|
||||
3. **Implement #1313 generation-bound read leases.** The lease registry,
|
||||
cross-node visibility, TTL, and crash recovery must exist before old-dir GC
|
||||
can claim the full snapshot-lifetime guarantee. #1325 supplies the required
|
||||
multi-node failure tests.
|
||||
4. **Bind #1314 prepared reads.** A bundle binds the exact selected generation
|
||||
within its source pool. Cross-pool ordering is forbidden until a common
|
||||
authority is demonstrated. Validate rebalance and mixed-version fallback in
|
||||
the #1325 multi-pool harness.
|
||||
5. **Reconcile #1318 quota fencing.** Either bind reserve/settle/reconcile to
|
||||
the selected object generation or document and prove that its independent
|
||||
snapshot-lease fence is a separate arbitration domain that cannot settle a
|
||||
different generation.
|
||||
6. **Re-audit #1323 cleanup.** The existing UUID receipt remains valid crash
|
||||
cleanup, but full closure against active readers requires the #1313 lease
|
||||
check and the selected generation semantics.
|
||||
|
||||
This document fixes the transport, encoding, proto, and gate constraints, but it is not yet a complete implementable algorithm. The following must be decided and written down before any of the five consumers is coded (per the #1307 maintainer re-review, issuecomment-4992956256):
|
||||
## Open design decisions (pin before contract closure)
|
||||
|
||||
- **Epoch type and total order.** The concrete token type and its total-order rule — a term+counter tuple, its persistence, and overflow behavior. Whether monotonicity is global or strictly per-object.
|
||||
- **Never-regress on lock-service restart / minority recovery.** The epoch source must survive a lock-service restart or minority-quorum recovery without ever handing out an epoch lower than one already persisted on disk (an in-memory counter reset to zero is a fencing inversion). This is the same requirement as "Monotonicity persistence semantics" above, elevated to a hard, tested acceptance.
|
||||
- **Complete xl.meta-writer coverage.** Every code path that writes xl.meta (commit rename, rollback delete/metadata restore, old-dir cleanup, heal, transition) must be enumerated and shown to compare or carry the epoch. A single unfenced writer voids the guarantee.
|
||||
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure rollback at `io_primitives.rs:2646-2691` restores a metadata backup, not just a per-writer tmp delete, so a late rollback by writer A can overwrite writer B's committed xl.meta. Rollback must execute only when `stored_epoch == failed_writer_epoch`; a higher stored epoch must abort the rollback. Task panic / cancel / timeout at `io_primitives.rs:2602-2605` must be reaped into the coordinator's state machine, never bubble out via `?` and skip convergence.
|
||||
The following decisions remain blockers for calling the contract implemented:
|
||||
|
||||
- **Authority mode.** Choose total order or opaque exact-CAS. If total order is
|
||||
selected, define the type, per-object scope, persistence, overflow, and
|
||||
never-regress restart/minority-recovery tests. If opaque identity is selected,
|
||||
define the atomic expected-generation CAS and remove all ordered wording.
|
||||
- **Complete xl.meta-writer coverage.** Enumerate commit rename, rollback
|
||||
restore/delete, cleanup, heal, transition, restore, replication, and data
|
||||
movement. Each path must compare/carry the selected generation or be proved
|
||||
incapable of replacing the authoritative object identity.
|
||||
- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure
|
||||
rollback in `rename_data` can restore backup metadata, not just remove a
|
||||
writer-private temporary file. It must execute only when the stored generation
|
||||
still matches the failed writer's expected generation. Panic, cancel, and
|
||||
timeout outcomes must be reaped into coordinator convergence rather than skip
|
||||
rollback through an early return.
|
||||
- **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe.
|
||||
- **Read-lease and GC crash recovery.** Lease registry location (local vs cross-node), TTL reclamation, and crash recovery for both the lease holder and the GC executor.
|
||||
- **Quota reserve → commit → settle idempotency.** The cross-stage reconcile / idempotency story for #1318, including owner-crash reconciliation, so a reservation is neither lost nor double-counted.
|
||||
- **Generation capability proof.** Decide whether to extend the current
|
||||
authenticated fleet proof or the runtime capability contract. It must prove
|
||||
authority version, mutation coverage, topology/process epoch, and RPC strict
|
||||
convergence in one revalidatable token.
|
||||
- **Read-lease and GC crash recovery.** Select the cross-node registry, TTL
|
||||
reclamation, lease-holder crash behavior, and GC-executor recovery. The
|
||||
current cleanup receipt equality check does not answer these questions.
|
||||
- **Quota reserve → commit → settle binding.** The durable ledger's idempotency
|
||||
exists, but its independent mutation tokens must be related to the selected
|
||||
object generation with a concrete late-settle rejection test.
|
||||
- **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited.
|
||||
- **Hot-path cost is a blocking metric.** If per-PUT fencing grant, quota reserve, or cleanup journal adds a consensus write / fsync / centralized serialization point, it must be measured under 4KiB and high-concurrency hot-key / hot-bucket A/B as a blocking gate, not accepted by default.
|
||||
- **Hot-path cost is a blocking metric.** Measure any additional consensus
|
||||
write, fsync, fleet-proof lookup, lease operation, or centralized serialization
|
||||
under 4 KiB and high-concurrency hot-key/hot-bucket A/B.
|
||||
- **Test infrastructure.** #1325 still lacks the complete 4-node × 4-drive,
|
||||
2-pool, directed network-fault, and large-object budget needed for restart,
|
||||
mixed-version, and cross-node lease acceptance.
|
||||
|
||||
## Acceptance for this contract
|
||||
|
||||
- #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this unified
|
||||
generation and no longer define their own token.
|
||||
- The five constraints — transport signature, encoding, proto presence,
|
||||
mixed-version gate direction, and capability negotiation — are pinned here
|
||||
once; each implementation sub-issue follows them rather than re-deciding.
|
||||
- [x] Architecture document exists and is linked from the architecture index.
|
||||
- [x] Transport signature, encoding, proto presence, mixed-version direction,
|
||||
and capability-proof requirements are defined once.
|
||||
- [x] Current implementations are separated from target guarantees; a closed
|
||||
child issue is not treated as proof of unified generation binding.
|
||||
- [ ] Authority mode and atomic comparison semantics are selected and tested.
|
||||
- [ ] #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this document and
|
||||
use the selected authority terminology.
|
||||
- [ ] #1313 and #1314 bind the selected generation and pass #1325 multi-node /
|
||||
multi-pool failure tests.
|
||||
- [ ] #1318 either binds reserve/settle to the selected generation or provides
|
||||
an accepted proof that its separate fence cannot cross-settle generations.
|
||||
- [ ] #1323 reconciliation checks both committed generation and active
|
||||
generation-bound leases.
|
||||
- [ ] Generation strict enablement is backed by one live proof that includes RPC
|
||||
signature/body/replay strict convergence and per-disk comparison support.
|
||||
|
||||
Reference in New Issue
Block a user