38 KiB
Object Generation Authority And Recovery Contract
Use this when: changing object commit fencing, rollback, old-directory cleanup, prepared reads, quota settlement, or the metadata and RPC fields used by those operations.
Source of truth: crates/ecstore/src/set_disk/ops/object.rs (assign_object_transaction_epoch, verify_object_transaction_epoch_fence); crates/ecstore/src/set_disk/core/io_primitives.rs (rename_data_owned_with_fence, commit_rename_data_dir); crates/ecstore/src/disk/local.rs (rename_data, write_all_meta); crates/lock/src/distributed_lock.rs (DistributedLockGuard, LockLostSignal). The implementation boundary below distinguishes existing behavior from the selected design.
Decision And Implementation Boundary
The selected minimum authority is a durable, ordered per-object decision protocol attached to the existing namespace-lock participant group. The object transaction UUID remains an opaque operation/idempotency identifier. It is not an ordered lock epoch. Extending the existing lock group requires durable promises, accepted values, quorum decisions, and recovery; adding a counter to today's lock response is insufficient.
An independent service holding every object's full manifest is not selected. It would add a new routing, membership, availability, and metadata ownership system and require a wider read/write migration. The selected protocol stores the current decision and recoverable outstanding successor with the existing lock participants; object payload and prepared metadata remain on the existing storage disks. This is still new consensus and persistence work, not a small RenameData patch.
Implementation status: this document does not implement or claim distributed generation authority. Existing fencing remains an opt-in coordinator equality recheck. rustfs/backlog#2251 cannot be completed by forwarding the UUID to disks and adding local CAS. Its implementation must be split at the protocol boundaries in Required Implementation Boundaries, with the availability and rollout changes reviewed before strict activation. The two original requirements “commit with a quorum while a disk is unreachable” and “every disk immediately rejects every older request” cannot both hold; the precise target below preserves quorum availability.
Related contracts remain authoritative for their domains: erasure-coding.md defines data durability and voting, heal-concurrency-model.md defines namespace-lock scope, placement-repair-invariants.md defines placement and repair admission, and minio-file-format-compat.md defines format interoperability.
Current Guarantee And Counterexamples
assign_object_transaction_epoch generates a random UUID for gated PUT and CompleteMultipartUpload. FileInfo::set_object_transaction_epoch in crates/filemeta/src/fileinfo.rs stores it under both internal metadata prefixes. verify_object_transaction_epoch_fence re-reads quorum metadata before the rename fanout, outside the eventual per-disk mutation critical section. ObjectTransactionEpochFence::Absent currently covers both an absent object and existing metadata without a UUID. Cleanup receipts compare UUID equality. None of these operations is a durable distributed CAS.
The lock implementation already bounds lease validity. LockLostSignal::is_lost includes the conservative deadline; DistributedLockGuard::run_heartbeat retains prior deadlines after transient RPC failure and reports loss when refresh quorum is no longer valid. LocalClient in crates/lock/src/client/local.rs keeps LocalGuardEntry in an in-memory map; LockResponse in crates/lock/src/types.rs contains no persisted ballot or accepted object decision. Restarting a lock participant therefore cannot supply the durable order required here.
These schedules disprove a local UUID-CAS replacement, even assuming a perfect local mutex and atomic metadata replacement. They are protocol counterexamples, not claims that a multi-node fault test has already been run.
| Schedule | Result and implication |
|---|---|
| Four disks start at X; A captures expected X and stalls. B commits B on d1–d3, satisfying W=3. d4 has not heard from B. A reaches d4 with expected X. | d4's exact-CAS accepts A. A durable local highest_ballot also accepts if d4 never received B's ballot. Neither mechanism proves rejection on every disk after a quorum commit. |
| A changes d1,d2 from X to A; after lock loss B changes d3,d4 from X to B. Each refuses the other's disks because expected X no longer matches. | Neither reaches W=3, even with all disks now reachable. Equality-CAS alone has no rule for choosing recovery, retaining uncertain work, or safely retiring it. This is a recovery/liveness counterexample, not proof of two successful intersecting write quorums. |
| A reaches W=3; its reply is lost. A's coordinator restarts and sees a partial or changed disk view. A's rollback runs after B has replaced A. | Timeout is not evidence of abort. Restoring A's backup can erase B unless rollback names its own committed effect and consults a durable decision. |
| A checks a lease/UUID; B later commits; A's blocking syscall resumes. | Another coordinator-side check, cancellation token, or process-local mutex cannot establish an atomic cross-node order. The mutation itself must consume the protocol state. |
A lower-ballot write on an isolated stale disk cannot become the authoritative object. A disk that has applied B must never replace B with A. An uncontacted disk may retain older committed materialization until recovery; it must not vote that state as a newer decision or authorize cleanup. Requiring all disks to learn B before acknowledging it would change W to N or require successful isolation of every unreachable disk. That availability change is rejected for the selected design and must not be hidden inside E03's tests.
Authority, Identity, And State
The authority key is (bucket incarnation, bucket, object key), covering the whole object version set. It is not the S3 version ID. Deleting a noncurrent version, updating tags, or recording replication status can change the authoritative revision while the current S3 version remains the same. The bucket incarnation prevents reuse after bucket deletion/recreation.
The protocol has separate typed values:
Ballot = (configuration epoch, counter, durable proposer ID), compared lexicographically only within the specified authority configuration. The proposer persists a counter before use and raises it above every observed promise. Restart never resets it; exhaustion is an error. A different process boot gets a new transport epoch, not permission to reuse a ballot for different bytes.Generation = (object revision, operation UUID), allocated by a chosen successor decision. Revisions increase from the committed predecessor; UUIDs are compared for equality only. Never infer generation order from modification time, version ID, or UUID bytes.DecisionValue = (authority key, predecessor generation, successor generation, operation kind, semantic metadata digest, per-disk prepared metadata/data receipts, outcome identity). Disk-specific erasure indices, checksums, and metadata blobs are bound by individual receipts, not assumed byte-identical across disks. The semantic digest includes the full version set and relevant metadata, including fields omitted from ordinary read voting.
A decision does not require a new FileInfo positional field. The operation UUID stays in the existing metadata map. Durable authority records and local recovery records carry the revision, predecessor, ballot, and complete decision identity. They use separately versioned records; they cannot be inferred from a version's UUID alone. xl.meta remains a recoverable materialization of the chosen decision in strict mode.
| Input/state | Required treatment |
|---|---|
| Object never present | An explicit absent bootstrap state, established during strict cutover; no client-supplied None may authorize creation. |
| Object deleted, including deletion of its last version | A durable tombstone head with its own revision; never revert to never-present. This prevents ABA and resurrection after a laggard rejoins. |
| Null version | A real version-set member, distinct from absent. Replacing/removing it changes the object head. |
| Delete marker | A real version-set member and, when latest, a deleted-current state. Preserve marker type and existing S3 behavior. |
| Noncurrent version mutation | Compare the whole-object predecessor and the selected version's identity; commit a successor even if the latest S3 version is unchanged. |
| Valid legacy metadata with no UUID | Import once during fenced cutover after ordinary quorum/format validation; assign a bootstrap generation in the authority. Never equate this with absent or silently import while old writers are still admitted. |
| Missing, malformed, nil, or conflicting dual generation keys after strict enrollment | Typed corrupt/unsupported-state error; retain data for repair. No default to zero, absent, or a new UUID. |
| Missing/lagging/replacement disk | A non-authoritative materialization target. Recover the chosen decision and validate its data before it can contribute; do not demand that it already equals the predecessor or blindly overwrite it. |
Durable Decision Protocol
Use the existing namespace-lock participant identities and routing, with an explicitly persisted authority configuration. Lock voters and erasure disks are different sets: Qlock = floor(lock_participants / 2) + 1 decides authority; the existing per-operation Wdata decides recoverable object durability. A lock vote is not a shard receipt. One-node deployments still persist their single voter's state. Membership changes cannot be inferred from whichever RPC endpoints answered.
Each participant persists, per authority key and successor slot, its highest promise, highest accepted (ballot, DecisionValue), and the last learned committed head. Promises and accepted records survive unlock, TTL expiry, restart, and log compaction. Durable records are written through a storage boundary below the object API; writing them through PUT would recursively acquire the same authority. LocalClient must not become a filesystem implementation: the lock crate consumes an injected durability interface, while the storage owner implements it.
The following is a protocol contract, not pseudocode to paste into the current rename implementation:
- Acquire the existing object namespace write lock for admission. Read/recover the latest authority head through a fresh quorum promise/recovery barrier; reading only cached learned-head markers is insufficient because a quorum may have accepted a value before its commit notification arrived. Resolve any accepted successor before returning a head or allocating another slot. An unavailable decision quorum is an explicit failure. The lock still prevents ordinary competing work, but it is not the safety proof after lease loss.
- Stage the new shards and exact replacement metadata in transaction-owned paths. Obtain Wdata receipts only after the required file and directory syncs. Receipts bind disk identity/incarnation, authority configuration, key, operation UUID, blob digests, and data directories. Preparing must not replace live metadata, remove old directories, or reuse a winner's directory. Metadata-only/delete operations also stage a recoverable replacement version set.
- For successor slot
head.revision + 1, obtain Qlock durable promises for a unique ballot. Each response returns its accepted value, if any. Adopt the value with the highest accepted ballot among the promise quorum. Only if none was accepted may the proposer offer its own candidate with that predecessor. A recovered candidate is not replaced merely because its coordinator timed out or its lease expired. - Validate the candidate's Wdata preparation receipts and predecessor, then obtain Qlock durable accepts of the same value at that ballot. A participant accepts only at or above its promise and must reject a different value at the same ballot. A value becomes chosen at Qlock acceptance. Quorum intersection plus adoption of the highest accepted value prevents a different value from being chosen in that slot. Retrying the same operation cannot allocate a second successor.
- Learn/persist the chosen decision and publish it on storage disks through the guarded local recovery protocol below. Return S3 success only after both a durable chosen decision and Wdata durable materializations satisfy the existing operation's rules. Decision chosen but publication incomplete is
OutcomeUnknown/PendingRecovery, never authorization to roll back the decision. A later proposer first resolves the prior slot before allocating the next one. - On a lost ACK, resolve by operation UUID and exact request digest. The results are
NotChosen,ChosenPendingPublication,Committed, orSupersededAfterCommit. Reusing a UUID for different input is invalid. A timeout without a recovered decision stays unknown. Record idempotency outcomes until the client retry horizon and all dependent cleanup/accounting records have passed a durable retirement watermark; old requests then fail as expired rather than being treated as new.
The decision and payload retention lifetimes are coupled. Accepted/staged data is not garbage merely because no coordinator is alive. A new promise quorum can adopt a previously accepted value and finish it. If the required payload has been physically lost, fail closed and repair; never choose a different value for an already chosen slot. Persisted voter state that is lost or corrupt requires catch-up/replacement, not an empty voter with the same identity.
This is the minimum extension that turns the lock grant into a recoverable authority. A promise-only grant lacks outcome recovery; per-disk promises without a common decision allow minority uncertainty to escape into reads. The protocol requires review and executable state-machine tests before production integration. It does not imply that the current lock RPC is already a consensus implementation.
Single-Disk Publication And Recovery
Local and remote disks execute the same guarded primitive. A remote RPC handler must decode and authenticate the request, then call that primitive; checking only in the RPC handler leaves local callers and deferred syscalls uncovered. All operations touching the object's metadata, backup, or referenced directories participate.
Under one object mutation guard, re-read local durable recovery state, verify the decision/configuration and local nonregression condition, record a write-ahead intent, sync it, perform data-directory rename and atomic metadata replacement, sync affected directories, then persist applied outcome. The guard, including ownership of any namespace/deletion lease, stays with the blocking syscall until it completes, even if its caller is cancelled. An async task disappearing must not release a guard while its syscall still runs.
A single filesystem rename does not atomically commit a sidecar plus xl.meta. The write-ahead record binds predecessor/successor identities and exact metadata bytes; startup recovery runs before disk readiness. Recovery replays a chosen intent forward and completes syncs. An unchosen staged operation stays private until authority recovery makes its retirement safe. Old snapshots never overwrite a newer applied local revision. Conflicting bytes for the same decision are corruption. Treat write/fsync errors and torn records as unknown until decoded and reconciled, not as successful rollback.
A laggard need not contain the predecessor. Recovery fetches the chosen decision and its verified metadata, reconstructs or validates its shards under erasure-coding.md, and installs that state. An empty replacement uses a fresh disk incarnation and cannot reuse old preparation receipts. A disk whose durable state claims a later decision than the supplied one rejects the operation; a conflicting same-revision digest is quarantined. Never erase a divergent disk merely because it is in the minority.
rollback_committed_rename_std, rollback_inline_metadata_commit_std, and restore_metadata_backup in crates/ecstore/src/disk/local.rs must become decision-aware before strict mode includes them. The permitted rollback is limited to a transaction's unchosen private preparation, or restoration proven by recovery to be necessary before any newer local effect. A chosen operation is repaired forward. Neither a client timeout nor a rename-tail error permits reverting an acknowledged decision.
RenameConvergence remains a post-publication repair signal. PartialCommit, SignatureDivergent, and Unknown do not decide which transaction won. Keep their diagnostics and quorum accounting; resolve authority first. Early ACK may still precede minority-tail completion after the two quorum conditions hold. Tests that inspect all disks must synchronize the tail or assert the permitted minority residue separately.
Writer Participation
Every semantic metadata change advances the whole-object generation, including metadata-only writes. A physical repair that reproduces exactly the already chosen bytes preserves the generation and consumes that chosen decision; it must not create a new semantic value. The table specifies participation, not a generated inventory of every call site.
| Writer and current code boundary | Required generation behavior |
|---|---|
PUT / data COPY: put_object_with_old_current_size_inner, copy_object in crates/ecstore/src/set_disk/ops/object.rs |
Stage, choose, publish a successor; preserve source read protection. A metadata-only COPY is also a semantic successor, even if data directories are shared. |
MPU: complete_multipart_upload, new_multipart_upload, abort_multipart_upload in crates/ecstore/src/set_disk/ops/multipart.rs |
Complete chooses the destination object's successor. Part staging/upload metadata and abort remain in the upload namespace; they cannot delete a directory transferred to a chosen object decision. |
DELETE, batch DELETE, null/marker removal: delete_object, delete_objects_with_accounting, delete_object_version in crates/ecstore/src/set_disk/ops/object.rs; lifecycle callers in crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs |
Each object has its own predecessor/decision; retain tombstone authority after the last version. Prefix deletion must enumerate decisions or prove a bucket-incarnation retirement barrier; a recursive bypass is forbidden in strict mode. |
Heal: heal_object_with_explicit_version_regen in crates/ecstore/src/set_disk/ops/heal.rs |
Exact repair preserves the chosen generation and verifies full metadata identity. A version-list or semantic metadata change needs a successor. no_lock may skip admission only; it cannot bypass authority. In-place directory repair cannot remove a reader's live directory. |
Transition / restore: transition_object, restore_transitioned_object, put_object_metadata in crates/ecstore/src/set_disk/ops/object.rs; finalize_restore_metadata, update_restore_metadata in crates/ecstore/src/set_disk/replication.rs |
Each metadata transition is a successor, preserving existing operation-ID, remote tuple, and tier lease checks. Bind the transition transaction to the exact predecessor/successor; a late finalizer cannot rebase onto another restore operation. |
Replication status and metadata/tag/retention writeback: put_object_metadata, put_object_tags, delete_object_tags, merge_replication_metadata_lww in crates/ecstore/src/set_disk/ops/object.rs; callers in crates/ecstore/src/bucket/replication/replication_resyncer.rs |
Commit a field-scoped successor conditional on the exact version/content identity. On conflict, reload and revalidate the mutation; never replay a full stale FileInfo. Existing LWW category rules remain applicable within that validation. |
Rebalance/decommission: migrate_entry_version in crates/ecstore/src/services/rebalance/migration.rs; decommission_tier_free_version, decommission_tiered_object in crates/ecstore/src/set_disk/mod.rs; crates/ecstore/src/data_movement/mod.rs |
The authority key and lock group remain stable across pools. Stage the destination, choose the location/ownership successor, then retire the exact source receipt. Do not mint independent source and destination authorities. Existing placement and tier ownership fences remain required. |
Generic metadata entry points: write_unique_file_info, update_object_meta_with_opts in crates/ecstore/src/set_disk/core/io_primitives.rs; LocalDisk::write_metadata, update_metadata, delete_version, delete_versions_internal, write_all_meta in crates/ecstore/src/disk/local.rs |
Consume a validated decision/recovery context or reject strict writes to enrolled objects. None may invent a generation, reset it through fresh, replace corrupt metadata with an unproven empty version set, or bypass durability with no_persistence. |
Rollback and GC: rename_data_owned_with_fence, commit_rename_data_dir, reclaim_orphan_data_dirs in crates/ecstore/src/set_disk/core/io_primitives.rs; reconcile_old_data_cleanup_receipts in crates/ecstore/src/set_disk/ops/object.rs |
Consume the owning decision and exact directory references. Cleanup does not change object contents or grant a new semantic generation. It must use a durable retirement decision and local reader/deletion guards. |
Strict capability is withheld until every raw writer in DiskAPI, its local implementation, DiskStore, remote adapters, and server handlers has an enforced path. Internal authority persistence must use its own narrow storage primitive, not evade this rule by recursively calling generic object metadata writes.
Reads, Garbage Collection, And Accounting
Strict reads need the chosen head, not a majority of arbitrary prepared/live UUIDs. Resolve the decision under the namespace read lock before accepting object metadata; validate the selected current or explicit version against it, and wait for or repair missing materialization. HEAD, GET, ListObjects/ListObjectVersions, scanner reads used for deletion, and prepared pool reads all need this distinction. A query may return an error while a chosen write is recovering; it must not expose an unchosen candidate or resurrect a retired version. This read-decision adapter is part of the strict-mode scope and is a reason E03 is larger than disk CAS.
Keep the current namespace read lock through EOF/drop, including multipart part boundaries. This design does not replace it with a new cross-node generation lease registry. A strict implementation must also bind every deferred local/remote part open to the resolved generation and acquire protection on the disk that owns the directory before handing the read capability out. A reader that loses authority or cannot renew its disk protection must fail before another open; it cannot continue on an unvalidated cached pathname.
LocalDisk::acquire_snapshot_lease, renew_snapshot_lease, release_snapshot_lease, and delete_data_dir in crates/ecstore/src/disk/local.rs provide disk-local path protection and deletion deferral. They are not proof of a fleet-wide object generation. Before reuse, their token must bind disk incarnation and exact generation/directory, and strict reads must reject a pre-restart token. Existing open file descriptors may finish reading an unlinked inode, but later part opens need a valid protected generation. This preserves streaming behavior without assuming a local mutex protects another node.
GC consumes a durable retirement authorization for exact directories no longer referenced by any retained version or pending accepted decision. Include retirement in the chosen successor that removes the last reference; if it was not recorded there, choose a metadata-neutral successor that records it before deletion. That successor advances the authority revision while preserving the S3 version contents; the cleanup syscall itself never mints an identity. Retirement prevents future repairs/reads from creating new references; a new reference requires a new decision and cannot revive a retired directory. At the destructive syscall, hold the object/directory guard, recheck the local chosen metadata references, old_dir != committed_dir, retirement/configuration identity, and local snapshot protection. Any uncertainty defers deletion. A stale cleanup receipt matching an earlier UUID is not enough. Across executor restart, replay the same retirement ID idempotently; do not convert a lost reply into a broader recursive delete.
Prepared pool reads remain a separate optimization domain. In crates/ecstore/src/store/rebalance.rs, prepare_latest_object_metadata_with_idx collects candidates and revalidates a refetched winner with validate_prepared_pool_refetch_identity from crates/ecstore/src/store/rebalance/support.rs. PreparedPoolReadFallbackBarrier is a #[cfg(test)] scheduling fixture, not a production identity. Keep the all-pool resolution rules until a chosen location decision supplies equivalent evidence. The prepared bundle binds authority generation plus pool identity; mismatch requires a complete refetch or a typed failure, never selection of a different generation using the old bundle.
Quota remains a separate per-bucket arbitration domain. QuotaLedger and settle in crates/ecstore/src/bucket/quota/reservation.rs key reservations by operation UUID, validate object/size, and update under the ledger fence. A late settlement cannot remove a different reservation key; an absent/mismatched key fails or follows the existing idempotent abort rule. That proves key isolation, not that current generation A committed or that a ledger storage write is immune to stale-disk mutation.
For strict mode, add the decision identity to the reservation/settlement binding. Settle a committed historical decision even if it has since been superseded, but only against its original reservation and recorded old/new sizes; demanding that it still be current would leak valid reservations. Abort only a recovered unchosen/retired operation. Unknown outcomes stay reserved and reconcile. Retain the existing conservative usage floor and commit_started recovery behavior. Ledger writes themselves require the authority protocol, with a documented lock order and no recursion through their own reservation path. No quota token is compared numerically to an object ballot or used to revoke a newer object's leases.
Restart, Membership, And Rollout
Authority configuration is durable and includes participant identities, routing, bucket incarnation, protocol version, and quorum rules. Changing storage pool placement must not remap the authority key. A replaced voter starts as a non-voter, catches up durable promise/accepted/chosen state, and only joins through a quorum-approved configuration transition. Configuration change requires intersecting old/new decision quorums; losing the old quorum is a recovery incident, not permission to bootstrap a new empty authority. Offline data disks rejoin through generation-aware catch-up, independent of voter admission.
A live capability proof must bind the authority configuration, topology, every participant process boot epoch, disk incarnations, writer/read/recovery protocol support, encoding version, and RPC signature/body/replay strictness. Restart, membership change, disk replacement, protocol downgrade, or a strict-transport setting change revokes it. Receivers revalidate before entering the publication critical section; accepted durable decisions survive proof revocation and are recovered under a fresh valid proof, never replayed as unvalidated requests. The remote-version-state fleet proof does not prove any of these generation capabilities.
The existing RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE and RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED flags in crates/config/src/constants/object.rs retain their current default-off behavior. They do not become a claim that the new protocol exists. If generation strictness is explicitly selected, missing capability is an error, never silent downgrade. No new environment variable is introduced by this document; a production gate must be documented with its implementation.
Strict enrollment requires quiescing old writers and readers for the enrolled namespace, recovering ambiguous operations, validating/importing legacy heads, persisting a strict-format/protocol marker, and enabling the complete fleet. New disks reject unbound legacy mutation RPCs for that namespace. Old binaries must be prevented from opening a strict-enrolled drive by a startup compatibility gate they understand before enrollment; an environment flag known only to new binaries is insufficient. Until that prerequisite is deployed, do not activate strict mode in a mixed fleet. Disabling flags after enrollment cannot drop durable authority; downgrade requires a separately verified quiescent materialization/export operation. Ordinary un-enrolled compatibility deployments keep their current behavior.
Encoding And Transport
- Do not bump
XL_META_VERSIONorXL_HEADER_VERSIONincrates/filemeta/src/filemeta.rs. Do not add fields to positional-msgpackFileInfo; carry the UUID through the metadata map and protocol records through explicit versioned envelopes. - Write RustFS/MinIO internal metadata dual keys using
crates/utils/src/http/metadata_compat.rs. Reject conflicting, nil, or malformed generation values; validate every persisted/RPC record again at consumption. - New proto values in
crates/protos/src/node.protorequire explicit presence (optionalscalars or a present message), including absent/tombstone state. Bind expected/new generation, ballot, receipts, configuration, and outcome identity in the canonical body. The JSON and msgpack representations must carry identical semantics; absent data from an old peer cannot decode as a valid zero ballot. crates/ecstore/src/cluster/rpc/remote_disk.rsandrustfs/src/storage/rpc/node_service/disk.rsmust share local protocol behavior. Extend canonical encoders for every affected mutation, not onlyRenameData. Authenticate both compatibility representations and reject disagreement rather than falling back to a weaker JSON twin.crates/ecstore/src/cluster/rpc/http_auth.rssupplies signature, canonical-body, and replay-scope checks. Strict generation capability requires fleet convergence ofRUSTFS_INTERNODE_RPC_SIGNATURE_STRICT,RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT, andRUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICTfromcrates/config/src/constants/internode.rs. Cover method substitution, body tamper, stripped metadata, consumed nonce replay, and receiver restart.- Preserve real-MinIO metadata fixture decoding and supported old-RustFS compatibility before strict enrollment. Do not promise that a live MinIO binary can start a RustFS-written drive set; minio-file-format-compat.md explicitly excludes that direction. Strict authority records also impose a new deployment boundary even though the
xl.metacontainer version is unchanged.
Required Failure Outcomes
Every scenario must first prove its intended barrier, quorum, or crash point was reached, then inspect authority records, decoded metadata, directory references, return/error categories, and full GET bytes. Single-process barriers are insufficient evidence for the network-partition cases.
| Interleaving | Unique permitted outcome |
|---|---|
| A pauses after coordinator verification; A loses lock quorum while data RPC remains reachable; B chooses and publishes; A resumes rename. | A cannot become the chosen successor for B's predecessor or overwrite B on a disk that applied B. Isolated older materialization cannot vote as current. Read B exactly; reconcile laggards under B's decision. |
| A's publication result is unknown; B succeeds; A's rollback/cleanup resumes. | Recover A's decision. Never restore/delete B's metadata or referenced data; retire only A-owned unchosen paths or separately authorized dead directories. |
| Reader acquires generation G, consumes part 1; replacement retires G; GC attempts to delete part 2's directory. | The reader's valid disk protection defers GC; otherwise the reader fails before its next open. Never silently serve another generation or delete a directory still covered by valid protection. Repeat across reader and disk restarts. |
| A and B each stage or partially publish on two of four disks; one coordinator dies; all disks return. | The promise/accept recovery rule preserves any chosen value or adopts the highest accepted candidate and finishes that slot. No guessing from UUID order, no permanent exact-CAS split, no replacement of a chosen value. |
| Process dies after staging sync, intent sync, data rename, metadata replace, directory sync, or accepted/decision reply. | Reopen durable records before readiness. Unchosen work stays private; chosen work is replayed forward; lost ACK resolves to the original operation identity. Torn/insufficient evidence fails closed. |
| Null version becomes a delete marker; a noncurrent version is deleted; delayed heal/metadata write resumes. | Whole-object predecessor no longer matches. Exact repair uses the chosen version set; no resurrection, marker-to-object conversion, or reset to never-present. |
| Old coordinator reaches a new strict disk, new coordinator reaches an old disk, voter restarts, or transport strictness changes. | Compatibility behavior only in an un-enrolled namespace. Strict admission fails until a fresh complete proof and supported startup gate exist; no zero/missing-field fallback. |
| Replacement disk is empty, or a restored minority disk has an old promise and old metadata. | It cannot vote as an initialized authority. Catch up chosen state and reconstruct data under a fresh incarnation; stale requests cannot bypass enrollment by presenting absent metadata. |
| Quota settlement for A arrives after B commits; prepared pool refetch sees B instead of A. | Only A's original chosen outcome may settle A's reservation; B is unaffected. Prepared A cannot supply metadata/data for B without a new validated preparation. |
Required Implementation Boundaries
These are durable ownership and acceptance boundaries, not permission to close the disk-fencing work before the protocol exists. The decision-model and availability changes require architecture review before production implementation. No unrelated external consensus service or full-manifest rewrite is authorized by this contract.
| Boundary | Required implementation and exit evidence |
|---|---|
| Durable authority substrate | Injected lock-participant persistence; typed ballot/configuration/decision values; prepare/accept/recover state machine; restart-safe proposer identities; corruption and voter replacement handling. Model/exhaustively test two competing proposers, lost replies, minority recovery, and every durable transition. The same-slot different-value property must be impossible. |
| Disk publication boundary | Separate private preparation from publication; implement write-ahead intent, decision receipts, atomic guarded mutation, idempotent recovery, and directory retirement. Include inline/non-inline, every crash point, canceled blocking syscalls, ACK loss, and empty/lagging disks. Existing rollback helpers cannot remain an unguarded alternate route. |
| Writer and read integration | Route every writer in the table and every strict read/scan decision through the authority; preserve data quorum and S3 version semantics. Bind prepared reads, MPU ownership transfer, tier operations, and quota outcomes. Demonstrate no raw metadata entry point bypasses strict mode. |
| Fleet activation | Deploy the startup downgrade barrier first; import legacy/absent heads during quiescence; implement configuration/proof revocation and both RPC encodings. Run real multi-node lock/data-plane partitions and mixed-binary/restart tests. Only then can strict E03 acceptance run and activation be considered. |
The conservative immediate action is to keep the existing compatibility behavior and improve its local convergence/recovery independently. Those fixes must describe their smaller guarantee and must not advertise E03's distributed safety. A strict-only local CAS helper can be built behind the inactive capability boundary, but it cannot enable the feature or close the authority work.
Performance And Activation Criteria
Measure the existing implementation and the full proposed path on identical machines, disk/filesystem, durability settings, network, object population, concurrency, and warmup. Include single hot-key and many-key 4 KiB PUT, 1 MiB PUT, metadata-only writes, and CompleteMultipartUpload with fixed part counts. Report throughput, p50/p95/p99, peak retained preparation/recovery bytes, recovery time, per-operation RPCs/fsyncs, and the object mutation critical-section duration. Include a slow minority disk, one voter loss, and restart recovery; a throughput result alone is insufficient.
The unoptimized proposal adds a lock-quorum promise round and an accept round with durable writes, plus decision learning/publication and local intent/applied-state persistence. Wdata staging remains separate. Read resolution may add an authority quorum round. Record actual overlapping rounds and fsync group commits; do not claim these costs disappear because the existing lock RPC is reused. Never hold a global lock across shard I/O, wait for all disks on the successful path, or weaken fsync/bitrot/quorum to recover throughput.
Activation requires all failure scenarios to pass with no acknowledged-data loss or wrong-generation read; no unexplained RPC/fsync amplification beyond the implemented phase budget; and an explicit performance acceptance recorded with the review. Use a conservative review trigger of more than 10% throughput loss or 15% p99 growth in any fixed-workload comparison: exceeding it blocks default activation until the architecture/operations owners accept the measured tradeoff or the implementation removes it. These are proposed rollout budgets, not measurements or performance claims. Without a reproducible baseline, leave strict mode unavailable.