Chris
77a36d92a3
* fix(decommission): let a failed start roll back its own activation A fresh decommission start whose peer reload failed could not roll back: the pool metadata stale-write guard from #6604 refuses any save that clears a persisted active or queued decommission, so the pool stayed active with no worker until the watchdog or a leader restart picked it up. The rollback save now carries the activation that start committed. The guard exempts a pool only while its persisted record still equals that activation field for field, so a worker checkpoint or a later start keeps the rejection. A canceled record is not an active decommission and is cleared by the rollback as well. Every other save path passes no activation and is guarded as before. A peer may already have reloaded the activation and started a worker before the rollback. Saves are now fenced by decommission generation: a write carrying a start_time is rejected when the persisted record belongs to a later start, or has no decommission state and was updated after that start_time, so such a worker cannot make the pool active again. Its queued promotion and progress checkpoint also adopt the persisted record and stop the worker. Refs rustfs/backlog#2873 * fix(decommission): fence terminal writes of a withdrawn decommission Terminal records carry no start_time, so the generation fence did not cover them: after a start rollback, a peer worker of the rolled back generation that failed (for example on a bucket completion) could still persist a failed record over the restored pool. The merge now rejects a failed, canceled or complete record over a persisted record without decommission state; every attempt persists decommission state first, so no legitimate path writes a terminal record there. Peers also drop a withdrawn decommission sooner. A pool metadata reload replaces a running decommission with a newer record that withdrew it, even under an active worker; bucket completion adopts such a record like promotion and progress checkpoints do; and a withdrawn progress checkpoint stops the worker instead of logging a failed save and moving more entries. Tests cover the peer failure mark, a rollback retry that sees a worker checkpoint land between attempts, and the reload withdrawal. A heal fixture that wrote complete records over a fresh pool.bin now persists the active attempt first. Refs rustfs/backlog#2873 * fix(decommission): fence peer writes after a retry start rollback A retry start replaces a failed or canceled record that keeps its unresolved ledger, so its rollback restores that terminal record rather than clearing the pool. The withdrawal checks only recognized a record without decommission state, and the generation fence ignored a persisted record without a start_time. A peer worker of the rolled back retry could therefore checkpoint the pool back to active. The generation fence now compares the persisted last_update whenever the persisted record has no start_time of its own (cleared, terminal or queued). The progress checkpoint also checks the generation against the record read under the pool metadata lock, as bucket completion already does, and the save-path withdrawal check treats a terminal record newer than the running view as a withdrawal, so the peer adopts it and stops. A reload still never lets a newer terminal record replace an active worker's view, so after a retry's rollback a peer worker stops at its next bucket completion or progress checkpoint rather than on reload. Tests cover the retry rollback against a peer checkpoint and direct write, and a peer bucket completion after a fresh start's rollback. Refs rustfs/backlog#2873 * fix(decommission): keep a retry rollback's restored record intact After a retry start rolls back, the restored record is failed, so a peer worker's own failure mark passed every merge guard and replaced it. A failure mark now saves through save_current_pool_meta_checked, which runs the withdrawal check against the record read under the pool metadata lock and adopts that record instead of writing. The withdrawal check compared the restored record with the worker's in-memory last_update, which a rejected save had already raised, so the peer never adopted the rollback and the watchdog kept resuming it. It now compares against the running generation (start_time), and a rejected unresolved-entry save undoes its in-memory change. save_current_pool_meta_for_decommission_start is also compiled for the test-util feature again; a test-util helper still calls it, which broke the workspace clippy and Connect builds. Refs rustfs/backlog#2873 * fix(decommission): keep an unresolved entry when its own save fails The rollback of a rejected unresolved-entry save also dropped the entry on transient failures such as a quorum or lock-fence error, so the following failure mark persisted a failed record without it and an operator could clear a decommission that skipped an unreadable object. Keep the entry and undo only the last_update bump, which is all a withdrawn worker needs to still see the withdrawal. Refs rustfs/backlog#2873
Documentation
Use the focused indexes rather than treating this directory as an unordered collection:
Operations
- Table maintenance periodic queueing — opt-in scheduling, limits, worker handoff, and recovery.
For PUT cleanup admission, blocking-pool isolation, and optional CPU affinity, see PUT cleanup isolation.
-
External certificate issuance and TLS rotation — publication, native reload, trust separation and failure handling.
-
Architecture guard troubleshooting — diagnose repository guard failures.
-
Logging governance — broad event audits, migrations, and guardrail changes.
-
Multipart upload memory diagnosis — allocator attribution, Docker reproduction and sustained upload checks.
For the logical per-operation io_uring read cap, see io_uring read chunk size.
For bounded local read-backend startup and cancellation, see io_uring backend initialization.
For legacy protection assessment and bounded protected copies, see Object integrity inventory, audit, and migration.
For encrypted listing compatibility and older stored fingerprints, see Encrypted object ETags.
For durable hard quota accounting, node-outage behavior, and fence protocol upgrades, see Durable hard quotas.
Operational runbooks live under operations/. Replication
operators should start with:
| Runbook | Use it for |
|---|---|
| Site replication operations | Health fields, pending operations, outage recovery, re-pair admission, IAM/SSE boundaries, and upgrades. |
| Replication target check | Validating an S3 destination and version fidelity before enabling replication. |
| Replication object size limits | Multipart routing, large-object limits, and retry characteristics. |
| Replication outbound transport | Integrity headers, generic target behavior, and transport knobs. |
For the erasure-coded cluster lifecycle (planning, parity and EC:0,
expansion, rebalance, decommission, heal, drive replacement, restart
recovery, and the rc CLI mapping), start with
Cluster and erasure-coding lifecycle operations.
For concurrent bucket creation and deletion, see Bucket operation admission.
For persisted administrator bucket tasks and bucket recreation, see Bucket heal recovery.
For interrupted object deletions and returning drives, see Delete transaction recovery.
For disk replacement across VM restarts and schema 5/6 maintenance migration, see Replacement generation recovery.
For historical GET timeouts during PUT or Heal, see Object lock contention diagnostics.
Other runbooks remain grouped by filename in operations/;
architecture pages link to the relevant runbook where a cross-boundary
procedure is required.
For storage dashboards, see Storage metrics and observer selection: drive ownership, snapshot freshness, counter queries, and rolling upgrades.
For per-bucket capacity and object counts, see Bucket usage metrics: OTLP export, cluster deduplication, accounting semantics, and scanner freshness.
For optional shard commitments, see Independent shard integrity rollout: activation, legacy repair results, multipart mode changes, and rollback limits.
For crates.io publication of workspace crates, see Workspace Cargo Publish: dependency ordering, dry-run, publish, and failure handling.