test(heal): cover privileged mount readiness

Add Linux-only ignored replacement readiness tests for independent mount admission and same-device sibling rejection.

Co-Authored-By: heihutu <heihutu@gmail.com>

.agents/skills/adversarial-validation/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: adversarial-validation
-description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the applicable reviewer roles with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, design proposal, or agent-instruction change that alters execution before declaring it done.
 ---
 
 # Adversarial Validation Playbooks
@@ -61,14 +61,15 @@ Null report example: "Attacked quorum-1 error reduction, exact max-keys listing
 
 ### Simplicity adversary
 
-- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+- Smaller-diff attack: inspect production growth separately from tests, fixtures, generated code, and documentation; test additions have no growth budget. Rewrite the production diff mentally (or in scratch) as the minimal equivalent edit. Report a finding only with a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries; fewer lines alone are not evidence.
   - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
-  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
-- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
-  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (conditional extraction rule, preserve sensitive control flow, canonical modules) and 'Reuse Before You Write'; the Adversarial Validation roles list charters this attack.
+- Reuse-and-necessity attack: for each new helper, search `crates/utils`, `crates/common`, the touched crate, the likely domain owner, and relevant direct dependencies. A reimplementation is a finding, but forced reuse with mismatched normalization, error, backoff, or durability semantics is also a finding. Demand a nameable trigger for new defensive branches. Tests remain subject to validity and near-duplicate coverage review, never a size limit.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests
   - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
+- Replacement-and-comment attack: when the diff introduces a replacement path or representation, trace all callers and flag a superseded in-scope path left behind without a compatibility requirement. Keep one canonical core behind compatibility adapters. Comments must state non-obvious invariants completely without narration or change history. Never demand unrelated deletion or trade away correctness, compatibility, or readability to reduce the diff.
 
-Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
+Null report example: "Separated production growth from tests/docs, tested a smaller equivalent, checked helper reuse and superseded paths, and found no break."
 
 ### Security reviewer
 
@@ -195,9 +196,9 @@ Null report example: "Attacked dual-key metadata writes/removals against MinIO-o
 
 ### Performance reviewer
 
-- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
+- For each `.clone()` or allocation added to a per-request/per-object path, identify the copied data and execution frequency. Report a finding only for a concrete repeated cost or benchmark regression. Recommend borrowing, moving, `Bytes`/`Arc`, `Cow`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs.
   - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
-  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
+  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths'; .agents/skills/rust-code-quality/SKILL.md requires a concrete hot-path cost rather than a proxy metric
 - For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
   - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
   - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
@@ -230,12 +231,12 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 
 ### Test-coverage skeptic
 
-- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
+- For every testable behavior claim in the PR description, revert that hunk and name the focused test or executable check that detects the revert. If no reasonable check exists, require the reason and residual risk from the validation floor. Especially verify the check exercises the real production path, not a lookalike helper.
   - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
-  - Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
+  - Evidence: AGENTS.md testable-behavior exit criterion. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
 - Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
   - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
-  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
+  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md requires an observable failure criterion; .agents/skills/security-advisory-lessons/SKILL.md asks whether the exploit form is denied.
 - When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
   - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
   - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
@@ -260,7 +261,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 - For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
   - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
   - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
-- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
+- A green focused test is evidence only for the targets it builds. Follow the `AGENTS.md` validation tier: add package-scoped Clippy or broader test-target compilation only when changed targets, features, or dependents remain uncovered; do not require a workspace-wide build by default.
   - Where: All crates; especially concurrent-branch merges into crates/ecstore
   - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
 
@@ -271,7 +272,6 @@ Null report example: "Attacked revert-detection for all 3 claimed behaviors (eac
 Probes are distilled from shipped bugs in git history (commit/PR references
 above), GitHub security advisories (see the security-advisory-lessons
 skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
-and `docs/operations/`. Line numbers drift; when a cited location no longer
-matches, trust the invariant and re-locate the code. When a new bug class
-ships, add a probe with its evidence here rather than growing the policy
-section in `AGENTS.md`.
+and `docs/operations/`. Line numbers drift; re-locate the invariant. Merge
+new incidents into an existing probe when they share a failure class; add a
+new probe only for a distinct attack, rather than growing the root policy.

.agents/skills/pr-creation-checker/SKILL.md

@@ -24,15 +24,17 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
 
 2. Inspect change scope
 - Review the diff and summarize what changed.
+- Inspect `git diff --stat` and `git diff --numstat`; assess production-code growth separately. Tests, fixtures, generated code, and documentation have no growth budget. Treat line counts as signals, not quotas.
 - Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
 - Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+- Use the simplicity-adversary verdict instead of producing a per-symbol inventory. Block growth only when the review identifies duplication or gives a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
+- Confirm replacement implementations remove the superseded in-scope path or adapt compatibility at the boundary to one canonical core.
 - Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
 - Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
 
 3. Verify readiness requirements
-- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
-- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
-- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
+- Select checks from `AGENTS.md` "Verification Before PR" based on the final diff's risk tier. Do not replace a focused behavioral test with `make pre-commit`, or a required high-risk `make pre-pr` with a narrower gate.
+- For focused verification, state why the selected tier is sufficient and list the scope-specific commands in the PR body.
 - If `make` is unavailable, use the equivalent commands from `.config/make/`.
 - Add scope-specific verification commands when the changed area needs more than the baseline.
 - If required checks fail, stop and return `BLOCKED`.
@@ -81,13 +83,14 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
 
 ## Blocker rules
 
-- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
+- Return `BLOCKED` if the checks required by the `AGENTS.md` validation tier have not passed.
 - Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
 - Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
 - Return `BLOCKED` if required template sections are missing.
 - Return `BLOCKED` if the title/body is not in English.
 - Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
 - Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
+- Return `BLOCKED` for production-code growth only when the review identifies a duplicated or superseded implementation, or supplies a concrete smaller design with equivalent semantics. Fewer lines alone are not evidence.
 
 ## Reference
 

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -3,8 +3,8 @@
 - Confirm the branch is based on current `main`.
 - Confirm the diff matches the stated scope.
 - Confirm no secrets, logs, temp files, or unrelated refactors are included.
-- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
-- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
+- Confirm the checks required by the `AGENTS.md` validation tier passed.
+- For focused verification, confirm it covered the changed surface and the PR body explains why the selected tier is sufficient.
 - Confirm extra verification commands are listed for risky changes.
 - Confirm the PR title uses Conventional Commits and stays within 72 characters.
 - Confirm the PR title does not use tool-specific prefixes such as `[codex]`.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rust-code-quality
-description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+description: Enforce Rust-specific code quality rules on every Rust change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
 ---
 
 # Rust Code Quality Gate
@@ -12,27 +12,29 @@ Use this skill on every Rust code change to enforce quality rules that `cargo cl
 1. Identify changed `.rs` files.
 2. Run automated checks on changed files.
 3. Run manual review checklist on the diff.
-4. Report findings; block merge if P0/P1 issues exist.
+4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
 
 ## Automated Checks
 
-Run these on every changed `.rs` file (excluding test modules):
+Use these searches to find candidates in changed `.rs` files. Inspect syntax,
+`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
+filters do not reliably distinguish production code from tests.
 
 ```bash
-# 1. unwrap/expect in production code
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
+# 1. unwrap/expect candidates
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
 
 # 2. Silent type truncation via `as` cast
 rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
 
 # 3. String as error type
-rg -n 'Result<.*String>' <changed-files> | grep -v test
+rg -n 'Result<.*String>' <changed-files>
 
 # 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
+rg -n 'Box<dyn.*Error' <changed-files>
 
 # 5. println/eprintln in production
-rg -n 'println!\|eprintln!' <changed-files> | grep -v test
+rg -n 'println!\|eprintln!' <changed-files>
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
@@ -46,37 +48,35 @@ rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 For every Rust code change, verify:
 
 ### Error Handling
-- [ ] No `unwrap()` or `expect()` in production code without justification comment
+- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
 - [ ] No `Result<_, String>` in public API signatures
-- [ ] No `Box<dyn Error>` in public trait/struct methods
+- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
 - [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable (what failed, with what input)
+- [ ] Error messages are actionable without exposing secret input
 
 ### Type Safety
 - [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] `try_into()` or explicit clamping used for numeric conversions
-- [ ] No `f64 as usize` without prior clamping
+- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
+- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
 
 ### Concurrency
 - [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
 - [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
+- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
 ### Memory and Performance
-- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
-- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
-- [ ] Large buffers wrapped in `Arc` rather than cloned
-- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
+- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
+- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
 
 ### Recursion Safety
-- [ ] Recursive functions have a depth limit or use iterative traversal
+- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
 - [ ] Tree/cache traversals handle corrupted/cyclic input safely
 
 ### Testing
-- [ ] Every test function has at least one `assert!`
-- [ ] Tests use `.expect("context")` not bare `.unwrap()`
-- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
+- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
+- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
+- [ ] Test volume and line count are never treated as production-code growth
 
 ### Serde
 - [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
@@ -88,18 +88,18 @@ For every Rust code change, verify:
 - [ ] New string literals don't duplicate existing constants
 
 ### Reuse and Necessity
-- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
+- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
 - [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
 - [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
+- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
 - [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
 
 ## Severity Classification
 
-- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
-- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
-- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
-- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
+- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
+- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
+- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
+- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
 
 ## Output Template
 
@@ -107,10 +107,10 @@ For every Rust code change, verify:
 ## Rust Code Quality Report
 
 ### Automated Scan
-- unwrap/expect in production: N found
-- as casts: N found
-- String errors: N found
-- println/eprintln: N found
+- unwrap/expect candidates inspected: N
+- numeric-cast candidates inspected: N
+- error-type candidates inspected: N
+- output-macro candidates inspected: N
 
 ### Findings
 - [P1] `path:line` — description

.agents/skills/rust-code-quality/references/checklist.md

@@ -1,52 +0,0 @@
-# Rust Code Quality Checklist
-
-Use this as a quick pre-merge checklist for every Rust code change.
-
-## Critical (P0 — block merge)
-
-| Check | Command |
-|-------|---------|
-| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
-| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
-| Lock order consistent across call sites | Manual: trace all lock acquisitions |
-| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
-| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
-
-## High (P1 — must fix)
-
-| Check | Command |
-|-------|---------|
-| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
-| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
-| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
-| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
-| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
-
-## Medium (P2 — should fix)
-
-| Check | Command |
-|-------|---------|
-| Tests have assertions | Manual: check for `assert` in test functions |
-| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
-| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
-| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
-
-## Low (P3 — nice to fix)
-
-| Check | Command |
-|-------|---------|
-| No camelCase statics | `rg 'static ref [a-z]' <files>` |
-| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
-| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
-
-## Quick One-Liner
-
-```bash
-# Run all automated checks on changed files
-CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
-echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
-echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
-echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
-echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
-echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
-```

.agents/skills/security-advisory-lessons/SKILL.md

@@ -66,14 +66,23 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 
 ### STS, OIDC, and federation flows
 - Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
+- For web identity, the JWT is the credential; exemption from SigV4 is not itself an authentication bypass. Treat pre-verification claims only as untrusted routing hints, bound token size, normalize public failures, rate-limit discovery, and issue credentials only after signature, issuer, audience, and expiration checks.
 - JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
 - JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
 - Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
 - OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
 
-### S3 copy, multipart, and presigned POST
+### IAM policy conditions and plugins
+- Treat request headers as attacker-controlled even after SigV4; callers sign their own spoofed headers. Do not merge them into server-derived condition keys such as identity, groups, version ID, signature version, JWT, or LDAP claims.
+- Keep the condition-key namespace explicit. Reserved server-derived keys must reject or ignore colliding headers, while intentional request-header keys such as `s3:x-amz-*` remain available.
+- Quantified IAM condition tests need partially overlapping multi-value sets. Fully contained and fully disjoint sets cannot distinguish `ForAllValues` from `ForAnyValue` bugs.
+- External policy plugins must receive the same security context as built-in policy evaluation. If OPA or another plugin depends on existing object tags, load and pass `ExistingObjectTag/*` before the plugin decision.
+
+### S3 object actions, copy, multipart, and presigned POST
+- Version-aware object requests need version-aware actions. Explicit `versionId` reads and copy sources must authorize `s3:GetObjectVersion`, not only `s3:GetObject`.
 - Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
 - Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Fallbacks from version actions to non-version actions must still pass the same public-access-block, anonymous-deny, and post-authorization gates as a direct allow.
 - Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
 
 ### Protocol frontends and IAM parity
@@ -132,6 +141,11 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
 - Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
 
+### Object Lock and retention invariants
+- Object Lock state must fail closed when bucket metadata is unreadable, fabricated, or unparsable. Only a confirmed absence of Object Lock configuration may permit unprotected deletes or writes.
+- Do not collapse metadata read faults, missing persisted metadata, parse failures, and genuinely absent Object Lock config into one "not configured" result.
+- Retention enforcement must cover foreground deletes, batch deletes, force-delete helpers, default-retention materialization on PUT, lifecycle expiry, scanner sweeps, and all-versions expiry.
+
 ## Review Prompts
 
 Use these prompts while reviewing a diff:
@@ -148,5 +162,9 @@ Use these prompts while reviewing a diff:
 - Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
 - Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does an explicit object version, fallback action, or plugin authorization path pass through the same action and post-authorization gates as the direct S3 path?
+- Can a caller-controlled header populate a condition key that should be derived only by the server?
+- Do condition tests include partially overlapping multi-value inputs for quantified operators?
+- Does unreadable bucket metadata make Object Lock or retention enforcement fail closed rather than disappear?
 - Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
 - Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -35,12 +35,21 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### STS, OIDC, and federation flows
 
 - `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
+- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
+- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
-### S3 copy, multipart, and upload policy validation
+### IAM policy conditions and external policy plugins
 
+- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
+- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
+- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
+
+### S3 object actions, copy, multipart, and upload policy validation
+
+- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
+- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
 - `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
 - `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
@@ -59,7 +68,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -92,6 +101,10 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
+### Object Lock and retention invariants
+
+- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
+
 ### Serde deserialization and input validation
 
 - No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
@@ -107,11 +120,13 @@ Use these targeted searches when a diff touches security-sensitive code:
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
+rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
 rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
 rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
 rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
@@ -121,9 +136,12 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
+- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
+- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
+- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.

.config/make/tests.mak

@@ -25,6 +25,7 @@ TEST_THREADS ?= 1
 script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
+	./scripts/test_docker_runtime_timezone.sh
 	./scripts/test_entrypoint_credentials.sh
 	./scripts/test_internode_grpc_ab_bench.sh
 	./scripts/test_object_batch_bench_enhanced.sh

.config/nextest.toml

@@ -280,10 +280,12 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 # tests that are unfit for the per-PR e2e-smoke gate:
 #
 #   * 2 remote-target TLS validation tests.
-#   * 13 bucket-replication data-plane/helper tests — they PUT/delete objects
+#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
 #     and poll until source and target converge; two replicate over HTTPS,
-#     four pin active SSE fail-closed contracts (SSE-C, SSE-S3, SSE-KMS, and
-#     the SSE-S3 resync path), and one guards event/history observers.
+#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
+#     the target incl. multipart and the resync path, SSE-C and
+#     target-without-KMS stay fail-closed), and one guards event/history
+#     observers.
 #   * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
 #     servers and drives the cross-process site-replication control plane.
 #   * 1 `_real_three_node` site-replication test.

.github/actions/setup/action.yml

@@ -57,7 +57,7 @@ runs:
   using: "composite"
   steps:
     # protobuf-compiler is deliberately absent: the setup-protoc step below
-    # installs 34.1 into the tool cache and prepends it to PATH, so the apt
+    # installs 35.1 into the tool cache and prepends it to PATH, so the apt
     # build (older, and never version-matched) was shadowed on every run and
     # simply never used.
     - name: Install system dependencies (Ubuntu)
@@ -81,7 +81,7 @@ runs:
     - name: Install protoc
       uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
       with:
-        version: "34.1"
+        version: "35.1"
         repo-token: ${{ github.token }}
 
     - name: Install flatc

.github/workflows/e2e-replication-nightly.yml

@@ -14,25 +14,24 @@
 
 # Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
 #
-# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
-# FAST replication tests. This scheduled lane runs the remaining 27
-# heavier replication e2e tests that are unfit for a per-PR gate:
-#
-#   * 2 remote-target TLS validation tests.
-#   * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
-#     convergence; two replicate over HTTPS, two pin active SSE failure
-#     contracts, and one guards event/history observers). The SSE-S3 contract
-#     remains ignored under backlog#1291.
-#   * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
-#     servers and drives the cross-process site-replication control plane).
-#   * 1 `_real_three_node` site-replication test.
-#   * 1 `_real_single_node` service-account round-trip test.
+# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the
+# FAST replication tests. This scheduled lane runs the remaining heavier
+# replication e2e tests that are unfit for a per-PR gate: remote-target TLS
+# validation, bucket-replication data-plane/helper tests (PUT/delete + poll
+# for convergence, HTTPS targets, active SSE failure contracts, event/history
+# observers), and the `_real_dual_node` / `_real_three_node` /
+# `_real_single_node` site-replication tests that each spawn full rustfs
+# server processes.
 #
 # The selection is the [profile.e2e-repl-nightly] default-filter in
 # .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
-# add ad-hoc cargo-test steps here; change the filterset instead.
+# add ad-hoc cargo-test steps here; change the filterset instead. The
+# authoritative membership and count come from
+# `cargo nextest list -p e2e_test --profile e2e-repl-nightly`; the PR/nightly
+# count invariant is maintained next to the filtersets in .config/nextest.toml
+# (deliberately not duplicated here).
 #
-# Explicit division of labor: these 27 tests run ONLY here, never double-run
+# Explicit division of labor: the nightly subset runs ONLY here, never double-run
 # in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
 # consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
 # into it rather than growing a second scheduled entrypoint.

.github/workflows/package.yml

@@ -225,7 +225,9 @@ jobs:
           VERSION="${{ needs.resolve.outputs.version }}"
           DEB_ARCH="${{ matrix.deb_arch }}"
           # DEB version: replace - with ~ (1.0.0-beta.12 -> 1.0.0~beta.12)
-          DEB_VERSION="${VERSION/-/~}"
+          # Use a variable for ~ to prevent tilde expansion by bash
+          TILDE='~'
+          DEB_VERSION="${VERSION/-/$TILDE}"
           PKG_DIR="rustfs_${DEB_VERSION}_${DEB_ARCH}"
 
           echo "Building DEB: ${PKG_DIR}.deb"

.github/workflows/windows-filesystem.yml

@@ -69,6 +69,10 @@ jobs:
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
+      - name: Check production Windows dependencies
+        shell: pwsh
+        run: cargo check -p rustfs-ecstore --lib
+
       - name: Test guarded rename publication
         shell: pwsh
         run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture

AGENTS.md

@@ -51,26 +51,25 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 
 ## Change Style for Existing Logic
 
-- Prefer direct, local code over extracting one-off helpers.
-- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
+- Start with the smallest direct, local edit. Add production files, types, traits, helpers, wrappers, or abstraction layers only when current behavior requires them. Extraction must remove present duplication, enforce a real boundary, or materially clarify a non-trivial flow; anticipated reuse is not enough.
 - Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`).
   Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead.
   If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work.
 - Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
 - Prefer editing existing code over rewriting files or reshaping unrelated logic.
-- Modify only what is required and remove only artifacts introduced by your own changes.
+- Modify only what is required. Remove any in-scope path or representation superseded by the change. If compatibility or rollback requires retention, adapt at the boundary to one canonical core and follow the repository's `RUSTFS_COMPAT_TODO` removal policy; never delete unrelated code merely to improve addition/deletion statistics.
 - Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
 - Do not refactor existing code only to make it easier to unit test.
 - Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
-- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
-- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
+- Keep code elegant, concise, and direct. Prefer the smallest readable design and existing abstractions over parallel managers, factories, adapters, or wrappers added only to make the design look extensible.
+- Comments state non-obvious reasons, assumptions, and invariants in the shortest complete form. Their length follows the invariant's complexity: `SAFETY`, lock ordering, durability, and compatibility contracts may need a short list of conditions. Never narrate the next line, restate a signature, or record change history; move durable design rationale to architecture or operations documentation.
 - Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
 ## Reuse Before You Write
 
 Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
 
-- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
+- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, and relevant direct workspace dependencies from `Cargo.toml`. Search snake_case signatures with a focused term. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing dependency already provides — is a review finding, not a style preference.
 - **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
 - **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
 - **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
@@ -79,6 +78,7 @@ Search for an existing implementation before writing a new one; extend what exis
 
 Net-new code — files, types, branches, comments — is cost to justify, not progress:
 
+- Inspect production-code additions separately. Tests, fixtures, generated code, and documentation do not count as production-code growth. Line counts are signals, not quotas: new production structures must map to a current requirement, and a blocker requires a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
 - Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
 - The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
 - Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
@@ -218,9 +218,10 @@ not to bless it.
 
 Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 
-- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
-  no runtime surface. Skip this section.
-- **Mechanical:** pure renames, file moves, test-only or tooling changes —
+- **Exempt:** docs/comments, formatting, and typos that cannot affect runtime,
+  builds, tests, or agent execution. Skip this section.
+- **Mechanical:** pure renames, file moves, test-only or tooling changes, and
+  agent-instruction changes that alter execution —
   correctness and simplicity adversaries only.
 - **Standard (the default):** any change that affects behavior.
 - **High risk:** touches locking, erasure coding, quorum/heal, replication,
@@ -242,7 +243,7 @@ encode this repo's shipped bugs.
 - **Correctness adversary** — construct a concrete input/state/interleaving
   that yields wrong output, data loss, or a crash. Probe error paths and edge
   values (empty, nil UUID, zero-length, quorum−1, missing version).
-- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
+- **Simplicity adversary** — same behavior, less code. Hunt reimplemented helpers, rewrites where an in-place edit suffices, speculative abstractions, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, and narration comments. A one-caller helper is a finding only when it merely forwards or splits a short linear flow without adding domain naming, boundary isolation, an invariant, or useful error context. Report a concrete smaller replacement; fewer lines alone are not evidence.
 - **Security reviewer** — authn/authz bypass, injection, secret leakage,
   untrusted deserialization (see Serde Safety), path traversal, timing leaks.
 - **Concurrency/durability reviewer** — lock ordering, races, cancellation,
@@ -253,10 +254,11 @@ encode this repo's shipped bugs.
   time across IO, sync or CPU-heavy work on async runtime threads, added
   fsync/flush outside the durability gate, hot-path logging noise. A
   measurable regression on a per-request or per-object path is a finding.
-- **Test-coverage skeptic** — for each claimed behavior, name the test that
-  fails if the change is reverted; then name a changed line that could be
-  wrong while all tests stay green — if one exists, coverage is insufficient.
-  A missing test is a finding, not a note.
+- **Test-coverage skeptic** — for each testable behavior claim, name the test
+  or executable check that detects a revert; then name a changed line that
+  could be wrong while all checks stay green. If a focused check is not
+  reasonable, require the reason and residual risk from the validation floor.
+  Test additions have no line-count or growth budget.
 
 Standard tier: correctness adversary + simplicity adversary + test-coverage
 skeptic, plus every role whose domain the diff touches (async or
@@ -282,7 +284,9 @@ High risk: all seven roles.
 
 - Every applicable role has run; every finding is fixed or rebutted with
   evidence.
-- Every behavior change has a test that fails without it.
+- Every testable behavior change has a focused regression check. Exceptions
+  follow the validation floor and state why a check is impractical and what
+  risk remains.
 - The Verification Before PR gates pass — adversarial review supplements
   those gates, never replaces them.
 - High risk only: record a one-line verdict per role in the PR description.

Cargo.lock

@@ -2006,17 +2006,6 @@ version = "1.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
 
-[[package]]
-name = "clocksource"
-version = "0.8.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "46a4f8c23584e9dc6e40de1406e8c776ae727c49f7cb85c0bb23fb8c2096f7e0"
-dependencies = [
- "libc",
- "time",
- "winapi",
-]
-
 [[package]]
 name = "cmake"
 version = "0.1.58"
@@ -4033,7 +4022,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
 dependencies = [
  "libc",
- "windows-sys 0.52.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -5479,7 +5468,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
 dependencies = [
  "hermit-abi",
  "libc",
- "windows-sys 0.52.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -7974,7 +7963,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
 dependencies = [
  "heck",
- "itertools 0.10.5",
+ "itertools 0.14.0",
  "log",
  "multimap",
  "once_cell",
@@ -7994,7 +7983,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
 dependencies = [
  "heck",
- "itertools 0.10.5",
+ "itertools 0.14.0",
  "log",
  "multimap",
  "petgraph 0.8.3",
@@ -8015,7 +8004,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
 dependencies = [
  "anyhow",
- "itertools 0.10.5",
+ "itertools 0.14.0",
  "proc-macro2",
  "quote",
  "syn 2.0.119",
@@ -8028,7 +8017,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
 dependencies = [
  "anyhow",
- "itertools 0.10.5",
+ "itertools 0.14.0",
  "proc-macro2",
  "quote",
  "syn 2.0.119",
@@ -8245,7 +8234,7 @@ dependencies = [
  "once_cell",
  "socket2",
  "tracing",
- "windows-sys 0.52.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -8394,12 +8383,10 @@ dependencies = [
 
 [[package]]
 name = "ratelimit"
-version = "0.10.1"
+version = "2.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5dc94ed8e3de45f6d8d052869d48c0dbeebcaa7a6c345ec7f0f917e10347428e"
+checksum = "e78b08065c51c82ff8c4a0d88e3dce3edfce39c375e946c3464210fff3433fb8"
 dependencies = [
- "clocksource",
- "parking_lot",
  "thiserror 2.0.20",
 ]
 
@@ -9391,6 +9378,7 @@ dependencies = [
  "thiserror 2.0.20",
  "time",
  "tokio",
+ "tokio-stream",
  "tokio-util",
  "tonic",
  "tower",
@@ -9469,6 +9457,7 @@ dependencies = [
  "futures",
  "hotpath",
  "http 1.5.0",
+ "libc",
  "metrics",
  "rustfs-common",
  "rustfs-concurrency",
@@ -10130,6 +10119,7 @@ dependencies = [
  "tracing",
  "transform-stream",
  "url",
+ "uuid",
 ]
 
 [[package]]
@@ -10145,6 +10135,7 @@ dependencies = [
  "hotpath",
  "parking_lot",
  "rustfs-s3select-api",
+ "rustfs-test-utils",
  "s3s",
  "tokio",
  "tracing",
@@ -10469,7 +10460,7 @@ dependencies = [
  "errno",
  "libc",
  "linux-raw-sys",
- "windows-sys 0.52.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -10542,7 +10533,7 @@ dependencies = [
  "security-framework",
  "security-framework-sys",
  "webpki-root-certs",
- "windows-sys 0.52.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -10599,7 +10590,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
 [[package]]
 name = "s3s"
 version = "0.14.1"
-source = "git+https://github.com/cxymds/s3s.git?rev=fe3941d91fa1c69956f209a9145995c9f0235bff#fe3941d91fa1c69956f209a9145995c9f0235bff"
+source = "git+https://github.com/rustfs/s3s.git?rev=d7028511a53f69d41ed3c69f36899f9b1aede647#d7028511a53f69d41ed3c69f36899f9b1aede647"
 dependencies = [
  "arc-swap",
  "arrayvec",
@@ -11751,7 +11742,7 @@ dependencies = [
  "getrandom 0.4.3",
  "once_cell",
  "rustix",
- "windows-sys 0.52.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -12853,7 +12844,7 @@ version = "0.1.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
 dependencies = [
- "windows-sys 0.52.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]

Cargo.toml

@@ -278,7 +278,7 @@ percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.2" }
-ratelimit = "0.10.1"
+ratelimit = "2.0.0"
 rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
 reed-solomon-simd = "3.1.0"
@@ -289,7 +289,7 @@ rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/cxymds/s3s.git", rev = "fe3941d91fa1c69956f209a9145995c9f0235bff" }
+s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d7028511a53f69d41ed3c69f36899f9b1aede647" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"

Dockerfile

@@ -91,7 +91,12 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
 RUN apk upgrade --no-cache && \
-    apk add --no-cache ca-certificates coreutils curl
+    apk add --no-cache \
+    ca-certificates \
+    coreutils \
+    curl \
+    tzdata \
+    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs

Dockerfile.glibc

@@ -96,9 +96,11 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
 RUN apt-get update && apt-get upgrade -y \
-    && apt-get install -y --no-install-recommends \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
+    tzdata \
+    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=build /build/rustfs /usr/bin/rustfs

crates/e2e_test/src/admin_iam_crud_test.rs

@@ -26,7 +26,9 @@
 //! Later batches tracked on backlog#1154: config get/set, info, pools status,
 //! group lifecycle, import/export IAM.
 
-use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
+use crate::common::{
+    RustFSTestEnvironment, admin_ok, admin_request, admin_request_with_session_token, build_test_sts_client, init_logging,
+};
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::{Client, Config};
@@ -87,6 +89,262 @@ fn bucket_rw_policy(bucket: &str) -> String {
     .to_string()
 }
 
+async fn create_user_with_service_account_update_policy(
+    env: &RustFSTestEnvironment,
+    user: &str,
+    secret: &str,
+    policy: &str,
+) -> TestResult {
+    admin_ok(
+        env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
+        Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
+    )
+    .await?;
+    admin_ok(
+        env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
+        Some(
+            serde_json::json!({
+                "Version": "2012-10-17",
+                "Statement": [
+                    {
+                        "Effect": "Allow",
+                        "Action": ["admin:UpdateServiceAccount"]
+                    },
+                    {
+                        "Effect": "Allow",
+                        "Action": ["sts:AssumeRole"],
+                        "Resource": ["arn:aws:s3:::*"]
+                    }
+                ]
+            })
+            .to_string(),
+        ),
+    )
+    .await?;
+    admin_ok(
+        env,
+        http::Method::POST,
+        "/rustfs/admin/v3/idp/builtin/policy/attach",
+        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
+    )
+    .await?;
+    Ok(())
+}
+
+async fn create_service_account_for(
+    env: &RustFSTestEnvironment,
+    parent: &str,
+) -> Result<(String, String), Box<dyn Error + Send + Sync>> {
+    let response = admin_ok(
+        env,
+        http::Method::PUT,
+        "/rustfs/admin/v3/add-service-accounts",
+        Some(serde_json::json!({ "targetUser": parent }).to_string()),
+    )
+    .await?;
+    let response: serde_json::Value = serde_json::from_str(&response)?;
+    let access_key = response["credentials"]["accessKey"]
+        .as_str()
+        .ok_or("service account response should contain credentials.accessKey")?
+        .to_owned();
+    let secret_key = response["credentials"]["secretKey"]
+        .as_str()
+        .ok_or("service account response should contain credentials.secretKey")?
+        .to_owned();
+    Ok((access_key, secret_key))
+}
+
+async fn assert_admin_status(
+    env: &RustFSTestEnvironment,
+    credentials: (&str, &str, Option<&str>),
+    path: &str,
+    body: String,
+    expected: StatusCode,
+    context: &str,
+) -> TestResult {
+    let (access_key, secret_key, session_token) = credentials;
+    let (status, response) =
+        admin_request_with_session_token(&env.url, http::Method::POST, path, Some(body), access_key, secret_key, session_token)
+            .await?;
+    assert_eq!(status, expected, "{context}: got {status}: {response}");
+    if expected == StatusCode::FORBIDDEN {
+        assert!(response.contains("AccessDenied"), "{context}: expected AccessDenied body, got {response}");
+    }
+    Ok(())
+}
+
+#[tokio::test]
+#[serial]
+async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let parent = "updateparent";
+    let parent_secret = "updateparentsecret";
+    let outsider = "updateoutsider";
+    let outsider_secret = "updateoutsidersecret";
+    let ordinary = "updateordinary";
+    let ordinary_secret = "updateordinarysecret";
+    create_user_with_service_account_update_policy(&env, parent, parent_secret, "update-parent-policy").await?;
+    create_user_with_service_account_update_policy(&env, outsider, outsider_secret, "update-outsider-policy").await?;
+    admin_ok(
+        &env,
+        http::Method::POST,
+        "/rustfs/admin/v3/idp/builtin/policy/attach",
+        Some(serde_json::json!({ "policies": ["consoleAdmin"], "user": outsider }).to_string()),
+    )
+    .await?;
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-user?accessKey={ordinary}"),
+        Some(serde_json::json!({ "secretKey": ordinary_secret, "status": "enabled" }).to_string()),
+    )
+    .await?;
+
+    let (target_access_key, _) = create_service_account_for(&env, parent).await?;
+    let target_path = format!("/rustfs/admin/v3/update-service-account?accessKey={target_access_key}");
+
+    assert_admin_status(
+        &env,
+        (&env.access_key, &env.secret_key, None),
+        &target_path,
+        serde_json::json!({}).to_string(),
+        StatusCode::NO_CONTENT,
+        "root no-op update across parents must succeed",
+    )
+    .await?;
+
+    let custom_policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:GetObject"],
+            "Resource": ["arn:aws:s3:::update-scope/*"]
+        }]
+    });
+    assert_admin_status(
+        &env,
+        (&env.access_key, &env.secret_key, None),
+        &target_path,
+        serde_json::json!({ "newPolicy": custom_policy }).to_string(),
+        StatusCode::NO_CONTENT,
+        "root implied-to-custom update across parents must succeed",
+    )
+    .await?;
+
+    assert_admin_status(
+        &env,
+        (parent, parent_secret, None),
+        &target_path,
+        serde_json::json!({ "newDescription": "updated by parent" }).to_string(),
+        StatusCode::NO_CONTENT,
+        "parent with UpdateServiceAccount may update its own service account",
+    )
+    .await?;
+
+    let takeover = serde_json::json!({
+        "newSecretKey": "cross-parent-takeover-secret",
+        "newDescription": "cross-parent takeover"
+    })
+    .to_string();
+    assert_admin_status(
+        &env,
+        (ordinary, ordinary_secret, None),
+        &target_path,
+        takeover.clone(),
+        StatusCode::FORBIDDEN,
+        "ordinary user must not update another parent's service account",
+    )
+    .await?;
+    assert_admin_status(
+        &env,
+        (outsider, outsider_secret, None),
+        &target_path,
+        takeover.clone(),
+        StatusCode::FORBIDDEN,
+        "non-owner consoleAdmin must not update across parents",
+    )
+    .await?;
+
+    let (derived_access_key, derived_secret_key) = create_service_account_for(&env, outsider).await?;
+    assert_admin_status(
+        &env,
+        (&derived_access_key, &derived_secret_key, None),
+        &target_path,
+        takeover.clone(),
+        StatusCode::FORBIDDEN,
+        "service-account credential must not update across parents",
+    )
+    .await?;
+
+    let assumed = build_test_sts_client(&env.url, outsider, outsider_secret, None, "e2e-admin-update-service-account")
+        .assume_role()
+        .role_arn("arn:aws:iam::123456789012:role/update-service-account")
+        .role_session_name("update-service-account-scope")
+        .send()
+        .await?;
+    let temporary = assumed
+        .credentials()
+        .ok_or("AssumeRole response should contain credentials")?;
+    assert_admin_status(
+        &env,
+        (temporary.access_key_id(), temporary.secret_access_key(), Some(temporary.session_token())),
+        &target_path,
+        takeover,
+        StatusCode::FORBIDDEN,
+        "temporary credential must not update across parents",
+    )
+    .await?;
+
+    let info = admin_ok(
+        &env,
+        http::Method::GET,
+        &format!("/rustfs/admin/v3/info-service-account?accessKey={target_access_key}"),
+        None,
+    )
+    .await?;
+    let info: serde_json::Value = serde_json::from_str(&info)?;
+    assert_eq!(
+        info["impliedPolicy"].as_bool(),
+        Some(false),
+        "root update must replace the implied policy with a custom policy"
+    );
+    assert!(
+        info["policy"].as_str().is_some_and(|policy| policy.contains("s3:GetObject")),
+        "custom policy must round-trip through the handler: {info}"
+    );
+    assert_eq!(
+        info["description"].as_str(),
+        Some("updated by parent"),
+        "denied takeover attempts must not mutate target"
+    );
+
+    let (missing_status, missing_body) = admin_request(
+        &env.url,
+        http::Method::POST,
+        "/rustfs/admin/v3/update-service-account?accessKey=missing-service-account",
+        Some(serde_json::json!({}).to_string()),
+        &env.access_key,
+        &env.secret_key,
+    )
+    .await?;
+    assert_eq!(missing_status, StatusCode::NOT_FOUND, "missing target must fail closed: {missing_body}");
+    assert!(
+        missing_body.contains("NoSuchResource"),
+        "missing target must preserve the lookup error: {missing_body}"
+    );
+
+    env.stop_server();
+    Ok(())
+}
+
 /// Full user -> policy -> service-account lifecycle, proving each management
 /// call takes effect on the data plane, not just that the endpoint answers 200.
 #[tokio::test]

crates/e2e_test/src/chaos.rs

@@ -40,6 +40,7 @@ use http::header::{CONTENT_TYPE, HOST};
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
 use s3s::Body;
+use std::collections::BTreeSet;
 use std::error::Error;
 use std::path::{Path, PathBuf};
 use tracing::info;
@@ -48,6 +49,34 @@ use walkdir::WalkDir;
 
 type ChaosResult<T> = Result<T, Box<dyn Error + Send + Sync>>;
 
+/// Physical `xl.meta` and shard-file census for one object version on one disk.
+///
+/// A successful S3 GET only proves that a quorum can serve an object. Replacement
+/// tests need this lower-level record to prove that the rebuilt target holds the
+/// `xl.meta` selected for a specific version and every `part.N` it declares.
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub(crate) struct VersionShardCensus {
+    pub version_id: Option<String>,
+    pub has_xl_meta: bool,
+    pub data_dir: Option<String>,
+    pub expected_part_numbers: BTreeSet<usize>,
+    pub present_part_numbers: BTreeSet<usize>,
+}
+
+impl VersionShardCensus {
+    pub(crate) fn is_complete(&self) -> bool {
+        self.has_xl_meta && self.expected_part_numbers == self.present_part_numbers
+    }
+
+    pub(crate) fn matches_manifest(&self, manifest: &Self) -> bool {
+        self.version_id == manifest.version_id
+            && self.is_complete()
+            && manifest.is_complete()
+            && self.data_dir == manifest.data_dir
+            && self.expected_part_numbers == manifest.expected_part_numbers
+    }
+}
+
 /// Single-node RustFS server with `disk_count` local volume directories that
 /// can be faulted individually while the server is running.
 pub struct DiskFaultHarness {
@@ -219,6 +248,78 @@ impl DiskFaultHarness {
     pub fn object_metadata_exists_on_disk(&self, disk_index: usize, bucket: &str, key: &str) -> bool {
         self.disks[disk_index].join(bucket).join(key).join("xl.meta").is_file()
     }
+
+    /// Census the physical files selected by `version_id` on one disk.
+    ///
+    /// Missing metadata and missing shard files are represented in the returned
+    /// census rather than as an error so callers can poll replacement progress.
+    /// Invalid metadata or an unknown requested version remains an error: treating
+    /// either as an incomplete rebuild would hide corruption or a wrong-version
+    /// recovery result.
+    pub(crate) fn census_object_version(
+        &self,
+        disk_index: usize,
+        bucket: &str,
+        key: &str,
+        version_id: Option<&str>,
+    ) -> ChaosResult<VersionShardCensus> {
+        census_object_version_on_disk(&self.disks[disk_index], bucket, key, version_id)
+    }
+}
+
+/// Census one physical object version without requiring a single-node harness.
+/// Cluster replacement tests use the same evidence as the disk-fault tests.
+pub(crate) fn census_object_version_on_disk(
+    disk: &Path,
+    bucket: &str,
+    key: &str,
+    version_id: Option<&str>,
+) -> ChaosResult<VersionShardCensus> {
+    let version_id = version_id.map(str::to_owned);
+    let object_dir = disk.join(bucket).join(key);
+    let meta_path = object_dir.join("xl.meta");
+    if !meta_path.is_file() {
+        return Ok(VersionShardCensus {
+            version_id,
+            has_xl_meta: false,
+            data_dir: None,
+            expected_part_numbers: BTreeSet::new(),
+            present_part_numbers: BTreeSet::new(),
+        });
+    }
+
+    let metadata = rustfs_filemeta::FileMeta::load(&std::fs::read(&meta_path)?)?;
+    let file_info = metadata.into_fileinfo(bucket, key, version_id.as_deref().unwrap_or_default(), true, false, true)?;
+    let expected_part_numbers = if file_info.inline_data() {
+        BTreeSet::new()
+    } else {
+        file_info.parts.iter().map(|part| part.number).collect()
+    };
+    let data_dir = file_info.data_dir.map(|id| id.to_string());
+    let part_dir = data_dir.as_ref().map_or_else(|| object_dir.clone(), |id| object_dir.join(id));
+    let present_part_numbers = match std::fs::read_dir(&part_dir) {
+        Ok(entries) => entries
+            .filter_map(Result::ok)
+            .filter_map(|entry| {
+                entry
+                    .file_type()
+                    .ok()
+                    .filter(|kind| kind.is_file())
+                    .and_then(|_| entry.file_name().to_str().map(str::to_owned))
+            })
+            .filter_map(|name| name.strip_prefix("part.").and_then(|number| number.parse::<usize>().ok()))
+            .collect(),
+        Err(error) if error.kind() == std::io::ErrorKind::NotFound => BTreeSet::new(),
+        Err(error) => return Err(error.into()),
+    };
+
+    Ok(VersionShardCensus {
+        version_id,
+        has_xl_meta: true,
+        data_dir,
+        expected_part_numbers,
+        present_part_numbers,
+    })
 }
 
 /// `POST` a signed (SigV4, service `s3`) admin request without relying on the

crates/e2e_test/src/common.rs

@@ -137,6 +137,18 @@ pub(crate) async fn signed_s3_request(
     content_type: Option<&str>,
     access_key: &str,
     secret_key: &str,
+) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
+    signed_s3_request_with_session_token(method, url, body, content_type, access_key, secret_key, None).await
+}
+
+async fn signed_s3_request_with_session_token(
+    method: http::Method,
+    url: &str,
+    body: Option<String>,
+    content_type: Option<&str>,
+    access_key: &str,
+    secret_key: &str,
+    session_token: Option<&str>,
 ) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
     let uri = url.parse::<http::Uri>()?;
     let authority = uri.authority().ok_or("S3 URL missing authority")?.to_string();
@@ -150,7 +162,14 @@ pub(crate) async fn signed_s3_request(
     }
 
     let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "S3 request body is too large")?;
-    let signed = sign_v4(request.body(Body::empty())?, content_length, access_key, secret_key, "", "us-east-1");
+    let signed = sign_v4(
+        request.body(Body::empty())?,
+        content_length,
+        access_key,
+        secret_key,
+        session_token.unwrap_or_default(),
+        "us-east-1",
+    );
 
     let mut request = local_http_client().request(method, url);
     for (name, value) in signed.headers() {
@@ -170,10 +189,23 @@ pub(crate) async fn admin_request(
     body: Option<String>,
     access_key: &str,
     secret_key: &str,
+) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
+    admin_request_with_session_token(base_url, method, path_and_query, body, access_key, secret_key, None).await
+}
+
+pub(crate) async fn admin_request_with_session_token(
+    base_url: &str,
+    method: http::Method,
+    path_and_query: &str,
+    body: Option<String>,
+    access_key: &str,
+    secret_key: &str,
+    session_token: Option<&str>,
 ) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
     let url = format!("{base_url}{path_and_query}");
     let content_type = body.as_ref().map(|_| "application/json");
-    let response = signed_s3_request(method, &url, body, content_type, access_key, secret_key).await?;
+    let response =
+        signed_s3_request_with_session_token(method, &url, body, content_type, access_key, secret_key, session_token).await?;
     let status = response.status();
     let body = response.text().await?;
     Ok((status, body))

crates/e2e_test/src/fake_s3_target/mod.rs

@@ -134,6 +134,7 @@ pub struct RequestRecord {
 #[derive(Default)]
 struct ControlState {
     scripts: HashMap<Operation, VecDeque<FaultAction>>,
+    keyed_scripts: HashMap<(Operation, String), VecDeque<FaultAction>>,
     requests: VecDeque<RequestRecord>,
     next_sequence: u64,
 }
@@ -352,25 +353,44 @@ impl FakeS3Target {
         state.buckets.entry(bucket).or_default();
     }
 
+    /// Remove all retained object versions while preserving the bucket.
+    pub fn clear_bucket_objects(&self, bucket: &str) {
+        let mut state = lock(&self.backend.store);
+        let (removed_versions, removed_bytes) = state
+            .buckets
+            .get_mut(bucket)
+            .expect("fake target bucket must exist")
+            .objects
+            .drain()
+            .flat_map(|(_, versions)| versions)
+            .fold((0usize, 0usize), |(count, bytes), version| (count + 1, bytes + version.body.len()));
+        state.total_versions = state
+            .total_versions
+            .checked_sub(removed_versions)
+            .expect("fake target version accounting must not underflow");
+        state.total_bytes = state
+            .total_bytes
+            .checked_sub(removed_bytes)
+            .expect("fake target byte accounting must not underflow");
+    }
+
+    pub fn has_object(&self, bucket: &str, key: &str) -> bool {
+        lock(&self.backend.store)
+            .buckets
+            .get(bucket)
+            .and_then(|bucket| bucket.objects.get(key))
+            .and_then(|versions| versions.last())
+            .is_some_and(|version| !version.delete_marker)
+    }
+
     /// Queue `times` copies of a fault for one operation.
     pub fn inject(&self, operation: Operation, action: FaultAction, times: usize) {
         if times == 0 {
             return;
         }
-        if let FaultAction::SlowDrain { chunk_bytes: 0, .. } = action {
-            panic!("slow-drain chunk size must be non-zero");
-        }
-        match &action {
-            FaultAction::Delay(duration) if *duration > MAX_FAULT_DURATION => {
-                panic!("fault delay must not exceed 30 seconds");
-            }
-            FaultAction::SlowDrain { delay, .. } if *delay >= MAX_FAULT_DURATION => {
-                panic!("slow-drain slice delay must be below 30 seconds");
-            }
-            _ => {}
-        }
+        validate_fault_action(&action);
         let mut state = lock(&self.control);
-        let queued = state.scripts.values().map(VecDeque::len).sum::<usize>();
+        let queued = queued_fault_count(&state);
         if queued.checked_add(times).is_none_or(|total| total > MAX_SCRIPTED_FAULTS) {
             panic!("fake target queues at most 4096 scripted faults");
         }
@@ -381,8 +401,28 @@ impl FakeS3Target {
             .extend(std::iter::repeat_n(action, times));
     }
 
+    /// Queue faults for one exact object key without affecting concurrent requests.
+    pub fn inject_for_key(&self, operation: Operation, key: impl Into<String>, action: FaultAction, times: usize) {
+        if times == 0 {
+            return;
+        }
+        validate_fault_action(&action);
+        let mut state = lock(&self.control);
+        let queued = queued_fault_count(&state);
+        if queued.checked_add(times).is_none_or(|total| total > MAX_SCRIPTED_FAULTS) {
+            panic!("fake target queues at most 4096 scripted faults");
+        }
+        state
+            .keyed_scripts
+            .entry((operation, key.into()))
+            .or_default()
+            .extend(std::iter::repeat_n(action, times));
+    }
+
     pub fn clear_faults(&self) {
-        lock(&self.control).scripts.clear();
+        let mut state = lock(&self.control);
+        state.scripts.clear();
+        state.keyed_scripts.clear();
     }
 
     pub fn requests(&self) -> Vec<RequestRecord> {
@@ -420,6 +460,25 @@ fn lock<T>(mutex: &Mutex<T>) -> MutexGuard<'_, T> {
     mutex.lock().unwrap_or_else(std::sync::PoisonError::into_inner)
 }
 
+fn validate_fault_action(action: &FaultAction) {
+    if let FaultAction::SlowDrain { chunk_bytes: 0, .. } = action {
+        panic!("slow-drain chunk size must be non-zero");
+    }
+    match action {
+        FaultAction::Delay(duration) if *duration > MAX_FAULT_DURATION => {
+            panic!("fault delay must not exceed 30 seconds");
+        }
+        FaultAction::SlowDrain { delay, .. } if *delay >= MAX_FAULT_DURATION => {
+            panic!("slow-drain slice delay must be below 30 seconds");
+        }
+        _ => {}
+    }
+}
+
+fn queued_fault_count(state: &ControlState) -> usize {
+    state.scripts.values().map(VecDeque::len).sum::<usize>() + state.keyed_scripts.values().map(VecDeque::len).sum::<usize>()
+}
+
 #[async_trait]
 impl S3Access for FaultAccess {
     async fn check(&self, context: &mut S3AccessContext<'_>) -> S3Result<()> {
@@ -492,7 +551,12 @@ fn record_request(
     content_length: Option<u64>,
 ) -> Option<RequestFault> {
     let mut state = lock(control);
-    let action = state.scripts.get_mut(&operation).and_then(VecDeque::pop_front);
+    let action = parsed
+        .key
+        .as_ref()
+        .and_then(|key| state.keyed_scripts.get_mut(&(operation, key.clone())))
+        .and_then(VecDeque::pop_front)
+        .or_else(|| state.scripts.get_mut(&operation).and_then(VecDeque::pop_front));
     state.next_sequence += 1;
     let sequence = state.next_sequence;
     if state.requests.len() == MAX_REQUEST_RECORDS {

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -431,4 +431,104 @@ mod tests {
         )
         .into())
     }
+
+    /// Issue #5850: `background-heal/status` must answer while a peer is down.
+    ///
+    /// Exercises the production path in `read_cluster_heal_status` end to end,
+    /// which the unit tests around `merge_peer_heal_statuses` cannot: with one
+    /// node stopped, the endpoint must return 200 with
+    /// `clusterStatusComplete: false` and an explicit `degraded` (or, when
+    /// heal work is known active, `active`) state — never the previous
+    /// cluster-wide 500 — and must return to a complete, non-degraded answer
+    /// once the node rejoins. Reverting either all-or-nothing gate (the
+    /// topology early-return or the merge hard-fail) turns the down-window
+    /// response into a 500 and fails this test.
+    #[tokio::test]
+    #[serial]
+    async fn test_background_heal_status_degrades_while_peer_down_and_recovers_after_rejoin()
+    -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        info!("Issue #5850: background-heal/status must degrade, not 500, while a peer is down");
+
+        let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+        cluster.set_env("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true");
+        cluster.start().await?;
+
+        let status_url = format!("{}/rustfs/admin/v3/background-heal/status", cluster.nodes[0].url);
+        // Owned copies: the closure must not borrow `cluster`, which
+        // stop_node/start_node need mutably between polls.
+        let access_key = cluster.access_key.clone();
+        let secret_key = cluster.secret_key.clone();
+        let fetch_status = || async {
+            let body = signed_admin_post(&status_url, None, &access_key, &secret_key).await?;
+            let json: serde_json::Value =
+                serde_json::from_str(&body).map_err(|err| format!("heal status response is not JSON ({err}): {body}"))?;
+            Ok::<serde_json::Value, Box<dyn Error + Send + Sync>>(json)
+        };
+
+        // Healthy cluster: the answer must be definitive. Poll briefly — the
+        // peer grid may still be settling right after start().
+        let mut healthy = fetch_status().await?;
+        for _ in 0..30 {
+            if healthy["clusterStatusComplete"] == serde_json::Value::Bool(true) {
+                break;
+            }
+            sleep(Duration::from_secs(1)).await;
+            healthy = fetch_status().await?;
+        }
+        assert_eq!(
+            healthy["clusterStatusComplete"],
+            serde_json::Value::Bool(true),
+            "healthy cluster should report a complete heal status: {healthy}"
+        );
+
+        cluster.stop_node(1)?;
+
+        // While the peer is down every response must stay 200 (signed_admin_post
+        // fails on any non-2xx, so the old 500 fails the test immediately) and
+        // must degrade to an explicitly-partial answer. The peer query timeout
+        // is 5 s, so a couple of polls are enough for the dead peer to surface.
+        let mut degraded = serde_json::Value::Null;
+        for _ in 0..30 {
+            degraded = fetch_status().await?;
+            if degraded["clusterStatusComplete"] == serde_json::Value::Bool(false) {
+                break;
+            }
+            sleep(Duration::from_secs(1)).await;
+        }
+        assert_eq!(
+            degraded["clusterStatusComplete"],
+            serde_json::Value::Bool(false),
+            "heal status must mark itself partial while a peer is down: {degraded}"
+        );
+        let state = degraded["state"].as_str().unwrap_or_default();
+        assert!(
+            state == "degraded" || state == "active",
+            "a partial answer must be labeled degraded (or active for known work), got {state:?}: {degraded}"
+        );
+
+        cluster.start_node(1).await?;
+
+        // After the rejoin the endpoint must return to a definitive answer.
+        let mut recovered = serde_json::Value::Null;
+        for _ in 0..60 {
+            recovered = fetch_status().await?;
+            if recovered["clusterStatusComplete"] == serde_json::Value::Bool(true) {
+                break;
+            }
+            sleep(Duration::from_secs(1)).await;
+        }
+        assert_eq!(
+            recovered["clusterStatusComplete"],
+            serde_json::Value::Bool(true),
+            "heal status should be complete again after the node rejoined: {recovered}"
+        );
+        assert_ne!(
+            recovered["state"].as_str().unwrap_or_default(),
+            "degraded",
+            "a complete answer must not be labeled degraded: {recovered}"
+        );
+
+        Ok(())
+    }
 }

crates/e2e_test/src/reliability_disk_fault_test.rs

@@ -22,15 +22,16 @@
 
 #[cfg(test)]
 mod tests {
-    use crate::chaos::{DiskFaultHarness, signed_admin_post};
+    use crate::chaos::{DiskFaultHarness, VersionShardCensus, signed_admin_post};
     use crate::common::init_logging;
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+    use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
     use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::HashSet;
-    use tokio::time::{Duration, sleep, timeout};
+    use std::error::Error;
+    use tokio::time::{Duration, Instant, interval, timeout};
     use tracing::info;
 
     const GET_TIMEOUT: Duration = Duration::from_secs(60);
@@ -271,12 +272,17 @@ mod tests {
         put_and_record(&client, bucket, "heal/nested/large.bin", payload(2 * 1024 * 1024, 34), &mut manifest).await?;
 
         verify_manifest(&client, bucket, &manifest, "baseline before disk replacement").await?;
-        for (key, _) in &manifest {
-            assert!(
-                harness.object_metadata_exists_on_disk(0, bucket, key),
-                "disk0 should hold xl.meta for {key} before replacement"
-            );
-        }
+        let manifest_keys = manifest.iter().map(|(key, _)| key.clone()).collect::<Vec<_>>();
+        let target_manifest: Vec<(String, VersionShardCensus)> = manifest_keys
+            .iter()
+            .map(|key| {
+                let census = harness.census_object_version(0, bucket, key, None)?;
+                if !census.is_complete() {
+                    return Err(format!("disk 0 has incomplete physical census for {key}: {census:?}").into());
+                }
+                Ok((key.clone(), census))
+            })
+            .collect::<Result<_, Box<dyn Error + Send + Sync>>>()?;
 
         harness.kill_server();
         harness.replace_disk_with_empty(0)?;
@@ -287,21 +293,112 @@ mod tests {
         signed_admin_post(&heal_url, Some(heal_body), &harness.env.access_key, &harness.env.secret_key).await?;
 
         let client = harness.env.create_s3_client();
-        let mut remaining: HashSet<String> = manifest.iter().map(|(key, _)| key.clone()).collect();
+        let mut remaining: HashSet<String> = manifest_keys.iter().cloned().collect();
         let heal_timeout_secs = std::env::var("RUSTFS_RELIABILITY_HEAL_TIMEOUT_SECS")
             .ok()
             .and_then(|value| value.parse::<u64>().ok())
             .unwrap_or(120);
+        let deadline = Instant::now() + Duration::from_secs(heal_timeout_secs);
+        let mut retry = interval(Duration::from_secs(1));
 
-        for _ in 0..heal_timeout_secs {
-            remaining.retain(|key| !harness.object_metadata_exists_on_disk(0, bucket, key));
+        loop {
+            remaining.retain(|key| {
+                let expected = target_manifest
+                    .iter()
+                    .find(|(manifest_key, _)| manifest_key == key)
+                    .map(|(_, manifest)| manifest)
+                    .expect("every key has a physical manifest");
+                harness
+                    .census_object_version(0, bucket, key, None)
+                    .map(|census| !census.matches_manifest(expected))
+                    .unwrap_or(true)
+            });
             if remaining.is_empty() {
                 verify_manifest(&client, bucket, &manifest, "after fresh-disk heal completed").await?;
                 return Ok(());
             }
-            sleep(Duration::from_secs(1)).await;
+            if Instant::now() >= deadline {
+                break;
+            }
+            retry.tick().await;
         }
 
         Err(format!("fresh-disk heal did not rebuild {remaining:?} on the replaced disk within {heal_timeout_secs}s").into())
     }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_versioned_shard_census_selects_each_version_data_dir() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        info!("Reliability: physical shard census selects the requested object version");
+
+        let mut harness = DiskFaultHarness::new(4).await?;
+        harness.start_server().await?;
+        let client = harness.env.create_s3_client();
+        let bucket = "reliability-versioned-census";
+        let key = "versions/large.bin";
+        client.create_bucket().bucket(bucket).send().await?;
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await?;
+
+        let first = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from(payload(256 * 1024, 41)))
+            .send()
+            .await?;
+        let first_version = first.version_id().ok_or("first PUT did not return a version ID")?;
+        let second = client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from(payload(256 * 1024, 42)))
+            .send()
+            .await?;
+        let second_version = second.version_id().ok_or("second PUT did not return a version ID")?;
+        let delete = client.delete_object().bucket(bucket).key(key).send().await?;
+        let delete_version = delete.version_id().ok_or("delete marker did not return a version ID")?;
+
+        let first_census = harness.census_object_version(0, bucket, key, Some(first_version))?;
+        let second_census = harness.census_object_version(0, bucket, key, Some(second_version))?;
+        let delete_census = harness.census_object_version(0, bucket, key, Some(delete_version))?;
+        assert!(
+            first_census.is_complete(),
+            "first version physical census is incomplete: {first_census:?}"
+        );
+        assert!(
+            second_census.is_complete(),
+            "second version physical census is incomplete: {second_census:?}"
+        );
+        assert_ne!(
+            first_census.data_dir, second_census.data_dir,
+            "distinct object versions must select distinct physical data directories"
+        );
+        assert_eq!(
+            first_census.expected_part_numbers, second_census.expected_part_numbers,
+            "same single-part shape should expose the same part numbers"
+        );
+        assert!(
+            delete_census.is_complete(),
+            "delete marker physical census is incomplete: {delete_census:?}"
+        );
+        assert!(
+            delete_census.expected_part_numbers.is_empty(),
+            "delete marker must not declare object shards: {delete_census:?}"
+        );
+        assert!(
+            delete_census.present_part_numbers.is_empty(),
+            "delete marker must not select stale object shards: {delete_census:?}"
+        );
+        Ok(())
+    }
 }

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -848,6 +848,13 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
+    async fn replacement_recovery_status(
+        &self,
+        _request: Request<rustfs_protos::proto_gen::node_service::ReplacementRecoveryStatusRequest>,
+    ) -> Result<Response<rustfs_protos::proto_gen::node_service::ReplacementRecoveryStatusResponse>, Status> {
+        Err(Status::unimplemented("lock-only test server"))
+    }
+
     async fn get_metacache_listing(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::GetMetacacheListingRequest>,

crates/e2e_test/src/replication_extension_test.rs

@@ -16,7 +16,9 @@ use crate::common::{
     RustFSTestEnvironment, awscurl_available, awscurl_post_sts_form_urlencoded, init_logging, local_http_client,
     replication_fast_env, rustfs_binary_path,
 };
-use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, Operation as FakeTargetOperation};
+use crate::fake_s3_target::{
+    FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation,
+};
 use crate::kms::common::{create_key_with_specific_id, sse_customer_key_md5_base64};
 use crate::storage_api::replication_extension::BucketTargetSys;
 use aws_sdk_s3::config::{Credentials, Region};
@@ -379,6 +381,10 @@ struct ReplicationResetStatusTarget {
     reset_id: String,
     #[serde(rename = "resyncStatus", default)]
     status: String,
+    #[serde(rename = "replicationCount", default)]
+    replicated_count: i64,
+    #[serde(rename = "object", default)]
+    object: String,
 }
 
 async fn signed_request(
@@ -1447,10 +1453,6 @@ where
     }
 }
 
-async fn wait_for_replication_failure_event(response: reqwest::Response, expected_key: &str) -> TestResult {
-    wait_for_replication_failure_event_stream(response.bytes_stream(), expected_key, Duration::from_secs(30)).await
-}
-
 fn target_history_contains_key(output: &ListObjectVersionsOutput, key: &str) -> bool {
     output.versions().iter().any(|version| version.key() == Some(key))
         || output.delete_markers().iter().any(|marker| marker.key() == Some(key))
@@ -1507,41 +1509,24 @@ async fn assert_failed_replication_stays_absent_for(
     }
 }
 
-async fn subscribe_to_replication_failure(
-    env: &RustFSTestEnvironment,
-    bucket: &str,
-    key: &str,
-) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
-    let url = format!(
-        "{}/{bucket}?events={}&prefix={}&ping=1",
-        env.url,
-        urlencoding::encode(REPLICATION_FAILED_EVENT),
-        urlencoding::encode(key)
-    );
-    let response = timeout(
-        Duration::from_secs(30),
-        signed_request(http::Method::GET, &url, &env.access_key, &env.secret_key, None, None),
-    )
-    .await
-    .map_err(|_| "replication failure event subscription did not respond within 30 seconds")??;
-    if response.status() != StatusCode::OK {
-        return Err(format!("failed to subscribe to replication failure events: {}", response.status()).into());
-    }
-    Ok(response)
-}
-
 async fn build_sse_replication_pair(
     label: &str,
-    enable_kms: bool,
+    source_kms: bool,
+    target_kms: bool,
 ) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment, String, String), Box<dyn Error + Send + Sync>> {
     let mut source_env = RustFSTestEnvironment::new().await?;
     let mut target_env = RustFSTestEnvironment::new().await?;
     let source_kms_key_dir = format!("{}/kms-keys", source_env.temp_dir);
     let target_kms_key_dir = format!("{}/kms-keys", target_env.temp_dir);
-    if enable_kms {
+    if source_kms {
         fs::create_dir_all(&source_kms_key_dir).await?;
-        fs::create_dir_all(&target_kms_key_dir).await?;
         create_key_with_specific_id(&source_kms_key_dir, REPL17_KMS_KEY_ID).await?;
+    }
+    // The two sites share a key id but never key material: each side generates
+    // its own key, which is exactly the independent-KMS topology managed-SSE
+    // replication must survive (target re-encrypts with its own envelope).
+    if target_kms {
+        fs::create_dir_all(&target_kms_key_dir).await?;
         create_key_with_specific_id(&target_kms_key_dir, REPL17_KMS_KEY_ID).await?;
     }
 
@@ -1549,7 +1534,7 @@ async fn build_sse_replication_pair(
     source_process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
     source_process_env.extend_from_slice(FAST_SCANNER_ENV);
     source_process_env.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
-    if enable_kms {
+    if source_kms {
         source_process_env.extend_from_slice(&[
             ("RUSTFS_KMS_ENABLE", "true"),
             ("RUSTFS_KMS_BACKEND", "local"),
@@ -1557,15 +1542,15 @@ async fn build_sse_replication_pair(
             ("RUSTFS_KMS_DEFAULT_KEY_ID", REPL17_KMS_KEY_ID),
             ("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true"),
             // Per-key KMS authorization is on so this contract is pinned in the
-            // configuration replication will eventually ship with: the replication
-            // worker carries no request identity and must stay exempt.
+            // configuration replication ships with: the replication worker
+            // carries no request identity and must stay exempt.
             ("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"),
         ]);
     }
     source_env.start_rustfs_server_with_env(vec![], &source_process_env).await?;
 
     let mut target_process_env = vec![("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")];
-    if enable_kms {
+    if target_kms {
         target_process_env.extend_from_slice(&[
             ("RUSTFS_KMS_ENABLE", "true"),
             ("RUSTFS_KMS_BACKEND", "local"),
@@ -1593,13 +1578,12 @@ async fn build_sse_replication_pair(
     Ok((source_env, target_env, source_bucket, target_bucket))
 }
 
-async fn assert_managed_sse_replication_fails_explicitly(label: &str, kms: bool) -> TestResult {
-    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair(label, true).await?;
+async fn assert_managed_sse_replicates_and_reencrypts(label: &str, kms: bool) -> TestResult {
+    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair(label, true, true).await?;
     let source_client = source_env.create_s3_client();
     let target_client = target_env.create_s3_client();
     let key = format!("{label}-contract.txt");
     let body = format!("repl-17 {label} payload").into_bytes();
-    let failure_events = subscribe_to_replication_failure(&source_env, &source_bucket, &key).await?;
 
     let encryption = if kms {
         ServerSideEncryption::AwsKms
@@ -1621,20 +1605,41 @@ async fn assert_managed_sse_replication_fails_explicitly(label: &str, kms: bool)
 
     let source = source_client.get_object().bucket(&source_bucket).key(&key).send().await?;
     assert_eq!(source.server_side_encryption(), Some(&encryption));
+    let source_etag = source.e_tag().map(str::to_string);
     assert_eq!(source.body.collect().await?.into_bytes().as_ref(), body.as_slice());
 
-    wait_for_replication_failure_event(failure_events, &key).await?;
-    wait_for_source_replication_status(&source_client, &source_bucket, &key, "FAILED", false).await?;
-    assert_failed_replication_stays_absent_for(
-        &source_client,
-        &source_bucket,
-        &target_client,
-        &target_bucket,
-        &key,
-        false,
-        Duration::from_secs(5),
-    )
-    .await?;
+    wait_for_source_replication_status(&source_client, &source_bucket, &key, "COMPLETED", false).await?;
+
+    // The target sits on an independent KMS (same key id, different material),
+    // so a successful plain GET proves the replica's envelope belongs to the
+    // target's KMS: a forwarded source envelope could never unwrap here.
+    let replica = target_client.get_object().bucket(&target_bucket).key(&key).send().await?;
+    assert_eq!(replica.server_side_encryption(), Some(&encryption));
+    let replica_version_id = replica.version_id().map(str::to_string);
+    let replica_etag = replica.e_tag().map(str::to_string);
+    assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
+
+    // The replica must keep the source ETag; otherwise every replication HEAD
+    // comparison sees a mismatch and re-replicates the object forever.
+    assert_eq!(replica_etag, source_etag, "replica ETag must match the source ETag");
+
+    // Spanning several fast-scanner cycles, the replica must stay the same
+    // version: a second version appearing here means the ETag comparison did
+    // not converge and the scanner is re-driving the object.
+    sleep(Duration::from_secs(5)).await;
+    let versions = target_client
+        .list_object_versions()
+        .bucket(&target_bucket)
+        .prefix(&key)
+        .send()
+        .await?;
+    let replica_versions: Vec<_> = versions.versions().iter().filter(|v| v.key() == Some(key.as_str())).collect();
+    assert_eq!(replica_versions.len(), 1, "replica must not accumulate versions from re-replication");
+    assert_eq!(
+        replica_versions[0].version_id().map(str::to_string),
+        replica_version_id,
+        "replica version must stay stable across scanner cycles"
+    );
 
     Ok(())
 }
@@ -4294,6 +4299,8 @@ async fn test_single_bucket_multipart_replication_fans_out_to_multiple_targets()
         .create_multipart_upload()
         .bucket(source_bucket)
         .key(object_key)
+        .content_type("application/x-fanout")
+        .metadata("app", "fanout")
         .send()
         .await?;
     let upload_id = created.upload_id().ok_or("missing multipart upload id")?.to_string();
@@ -4341,15 +4348,17 @@ async fn test_single_bucket_multipart_replication_fans_out_to_multiple_targets()
         wait_for_replicated_sha256(&target_client_b, target_bucket_b, object_key, expected_sha256),
     )?;
 
-    let target_etag_a = target_client_a
+    let target_head_a = target_client_a
         .head_object()
         .bucket(target_bucket_a)
         .key(object_key)
         .send()
-        .await?
-        .e_tag()
-        .ok_or("first target omitted ETag")?
-        .to_string();
+        .await?;
+    // Multipart replicas carry their metadata through CreateMultipartUpload;
+    // this pins the plaintext side of the multipart header fix.
+    assert_eq!(target_head_a.content_type(), Some("application/x-fanout"));
+    assert_eq!(target_head_a.metadata().and_then(|m| m.get("app").map(String::as_str)), Some("fanout"));
+    let target_etag_a = target_head_a.e_tag().ok_or("first target omitted ETag")?.to_string();
     let target_etag_b = target_client_b
         .head_object()
         .bucket(target_bucket_b)
@@ -4411,22 +4420,23 @@ async fn test_repl17_failure_observation_helpers() -> TestResult {
     Ok(())
 }
 
-/// backlog#1147 repl-17: SSE-C currently fails replication explicitly. Pin the
-/// observed contract: the source remains decryptable with its customer key,
-/// reports FAILED, emits the standard failure event, and leaves no target data.
+/// backlog#1147 repl-17 / backlog#1783: SSE-C objects replicate as ciphertext
+/// passthrough — the source cannot decrypt them (no customer key server-side),
+/// so the stored ciphertext and its encryption metadata travel verbatim and
+/// the replica is decryptable only with the original customer key. The
+/// backlog#1291 property still holds: never a silent plaintext replica.
 #[tokio::test]
 #[serial]
 async fn test_bucket_replication_sse_c_contract() -> TestResult {
     init_logging();
 
-    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair("ssec", false).await?;
+    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair("ssec", false, false).await?;
     let source_client = source_env.create_s3_client();
     let target_client = target_env.create_s3_client();
     let key = "ssec-contract.txt";
     let body = b"repl-17 SSE-C payload";
     let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
     let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
-    let failure_events = subscribe_to_replication_failure(&source_env, &source_bucket, key).await?;
 
     source_client
         .put_object()
@@ -4448,73 +4458,228 @@ async fn test_bucket_replication_sse_c_contract() -> TestResult {
         .sse_customer_key_md5(&customer_key_md5)
         .send()
         .await?;
+    let source_etag = source.e_tag().map(str::to_string);
     assert_eq!(source.body.collect().await?.into_bytes().as_ref(), body);
 
-    wait_for_replication_failure_event(failure_events, key).await?;
-    wait_for_source_replication_status(&source_client, &source_bucket, key, "FAILED", true).await?;
+    wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", true).await?;
+
+    // The replica is readable only with the original customer key.
+    let replica = target_client
+        .get_object()
+        .bucket(&target_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
+    let replica_etag = replica.e_tag().map(str::to_string);
+    assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body);
+    assert_eq!(replica_etag, source_etag, "replica ETag must match the source ETag");
+
+    // Without the customer key the replica must not be readable — the direct
+    // detection point for a silent-plaintext replica (backlog#1291).
+    let plain_read = target_client.get_object().bucket(&target_bucket).key(key).send().await;
+    assert!(plain_read.is_err(), "SSE-C replica must not be readable without the customer key");
+
+    // A wrong customer key must fail too.
+    let wrong_key = BASE64_STANDARD.encode("99999999999999999999999999999999");
+    let wrong_key_md5 = sse_customer_key_md5_base64("99999999999999999999999999999999");
+    let wrong_read = target_client
+        .get_object()
+        .bucket(&target_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&wrong_key)
+        .sse_customer_key_md5(&wrong_key_md5)
+        .send()
+        .await;
+    assert!(wrong_read.is_err(), "SSE-C replica must reject a wrong customer key");
+
+    Ok(())
+}
+
+/// backlog#1783: SSE-C multipart objects pass through as ciphertext part by
+/// part — part boundaries and the encrypted-multipart marker survive so the
+/// replica decrypts each part with its part-derived nonce.
+#[tokio::test]
+#[serial]
+async fn test_bucket_replication_sse_c_multipart_passthrough() -> TestResult {
+    init_logging();
+
+    const PART_SIZE: usize = 5 * 1024 * 1024;
+    const PART_COUNT: usize = 3;
+
+    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair("ssec-mp", false, false).await?;
+    let source_client = source_env.create_s3_client();
+    let target_client = target_env.create_s3_client();
+    let key = "ssec-mp-contract.bin";
+    let customer_key = BASE64_STANDARD.encode(REPL17_SSEC_KEY);
+    let customer_key_md5 = sse_customer_key_md5_base64(REPL17_SSEC_KEY);
+
+    let created = source_client
+        .create_multipart_upload()
+        .bucket(&source_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    let upload_id = created.upload_id().ok_or("missing multipart upload id")?.to_string();
+
+    let mut completed_parts = Vec::with_capacity(PART_COUNT);
+    let mut payload = Vec::with_capacity(PART_SIZE * PART_COUNT);
+    for part_number in 1..=PART_COUNT {
+        let part = vec![u8::try_from(part_number)?; PART_SIZE];
+        payload.extend_from_slice(&part);
+        let uploaded = source_client
+            .upload_part()
+            .bucket(&source_bucket)
+            .key(key)
+            .upload_id(&upload_id)
+            .part_number(i32::try_from(part_number)?)
+            .body(ByteStream::from(part))
+            .sse_customer_algorithm("AES256")
+            .sse_customer_key(&customer_key)
+            .sse_customer_key_md5(&customer_key_md5)
+            .send()
+            .await?;
+        completed_parts.push(
+            CompletedPart::builder()
+                .part_number(i32::try_from(part_number)?)
+                .set_e_tag(uploaded.e_tag().map(str::to_string))
+                .build(),
+        );
+    }
+    source_client
+        .complete_multipart_upload()
+        .bucket(&source_bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+
+    wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", true).await?;
+
+    let source_head = source_client
+        .head_object()
+        .bucket(&source_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    let replica = target_client
+        .get_object()
+        .bucket(&target_bucket)
+        .key(key)
+        .sse_customer_algorithm("AES256")
+        .sse_customer_key(&customer_key)
+        .sse_customer_key_md5(&customer_key_md5)
+        .send()
+        .await?;
+    // The replica must carry the SSE-C marker and the source's multipart ETag
+    // (its -N suffix also pins that the part structure survived).
+    assert_eq!(replica.sse_customer_algorithm(), Some("AES256"));
+    assert_eq!(replica.e_tag(), source_head.e_tag(), "replica must keep the source multipart ETag");
+    let replica_version_id = replica.version_id().map(str::to_string);
+    assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), payload.as_slice());
+
+    let plain_read = target_client.get_object().bucket(&target_bucket).key(key).send().await;
+    assert!(
+        plain_read.is_err(),
+        "SSE-C multipart replica must not be readable without the customer key"
+    );
+
+    // Stability across scanner cycles: convergence must hold for passthrough.
+    sleep(Duration::from_secs(5)).await;
+    let versions = target_client
+        .list_object_versions()
+        .bucket(&target_bucket)
+        .prefix(key)
+        .send()
+        .await?;
+    let replica_versions: Vec<_> = versions.versions().iter().filter(|v| v.key() == Some(key)).collect();
+    assert_eq!(replica_versions.len(), 1, "SSE-C replica must not accumulate versions");
+    assert_eq!(replica_versions[0].version_id().map(str::to_string), replica_version_id);
+
+    Ok(())
+}
+
+/// backlog#1147 repl-17 / backlog#1783: SSE-S3 objects replicate by decrypting
+/// at the source and re-encrypting on the target with the target's own KMS.
+/// The property backlog#1291 pinned — never a silent plaintext replica — still
+/// holds, but the expectation flips from FAILED to a converged, decryptable
+/// replica: COMPLETED status, byte-identical plain GET on the target
+/// (independent KMS, so success proves target-owned envelopes), preserved
+/// source ETag, and a version that stays stable across scanner cycles.
+#[tokio::test]
+#[serial]
+async fn test_bucket_replication_sse_s3_contract() -> TestResult {
+    init_logging();
+    assert_managed_sse_replicates_and_reencrypts("sse-s3", false).await
+}
+
+/// backlog#1783: when the target site has no KMS, managed-SSE replication must
+/// fail closed — replication FAILED, and no plaintext (or any) replica ever
+/// materializes on the target.
+#[tokio::test]
+#[serial]
+async fn test_bucket_replication_sse_s3_fails_closed_without_target_kms() -> TestResult {
+    init_logging();
+
+    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair("sse-nokms", true, false).await?;
+    let source_client = source_env.create_s3_client();
+    let target_client = target_env.create_s3_client();
+    let key = "sse-nokms-contract.txt";
+    let body = b"repl-17 sse target-without-kms payload".to_vec();
+
+    source_client
+        .put_object()
+        .bucket(&source_bucket)
+        .key(key)
+        .body(ByteStream::from(body.clone()))
+        .server_side_encryption(ServerSideEncryption::Aes256)
+        .send()
+        .await?;
+
+    wait_for_source_replication_status(&source_client, &source_bucket, key, "FAILED", false).await?;
     assert_failed_replication_stays_absent_for(
         &source_client,
         &source_bucket,
         &target_client,
         &target_bucket,
         key,
-        true,
+        false,
         Duration::from_secs(5),
     )
     .await?;
 
-    target_client
-        .put_object()
-        .bucket(&target_bucket)
-        .key(key)
-        .body(ByteStream::from_static(b"observer negative-path fixture"))
-        .send()
-        .await?;
-    target_client.delete_object().bucket(&target_bucket).key(key).send().await?;
-    let history_error = assert_failed_replication_stays_absent_for(
-        &source_client,
-        &source_bucket,
-        &target_client,
-        &target_bucket,
-        key,
-        true,
-        Duration::ZERO,
-    )
-    .await
-    .expect_err("target history must violate the failed replication contract");
-    assert!(history_error.to_string().contains("created target history"));
+    let source = source_client.get_object().bucket(&source_bucket).key(key).send().await?;
+    assert_eq!(source.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
+    assert_eq!(source.body.collect().await?.into_bytes().as_ref(), body.as_slice());
 
     Ok(())
 }
 
-/// backlog#1147 repl-17 / backlog#1291: SSE-S3 must fail closed until managed
-/// encryption is supported on the target. The silent plaintext replication
-/// that originally kept this test ignored was fixed by the fail-closed gate in
-/// `crates/ecstore/src/bucket/replication/replication_target_boundary.rs`
-/// (all replication modes route through it), so this now pins the current
-/// fail-closed contract: FAILED status, failure event, readable source, and a
-/// stable absence of all target versions.
+/// P1-22 stage 0 → backlog#1783: the existing-object resync path re-drives
+/// managed-SSE objects through the same target boundary as live replication.
+/// After the live pass completes, a resync over the bucket must converge —
+/// the ETag comparison sees the preserved source ETag on the replica and does
+/// not rewrite it, so the replica's version stays stable through the resync.
 #[tokio::test]
 #[serial]
-async fn test_bucket_replication_sse_s3_contract() -> TestResult {
-    init_logging();
-    assert_managed_sse_replication_fails_explicitly("sse-s3", false).await
-}
-
-/// P1-22 stage 0: the existing-object resync path must fail closed for
-/// managed-SSE objects exactly like inline replication (which
-/// `test_bucket_replication_sse_s3_contract` pins, including the scanner heal
-/// re-drive). Resync re-drives every object version through the same
-/// fail-closed target boundary, so a resync over an encrypted bucket must
-/// terminate without ever materializing a plaintext (or unreadable) replica;
-/// the post-resync stays-absent window also spans further fast-scanner heal
-/// cycles.
-#[tokio::test]
-#[serial]
-async fn test_bucket_replication_sse_s3_resync_stays_fail_closed() -> TestResult {
+async fn test_bucket_replication_sse_s3_resync_converges() -> TestResult {
     init_logging();
 
-    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair("sse-resync", true).await?;
+    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair("sse-resync", true, true).await?;
     let source_client = source_env.create_s3_client();
     let target_client = target_env.create_s3_client();
     let key = "sse-resync-contract.txt";
@@ -4528,29 +4693,33 @@ async fn test_bucket_replication_sse_s3_resync_stays_fail_closed() -> TestResult
         .server_side_encryption(ServerSideEncryption::Aes256)
         .send()
         .await?;
-    wait_for_source_replication_status(&source_client, &source_bucket, key, "FAILED", false).await?;
+    wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", false).await?;
 
-    // Resync: drive the existing-object resync path over the failed object.
+    let replica = target_client.get_object().bucket(&target_bucket).key(key).send().await?;
+    assert_eq!(replica.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
+    let replica_version_id = replica.version_id().map(str::to_string);
+    assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body.as_slice());
+
+    // Resync: drive the existing-object resync path over the replicated object.
     let (target_arn, reset_id) = start_bucket_replication_reset(&source_env, &source_bucket).await?;
     let terminal = wait_for_replication_reset_target(&source_env, &source_bucket, &target_arn, |target| {
         target.reset_id == reset_id && matches!(target.status.as_str(), "Completed" | "Failed")
     })
     .await?;
     assert_eq!(terminal.reset_id, reset_id);
+    assert_eq!(terminal.status, "Completed", "resync over a managed-SSE bucket must complete");
+
+    // The resync pass must not have rewritten the converged replica.
+    let versions = target_client
+        .list_object_versions()
+        .bucket(&target_bucket)
+        .prefix(key)
+        .send()
+        .await?;
+    let replica_versions: Vec<_> = versions.versions().iter().filter(|v| v.key() == Some(key)).collect();
+    assert_eq!(replica_versions.len(), 1, "resync must not create additional replica versions");
+    assert_eq!(replica_versions[0].version_id().map(str::to_string), replica_version_id);
 
-    // The resync pass must have failed closed: still no target version (the
-    // window also spans further scanner heal cycles), and the source object
-    // stays readable and encrypted.
-    assert_failed_replication_stays_absent_for(
-        &source_client,
-        &source_bucket,
-        &target_client,
-        &target_bucket,
-        key,
-        false,
-        Duration::from_secs(5),
-    )
-    .await?;
     let source = source_client.get_object().bucket(&source_bucket).key(key).send().await?;
     assert_eq!(source.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
     assert_eq!(source.body.collect().await?.into_bytes().as_ref(), body.as_slice());
@@ -4558,14 +4727,106 @@ async fn test_bucket_replication_sse_s3_resync_stays_fail_closed() -> TestResult
     Ok(())
 }
 
-/// backlog#1147 repl-17: SSE-KMS currently fails closed rather than creating an
-/// unreadable replica; the shared helper verifies FAILED, the failure event,
-/// source readability, and a stable absence of all target versions.
+/// backlog#1147 repl-17 / backlog#1783: SSE-KMS replicates like SSE-S3 — the
+/// source key id never crosses sites (only the aws:kms intent), and the target
+/// re-encrypts under its own default key. The independent-KMS pair proves the
+/// replica's envelope is target-owned.
 #[tokio::test]
 #[serial]
-async fn test_bucket_replication_sse_kms_failure_contract() -> TestResult {
+async fn test_bucket_replication_sse_kms_contract() -> TestResult {
     init_logging();
-    assert_managed_sse_replication_fails_explicitly("sse-kms", true).await
+    assert_managed_sse_replicates_and_reencrypts("sse-kms", true).await
+}
+
+/// backlog#1783: managed-SSE multipart objects keep their part structure and
+/// their metadata through replication. CreateMultipartUpload on the target
+/// carries the full header set (SSE intent, content-type, user metadata) and
+/// the completed replica preserves the source's multipart ETag.
+#[tokio::test]
+#[serial]
+async fn test_bucket_replication_sse_s3_multipart_reencrypts() -> TestResult {
+    init_logging();
+
+    const PART_SIZE: usize = 5 * 1024 * 1024;
+    const PART_COUNT: usize = 3;
+
+    let (source_env, target_env, source_bucket, target_bucket) = build_sse_replication_pair("sse-mp", true, true).await?;
+    let source_client = source_env.create_s3_client();
+    let target_client = target_env.create_s3_client();
+    let key = "sse-mp-contract.bin";
+
+    let created = source_client
+        .create_multipart_upload()
+        .bucket(&source_bucket)
+        .key(key)
+        .content_type("application/x-repl17")
+        .metadata("app", "repl17")
+        .server_side_encryption(ServerSideEncryption::Aes256)
+        .send()
+        .await?;
+    let upload_id = created.upload_id().ok_or("missing multipart upload id")?.to_string();
+
+    let mut completed_parts = Vec::with_capacity(PART_COUNT);
+    let mut payload = Vec::with_capacity(PART_SIZE * PART_COUNT);
+    for part_number in 1..=PART_COUNT {
+        let part = vec![u8::try_from(part_number)?; PART_SIZE];
+        payload.extend_from_slice(&part);
+        let uploaded = source_client
+            .upload_part()
+            .bucket(&source_bucket)
+            .key(key)
+            .upload_id(&upload_id)
+            .part_number(i32::try_from(part_number)?)
+            .body(ByteStream::from(part))
+            .send()
+            .await?;
+        completed_parts.push(
+            CompletedPart::builder()
+                .part_number(i32::try_from(part_number)?)
+                .set_e_tag(uploaded.e_tag().map(str::to_string))
+                .build(),
+        );
+    }
+    source_client
+        .complete_multipart_upload()
+        .bucket(&source_bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
+        .send()
+        .await?;
+
+    wait_for_source_replication_status(&source_client, &source_bucket, key, "COMPLETED", false).await?;
+
+    let source_head = source_client.head_object().bucket(&source_bucket).key(key).send().await?;
+    let replica = target_client.get_object().bucket(&target_bucket).key(key).send().await?;
+    assert_eq!(replica.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
+    assert_eq!(replica.e_tag(), source_head.e_tag(), "replica must keep the source multipart ETag");
+    assert_eq!(
+        replica.last_modified(),
+        source_head.last_modified(),
+        "replica must keep the source mtime or the multipart HEAD comparison never converges"
+    );
+    assert_eq!(replica.content_type(), Some("application/x-repl17"));
+    assert_eq!(replica.metadata().and_then(|m| m.get("app").map(String::as_str)), Some("repl17"));
+    let replica_version_id = replica.version_id().map(str::to_string);
+    assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), payload.as_slice());
+
+    // The multipart replica must also stay stable across scanner cycles: a
+    // rewritten or additional version means ETag/mtime convergence failed and
+    // the scanner keeps re-driving the object.
+    sleep(Duration::from_secs(5)).await;
+    let versions = target_client
+        .list_object_versions()
+        .bucket(&target_bucket)
+        .prefix(key)
+        .send()
+        .await?;
+    let replica_versions: Vec<_> = versions.versions().iter().filter(|v| v.key() == Some(key)).collect();
+    assert_eq!(replica_versions.len(), 1, "multipart replica must not accumulate versions");
+    assert_eq!(replica_versions[0].version_id().map(str::to_string), replica_version_id);
+
+    Ok(())
 }
 
 /// backlog#1147 repl-5, scenario (a) — target outage + recovery (rustfs#3421 / #2071).
@@ -5203,6 +5464,7 @@ async fn test_site_replication_resync_lifecycle_survives_real_server_restart() -
     init_logging();
     let resync_process_env = [
         ("RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET", "true"),
+        ("RUSTFS_REPL_RESYNC_POLL_MAX_MS", "100"),
         // Verbose server logging can block startup when this focused test is run
         // through a captured test process rather than nextest.
         ("RUST_LOG", "error"),
@@ -5219,6 +5481,7 @@ async fn test_site_replication_resync_lifecycle_survives_real_server_restart() -
         .await?;
 
     let source_bucket = "site-repl-resync-src";
+    const RESYNC_OBJECT_COUNT: usize = 128;
 
     let source_client = source_env.create_s3_client();
     let target_client = target_env.create_s3_client();
@@ -5263,12 +5526,13 @@ async fn test_site_replication_resync_lifecycle_survives_real_server_restart() -
     wait_for_bucket_on_target(&source_client, source_bucket).await?;
     let target_arn = wait_for_remote_target_arn(&source_env, source_bucket).await?;
 
-    for idx in 0..96 {
+    for idx in 0..RESYNC_OBJECT_COUNT {
+        let size = if idx == 0 { 8 * 1024 * 1024 } else { 8 * 1024 };
         source_client
             .put_object()
             .bucket(source_bucket)
             .key(format!("resync-object-{idx:02}"))
-            .body(ByteStream::from(vec![b'x'; 512 * 1024]))
+            .body(ByteStream::from(vec![b'x'; size]))
             .send()
             .await?;
     }
@@ -5327,6 +5591,22 @@ async fn test_site_replication_resync_lifecycle_survives_real_server_restart() -
     );
     let restarted_reset_id = restarted.resync_id.clone();
 
+    let partial_deadline = tokio::time::Instant::now() + Duration::from_secs(10);
+    loop {
+        let status = site_replication_resync_op(&source_env, "status", &remote_peer).await?;
+        let replicated = status.replicated_objects;
+        if status.resync_id == restarted_reset_id
+            && replicated > 0
+            && replicated < u64::try_from(RESYNC_OBJECT_COUNT).expect("test object count should fit u64")
+        {
+            break;
+        }
+        if status.state == "completed" || tokio::time::Instant::now() >= partial_deadline {
+            return Err(format!("resync did not expose a partial durable checkpoint before restart: {status:?}").into());
+        }
+        sleep(Duration::from_millis(10)).await;
+    }
+
     source_env.restart_server_preserving_data(vec![], &resync_process_env).await?;
     wait_for_site_replication_enabled(&source_env, 2).await?;
 
@@ -5360,6 +5640,41 @@ async fn test_site_replication_resync_lifecycle_survives_real_server_restart() -
     })?;
     assert_eq!(restarted_target.reset_id, restarted_reset_id);
 
+    let completion_deadline = tokio::time::Instant::now() + Duration::from_secs(60);
+    let completed = loop {
+        let status = site_replication_resync_op(&source_env, "status", &remote_peer).await?;
+        match status.state.as_str() {
+            "completed" => break status,
+            "failed" => return Err(format!("recovered resync failed: {status:?}").into()),
+            _ if tokio::time::Instant::now() < completion_deadline => sleep(Duration::from_millis(500)).await,
+            _ => return Err(format!("recovered resync did not complete in time: {status:?}").into()),
+        }
+    };
+    assert_eq!(
+        completed.replicated_objects,
+        u64::try_from(RESYNC_OBJECT_COUNT).expect("test object count should fit u64")
+    );
+
+    let replicated = futures::stream::iter(0..RESYNC_OBJECT_COUNT)
+        .map(|idx| {
+            let target_client = target_client.clone();
+            async move {
+                let key = format!("resync-object-{idx:02}");
+                let body = wait_for_object_on_target(&target_client, source_bucket, &key).await?;
+                let expected_size = if idx == 0 { 8 * 1024 * 1024 } else { 8 * 1024 };
+                if body != vec![b'x'; expected_size] {
+                    return Err(format!("recovered resync object body mismatch for {key}").into());
+                }
+                Ok::<(), Box<dyn Error + Send + Sync>>(())
+            }
+        })
+        .buffer_unordered(16)
+        .collect::<Vec<_>>()
+        .await;
+    for result in replicated {
+        result?;
+    }
+
     Ok(())
 }
 
@@ -6862,6 +7177,119 @@ async fn wait_for_target_request_version_id(
     }
 }
 
+#[tokio::test]
+#[serial]
+async fn test_bucket_resync_restart_revisits_objects_before_out_of_order_checkpoint() -> TestResult {
+    init_logging();
+
+    const OBJECT_COUNT: usize = 128;
+    let target = FakeS3Target::start().await?;
+    let target_bucket = "resync-checkpoint-dst";
+    target.create_bucket(target_bucket);
+
+    let mut source_env = RustFSTestEnvironment::new().await?;
+    let mut process_env = replication_fast_env();
+    process_env.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
+    process_env.extend_from_slice(&[
+        ("NO_PROXY", "127.0.0.1,localhost"),
+        ("HTTP_PROXY", ""),
+        ("HTTPS_PROXY", ""),
+        ("RUST_LOG", "error"),
+    ]);
+    source_env.start_rustfs_server_with_env(vec![], &process_env).await?;
+
+    let source_bucket = "resync-checkpoint-src";
+    let source_client = source_env.create_s3_client();
+    source_client.create_bucket().bucket(source_bucket).send().await?;
+    enable_bucket_versioning(&source_env, source_bucket).await?;
+    let target_arn = set_replication_target_with_options(
+        &source_env,
+        source_bucket,
+        ReplicationTargetOptions {
+            endpoint: &target.address(),
+            access_key: FAKE_ACCESS_KEY,
+            secret_key: FAKE_SECRET_KEY,
+            target_bucket,
+            secure: false,
+            skip_tls_verify: false,
+            ca_cert_pem: None,
+        },
+    )
+    .await?;
+    put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
+
+    for idx in 0..OBJECT_COUNT {
+        source_client
+            .put_object()
+            .bucket(source_bucket)
+            .key(format!("checkpoint-{idx:03}"))
+            .body(ByteStream::from(format!("checkpoint payload {idx}").into_bytes()))
+            .send()
+            .await?;
+    }
+    wait_for_source_replication_status(&source_client, source_bucket, "checkpoint-127", "COMPLETED", false).await?;
+    let initial_replication_deadline = tokio::time::Instant::now() + Duration::from_secs(30);
+    loop {
+        let replicated = target
+            .requests()
+            .into_iter()
+            .filter(|request| request.operation == FakeTargetOperation::PutObject)
+            .count();
+        if replicated >= OBJECT_COUNT {
+            break;
+        }
+        if tokio::time::Instant::now() >= initial_replication_deadline {
+            return Err(format!("initial replication only sent {replicated}/{OBJECT_COUNT} objects").into());
+        }
+        sleep(Duration::from_millis(100)).await;
+    }
+    target.clear_bucket_objects(target_bucket);
+    target.take_requests();
+    target.inject_for_key(
+        FakeTargetOperation::PutObject,
+        "checkpoint-000",
+        FakeTargetFault::Delay(Duration::from_secs(30)),
+        100,
+    );
+
+    let (reset_arn, reset_id) = start_bucket_replication_reset(&source_env, source_bucket).await?;
+    assert_eq!(reset_arn, target_arn);
+    let partial = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |status| {
+        status.reset_id == reset_id
+            && status.replicated_count > 0
+            && status.replicated_count < i64::try_from(OBJECT_COUNT).expect("test object count should fit i64")
+            && status.object.as_str() > "checkpoint-000"
+    })
+    .await
+    .map_err(|err| format!("{err}; target journal: {:?}", target.requests()))?;
+    assert!(target.requests().iter().any(|request| {
+        request.operation == FakeTargetOperation::PutObject
+            && request.key.as_deref() == Some("checkpoint-000")
+            && request.fault == Some(FakeTargetFault::Delay(Duration::from_secs(30)))
+    }));
+    assert!(!target.has_object(target_bucket, "checkpoint-000"));
+    source_env.stop_server();
+    target.clear_faults();
+
+    source_env.start_rustfs_server_without_cleanup_with_env(&process_env).await?;
+    let completed = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |status| {
+        status.reset_id == reset_id && status.status == "Completed"
+    })
+    .await?;
+    assert_eq!(
+        completed.replicated_count,
+        i64::try_from(OBJECT_COUNT).expect("test object count should fit i64")
+    );
+    assert!(
+        target.has_object(target_bucket, "checkpoint-000"),
+        "restart skipped failed object checkpoint-000 before persisted checkpoint {}",
+        partial.object
+    );
+
+    target.shutdown().await;
+    Ok(())
+}
+
 /// P0-5: MinIO derives the replicated version exclusively from the `versionId`
 /// query parameter (`putOptsFromReq`); the internal x-*-source-version-id
 /// headers do not exist there. Without the query, a MinIO target mints fresh
@@ -6975,3 +7403,87 @@ async fn test_replication_put_and_create_multipart_carry_source_version_id_query
     target.shutdown().await;
     Ok(())
 }
+
+/// P1-20: inbound replicas never cascade. An object replicated A->B carries
+/// x-amz-replication-status=REPLICA on B; `must_replicate` returns an empty
+/// decision for replicas, so even an ExistingObjectReplication=Enabled rule
+/// configured on B AFTER the replica landed (making it an "existing object"
+/// for that rule) must never push it onward — while B's own native objects
+/// flow to the onward bucket, proving B's outbound replication and scanner
+/// are live.
+#[tokio::test]
+#[serial]
+async fn test_scanner_never_cascades_inbound_replicas() -> TestResult {
+    init_logging();
+
+    let mut env_a = RustFSTestEnvironment::new().await?;
+    let mut env_a_vars = replication_fast_env();
+    env_a_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
+    env_a.start_rustfs_server_with_env(vec![], &env_a_vars).await?;
+
+    // B becomes a replication source itself, driven by its scanner.
+    let mut env_b = RustFSTestEnvironment::new().await?;
+    let mut env_b_vars = replication_fast_env();
+    env_b_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
+    env_b_vars.extend_from_slice(FAST_SCANNER_ENV);
+    env_b.start_rustfs_server_with_env(vec![], &env_b_vars).await?;
+
+    let bucket_a = "cascade-a-src";
+    let bucket_b = "cascade-b-mid";
+    let bucket_c = "cascade-a-third";
+    let client_a = env_a.create_s3_client();
+    let client_b = env_b.create_s3_client();
+    client_a.create_bucket().bucket(bucket_a).send().await?;
+    client_b.create_bucket().bucket(bucket_b).send().await?;
+    client_a.create_bucket().bucket(bucket_c).send().await?;
+    enable_bucket_versioning(&env_a, bucket_a).await?;
+    enable_bucket_versioning(&env_b, bucket_b).await?;
+    enable_bucket_versioning(&env_a, bucket_c).await?;
+
+    // A -> B first: the replica lands on B before B has any outbound rule.
+    let arn_ab = set_replication_target(&env_a, bucket_a, &env_b, bucket_b).await?;
+    put_bucket_replication(&env_a, bucket_a, &arn_ab).await?;
+
+    let replica_key = "replica-object.txt";
+    let replica_payload = "replica payload";
+    client_a
+        .put_object()
+        .bucket(bucket_a)
+        .key(replica_key)
+        .body(ByteStream::from_static(replica_payload.as_bytes()))
+        .send()
+        .await?;
+    wait_for_replicated_object(&client_b, bucket_b, replica_key, replica_payload).await?;
+
+    // Precondition for the anti-cascade contract: the inbound copy must carry
+    // REPLICA status on B. If this fails, the break is in inbound status
+    // stamping, not in the scanner guard.
+    let inbound = client_b.head_object().bucket(bucket_b).key(replica_key).send().await?;
+    assert_eq!(
+        inbound.replication_status().map(|status| status.as_str()),
+        Some("REPLICA"),
+        "inbound replica must be stamped REPLICA on the target"
+    );
+
+    // Now wire B's outbound rule; the replica is an "existing object" for it.
+    let arn_bc = set_replication_target(&env_b, bucket_b, &env_a, bucket_c).await?;
+    put_bucket_replication(&env_b, bucket_b, &arn_bc).await?;
+
+    // B's own native object flows onward through the live path.
+    let native_key = "native-control.txt";
+    let native_payload = "native control payload";
+    client_b
+        .put_object()
+        .bucket(bucket_b)
+        .key(native_key)
+        .body(ByteStream::from_static(native_payload.as_bytes()))
+        .send()
+        .await?;
+    wait_for_replicated_object(&client_a, bucket_c, native_key, native_payload).await?;
+
+    // With B's outbound proven, the inbound replica must stay put across
+    // multiple fast-scanner cycles.
+    assert_replication_key_absent(&client_a, bucket_c, replica_key, Duration::from_secs(6)).await?;
+
+    Ok(())
+}

crates/ecstore/Cargo.toml

@@ -181,6 +181,7 @@ path-absolutize = { workspace = true }
 rmp.workspace = true
 rmp-serde.workspace = true
 tokio-util = { workspace = true, features = ["io", "compat"] }
+tokio-stream = { workspace = true, features = ["sync"] }
 base64 = { workspace = true }
 hmac = { workspace = true }
 sha1 = { workspace = true }
@@ -239,7 +240,19 @@ rustfs-uring = "0.2.1"
 
 [target.'cfg(windows)'.dependencies]
 winapi-util.workspace = true
-windows-sys = { workspace = true, features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }
+windows-sys = { workspace = true, features = [
+    "Wdk_Foundation",
+    "Wdk_Storage_FileSystem",
+    "Win32_Foundation",
+    "Win32_Security",
+    "Win32_Storage_FileSystem",
+    "Win32_System_IO",
+    "Win32_System_SystemServices",
+    "Win32_System_WindowsProgramming",
+] }
+
+[target.'cfg(windows)'.dev-dependencies]
+windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
 
 [dev-dependencies]
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }

crates/ecstore/src/api/mod.rs

@@ -326,8 +326,8 @@ pub mod disk {
     pub use crate::disk::local::ScanGuard;
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
-        CheckPartsResp, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore,
-        FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
+        CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
+        DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
         PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
         STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
         validate_batch_read_version_item_count,
@@ -414,7 +414,10 @@ pub mod object {
         lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, register_object_mutation_hook,
         unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
     };
-    pub use crate::store::PreparedGetObjectReader;
+    pub use crate::store::{
+        PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
+        SnapshotConsistencyError,
+    };
 }
 
 pub mod rebalance {
@@ -439,16 +442,26 @@ pub mod rpc {
         ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
         check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
         gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
-        normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof,
-        tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
-        verify_put_file_auth_trailer, verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
-        verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
-        verify_tonic_rpc_signature_with_bootstrap,
+        normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_put_file_capability,
+        sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
+        tonic_rpc_auth_failure_reason, verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature,
+        verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest,
+        verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
     };
 }
 
 pub mod set_disk {
     pub use crate::set_disk::{DEFAULT_READ_BUFFER_SIZE, SetDisks, get_lock_acquire_timeout, is_valid_storage_class};
+
+    /// Return the canonical object-metadata identity used for read-quorum grouping.
+    pub fn file_info_quorum_hash(meta: &rustfs_filemeta::FileInfo) -> [u8; 32] {
+        crate::set_disk::SetDisks::file_info_quorum_hash(meta)
+    }
+
+    #[cfg(feature = "test-util")]
+    pub mod test_util {
+        pub use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
+    }
 }
 
 pub mod store_list {

crates/ecstore/src/bucket/bandwidth/monitor.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use crate::bucket::bandwidth::reader::BucketOptions;
-use ratelimit::{Error as RatelimitError, Ratelimiter};
+use ratelimit::{Clock, Error as RatelimitError, Ratelimiter};
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::sync::atomic::{AtomicU64, Ordering};
@@ -24,6 +24,33 @@ use tracing::warn;
 /// BETA_BUCKET is the weight used to calculate exponential moving average
 const BETA_BUCKET: f64 = 0.1;
 
+// ratelimit 2.0 stores tokens at six decimal places. Above this limit its
+// scaled capacity and token-cost calculations saturate instead of preserving
+// the configured bandwidth.
+const MAX_RATELIMIT_TOKENS: i64 = 18_446_744_073_709;
+
+fn consume_tokens<C: Clock>(limiter: &Ratelimiter<C>, n: u64) -> (u64, f64, u64) {
+    if n == 0 {
+        return (0, limiter.rate() as f64, 0);
+    }
+
+    let mut consumed = 0u64;
+    // Consuming one token also refills the bucket based on elapsed time, so
+    // the subsequent `available()` read reflects freshly accrued tokens.
+    if limiter.try_wait().is_ok() {
+        consumed = 1;
+    }
+    let available = limiter.available();
+    let to_consume = n - consumed;
+    let batch = to_consume.min(available);
+    if batch > 0 && limiter.try_wait_n(batch).is_ok() {
+        consumed += batch;
+    }
+    let deficit = n.saturating_sub(consumed);
+    let rate = limiter.rate() as f64;
+    (deficit, rate, consumed)
+}
+
 #[derive(Clone)]
 pub struct BucketThrottle {
     limiter: Arc<Mutex<Ratelimiter>>,
@@ -34,9 +61,9 @@ impl BucketThrottle {
     fn new(node_bandwidth_per_sec: i64) -> Result<Self, RatelimitError> {
         let node_bandwidth_per_sec = node_bandwidth_per_sec.max(1);
         let amount = node_bandwidth_per_sec as u64;
-        let limiter_inner = Ratelimiter::builder(amount, Duration::from_secs(1))
-            .max_tokens(amount)
-            .build()?;
+        // ratelimit 2.0's builder takes a per-second rate; the refill period
+        // defaults to one second, so `amount` tokens accrue per second.
+        let limiter_inner = Ratelimiter::builder(amount).max_tokens(amount).build()?;
         Ok(Self {
             limiter: Arc::new(Mutex::new(limiter_inner)),
             node_bandwidth_per_sec,
@@ -47,32 +74,21 @@ impl BucketThrottle {
         self.limiter.lock().unwrap_or_else(|e| e.into_inner()).max_tokens()
     }
 
-    /// The ratelimit crate (0.10.0) does not provide a bulk token consumption API.
-    /// try_wait() first to consume 1 token AND trigger the internal refill
-    /// mechanism (tokens are only refilled during try_wait/wait calls).
-    /// directly adjust available tokens via set_available() to consume the remaining amount.
+    /// Best-effort bulk token consumption: consume up to `n` tokens and report
+    /// how many were taken plus any shortfall.
+    ///
+    /// `try_wait_n` on the ratelimit crate is all-or-nothing, so we cannot ask
+    /// for `n` directly and still consume a partial amount. Instead we take one
+    /// token first (which also triggers the internal time-based refill), read
+    /// the now-current available count, and consume `min(remaining, available)`
+    /// in a single `try_wait_n` call — that batch never exceeds `available`, so
+    /// it always succeeds.
     pub(crate) fn consume(&self, n: u64) -> (u64, f64, u64) {
         let guard = self.limiter.lock().unwrap_or_else(|e| {
             warn!("bucket throttle mutex poisoned, recovering");
             e.into_inner()
         });
-        if n == 0 {
-            return (0, guard.rate(), 0);
-        }
-        let mut consumed = 0u64;
-        if guard.try_wait().is_ok() {
-            consumed = 1;
-        }
-        let available = guard.available();
-        let to_consume = n - consumed;
-        let batch = to_consume.min(available);
-        if batch > 0 {
-            let _ = guard.set_available(available - batch);
-            consumed += batch;
-        }
-        let deficit = n.saturating_sub(consumed);
-        let rate = guard.rate();
-        (deficit, rate, consumed)
+        consume_tokens(&guard, n)
     }
 }
 
@@ -329,6 +345,16 @@ impl Monitor {
                 "bandwidth limit too small for cluster size, per-node limit will clamp to 1 byte/s"
             );
         }
+        if limit_bytes > MAX_RATELIMIT_TOKENS {
+            warn!(
+                bucket = bucket,
+                arn = arn,
+                limit_bytes = limit_bytes,
+                max_limit_bytes = MAX_RATELIMIT_TOKENS,
+                "bandwidth limit exceeds ratelimiter capacity, throttling disabled for this target"
+            );
+            return;
+        }
         let opts = BucketOptions {
             name: bucket.to_string(),
             replication_arn: arn.to_string(),
@@ -375,6 +401,30 @@ mod tests {
     use super::*;
     use std::panic::{AssertUnwindSafe, catch_unwind};
 
+    #[derive(Clone)]
+    struct TestClock {
+        elapsed_ns: Arc<AtomicU64>,
+    }
+
+    impl TestClock {
+        fn new() -> Self {
+            Self {
+                elapsed_ns: Arc::new(AtomicU64::new(0)),
+            }
+        }
+
+        fn advance(&self, duration: Duration) {
+            let elapsed_ns = u64::try_from(duration.as_nanos()).unwrap_or(u64::MAX);
+            self.elapsed_ns.fetch_add(elapsed_ns, Ordering::Relaxed);
+        }
+    }
+
+    impl Clock for TestClock {
+        fn elapsed(&self) -> Duration {
+            Duration::from_nanos(self.elapsed_ns.load(Ordering::Relaxed))
+        }
+    }
+
     #[test]
     fn test_set_and_get_throttle_with_node_split() {
         let monitor = Monitor::new(4);
@@ -426,6 +476,15 @@ mod tests {
         assert!(!monitor.is_throttled("b1", "arn1"));
     }
 
+    #[test]
+    fn test_set_bandwidth_limit_rejects_unrepresentable_rate() {
+        let monitor = Monitor::new(1);
+
+        monitor.set_bandwidth_limit("b1", "arn1", MAX_RATELIMIT_TOKENS + 1);
+
+        assert!(!monitor.is_throttled("b1", "arn1"));
+    }
+
     #[test]
     fn test_consume_returns_deficit_when_tokens_exhausted() {
         let throttle = BucketThrottle::new(100).expect("test");
@@ -436,6 +495,23 @@ mod tests {
         assert!(rate > 0.0);
     }
 
+    #[test]
+    fn test_consume_refills_continuously() {
+        let clock = TestClock::new();
+        let limiter = Ratelimiter::with_clock(100, clock.clone());
+
+        assert_eq!(consume_tokens(&limiter, 100), (100, 100.0, 0));
+
+        clock.advance(Duration::from_millis(250));
+        assert_eq!(consume_tokens(&limiter, 100), (75, 100.0, 25));
+
+        clock.advance(Duration::from_millis(250));
+        assert_eq!(consume_tokens(&limiter, 100), (75, 100.0, 25));
+
+        clock.advance(Duration::from_millis(500));
+        assert_eq!(consume_tokens(&limiter, 100), (50, 100.0, 50));
+    }
+
     #[test]
     fn test_consume_no_deficit_when_tokens_sufficient() {
         let throttle = BucketThrottle::new(10000).expect("test");

crates/ecstore/src/bucket/bandwidth/reader.rs

@@ -306,7 +306,7 @@ mod tests {
     #[tokio::test]
     async fn test_monitored_reader_header_size_accounting() {
         let monitor = Monitor::new(1);
-        monitor.set_bandwidth_limit("b1", "arn1", 100);
+        monitor.set_bandwidth_limit("b1", "arn1", 1_000_000_000);
 
         let data = vec![0u8; 200];
         let inner = TestAsyncReader::new(&data);

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -1832,12 +1832,27 @@ impl TargetClient {
         object: &str,
         version_id: Option<String>,
     ) -> Result<HeadObjectOutput, SdkError<HeadObjectError>> {
+        // Announce the replication check so a RustFS target returns SSE-C
+        // object metadata (etag/size) without the customer key the replication
+        // worker cannot hold; otherwise SSE-C replicas never converge on HEAD.
+        let mut headers = HeaderMap::new();
+        insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
         match self
             .client
             .head_object()
             .bucket(bucket)
             .key(object)
             .set_version_id(version_id)
+            .customize()
+            .map_request(move |mut req| {
+                for (k, v) in headers.clone().into_iter() {
+                    if let Some(key_str) = k.map(|k| k.as_str().to_string()) {
+                        let value_str = v.to_str().unwrap_or("").to_string();
+                        req.headers_mut().insert(key_str, value_str);
+                    }
+                }
+                Result::<_, std::convert::Infallible>::Ok(req)
+            })
             .send()
             .await
         {
@@ -1922,14 +1937,11 @@ impl TargetClient {
         object: &str,
         opts: &PutObjectOptions,
     ) -> Result<String, S3ClientError> {
-        let mut headers = HeaderMap::new();
+        // Object metadata belongs to CreateMultipartUpload in S3 semantics;
+        // building only the source-version headers here used to drop user
+        // metadata, content-type, and the SSE intent for multipart replicas.
+        let headers = opts.header();
         let version_id = opts.internal.source_version_id.clone();
-        if !version_id.is_empty() {
-            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
-        }
-        if opts.internal.replication_request {
-            insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
-        }
         // The remote version of a multipart replication is decided at initiate
         // time; CompleteMultipartUpload does not read a versionId.
         let api_version_id = resolve_put_api_version_id(&version_id).map(ToOwned::to_owned);

crates/ecstore/src/bucket/object_lock/objectlock.rs

@@ -200,6 +200,29 @@ mod tests {
         assert!(retention.retain_until_date.is_some());
     }
 
+    /// backlog#1733 g-key-002: the persisted literal keys must still be read
+    /// through the current header constants, or WORM metadata fails open.
+    #[test]
+    fn persisted_compliance_lock_metadata_remains_effective() {
+        let mut meta = HashMap::new();
+        meta.insert("x-amz-object-lock-mode".to_string(), "COMPLIANCE".to_string());
+        meta.insert("x-amz-object-lock-retain-until-date".to_string(), "9999-01-01T00:00:00Z".to_string());
+        meta.insert("x-amz-object-lock-legal-hold".to_string(), "ON".to_string());
+
+        let retention = get_object_retention_meta(&meta);
+        assert_eq!(
+            retention.mode.as_ref().map(|mode| mode.as_str()),
+            Some(ObjectLockRetentionMode::COMPLIANCE)
+        );
+        assert!(retention.retain_until_date.is_some(), "persisted retention date must remain readable");
+
+        let legal_hold = get_object_legalhold_meta(&meta);
+        assert_eq!(
+            legal_hold.status.as_ref().map(|status| status.as_str()),
+            Some(ObjectLockLegalHoldStatus::ON)
+        );
+    }
+
     #[test]
     fn test_get_object_legalhold_meta_empty() {
         let meta = HashMap::new();

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -710,8 +710,8 @@ pub struct ReplicationPool<S: ReplicationStorage> {
     mrf_save_tx: Sender<MrfReplicateEntry>,
     mrf_save_rx: Mutex<Option<Receiver<MrfReplicateEntry>>>,
 
-    // Control channels
-    mrf_worker_kill_tx: Sender<()>,
+    // MRF worker lifecycle
+    mrf_worker_cancellations: Mutex<Vec<CancellationToken>>,
     mrf_stop_tx: Sender<()>,
 
     // Worker size tracking
@@ -734,7 +734,6 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
         // Create MRF channels
         let (mrf_replica_tx, mrf_replica_rx) = mpsc::channel(100000);
         let (mrf_save_tx, mrf_save_rx) = mpsc::channel(100000);
-        let (mrf_worker_kill_tx, _mrf_worker_kill_rx) = mpsc::channel(worker_counts.mrf_workers);
         let (mrf_stop_tx, _mrf_stop_rx) = mpsc::channel(1);
 
         let pool = Arc::new(Self {
@@ -752,7 +751,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
             mrf_replica_rx: Arc::new(Mutex::new(mrf_replica_rx)),
             mrf_save_tx,
             mrf_save_rx: Mutex::new(Some(mrf_save_rx)),
-            mrf_worker_kill_tx,
+            mrf_worker_cancellations: Mutex::new(Vec::with_capacity(worker_counts.mrf_workers)),
             mrf_stop_tx,
             mrf_worker_size: AtomicI32::new(0),
             task_handles: Mutex::new(Vec::new()),
@@ -896,12 +895,12 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
 
     /// Resizes the failed workers pool
     pub async fn resize_failed_workers(&self, n: i32) {
-        // Spawn workers up to n.  Each worker shares the receiver via Arc<Mutex<...>>.
-        // The mutex is held only while calling recv() — released before processing — so
-        // all workers process entries concurrently (the dequeue step is serialised but
-        // the replication I/O is not).
-        while self.mrf_worker_size.load(Ordering::SeqCst) < n {
-            self.mrf_worker_size.fetch_add(1, Ordering::SeqCst);
+        let target = mrf_worker_size_to_count(n);
+        let mut cancellations = self.mrf_worker_cancellations.lock().await;
+
+        while cancellations.len() < target {
+            let cancellation = CancellationToken::new();
+            cancellations.push(cancellation.clone());
 
             let active_counter = self.active_mrf_workers.clone();
             let stats = self.stats.clone();
@@ -910,7 +909,18 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
 
             let handle = tokio::spawn(async move {
                 loop {
-                    let operation = { mrf_rx.lock().await.recv().await };
+                    let operation = tokio::select! {
+                        biased;
+                        operation = async {
+                            let mut receiver = mrf_rx.lock().await;
+                            tokio::select! {
+                                biased;
+                                operation = receiver.recv() => operation,
+                                _ = cancellation.cancelled() => None,
+                            }
+                        } => operation,
+                        _ = cancellation.cancelled() => break,
+                    };
                     let Some(operation) = operation else { break };
 
                     let _active = ActiveWorkerGuard::new(active_counter.clone());
@@ -920,11 +930,13 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
             self.task_handles.lock().await.push(handle);
         }
 
-        // Remove workers if needed
-        while self.mrf_worker_size.load(Ordering::SeqCst) > n {
-            self.mrf_worker_size.fetch_sub(1, Ordering::SeqCst);
-            let _ = self.mrf_worker_kill_tx.try_send(());
+        while cancellations.len() > target {
+            if let Some(cancellation) = cancellations.pop() {
+                cancellation.cancel();
+            }
         }
+
+        self.mrf_worker_size.store(n.max(0), Ordering::SeqCst);
     }
 
     /// Resizes worker priority and counts
@@ -3350,7 +3362,6 @@ mod tests {
     ) -> Arc<ReplicationPool<LoadResyncNodeStore>> {
         let (mrf_replica_tx, mrf_replica_rx) = mpsc::channel(1);
         let (mrf_save_tx, mrf_save_rx) = mpsc::channel(mrf_save_capacity);
-        let (mrf_worker_kill_tx, _) = mpsc::channel(1);
         let (mrf_stop_tx, _) = mpsc::channel(1);
 
         Arc::new(ReplicationPool {
@@ -3368,7 +3379,7 @@ mod tests {
             mrf_replica_rx: Arc::new(Mutex::new(mrf_replica_rx)),
             mrf_save_tx,
             mrf_save_rx: Mutex::new(Some(mrf_save_rx)),
-            mrf_worker_kill_tx,
+            mrf_worker_cancellations: Mutex::new(Vec::new()),
             mrf_stop_tx,
             mrf_worker_size: AtomicI32::new(0),
             task_handles: Mutex::new(Vec::new()),
@@ -3971,6 +3982,54 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn resize_failed_workers_cancels_idle_workers() {
+        let shared = empty_resync_shared_state();
+        let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("mrf-resize", shared))).await;
+
+        pool.resize_failed_workers(4).await;
+        assert_eq!(pool.mrf_worker_cancellations.lock().await.len(), 4);
+        assert_eq!(pool.mrf_worker_size.load(Ordering::SeqCst), 4);
+
+        pool.resize_failed_workers(1).await;
+
+        tokio::time::timeout(Duration::from_secs(10), async {
+            loop {
+                let finished = pool
+                    .task_handles
+                    .lock()
+                    .await
+                    .iter()
+                    .filter(|handle| handle.is_finished())
+                    .count();
+                if finished == 3 {
+                    break;
+                }
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("canceled MRF workers should exit while the shared queue is idle");
+
+        assert_eq!(pool.mrf_worker_cancellations.lock().await.len(), 1);
+        assert_eq!(pool.mrf_worker_size.load(Ordering::SeqCst), 1);
+    }
+
+    #[tokio::test]
+    async fn resize_failed_workers_is_idempotent_across_growth_and_shrink() {
+        let shared = empty_resync_shared_state();
+        let pool = new_test_replication_pool(Arc::new(LoadResyncNodeStore::new("mrf-resize-repeat", shared))).await;
+
+        for target in [2, 4, 1, 4, 4] {
+            pool.resize_failed_workers(target).await;
+            assert_eq!(
+                pool.mrf_worker_cancellations.lock().await.len(),
+                usize::try_from(target).expect("test worker count should fit usize")
+            );
+            assert_eq!(pool.mrf_worker_size.load(Ordering::SeqCst), target);
+        }
+    }
+
     #[test]
     fn replicate_object_info_from_object_info_preserves_ssec_checksum() {
         let checksum = bytes::Bytes::from_static(b"ssec-checksum");

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -69,16 +69,17 @@ use rustfs_s3_types::EventName;
 use rustfs_utils::http::{
     AMZ_TAGGING_DIRECTIVE, SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_STATUS, has_internal_suffix, insert_str,
 };
-use rustfs_utils::{DEFAULT_SIP_HASH_KEY, sip_hash};
+use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash};
 #[cfg(test)]
 use s3s::dto::ReplicationConfiguration;
 use std::collections::HashMap;
 use std::fmt::Display;
 use std::sync::Arc;
+use std::sync::atomic::{AtomicBool, Ordering};
 use time::OffsetDateTime;
 use time::format_description::well_known::Rfc3339;
 use tokio::io::AsyncRead;
-use tokio::sync::RwLock;
+use tokio::sync::{OwnedSemaphorePermit, RwLock, Semaphore};
 use tokio::task::{JoinHandle, JoinSet};
 use tokio::time::Duration as TokioDuration;
 use tokio_util::io::ReaderStream;
@@ -86,6 +87,9 @@ use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, instrument, trace, warn};
 
 const BACKGROUND_WALKDIR_TIMEOUT: TokioDuration = TokioDuration::from_secs(60);
+const ENV_REPL_RESYNC_MAX_JOBS: &str = "RUSTFS_REPL_RESYNC_MAX_JOBS";
+const DEFAULT_REPL_RESYNC_MAX_JOBS: usize = 2;
+const MAX_REPL_RESYNC_MAX_JOBS: usize = 32;
 use uuid::Uuid;
 
 const EVENT_RESYNC_STATUS_UPDATE_SKIPPED: &str = "replication_resync_status_update_skipped";
@@ -256,11 +260,20 @@ fn resync_status_duration(
 
 type ResyncCancelKey = (String, String, String);
 
+fn configured_resync_max_jobs() -> usize {
+    bounded_resync_max_jobs(get_env_usize(ENV_REPL_RESYNC_MAX_JOBS, DEFAULT_REPL_RESYNC_MAX_JOBS))
+}
+
+fn bounded_resync_max_jobs(value: usize) -> usize {
+    value.clamp(1, MAX_REPL_RESYNC_MAX_JOBS)
+}
+
 #[derive(Debug)]
 pub struct ReplicationResyncer {
     pub status_map: Arc<RwLock<HashMap<String, BucketReplicationResyncStatus>>>,
     pub worker_size: usize,
     pub(crate) cancel_tokens: Arc<RwLock<HashMap<ResyncCancelKey, CancellationToken>>>,
+    resync_admission: Arc<Semaphore>,
 }
 
 impl ReplicationResyncer {
@@ -269,6 +282,14 @@ impl ReplicationResyncer {
             status_map: Arc::new(RwLock::new(HashMap::new())),
             worker_size: RESYNC_WORKER_COUNT,
             cancel_tokens: Arc::new(RwLock::new(HashMap::new())),
+            resync_admission: Arc::new(Semaphore::new(configured_resync_max_jobs())),
+        }
+    }
+
+    async fn acquire_resync_admission(&self, cancellation_token: &CancellationToken) -> Option<OwnedSemaphorePermit> {
+        tokio::select! {
+            permit = self.resync_admission.clone().acquire_owned() => permit.ok(),
+            _ = cancellation_token.cancelled() => None,
         }
     }
 
@@ -603,6 +624,10 @@ impl ReplicationResyncer {
             }
         };
 
+        let Some(_resync_admission_permit) = self.acquire_resync_admission(&cancellation_token).await else {
+            return;
+        };
+
         let cfg = match get_replication_config(&opts.bucket).await {
             Ok(cfg) => cfg,
             Err(err) => {
@@ -715,55 +740,36 @@ impl ReplicationResyncer {
         }
 
         let (tx, mut rx) = tokio::sync::mpsc::channel(100);
-
-        if let Err(err) = storage
-            .clone()
-            .walk(
-                cancellation_token.clone(),
-                &opts.bucket,
-                "",
-                tx.clone(),
-                WalkOptions::default().with_walkdir_timeouts(BACKGROUND_WALKDIR_TIMEOUT),
-            )
-            .await
-        {
-            error!(
-                event = EVENT_RESYNC_RUNTIME_SKIPPED,
-                component = LOG_COMPONENT_ECSTORE,
-                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
-                bucket = %opts.bucket,
-                arn = %opts.arn,
-                reason = "walk_failed",
-                error = %err,
-                "Replication resync bucket walk failed"
-            );
-            self.resync_bucket_mark_status(ResyncStatusType::ResyncFailed, opts.clone(), storage.clone())
-                .await;
-            return;
-        }
-        drop(tx);
-
-        let status = {
-            self.status_map
-                .read()
+        let walk_failed = Arc::new(AtomicBool::new(false));
+        let walk_failed_task = walk_failed.clone();
+        let walk_storage = storage.clone();
+        let walk_cancellation = cancellation_token.clone();
+        let walk_bucket = opts.bucket.clone();
+        let walk_arn = opts.arn.clone();
+        let walk_task = tokio::spawn(async move {
+            if let Err(err) = walk_storage
+                .walk(
+                    walk_cancellation,
+                    &walk_bucket,
+                    "",
+                    tx,
+                    WalkOptions::default().with_walkdir_timeouts(BACKGROUND_WALKDIR_TIMEOUT),
+                )
                 .await
-                .get(&opts.bucket)
-                .and_then(|status| status.targets_map.get(&opts.arn))
-                .cloned()
-                .unwrap_or_default()
-        };
-
-        // An empty checkpoint means no per-object progress was persisted before the
-        // interruption: resume from the beginning, otherwise `object.name != checkpoint`
-        // below would skip every object and mark the resync completed without work.
-        let mut last_checkpoint = if (status.resync_status == ResyncStatusType::ResyncStarted
-            || status.resync_status == ResyncStatusType::ResyncFailed)
-            && !status.object.is_empty()
-        {
-            Some(status.object)
-        } else {
-            None
-        };
+            {
+                walk_failed_task.store(true, Ordering::Relaxed);
+                error!(
+                    event = EVENT_RESYNC_RUNTIME_SKIPPED,
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                    bucket = %walk_bucket,
+                    arn = %walk_arn,
+                    reason = "walk_failed",
+                    error = %err,
+                    "Replication resync bucket walk failed"
+                );
+            }
+        });
 
         let mut worker_txs = Vec::new();
         // mpsc, not broadcast: a lagging broadcast receiver returns Err(Lagged) which
@@ -773,7 +779,7 @@ impl ReplicationResyncer {
         let opts_clone = opts.clone();
         let self_clone = self.clone();
 
-        let mut futures = Vec::new();
+        let mut futures = vec![walk_task];
 
         let results_fut = tokio::spawn(async move {
             while let Some(st) = results_rx.recv().await {
@@ -962,6 +968,8 @@ impl ReplicationResyncer {
                     error = %err,
                     "Failed to receive resync object info"
                 );
+                cancellation_token.cancel();
+                drop(rx);
                 let worker_failed = finish_resync_workers(worker_txs, results_tx, futures, false).await;
                 if worker_failed {
                     error!(
@@ -980,6 +988,7 @@ impl ReplicationResyncer {
             }
 
             if cancellation_token.is_cancelled() {
+                drop(rx);
                 finish_resync_workers(worker_txs, results_tx, futures, true).await;
                 self.resync_bucket_mark_status(ResyncStatusType::ResyncCanceled, opts.clone(), storage.clone())
                     .await;
@@ -990,14 +999,6 @@ impl ReplicationResyncer {
                 continue;
             };
 
-            if heal
-                && let Some(checkpoint) = &last_checkpoint
-                && &object.name != checkpoint
-            {
-                continue;
-            }
-            last_checkpoint = None;
-
             let roi = match get_heal_replicate_object_info(&object, &rcfg).await {
                 Ok(roi) => roi,
                 Err(err) => {
@@ -1011,6 +1012,8 @@ impl ReplicationResyncer {
                         error = %err,
                         "Failed to classify object for replication resync"
                     );
+                    cancellation_token.cancel();
+                    drop(rx);
                     let worker_failed = finish_resync_workers(worker_txs, results_tx, futures, false).await;
                     if worker_failed {
                         error!(
@@ -1033,6 +1036,7 @@ impl ReplicationResyncer {
             }
 
             if cancellation_token.is_cancelled() {
+                drop(rx);
                 finish_resync_workers(worker_txs, results_tx, futures, true).await;
                 self.resync_bucket_mark_status(ResyncStatusType::ResyncCanceled, opts.clone(), storage.clone())
                     .await;
@@ -1052,6 +1056,8 @@ impl ReplicationResyncer {
                     error = %err,
                     "Failed to send resync object to worker"
                 );
+                cancellation_token.cancel();
+                drop(rx);
                 let worker_failed = finish_resync_workers(worker_txs, results_tx, futures, false).await;
                 if worker_failed {
                     error!(
@@ -1072,7 +1078,7 @@ impl ReplicationResyncer {
 
         let worker_failed = finish_resync_workers(worker_txs, results_tx, futures, false).await;
         let target_failed = self.target_has_resync_failures(&opts).await;
-        let status = if worker_failed || target_failed {
+        let status = if walk_failed.load(Ordering::Relaxed) || worker_failed || target_failed {
             ResyncStatusType::ResyncFailed
         } else {
             ResyncStatusType::ResyncCompleted
@@ -2355,16 +2361,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             ..Default::default()
         };
 
-        if self.target_replication_status(&tgt_client.arn) == ReplicationStatusType::Completed
-            && !self.existing_obj_resync.is_empty()
-            && self.existing_obj_resync.must_resync_target(&tgt_client.arn)
-        {
-            rinfo.replication_status = ReplicationStatusType::Completed;
-            rinfo.replication_resynced = true;
-
-            return rinfo;
-        }
-
         if ReplicationTargetStore::target_is_offline(&tgt_client).await {
             debug!(
                 event = EVENT_RESYNC_RUNTIME_SKIPPED,
@@ -2394,6 +2390,9 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             version_suspended,
             versioned,
             replication_request: true,
+            // SSE-C passthrough reads the stored ciphertext verbatim; the
+            // decrypting reader cannot serve it (no customer key server-side).
+            raw_data_movement_read: self.ssec,
             ..Default::default()
         };
 
@@ -2455,6 +2454,9 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
                 return rinfo;
             }
         };
+        // SSE-C passthrough sends the stored ciphertext; the wire length is
+        // the stored size while rinfo keeps the logical size for metering.
+        let transfer_size = if self.ssec { object_info.size } else { size };
 
         if tgt_client.bucket.is_empty() {
             debug!(
@@ -2601,7 +2603,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             gr.stream = wrap_with_bandwidth_monitor(gr.stream, &put_opts, &bucket, &rinfo.arn);
             let byte_stream = async_read_to_bytestream(gr.stream);
             let result = tgt_client
-                .put_object(&tgt_client.bucket, &object, size, byte_stream, &put_opts)
+                .put_object(&tgt_client.bucket, &object, transfer_size, byte_stream, &put_opts)
                 .await
                 .map_err(|e| std::io::Error::other(e.to_string()));
             record_proxy_request(&bucket, "PutObject", result.is_err()).await;
@@ -2686,6 +2688,9 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             version_suspended,
             versioned,
             replication_request: true,
+            // SSE-C passthrough reads the stored ciphertext verbatim; the
+            // decrypting reader cannot serve it (no customer key server-side).
+            raw_data_movement_read: self.ssec,
             ..Default::default()
         };
 
@@ -2723,15 +2728,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
 
         rinfo.prev_replication_status = object_info.target_replication_status(&tgt_client.arn);
 
-        if rinfo.prev_replication_status == ReplicationStatusType::Completed
-            && !self.existing_obj_resync.is_empty()
-            && self.existing_obj_resync.must_resync_target(&tgt_client.arn)
-        {
-            rinfo.replication_status = ReplicationStatusType::Completed;
-            rinfo.replication_resynced = true;
-            return rinfo;
-        }
-
         let size = match object_info.get_actual_size() {
             Ok(size) => size,
             Err(e) => {
@@ -2755,8 +2751,9 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
                 return rinfo;
             }
         };
-
-        // TODO: SSE
+        // SSE-C passthrough sends the stored ciphertext; the wire length is
+        // the stored size while rinfo keeps the logical size for metering.
+        let transfer_size = if self.ssec { object_info.size } else { size };
 
         if tgt_client.bucket.is_empty() {
             debug!(
@@ -3013,7 +3010,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             gr.stream = wrap_with_bandwidth_monitor(gr.stream, &put_opts, &bucket, &rinfo.arn);
             let byte_stream = async_read_to_bytestream(gr.stream);
             let result = tgt_client
-                .put_object(&tgt_client.bucket, &object, size, byte_stream, &put_opts)
+                .put_object(&tgt_client.bucket, &object, transfer_size, byte_stream, &put_opts)
                 .await
                 .map_err(|e| std::io::Error::other(e.to_string()));
             record_proxy_request(&bucket, "PutObject", result.is_err()).await;
@@ -3152,10 +3149,17 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR
     let mut header_size = replication_put_object_header_size(&put_opts);
     let mut offset: i64 = 0;
     for part_info in object_info.parts.iter() {
+        // Ciphertext passthrough (raw read) ranges over the stored part
+        // bytes; decrypted reads range over the logical plaintext parts.
+        let part_size = if obj_opts.raw_data_movement_read {
+            part_info.size as i64
+        } else {
+            part_info.actual_size
+        };
         let part_plan = replication_multipart_part_plan(ReplicationMultipartPartInput {
             offset,
             part_number: part_info.number,
-            part_size: part_info.actual_size,
+            part_size,
         })
         .map_err(|err| std::io::Error::other(err.to_string()))?;
         let range_spec = HTTPRangeSpec {
@@ -3204,7 +3208,7 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR
         object,
         &upload_id,
         uploaded_parts,
-        &replication_complete_multipart_options(actual_size),
+        &replication_complete_multipart_options(actual_size, object_info.etag.clone().unwrap_or_default(), object_info.mod_time),
     )
     .await
     .map_err(|e| std::io::Error::other(e.to_string()))?;
@@ -3253,6 +3257,48 @@ mod tests {
         ReplicationTargetStore::register_test_target(target).await;
     }
 
+    #[test]
+    fn resync_admission_configuration_is_bounded() {
+        assert_eq!(ENV_REPL_RESYNC_MAX_JOBS, "RUSTFS_REPL_RESYNC_MAX_JOBS");
+        assert_eq!(bounded_resync_max_jobs(0), 1);
+        assert_eq!(bounded_resync_max_jobs(DEFAULT_REPL_RESYNC_MAX_JOBS), 2);
+        assert_eq!(bounded_resync_max_jobs(1000), MAX_REPL_RESYNC_MAX_JOBS);
+    }
+
+    #[tokio::test]
+    async fn resync_admission_limits_jobs_and_wait_is_cancelable() {
+        let resyncer = ReplicationResyncer {
+            resync_admission: Arc::new(Semaphore::new(2)),
+            ..ReplicationResyncer::new().await
+        };
+        let first = resyncer
+            .acquire_resync_admission(&CancellationToken::new())
+            .await
+            .expect("first resync should acquire admission");
+        let second = resyncer
+            .acquire_resync_admission(&CancellationToken::new())
+            .await
+            .expect("second resync should acquire admission");
+        let cancellation = CancellationToken::new();
+        let blocked = resyncer.acquire_resync_admission(&cancellation);
+        tokio::pin!(blocked);
+
+        assert!(
+            tokio::time::timeout(TokioDuration::from_millis(25), &mut blocked)
+                .await
+                .is_err()
+        );
+        cancellation.cancel();
+        assert!(
+            tokio::time::timeout(TokioDuration::from_secs(1), &mut blocked)
+                .await
+                .expect("canceled admission wait should finish")
+                .is_none()
+        );
+
+        drop((first, second));
+    }
+
     #[test]
     fn replication_target_offline_error_classifier_is_network_scoped() {
         assert!(is_replication_target_offline_error(&"put_object dispatch failure: connector error"));

crates/ecstore/src/bucket/replication/replication_target_boundary.rs

@@ -28,7 +28,7 @@ use rustfs_utils::http::{
     AMZ_STORAGE_CLASS, AMZ_TAG_COUNT, CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE,
     HeaderExt as _, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP,
     SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_TAGGING_TIMESTAMP, get_str, insert_header_map,
-    is_internal_key,
+    is_internal_key, is_object_encryption_marker, is_replication_stripped_encryption_key, ssec_replication_transport_header,
 };
 use time::OffsetDateTime;
 use time::format_description::well_known::Rfc3339;
@@ -62,24 +62,6 @@ static STANDARD_HEADERS: &[&str] = &[
     AMZ_SERVER_SIDE_ENCRYPTION,
 ];
 
-static VALID_SSE_REPLICATION_HEADERS: &[(&str, &str)] = &[
-    (
-        "X-Rustfs-Internal-Server-Side-Encryption-Sealed-Key",
-        "X-Rustfs-Replication-Server-Side-Encryption-Sealed-Key",
-    ),
-    (
-        "X-Rustfs-Internal-Server-Side-Encryption-Seal-Algorithm",
-        "X-Rustfs-Replication-Server-Side-Encryption-Seal-Algorithm",
-    ),
-    (
-        "X-Rustfs-Internal-Server-Side-Encryption-Iv",
-        "X-Rustfs-Replication-Server-Side-Encryption-Iv",
-    ),
-    ("X-Rustfs-Internal-Encrypted-Multipart", "X-Rustfs-Replication-Encrypted-Multipart"),
-    ("X-Rustfs-Internal-Actual-Object-Size", "X-Rustfs-Replication-Actual-Object-Size"),
-];
-
-const ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED: &str = "managed SSE replication requires target encryption support";
 const ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED: &str = "replication source contains unsupported encryption metadata";
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -105,15 +87,29 @@ fn classify_replication_source_encryption(metadata: &HashMap<String, String>) ->
     let kms_context = metadata_value(metadata, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT);
 
     if is_ssec {
-        return if sse.is_some() || kms_key_id.is_some() || kms_context.is_some() {
-            ReplicationSourceEncryption::Unsupported
-        } else {
+        // Stored SSE-C objects always carry x-amz-server-side-encryption=AES256
+        // alongside the customer-algorithm key; only KMS evidence marks a
+        // mixed, unsupported state.
+        let sse_compatible = sse.map(str::trim).is_none_or(|value| value.eq_ignore_ascii_case("AES256"));
+        return if sse_compatible && kms_key_id.is_none() && kms_context.is_none() {
             ReplicationSourceEncryption::SseC
+        } else {
+            ReplicationSourceEncryption::Unsupported
         };
     }
 
     match sse.map(str::trim) {
-        None if kms_key_id.is_none() && kms_context.is_none() => ReplicationSourceEncryption::Plaintext,
+        None if kms_key_id.is_none() && kms_context.is_none() => {
+            // Sealed material without any recognizable SSE marker (e.g. an
+            // object written by MinIO, which does not persist the x-amz SSE
+            // intent header) must fail closed: replicating it as plaintext
+            // ships ciphertext the target can never decrypt.
+            if metadata.keys().any(|key| is_object_encryption_marker(key)) {
+                ReplicationSourceEncryption::Unsupported
+            } else {
+                ReplicationSourceEncryption::Plaintext
+            }
+        }
         Some(value) if value.eq_ignore_ascii_case("AES256") && kms_key_id.is_none() && kms_context.is_none() => {
             ReplicationSourceEncryption::SseS3
         }
@@ -163,28 +159,38 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
     let source_encryption = classify_replication_source_encryption(&object_info.user_defined);
     let is_ssec = matches!(source_encryption, ReplicationSourceEncryption::SseC);
 
-    match source_encryption {
-        ReplicationSourceEncryption::Plaintext | ReplicationSourceEncryption::SseC => {}
-        ReplicationSourceEncryption::SseS3 | ReplicationSourceEncryption::SseKms => {
-            return Err(Error::other(ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED));
-        }
-        ReplicationSourceEncryption::Unsupported => {
-            return Err(Error::other(ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED));
-        }
+    if matches!(source_encryption, ReplicationSourceEncryption::Unsupported) {
+        return Err(Error::other(ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED));
     }
 
     for (key, value) in object_info.user_defined.iter() {
-        let has_valid_sse_header = valid_sse_replication_header(key).is_some();
-
-        if (!is_ssec || !has_valid_sse_header) && (is_internal_key(key) || is_standard_header(key)) {
+        if is_ssec && let Some(transport_header) = ssec_replication_transport_header(key) {
+            meta.insert(transport_header.to_string(), value.to_string());
             continue;
         }
 
-        if let Some(replication_header) = valid_sse_replication_header(key) {
-            meta.insert(replication_header.to_string(), value.to_string());
-        } else {
-            meta.insert(key.to_string(), value.to_string());
+        // Encryption metadata that is not remapped for SSE-C passthrough must
+        // never leave the source site: envelopes and intent headers are only
+        // meaningful to the source KMS.
+        if is_replication_stripped_encryption_key(key) {
+            continue;
         }
+
+        if is_internal_key(key) || is_standard_header(key) {
+            continue;
+        }
+
+        meta.insert(key.to_string(), value.to_string());
+    }
+
+    // Managed SSE replicates as plaintext (the replication reader decrypts via
+    // the object-encryption resolver) and re-encrypts on the target with the
+    // target's own KMS. Send only the encryption intent — never the source
+    // key id, whose meaning is local to the source site's KMS.
+    if matches!(source_encryption, ReplicationSourceEncryption::SseS3) {
+        meta.insert(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string());
+    } else if matches!(source_encryption, ReplicationSourceEncryption::SseKms) {
+        meta.insert(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "aws:kms".to_string());
     }
 
     let mut is_multipart = object_info.is_multipart();
@@ -195,6 +201,11 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
         if is_ssec {
             let encoded = BASE64_STANDARD.encode(checksum_data);
             insert_header_map(&mut meta, SUFFIX_REPLICATION_SSEC_CRC, encoded);
+        } else if object_info.is_encrypted() {
+            // Encrypted checksums cannot be exposed as plaintext headers, and
+            // decrypt_checksums reports is_multipart=false for them (a value
+            // the response path relies on). Keep the object's own multipart
+            // flag so encrypted objects stay on the multipart route.
         } else {
             let (checksum_meta, is_mp) = object_info.decrypt_checksums(0, &HeaderMap::new())?;
             is_multipart = is_mp;
@@ -394,13 +405,22 @@ pub(crate) fn replication_force_delete_remove_options() -> RemoveObjectOptions {
     }
 }
 
-pub(crate) fn replication_complete_multipart_options(actual_size: String) -> PutObjectOptions {
+pub(crate) fn replication_complete_multipart_options(
+    actual_size: String,
+    source_etag: String,
+    source_mtime: Option<OffsetDateTime>,
+) -> PutObjectOptions {
     let mut user_metadata = HashMap::new();
     insert_header_map(&mut user_metadata, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, actual_size);
 
     PutObjectOptions {
         user_metadata,
         internal: AdvancedPutOptions {
+            source_etag,
+            // AdvancedPutOptions::default() stamps now_utc(); an absent source
+            // mtime must degrade to epoch so header() suppresses the header
+            // instead of asserting the replication time as the object's mtime.
+            source_mtime: source_mtime.unwrap_or(OffsetDateTime::UNIX_EPOCH),
             replication_status: ReplicationStatusType::Replica,
             replication_request: true,
             ..Default::default()
@@ -413,20 +433,14 @@ fn is_standard_header(key: &str) -> bool {
     STANDARD_HEADERS.iter().any(|header| header.eq_ignore_ascii_case(key))
 }
 
-fn valid_sse_replication_header(key: &str) -> Option<&str> {
-    VALID_SSE_REPLICATION_HEADERS
-        .iter()
-        .find(|(internal, _)| key.eq_ignore_ascii_case(internal))
-        .map(|(_, replication)| *replication)
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
     use aws_smithy_types::DateTime;
     use rustfs_replication::content_matches_by_etag;
     use rustfs_utils::http::{
-        SSEC_ALGORITHM_HEADER, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, get_header_map,
+        SSEC_ALGORITHM_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
+        get_header_map,
     };
     use std::sync::Arc;
     use time::Duration;
@@ -571,7 +585,21 @@ mod tests {
 
     #[test]
     fn replication_complete_multipart_options_sets_actual_size() {
-        let options = replication_complete_multipart_options("1024".to_string());
+        let source_mtime = OffsetDateTime::from_unix_timestamp(1_716_170_000).expect("valid test timestamp");
+        let options = replication_complete_multipart_options(
+            "1024".to_string(),
+            "0123456789abcdef0123456789abcdef-3".to_string(),
+            Some(source_mtime),
+        );
+        assert_eq!(options.internal.source_etag, "0123456789abcdef0123456789abcdef-3");
+        assert_eq!(options.internal.source_mtime, source_mtime);
+
+        // Absent source mtime must degrade to epoch (header suppressed), not
+        // the AdvancedPutOptions default of now_utc() — that default would
+        // stamp the replication time as the replica's mtime and break the
+        // multipart HEAD convergence.
+        let options_no_mtime = replication_complete_multipart_options("1024".to_string(), String::new(), None);
+        assert_eq!(options_no_mtime.internal.source_mtime.unix_timestamp(), 0);
 
         assert_eq!(
             get_header_map(&options.user_metadata, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE).as_deref(),
@@ -583,11 +611,29 @@ mod tests {
 
     #[test]
     fn replication_put_options_filter_and_map_metadata() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_IV_HEADER, MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER, MINIO_INTERNAL_ENCRYPTION_IV_HEADER,
+            MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER, MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
+            REPLICATION_ENCRYPTED_MULTIPART_HEADER, REPLICATION_ENCRYPTION_IV_HEADER, REPLICATION_SSE_IV_HEADER,
+            REPLICATION_SSE_SEAL_ALGORITHM_HEADER, REPLICATION_SSE_SEALED_KEY_HEADER, REPLICATION_SSEC_ALGORITHM_HEADER,
+            REPLICATION_SSEC_KEY_MD5_HEADER, REPLICATION_SSEC_ORIGINAL_SIZE_HEADER, SSEC_ORIGINAL_SIZE_HEADER,
+        };
+
+        // The stored shape of a real SSE-C object: SSE marker plus customer
+        // material, per encryption_material_to_metadata. Every transport-table
+        // source key is present so each mapping is pinned individually.
         let mut metadata = HashMap::new();
         metadata.insert(CONTENT_TYPE.to_string(), "text/plain".to_string());
         metadata.insert("x-user-meta".to_string(), "value".to_string());
+        metadata.insert(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string());
         metadata.insert(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string());
-        metadata.insert("X-Rustfs-Internal-Server-Side-Encryption-Sealed-Key".to_string(), "sealed".to_string());
+        metadata.insert(SSEC_KEY_MD5_HEADER.to_string(), "md5-value".to_string());
+        metadata.insert(SSEC_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string());
+        metadata.insert(INTERNAL_ENCRYPTION_IV_HEADER.to_string(), "iv-direct".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_IV_HEADER.to_string(), "iv-minio".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), "DAREv2-HMAC-SHA256".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER.to_string(), "sealed".to_string());
+        metadata.insert(MINIO_INTERNAL_ENCRYPTION_MULTIPART_HEADER.to_string(), "true".to_string());
 
         let object_info = ObjectInfo {
             user_defined: Arc::new(metadata),
@@ -605,12 +651,40 @@ mod tests {
         assert!(!is_multipart);
         assert_eq!(options.user_metadata.get("x-user-meta"), Some(&"value".to_string()));
         assert!(!options.user_metadata.contains_key(CONTENT_TYPE));
+
+        // Every stored SSE-C material key is remapped onto its transport name.
+        assert_eq!(options.user_metadata.get(REPLICATION_SSEC_ALGORITHM_HEADER), Some(&"AES256".to_string()));
+        assert_eq!(options.user_metadata.get(REPLICATION_SSEC_KEY_MD5_HEADER), Some(&"md5-value".to_string()));
         assert_eq!(
-            options
-                .user_metadata
-                .get("X-Rustfs-Replication-Server-Side-Encryption-Sealed-Key"),
-            Some(&"sealed".to_string())
+            options.user_metadata.get(REPLICATION_SSEC_ORIGINAL_SIZE_HEADER),
+            Some(&"1024".to_string())
         );
+        assert_eq!(
+            options.user_metadata.get(REPLICATION_ENCRYPTION_IV_HEADER),
+            Some(&"iv-direct".to_string())
+        );
+        assert_eq!(options.user_metadata.get(REPLICATION_SSE_IV_HEADER), Some(&"iv-minio".to_string()));
+        assert_eq!(
+            options.user_metadata.get(REPLICATION_SSE_SEAL_ALGORITHM_HEADER),
+            Some(&"DAREv2-HMAC-SHA256".to_string())
+        );
+        assert_eq!(options.user_metadata.get(REPLICATION_SSE_SEALED_KEY_HEADER), Some(&"sealed".to_string()));
+        assert_eq!(
+            options.user_metadata.get(REPLICATION_ENCRYPTED_MULTIPART_HEADER),
+            Some(&"true".to_string())
+        );
+
+        // The stored keys themselves and the SSE intent header must not leave
+        // the source verbatim.
+        assert!(!options.user_metadata.contains_key(AMZ_SERVER_SIDE_ENCRYPTION));
+        assert!(!options.user_metadata.contains_key(SSEC_ALGORITHM_HEADER));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_IV_HEADER));
+        assert!(
+            !options
+                .user_metadata
+                .contains_key(MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER)
+        );
+
         assert_eq!(options.content_type, "text/plain");
         assert_eq!(options.content_encoding, "gzip");
         assert_eq!(options.user_tags.get("env"), Some(&"prod".to_string()));
@@ -620,6 +694,68 @@ mod tests {
         assert!(options.internal.replication_request);
     }
 
+    #[test]
+    fn replication_put_options_strip_encryption_metadata_from_plaintext_objects() {
+        use rustfs_utils::http::object_encryption_keys::{INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER, SSEC_ORIGINAL_SIZE_HEADER};
+
+        // Migration leftovers: original-size metadata is not an encryption
+        // marker (older plaintext objects can retain it), so the object still
+        // classifies as plaintext — but the keys must be stripped, never
+        // forwarded as plain user metadata (backlog#1783 D2). The SSE-C
+        // original-size key is also a transport-table source key, so this
+        // doubles as the guard for the is_ssec gate: without SSE-C
+        // classification it must be stripped, not remapped.
+        let metadata = HashMap::from([
+            ("x-user-meta".to_string(), "value".to_string()),
+            (INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string()),
+            (SSEC_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string()),
+        ]);
+        let object_info = ObjectInfo {
+            user_defined: Arc::new(metadata),
+            ..Default::default()
+        };
+
+        let (options, _) = replication_put_object_options("", &object_info).expect("build put options");
+
+        assert_eq!(options.user_metadata.get("x-user-meta"), Some(&"value".to_string()));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER));
+        assert!(!options.user_metadata.contains_key(SSEC_ORIGINAL_SIZE_HEADER));
+        assert!(
+            !options
+                .user_metadata
+                .keys()
+                .any(|key| key.to_ascii_lowercase().starts_with("x-rustfs-replication-")),
+            "non-SSE-C objects must never emit SSE replication transport keys"
+        );
+    }
+
+    #[test]
+    fn replication_put_options_fail_closed_on_sealed_material_without_sse_marker() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
+        };
+
+        // Sealed material without a recognizable SSE marker (MinIO-written
+        // objects, or corrupted metadata) must fail closed instead of
+        // replicating ciphertext as a plaintext object.
+        for sealed_key in [
+            INTERNAL_ENCRYPTION_KEY_HEADER,
+            MINIO_INTERNAL_ENCRYPTION_SSEC_SEALED_KEY_HEADER,
+        ] {
+            let object_info = ObjectInfo {
+                user_defined: Arc::new(HashMap::from([(sealed_key.to_string(), "sealed-envelope".to_string())])),
+                ..Default::default()
+            };
+
+            let err = match replication_put_object_options("", &object_info) {
+                Ok(_) => panic!("sealed material without an SSE marker must fail closed ({sealed_key})"),
+                Err(err) => err,
+            };
+            assert!(err.to_string().contains(ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED));
+            assert!(!err.to_string().contains("sealed-envelope"));
+        }
+    }
+
     #[test]
     fn replication_put_options_adds_ssec_checksum_metadata() {
         let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]);
@@ -658,6 +794,30 @@ mod tests {
             classify_replication_source_encryption(&HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())])),
             ReplicationSourceEncryption::SseC
         );
+        // Real stored SSE-C objects carry the AES256 SSE marker alongside the
+        // customer algorithm (encryption_material_to_metadata writes both).
+        assert_eq!(
+            classify_replication_source_encryption(&HashMap::from([
+                (SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string()),
+                ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
+            ])),
+            ReplicationSourceEncryption::SseC
+        );
+        // SSE-C material mixed with KMS evidence stays unsupported.
+        assert_eq!(
+            classify_replication_source_encryption(&HashMap::from([
+                (SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string()),
+                (AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID.to_string(), "key-1".to_string()),
+            ])),
+            ReplicationSourceEncryption::Unsupported
+        );
+        assert_eq!(
+            classify_replication_source_encryption(&HashMap::from([
+                (SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string()),
+                ("x-amz-server-side-encryption".to_string(), "aws:kms".to_string()),
+            ])),
+            ReplicationSourceEncryption::Unsupported
+        );
         assert_eq!(
             classify_replication_source_encryption(&HashMap::from([(
                 "x-amz-server-side-encryption".to_string(),
@@ -675,36 +835,75 @@ mod tests {
     }
 
     #[test]
-    fn replication_put_options_rejects_sse_s3_until_target_encryption_is_supported() {
-        let object_info = ObjectInfo {
-            user_defined: Arc::new(HashMap::from([(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string())])),
-            ..Default::default()
+    fn replication_put_options_sends_sse_s3_intent_without_source_material() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_ALGORITHM_HEADER, INTERNAL_ENCRYPTION_IV_HEADER, INTERNAL_ENCRYPTION_KEY_HEADER,
+            INTERNAL_ENCRYPTION_KEY_ID_HEADER, INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER,
         };
 
-        let err = match replication_put_object_options("", &object_info) {
-            Ok(_) => panic!("SSE-S3 replication should fail closed until target encryption headers are supported"),
-            Err(err) => err,
-        };
-
-        assert!(err.to_string().contains(ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED));
-    }
-
-    #[test]
-    fn replication_put_options_rejects_sse_kms_until_target_encryption_is_supported() {
+        // The stored shape of a managed SSE-S3 object per
+        // encryption_material_to_metadata: SSE marker plus envelope material.
         let object_info = ObjectInfo {
             user_defined: Arc::new(HashMap::from([
-                (AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "aws:kms".to_string()),
-                (AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID.to_string(), "key-1".to_string()),
+                (AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string()),
+                (INTERNAL_ENCRYPTION_KEY_ID_HEADER.to_string(), "default".to_string()),
+                (INTERNAL_ENCRYPTION_KEY_HEADER.to_string(), "sealed-envelope".to_string()),
+                (INTERNAL_ENCRYPTION_IV_HEADER.to_string(), "iv".to_string()),
+                (INTERNAL_ENCRYPTION_ALGORITHM_HEADER.to_string(), "AES256-GCM".to_string()),
+                (INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER.to_string(), "1024".to_string()),
+                ("x-user-meta".to_string(), "value".to_string()),
             ])),
             ..Default::default()
         };
 
-        let err = match replication_put_object_options("", &object_info) {
-            Ok(_) => panic!("SSE-KMS replication should fail closed until target encryption headers are supported"),
-            Err(err) => err,
+        let (options, _) = replication_put_object_options("", &object_info).expect("managed SSE-S3 must build put options");
+
+        assert_eq!(options.user_metadata.get(AMZ_SERVER_SIDE_ENCRYPTION), Some(&"AES256".to_string()));
+        assert_eq!(options.user_metadata.get("x-user-meta"), Some(&"value".to_string()));
+        // No envelope material and no key id may leave the source.
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_KEY_HEADER));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_KEY_ID_HEADER));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_IV_HEADER));
+        assert!(
+            !options.user_metadata.values().any(|value| value.contains("sealed-envelope")),
+            "source envelope material must never leave the source site"
+        );
+    }
+
+    #[test]
+    fn replication_put_options_sends_sse_kms_intent_without_source_key_id() {
+        use rustfs_utils::http::object_encryption_keys::{
+            INTERNAL_ENCRYPTION_KEY_HEADER, MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER,
         };
 
-        assert!(err.to_string().contains(ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED));
+        let object_info = ObjectInfo {
+            user_defined: Arc::new(HashMap::from([
+                (AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "aws:kms".to_string()),
+                (AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID.to_string(), "source-key-1".to_string()),
+                (INTERNAL_ENCRYPTION_KEY_HEADER.to_string(), "sealed-envelope".to_string()),
+                (MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER.to_string(), "ctx".to_string()),
+            ])),
+            ..Default::default()
+        };
+
+        let (options, _) = replication_put_object_options("", &object_info).expect("managed SSE-KMS must build put options");
+
+        // Intent only: the target encrypts with its own default KMS key.
+        assert_eq!(options.user_metadata.get(AMZ_SERVER_SIDE_ENCRYPTION), Some(&"aws:kms".to_string()));
+        assert!(!options.user_metadata.contains_key(AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID));
+        assert!(!options.user_metadata.contains_key(INTERNAL_ENCRYPTION_KEY_HEADER));
+        assert!(
+            !options
+                .user_metadata
+                .contains_key(MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER)
+        );
+        assert!(
+            !options
+                .user_metadata
+                .values()
+                .any(|value| value.contains("sealed-envelope") || value.contains("source-key-1")),
+            "source KMS identifiers and envelopes must never leave the source site"
+        );
     }
 
     #[test]

crates/ecstore/src/client/api_get_object_file.rs

@@ -57,7 +57,7 @@ fn build_part_path(file_path: &Path) -> PathBuf {
 
 async fn open_download_part_file(file_part_path: &Path) -> io::Result<tokio::fs::File> {
     let mut options = OpenOptions::new();
-    options.create(true).read(true).write(true);
+    options.create(true).truncate(false).read(true).write(true);
 
     #[cfg(not(windows))]
     options.mode(0o600);

crates/ecstore/src/cluster/rpc/http_auth.rs

@@ -29,7 +29,7 @@
 use crate::cluster::rpc::context_propagation::{inject_request_id_into_http_headers, inject_trace_context_into_http_headers};
 use crate::storage_api_contracts::internode::{
     NS_SCANNER_PROTOCOL_VERSION, PUT_FILE_AUTH_TRAILER_DIGEST_LEN, PUT_FILE_AUTH_TRAILER_LEN, PUT_FILE_AUTH_TRAILER_MAC_LEN,
-    PUT_FILE_AUTH_TRAILER_MAGIC,
+    PUT_FILE_AUTH_TRAILER_MAGIC, PUT_FILE_CAPABILITY_VERSION,
 };
 use base64::Engine as _;
 use base64::engine::general_purpose;
@@ -75,6 +75,7 @@ const RPC_RESPONSE_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-response-proof-v1\0";
 const RPC_REPLAY_SCOPE_DOMAIN: &[u8] = b"rustfs-rpc-replay-scope-v3\0";
 const RPC_BOOT_EPOCH_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-boot-epoch-proof-v1\0";
 const HTTP_PUT_FILE_AUTH_DOMAIN: &[u8] = b"rustfs-http-put-file-auth-v1\0";
+const HTTP_PUT_FILE_CAPABILITY_AUTH_DOMAIN: &[u8] = b"rustfs-http-put-file-capability-v1\0";
 const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
 const UNSIGNED_PAYLOAD_NONCE: &str = "unsigned";
 const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
@@ -83,7 +84,7 @@ const REPLAY_CACHE_RETENTION_SECS: usize = 601;
 const REPLAY_CACHE_ENTRY_BYTES_ESTIMATE: u64 = 128;
 const REPLAY_CACHE_AUTO_MEMORY_PERCENT: u64 = 8;
 const REPLAY_CACHE_AUTO_RPC_RPS_PER_CPU: usize = 2048;
-const REPLAY_CACHE_AUTO_MAX_CAPACITY: usize = 16_777_216;
+const REPLAY_CACHE_AUTO_MAX_CAPACITY: usize = 33_554_432;
 const NS_SCANNER_CAPABILITY_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-capability-v3";
 pub const TONIC_RPC_PREFIX: &str = "/node_service.NodeService";
 static INTERNODE_RPC_SIGNATURE_STRICT: LazyLock<bool> = LazyLock::new(|| {
@@ -583,6 +584,36 @@ pub fn verify_put_file_auth_trailer(url: &str, method: &Method, nonce: Uuid, tra
     Ok(body_sha256.to_string())
 }
 
+fn update_put_file_capability_mac(mac: &mut HmacSha256, challenge: Uuid, server_epoch: Uuid, version: u16) {
+    mac.update(HTTP_PUT_FILE_CAPABILITY_AUTH_DOMAIN);
+    mac.update(challenge.as_bytes());
+    mac.update(server_epoch.as_bytes());
+    mac.update(&version.to_be_bytes());
+}
+
+fn put_file_capability_mac(challenge: Uuid, server_epoch: Uuid, version: u16) -> std::io::Result<HmacSha256> {
+    if challenge.is_nil() || server_epoch.is_nil() || version != PUT_FILE_CAPABILITY_VERSION {
+        return Err(std::io::Error::other("Invalid put_file capability scope"));
+    }
+    let mut mac = HmacSha256::new_from_slice(get_shared_secret()?.as_bytes())
+        .map_err(|_| std::io::Error::other("Invalid RPC HMAC secret"))?;
+    update_put_file_capability_mac(&mut mac, challenge, server_epoch, version);
+    Ok(mac)
+}
+
+pub fn sign_put_file_capability(challenge: Uuid, server_epoch: Uuid, version: u16) -> std::io::Result<Vec<u8>> {
+    Ok(put_file_capability_mac(challenge, server_epoch, version)?
+        .finalize()
+        .into_bytes()
+        .to_vec())
+}
+
+pub fn verify_put_file_capability(challenge: Uuid, server_epoch: Uuid, version: u16, proof: &[u8]) -> std::io::Result<()> {
+    put_file_capability_mac(challenge, server_epoch, version)?
+        .verify_slice(proof)
+        .map_err(|_| std::io::Error::new(std::io::ErrorKind::PermissionDenied, "Invalid put_file capability proof"))
+}
+
 fn update_ns_scanner_capability_mac(mac: &mut HmacSha256, challenge: Uuid, server_epoch: Uuid) {
     mac.update(NS_SCANNER_CAPABILITY_AUTH_DOMAIN);
     mac.update(&NS_SCANNER_PROTOCOL_VERSION.to_be_bytes());
@@ -2331,6 +2362,20 @@ mod tests {
         assert_eq!(err.to_string(), "Invalid put_file auth trailer");
     }
 
+    #[test]
+    fn put_file_capability_proof_binds_challenge_epoch_and_version() {
+        ensure_test_rpc_secret();
+        let challenge = Uuid::parse_str("11111111-2222-4333-8444-555555555555").expect("challenge");
+        let server_epoch = Uuid::parse_str("aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee").expect("server epoch");
+        let proof = sign_put_file_capability(challenge, server_epoch, PUT_FILE_CAPABILITY_VERSION)
+            .expect("capability proof should build");
+
+        assert!(verify_put_file_capability(challenge, server_epoch, PUT_FILE_CAPABILITY_VERSION, &proof).is_ok());
+        assert!(verify_put_file_capability(Uuid::new_v4(), server_epoch, PUT_FILE_CAPABILITY_VERSION, &proof).is_err());
+        assert!(verify_put_file_capability(challenge, Uuid::new_v4(), PUT_FILE_CAPABILITY_VERSION, &proof).is_err());
+        assert!(verify_put_file_capability(challenge, server_epoch, PUT_FILE_CAPABILITY_VERSION + 1, &proof).is_err());
+    }
+
     #[test]
     fn tier_mutation_rpc_contract_requires_method_bound_v2_body_digest() {
         ensure_test_rpc_secret();
@@ -2460,7 +2505,7 @@ mod tests {
     }
 
     #[test]
-    fn replay_cache_capacity_auto_reaches_hotpath_verified_capacity_on_larger_nodes() {
+    fn replay_cache_capacity_auto_uses_resource_model_on_larger_nodes() {
         let gib = 1024_u64 * 1024 * 1024;
         let decision =
             replay_cache_capacity_decision(rustfs_utils::EnvParseOutcome::Absent, 16, Some(32 * gib), Some(MemoryBasis::Host));
@@ -2468,7 +2513,17 @@ mod tests {
         assert_eq!(decision.source, ReplayCacheCapacitySource::Auto);
         assert_eq!(decision.memory_based_capacity, 21_474_836);
         assert_eq!(decision.cpu_based_capacity, 19_693_568);
-        assert_eq!(decision.capacity, 16_777_216);
+        assert_eq!(decision.capacity, 19_693_568);
+    }
+
+    #[test]
+    fn replay_cache_capacity_auto_caps_extreme_nodes() {
+        let gib = 1024_u64 * 1024 * 1024;
+        let decision =
+            replay_cache_capacity_decision(rustfs_utils::EnvParseOutcome::Absent, 128, Some(512 * gib), Some(MemoryBasis::Host));
+
+        assert_eq!(decision.source, ReplayCacheCapacitySource::Auto);
+        assert_eq!(decision.capacity, REPLAY_CACHE_AUTO_MAX_CAPACITY);
     }
 
     #[test]

crates/ecstore/src/cluster/rpc/internode_data_transport.rs

@@ -12,15 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::cluster::rpc::{build_auth_headers, build_put_file_auth_trailer, verify_ns_scanner_capability};
+use crate::cluster::rpc::{
+    build_auth_headers, build_put_file_auth_trailer, verify_ns_scanner_capability, verify_put_file_capability,
+};
 use crate::disk::error::{Error, Result};
 use crate::disk::{FileReader, FileWriter};
 use crate::storage_api_contracts::internode::{
     NS_SCANNER_BODY_SHA256_QUERY, NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CYCLE_QUERY, NS_SCANNER_LEADER_EPOCH_QUERY,
     NS_SCANNER_PROTOCOL_VERSION, NS_SCANNER_PROTOCOL_VERSION_QUERY, NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY,
     NS_SCANNER_SESSION_ID_QUERY, NS_SCANNER_SESSION_SEQUENCE_QUERY, NsScannerCapabilityResponse, PUT_FILE_AUTH_QUERY,
-    PUT_FILE_AUTH_V1, PUT_FILE_NONCE_QUERY, WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_STREAM_COMPLETION_QUERY,
-    WALK_DIR_STREAM_COMPLETION_V1,
+    PUT_FILE_AUTH_V1, PUT_FILE_CAPABILITY_CHALLENGE_QUERY, PUT_FILE_CAPABILITY_QUERY, PUT_FILE_CAPABILITY_VERSION,
+    PUT_FILE_NONCE_QUERY, PUT_FILE_SERVER_EPOCH_QUERY, PutFileCapabilityResponse, WALK_DIR_BODY_SHA256_QUERY,
+    WALK_DIR_STREAM_COMPLETION_QUERY, WALK_DIR_STREAM_COMPLETION_V1,
 };
 use async_trait::async_trait;
 use http::{HeaderMap, HeaderValue, Method, header::CONTENT_TYPE};
@@ -30,20 +33,29 @@ use rustfs_config::{
 };
 use rustfs_rio::{HttpReader, HttpWriter};
 use sha2::{Digest, Sha256};
+use std::collections::HashMap;
+use std::future::Future;
 use std::pin::Pin;
-use std::sync::{Arc, OnceLock};
+use std::sync::{Arc, LazyLock, OnceLock};
 use std::task::{Context, Poll};
-use std::time::Duration;
+use std::time::{Duration, Instant};
 use tokio::io::{AsyncReadExt, AsyncWrite};
+use tokio::sync::OnceCell;
 use uuid::Uuid;
 
 static INTERNODE_DATA_TRANSPORT: OnceLock<std::result::Result<Arc<dyn InternodeDataTransport>, String>> = OnceLock::new();
 
 const READ_FILE_STREAM_PATH: &str = "/rustfs/rpc/read_file_stream";
 const PUT_FILE_STREAM_PATH: &str = "/rustfs/rpc/put_file_stream";
+const PUT_FILE_AUTH_STREAM_PATH: &str = "/rustfs/rpc/put_file_stream_v1";
+const PUT_FILE_CAPABILITY_PATH: &str = "/rustfs/rpc/put_file_capability";
 const WALK_DIR_PATH: &str = "/rustfs/rpc/walk_dir";
 const NS_SCANNER_PATH: &str = "/rustfs/rpc/ns_scanner";
 const NS_SCANNER_MAX_CAPABILITY_RESPONSE_SIZE: usize = 1024;
+const PUT_FILE_MAX_CAPABILITY_RESPONSE_SIZE: usize = 1024;
+const PUT_FILE_LEGACY_CAPABILITY_TTL: Duration = Duration::from_secs(30);
+const PUT_FILE_V1_CAPABILITY_TTL: Duration = Duration::from_secs(30);
+const PUT_FILE_CAPABILITY_PROBE_TIMEOUT: Duration = Duration::from_secs(5);
 const CONTENT_TYPE_JSON: &str = "application/json";
 const CONTENT_TYPE_MSGPACK: &str = "application/msgpack";
 
@@ -54,6 +66,73 @@ fn unsupported_transport_message(transport: &str) -> String {
     )
 }
 
+#[derive(Debug, Clone, Copy)]
+enum PutFileCapabilityState {
+    LegacyUntil(Instant),
+    V1 { server_epoch: Uuid, revalidate_after: Instant },
+}
+
+#[derive(Debug)]
+struct PutFileCapabilityProbeFailure(Error);
+
+impl PutFileCapabilityProbeFailure {
+    fn to_error(&self) -> Error {
+        match &self.0 {
+            Error::Io(error) => rustfs_rio::clone_internode_http_io_error(error)
+                .map(Error::Io)
+                .unwrap_or_else(|| self.0.clone()),
+            _ => self.0.clone(),
+        }
+    }
+}
+
+type PutFileCapabilityProbeOutcome = std::result::Result<Option<Uuid>, PutFileCapabilityProbeFailure>;
+
+#[derive(Debug, Clone)]
+struct PutFileCapabilityFlight {
+    generation: u64,
+    v1_was_pinned: bool,
+    outcome: Arc<OnceCell<PutFileCapabilityProbeOutcome>>,
+}
+
+#[derive(Debug, Default)]
+struct PutFileCapabilityCacheState {
+    cached: Option<PutFileCapabilityState>,
+    generation: u64,
+    in_flight: Option<PutFileCapabilityFlight>,
+}
+
+type PutFileCapabilityCacheEntry = Arc<tokio::sync::RwLock<PutFileCapabilityCacheState>>;
+
+static PUT_FILE_CAPABILITY_CACHE: LazyLock<parking_lot::RwLock<HashMap<String, PutFileCapabilityCacheEntry>>> =
+    LazyLock::new(|| parking_lot::RwLock::new(HashMap::new()));
+
+fn put_file_capability_cache_entry(endpoint: &str) -> PutFileCapabilityCacheEntry {
+    if let Some(entry) = PUT_FILE_CAPABILITY_CACHE.read().get(endpoint).cloned() {
+        return entry;
+    }
+    PUT_FILE_CAPABILITY_CACHE
+        .write()
+        .entry(endpoint.to_owned())
+        .or_insert_with(|| Arc::new(tokio::sync::RwLock::new(PutFileCapabilityCacheState::default())))
+        .clone()
+}
+
+fn fresh_put_file_capability(state: Option<PutFileCapabilityState>, now: Instant) -> Option<Option<Uuid>> {
+    match state {
+        Some(PutFileCapabilityState::V1 {
+            server_epoch,
+            revalidate_after,
+        }) if now < revalidate_after => Some(Some(server_epoch)),
+        Some(PutFileCapabilityState::LegacyUntil(expires_at)) if now < expires_at => Some(None),
+        Some(PutFileCapabilityState::V1 { .. }) | Some(PutFileCapabilityState::LegacyUntil(_)) | None => None,
+    }
+}
+
+fn put_file_capability_status_is_legacy(status: u16) -> bool {
+    status == 404
+}
+
 #[derive(Debug, Clone, Copy, Eq, PartialEq)]
 pub struct InternodeDataTransportCapabilities {
     /// Backend can open a streaming remote disk reader.
@@ -169,12 +248,16 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
     }
 
     async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter> {
-        let nonce = Uuid::new_v4();
-        let url = build_put_file_stream_url(&request, Some(nonce));
+        let server_epoch = self.put_file_auth_capability(&request.endpoint).await?;
+        let nonce = server_epoch.map(|_| Uuid::new_v4());
+        let url = build_put_file_stream_url(&request, nonce.zip(server_epoch));
         let mut headers = json_headers();
         build_auth_headers(&url, &Method::PUT, &mut headers)?;
         let writer = HttpWriter::new(url.clone(), Method::PUT, headers).await?;
-        Ok(Box::new(PutFileAuthWriter::new(writer, url, nonce)))
+        match nonce {
+            Some(nonce) => Ok(Box::new(PutFileAuthWriter::new(writer, url, nonce))),
+            None => Ok(Box::new(writer)),
+        }
     }
 
     async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader> {
@@ -228,6 +311,134 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
     }
 }
 
+impl TcpHttpInternodeDataTransport {
+    async fn put_file_auth_capability(&self, endpoint: &str) -> Result<Option<Uuid>> {
+        resolve_put_file_auth_capability(endpoint, || async {
+            tokio::time::timeout(PUT_FILE_CAPABILITY_PROBE_TIMEOUT, self.probe_put_file_auth(endpoint))
+                .await
+                .map_err(|_| {
+                    Error::from(rustfs_rio::internode_http_timeout_error(
+                        &Method::GET,
+                        &format!("{endpoint}{PUT_FILE_CAPABILITY_PATH}"),
+                    ))
+                })?
+        })
+        .await
+    }
+
+    async fn probe_put_file_auth(&self, endpoint: &str) -> Result<Option<Uuid>> {
+        let challenge = Uuid::new_v4();
+        let url = build_put_file_capability_url(endpoint, challenge);
+        let mut headers = msgpack_headers();
+        build_auth_headers(&url, &Method::GET, &mut headers)?;
+        let reader = match HttpReader::new(url, Method::GET, headers, None).await {
+            Ok(reader) => reader,
+            Err(err) => {
+                let err = Error::from(err);
+                if matches!(
+                    err.internode_http_error_kind(),
+                    Some(rustfs_rio::InternodeHttpErrorKind::HttpStatus(status))
+                        if put_file_capability_status_is_legacy(status.as_u16())
+                ) {
+                    return Ok(None);
+                }
+                return Err(err);
+            }
+        };
+        let mut body = Vec::new();
+        reader
+            .take(u64::try_from(PUT_FILE_MAX_CAPABILITY_RESPONSE_SIZE + 1).unwrap_or(u64::MAX))
+            .read_to_end(&mut body)
+            .await?;
+        Ok(Some(verify_put_file_capability_response(challenge, &body)?))
+    }
+}
+
+async fn resolve_put_file_auth_capability<F, Fut>(endpoint: &str, probe: F) -> Result<Option<Uuid>>
+where
+    F: FnOnce() -> Fut,
+    Fut: Future<Output = Result<Option<Uuid>>>,
+{
+    let entry = put_file_capability_cache_entry(endpoint);
+    {
+        let state = entry.read().await;
+        if let Some(cached) = fresh_put_file_capability(state.cached, Instant::now()) {
+            return Ok(cached);
+        }
+    }
+
+    let flight = {
+        let mut state = entry.write().await;
+        if let Some(cached) = fresh_put_file_capability(state.cached, Instant::now()) {
+            return Ok(cached);
+        }
+        if let Some(flight) = state.in_flight.clone() {
+            flight
+        } else {
+            state.generation = state
+                .generation
+                .checked_add(1)
+                .ok_or_else(|| Error::other("put_file capability probe generation exhausted"))?;
+            let flight = PutFileCapabilityFlight {
+                generation: state.generation,
+                v1_was_pinned: matches!(state.cached, Some(PutFileCapabilityState::V1 { .. })),
+                outcome: Arc::new(OnceCell::new()),
+            };
+            state.in_flight = Some(flight.clone());
+            flight
+        }
+    };
+
+    let outcome = flight
+        .outcome
+        .get_or_init(|| async { probe().await.map_err(PutFileCapabilityProbeFailure) })
+        .await;
+
+    {
+        let mut state = entry.write().await;
+        let is_current_flight = state
+            .in_flight
+            .as_ref()
+            .is_some_and(|current| current.generation == flight.generation && Arc::ptr_eq(&current.outcome, &flight.outcome));
+        if is_current_flight {
+            match outcome {
+                Ok(Some(server_epoch)) => {
+                    state.cached = Some(PutFileCapabilityState::V1 {
+                        server_epoch: *server_epoch,
+                        revalidate_after: Instant::now() + PUT_FILE_V1_CAPABILITY_TTL,
+                    });
+                }
+                Ok(None) if !flight.v1_was_pinned => {
+                    state.cached = Some(PutFileCapabilityState::LegacyUntil(Instant::now() + PUT_FILE_LEGACY_CAPABILITY_TTL));
+                }
+                Ok(None) | Err(_) => {}
+            }
+            state.in_flight = None;
+        }
+    }
+
+    match outcome {
+        Ok(Some(server_epoch)) => Ok(Some(*server_epoch)),
+        Ok(None) if flight.v1_was_pinned => Err(Error::other("remote put_file capability downgrade rejected")),
+        Ok(None) => Ok(None),
+        Err(failure) => Err(failure.to_error()),
+    }
+}
+
+fn verify_put_file_capability_response(challenge: Uuid, body: &[u8]) -> Result<Uuid> {
+    if body.is_empty() || body.len() > PUT_FILE_MAX_CAPABILITY_RESPONSE_SIZE {
+        return Err(Error::other("invalid remote put_file capability response size"));
+    }
+    let response: PutFileCapabilityResponse =
+        rmp_serde::from_slice(body).map_err(|_| Error::other("invalid remote put_file capability response"))?;
+    if response.version != PUT_FILE_CAPABILITY_VERSION || response.server_epoch.is_nil() {
+        return Err(Error::other("incompatible remote put_file capability response"));
+    }
+    verify_put_file_capability(challenge, response.server_epoch, response.version, &response.proof)
+        .map_err(|err| Error::other(format!("remote put_file capability authentication failed: {err}")))?;
+    Ok(response.server_epoch)
+}
+
 fn build_read_file_stream_url(request: &ReadStreamRequest) -> String {
     format!(
         "{}{}?disk={}&volume={}&path={}&offset={}&length={}",
@@ -241,26 +452,43 @@ fn build_read_file_stream_url(request: &ReadStreamRequest) -> String {
     )
 }
 
-fn build_put_file_stream_url(request: &WriteStreamRequest, auth_nonce: Option<Uuid>) -> String {
+fn build_put_file_stream_url(request: &WriteStreamRequest, auth_scope: Option<(Uuid, Uuid)>) -> String {
+    let stream_path = if auth_scope.is_some() {
+        PUT_FILE_AUTH_STREAM_PATH
+    } else {
+        PUT_FILE_STREAM_PATH
+    };
     let mut url = format!(
         "{}{}?disk={}&volume={}&path={}&append={}&size={}",
         request.endpoint,
-        PUT_FILE_STREAM_PATH,
+        stream_path,
         urlencoding::encode(&request.disk),
         urlencoding::encode(&request.volume),
         urlencoding::encode(&request.path),
         request.append,
         request.size
     );
-    if let Some(nonce) = auth_nonce {
+    if let Some((nonce, server_epoch)) = auth_scope {
         url.push_str(&format!(
-            "&{}={}&{}={}",
-            PUT_FILE_AUTH_QUERY, PUT_FILE_AUTH_V1, PUT_FILE_NONCE_QUERY, nonce
+            "&{}={}&{}={}&{}={}",
+            PUT_FILE_AUTH_QUERY, PUT_FILE_AUTH_V1, PUT_FILE_NONCE_QUERY, nonce, PUT_FILE_SERVER_EPOCH_QUERY, server_epoch
         ));
     }
     url
 }
 
+fn build_put_file_capability_url(endpoint: &str, challenge: Uuid) -> String {
+    format!(
+        "{}{}?{}={}&{}={}",
+        endpoint,
+        PUT_FILE_CAPABILITY_PATH,
+        PUT_FILE_CAPABILITY_QUERY,
+        PUT_FILE_CAPABILITY_VERSION,
+        PUT_FILE_CAPABILITY_CHALLENGE_QUERY,
+        challenge
+    )
+}
+
 struct PutFileAuthWriter<W> {
     inner: W,
     url: String,
@@ -450,6 +678,28 @@ pub fn build_internode_data_transport_from_env() -> Result<Arc<dyn InternodeData
 #[cfg(test)]
 mod tests {
     use super::*;
+    use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+    use tokio::sync::{Barrier, Notify};
+
+    async fn wait_for_capability_flight_waiters(entry: &PutFileCapabilityCacheEntry, waiters: usize) {
+        tokio::time::timeout(Duration::from_secs(5), async {
+            loop {
+                let strong_count = entry
+                    .read()
+                    .await
+                    .in_flight
+                    .as_ref()
+                    .map(|flight| Arc::strong_count(&flight.outcome))
+                    .unwrap_or_default();
+                if strong_count > waiters {
+                    return;
+                }
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("capability callers should join the in-flight probe");
+    }
 
     #[derive(Debug)]
     struct LegacyTestTransport;
@@ -578,6 +828,7 @@ mod tests {
     #[test]
     fn put_file_stream_url_advertises_auth_nonce_when_enabled() {
         let nonce = Uuid::parse_str("11111111-2222-4333-8444-555555555555").expect("nonce");
+        let server_epoch = Uuid::parse_str("aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee").expect("server epoch");
         let url = build_put_file_stream_url(
             &WriteStreamRequest {
                 endpoint: "http://node1:9000".to_string(),
@@ -587,19 +838,405 @@ mod tests {
                 append: false,
                 size: 4096,
             },
-            Some(nonce),
+            Some((nonce, server_epoch)),
         );
 
         assert_eq!(
             url,
             concat!(
-                "http://node1:9000/rustfs/rpc/put_file_stream?disk=http%3A%2F%2Fnode1%3A9000%2Fdata%2Frustfs0",
+                "http://node1:9000/rustfs/rpc/put_file_stream_v1?disk=http%3A%2F%2Fnode1%3A9000%2Fdata%2Frustfs0",
                 "&volume=bucket&path=object%2Fpart.1&append=false&size=4096",
-                "&put_file_auth=digest-trailer-v1&put_file_nonce=11111111-2222-4333-8444-555555555555"
+                "&put_file_auth=digest-trailer-v1&put_file_nonce=11111111-2222-4333-8444-555555555555",
+                "&put_file_server_epoch=aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee"
             )
         );
     }
 
+    #[test]
+    fn put_file_capability_url_binds_version_and_challenge() {
+        let challenge = Uuid::parse_str("11111111-2222-4333-8444-555555555555").expect("challenge");
+
+        assert_eq!(
+            build_put_file_capability_url("http://node1:9000", challenge),
+            concat!(
+                "http://node1:9000/rustfs/rpc/put_file_capability?put_file_capability=1",
+                "&put_file_challenge=11111111-2222-4333-8444-555555555555"
+            )
+        );
+    }
+
+    #[test]
+    fn put_file_capability_legacy_statuses_are_exact() {
+        assert!(put_file_capability_status_is_legacy(404));
+        for status in [200, 400, 401, 403, 405, 408, 426, 429, 500, 503] {
+            assert!(!put_file_capability_status_is_legacy(status));
+        }
+    }
+
+    #[test]
+    fn put_file_capability_timeout_is_retryable() {
+        let error = Error::from(rustfs_rio::internode_http_timeout_error(
+            &Method::GET,
+            "http://node:9000/rustfs/rpc/put_file_capability",
+        ));
+
+        assert_eq!(
+            error.internode_http_error_kind(),
+            Some(rustfs_rio::InternodeHttpErrorKind::ConnectTimeout)
+        );
+        assert!(error.is_retryable_internode_write_failure());
+    }
+
+    #[tokio::test]
+    async fn put_file_capability_cache_pins_v1_and_honors_live_legacy_ttl() {
+        let transport = TcpHttpInternodeDataTransport;
+        let v1_endpoint = format!("http://v1-{}.invalid", Uuid::new_v4());
+        let v1_entry = put_file_capability_cache_entry(&v1_endpoint);
+        let server_epoch = Uuid::new_v4();
+        v1_entry.write().await.cached = Some(PutFileCapabilityState::V1 {
+            server_epoch,
+            revalidate_after: Instant::now() + PUT_FILE_V1_CAPABILITY_TTL,
+        });
+        assert_eq!(
+            transport.put_file_auth_capability(&v1_endpoint).await.expect("v1 cache"),
+            Some(server_epoch)
+        );
+        let cache_probe_called = AtomicBool::new(false);
+        assert_eq!(
+            resolve_put_file_auth_capability(&v1_endpoint, || async {
+                cache_probe_called.store(true, Ordering::SeqCst);
+                Ok(None)
+            })
+            .await
+            .expect("live v1 cache"),
+            Some(server_epoch)
+        );
+        assert!(!cache_probe_called.load(Ordering::SeqCst));
+        v1_entry.write().await.cached = Some(PutFileCapabilityState::V1 {
+            server_epoch,
+            revalidate_after: Instant::now(),
+        });
+        assert!(
+            resolve_put_file_auth_capability(&v1_endpoint, || async { Ok(None) })
+                .await
+                .is_err()
+        );
+        let replacement_epoch = Uuid::new_v4();
+        assert_eq!(
+            resolve_put_file_auth_capability(&v1_endpoint, || async { Ok(Some(replacement_epoch)) })
+                .await
+                .expect("authenticated replacement should refresh the epoch"),
+            Some(replacement_epoch)
+        );
+
+        let legacy_endpoint = format!("http://legacy-{}.invalid", Uuid::new_v4());
+        let legacy_entry = put_file_capability_cache_entry(&legacy_endpoint);
+        legacy_entry.write().await.cached =
+            Some(PutFileCapabilityState::LegacyUntil(Instant::now() + PUT_FILE_LEGACY_CAPABILITY_TTL));
+        assert!(
+            transport
+                .put_file_auth_capability(&legacy_endpoint)
+                .await
+                .expect("legacy cache")
+                .is_none()
+        );
+
+        let expired_endpoint = format!("http://expired-legacy-{}.invalid", Uuid::new_v4());
+        let expired_entry = put_file_capability_cache_entry(&expired_endpoint);
+        expired_entry.write().await.cached = Some(PutFileCapabilityState::LegacyUntil(Instant::now()));
+        let reprobed = std::sync::atomic::AtomicBool::new(false);
+        assert_eq!(
+            resolve_put_file_auth_capability(&expired_endpoint, || async {
+                reprobed.store(true, std::sync::atomic::Ordering::SeqCst);
+                Ok(Some(server_epoch))
+            })
+            .await
+            .expect("expired legacy cache should reprobe"),
+            Some(server_epoch)
+        );
+        assert!(reprobed.load(std::sync::atomic::Ordering::SeqCst));
+    }
+
+    #[tokio::test]
+    async fn legacy_put_file_capability_omits_the_auth_trailer_protocol() {
+        let endpoint = format!("http://legacy-selection-{}.invalid", Uuid::new_v4());
+        let server_epoch = resolve_put_file_auth_capability(&endpoint, || async { Ok(None) })
+            .await
+            .expect("legacy capability result");
+        let auth_scope = server_epoch.map(|epoch| (Uuid::new_v4(), epoch));
+        let url = build_put_file_stream_url(
+            &WriteStreamRequest {
+                endpoint,
+                disk: "http://node1:9000/data/rustfs0".to_string(),
+                volume: "bucket".to_string(),
+                path: "object/part.1".to_string(),
+                append: false,
+                size: 4096,
+            },
+            auth_scope,
+        );
+
+        assert!(auth_scope.is_none());
+        assert!(!url.contains(PUT_FILE_AUTH_QUERY));
+        assert!(!url.contains(PUT_FILE_NONCE_QUERY));
+    }
+
+    #[tokio::test]
+    async fn put_file_capability_probe_is_singleflight_per_endpoint() {
+        let endpoint = format!("http://singleflight-{}.invalid", Uuid::new_v4());
+        let entry = put_file_capability_cache_entry(&endpoint);
+        let calls = Arc::new(AtomicUsize::new(0));
+        let release = Arc::new(Notify::new());
+        let start = Arc::new(Barrier::new(65));
+        let mut tasks = Vec::with_capacity(64);
+
+        for _ in 0..64 {
+            let endpoint = endpoint.clone();
+            let calls = Arc::clone(&calls);
+            let release = Arc::clone(&release);
+            let start = Arc::clone(&start);
+            tasks.push(tokio::spawn(async move {
+                start.wait().await;
+                resolve_put_file_auth_capability(&endpoint, || async move {
+                    calls.fetch_add(1, Ordering::SeqCst);
+                    release.notified().await;
+                    Err(Error::from(rustfs_rio::new_test_internode_http_io_error(
+                        rustfs_rio::InternodeHttpErrorKind::ConnectionRefused,
+                    )))
+                })
+                .await
+            }));
+        }
+
+        start.wait().await;
+        wait_for_capability_flight_waiters(&entry, 64).await;
+        assert_eq!(calls.load(Ordering::SeqCst), 1);
+        release.notify_waiters();
+
+        let results = tokio::time::timeout(Duration::from_secs(1), futures::future::join_all(tasks))
+            .await
+            .expect("all callers should finish within one probe window");
+        for result in results {
+            let error = result.expect("capability task should finish").expect_err("probe should fail");
+            assert_eq!(
+                error.internode_http_error_kind(),
+                Some(rustfs_rio::InternodeHttpErrorKind::ConnectionRefused)
+            );
+            assert!(error.is_retryable_internode_write_failure());
+        }
+        assert_eq!(calls.load(Ordering::SeqCst), 1);
+    }
+
+    #[tokio::test]
+    async fn put_file_capability_probe_recovers_when_initializer_is_cancelled() {
+        let endpoint = format!("http://cancelled-singleflight-{}.invalid", Uuid::new_v4());
+        let entry = put_file_capability_cache_entry(&endpoint);
+        let calls = Arc::new(AtomicUsize::new(0));
+        let initializer_started = Arc::new(Notify::new());
+        let never_release = Arc::new(Notify::new());
+
+        let first = {
+            let endpoint = endpoint.clone();
+            let calls = Arc::clone(&calls);
+            let initializer_started = Arc::clone(&initializer_started);
+            let never_release = Arc::clone(&never_release);
+            tokio::spawn(async move {
+                resolve_put_file_auth_capability(&endpoint, || async move {
+                    calls.fetch_add(1, Ordering::SeqCst);
+                    initializer_started.notify_one();
+                    never_release.notified().await;
+                    Ok(Some(Uuid::new_v4()))
+                })
+                .await
+            })
+        };
+        initializer_started.notified().await;
+
+        let replacement_epoch = Uuid::new_v4();
+        let second = {
+            let endpoint = endpoint.clone();
+            let calls = Arc::clone(&calls);
+            tokio::spawn(async move {
+                resolve_put_file_auth_capability(&endpoint, || async move {
+                    calls.fetch_add(1, Ordering::SeqCst);
+                    Ok(Some(replacement_epoch))
+                })
+                .await
+            })
+        };
+        wait_for_capability_flight_waiters(&entry, 2).await;
+        first.abort();
+        assert!(first.await.expect_err("initializer should be cancelled").is_cancelled());
+
+        assert_eq!(
+            second.await.expect("waiter should finish").expect("waiter should take over"),
+            Some(replacement_epoch)
+        );
+        assert_eq!(calls.load(Ordering::SeqCst), 2);
+    }
+
+    #[tokio::test]
+    async fn put_file_capability_probe_recovers_after_all_callers_cancel() {
+        let endpoint = format!("http://all-cancelled-{}.invalid", Uuid::new_v4());
+        let entry = put_file_capability_cache_entry(&endpoint);
+        let calls = Arc::new(AtomicUsize::new(0));
+        let initializer_started = Arc::new(Notify::new());
+        let never_release = Arc::new(Notify::new());
+
+        let first = {
+            let endpoint = endpoint.clone();
+            let calls = Arc::clone(&calls);
+            let initializer_started = Arc::clone(&initializer_started);
+            let never_release = Arc::clone(&never_release);
+            tokio::spawn(async move {
+                resolve_put_file_auth_capability(&endpoint, || async move {
+                    calls.fetch_add(1, Ordering::SeqCst);
+                    initializer_started.notify_one();
+                    never_release.notified().await;
+                    Ok(None)
+                })
+                .await
+            })
+        };
+        initializer_started.notified().await;
+        let second = {
+            let endpoint = endpoint.clone();
+            let calls = Arc::clone(&calls);
+            tokio::spawn(async move {
+                resolve_put_file_auth_capability(&endpoint, || async move {
+                    calls.fetch_add(1, Ordering::SeqCst);
+                    Ok(None)
+                })
+                .await
+            })
+        };
+        wait_for_capability_flight_waiters(&entry, 2).await;
+        first.abort();
+        second.abort();
+        assert!(first.await.expect_err("initializer should be cancelled").is_cancelled());
+        assert!(second.await.expect_err("waiter should be cancelled").is_cancelled());
+
+        let server_epoch = Uuid::new_v4();
+        assert_eq!(
+            resolve_put_file_auth_capability(&endpoint, || async {
+                calls.fetch_add(1, Ordering::SeqCst);
+                Ok(Some(server_epoch))
+            })
+            .await
+            .expect("later caller should initialize the abandoned flight"),
+            Some(server_epoch)
+        );
+        assert_eq!(calls.load(Ordering::SeqCst), 2);
+    }
+
+    #[tokio::test]
+    async fn put_file_capability_failed_wave_can_retry_immediately() {
+        let endpoint = format!("http://retry-after-failure-{}.invalid", Uuid::new_v4());
+        let first = resolve_put_file_auth_capability(&endpoint, || async { Err(Error::Timeout) }).await;
+        assert!(matches!(first, Err(Error::Timeout)));
+
+        let server_epoch = Uuid::new_v4();
+        assert_eq!(
+            resolve_put_file_auth_capability(&endpoint, || async { Ok(Some(server_epoch)) })
+                .await
+                .expect("new request should reprobe"),
+            Some(server_epoch)
+        );
+    }
+
+    #[tokio::test]
+    async fn put_file_capability_probes_different_endpoints_in_parallel() {
+        let first_endpoint = format!("http://parallel-a-{}.invalid", Uuid::new_v4());
+        let second_endpoint = format!("http://parallel-b-{}.invalid", Uuid::new_v4());
+        let probes_started = Arc::new(Barrier::new(2));
+        let first_barrier = Arc::clone(&probes_started);
+        let second_barrier = Arc::clone(&probes_started);
+
+        let results = tokio::time::timeout(Duration::from_secs(5), async {
+            tokio::join!(
+                resolve_put_file_auth_capability(&first_endpoint, || async move {
+                    first_barrier.wait().await;
+                    Ok(None)
+                }),
+                resolve_put_file_auth_capability(&second_endpoint, || async move {
+                    second_barrier.wait().await;
+                    Ok(None)
+                })
+            )
+        })
+        .await
+        .expect("different endpoints should not serialize");
+        assert!(results.0.expect("first result").is_none());
+        assert!(results.1.expect("second result").is_none());
+    }
+
+    #[tokio::test]
+    async fn stale_put_file_capability_flight_cannot_overwrite_newer_state() {
+        let endpoint = format!("http://stale-flight-{}.invalid", Uuid::new_v4());
+        let entry = put_file_capability_cache_entry(&endpoint);
+        let probe_started = Arc::new(Notify::new());
+        let release = Arc::new(Notify::new());
+        let stale_epoch = Uuid::new_v4();
+        let newer_epoch = Uuid::new_v4();
+
+        let task = {
+            let endpoint = endpoint.clone();
+            let probe_started = Arc::clone(&probe_started);
+            let release = Arc::clone(&release);
+            tokio::spawn(async move {
+                resolve_put_file_auth_capability(&endpoint, || async move {
+                    probe_started.notify_one();
+                    release.notified().await;
+                    Ok(Some(stale_epoch))
+                })
+                .await
+            })
+        };
+        probe_started.notified().await;
+        {
+            let mut state = entry.write().await;
+            state.generation = state.generation.checked_add(1).expect("test generation should advance");
+            state.cached = Some(PutFileCapabilityState::V1 {
+                server_epoch: newer_epoch,
+                revalidate_after: Instant::now() + PUT_FILE_V1_CAPABILITY_TTL,
+            });
+            state.in_flight = None;
+        }
+        release.notify_one();
+        assert_eq!(
+            task.await.expect("stale task should finish").expect("stale probe result"),
+            Some(stale_epoch)
+        );
+        assert_eq!(
+            fresh_put_file_capability(entry.read().await.cached, Instant::now()),
+            Some(Some(newer_epoch))
+        );
+    }
+
+    #[test]
+    fn put_file_capability_response_fails_closed_on_malformed_or_unbound_data() {
+        let _ = rustfs_credentials::set_global_rpc_secret("put-file-capability-response-test-secret".to_string());
+        let challenge = Uuid::parse_str("11111111-2222-4333-8444-555555555555").expect("challenge");
+        let server_epoch = Uuid::parse_str("aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee").expect("server epoch");
+        let proof = crate::cluster::rpc::sign_put_file_capability(challenge, server_epoch, PUT_FILE_CAPABILITY_VERSION)
+            .expect("proof should build");
+        let response = PutFileCapabilityResponse {
+            version: PUT_FILE_CAPABILITY_VERSION,
+            server_epoch,
+            proof,
+        };
+        let body = rmp_serde::to_vec_named(&response).expect("response should encode");
+
+        assert_eq!(
+            verify_put_file_capability_response(challenge, &body).expect("response should verify"),
+            server_epoch
+        );
+        assert!(verify_put_file_capability_response(Uuid::new_v4(), &body).is_err());
+        assert!(verify_put_file_capability_response(challenge, &body[..body.len() - 1]).is_err());
+        assert!(verify_put_file_capability_response(challenge, &[]).is_err());
+        assert!(verify_put_file_capability_response(challenge, &vec![0_u8; PUT_FILE_MAX_CAPABILITY_RESPONSE_SIZE + 1]).is_err());
+    }
+
     #[tokio::test]
     async fn put_file_auth_writer_appends_trailer_on_shutdown() {
         use tokio::io::AsyncWriteExt;

crates/ecstore/src/cluster/rpc/mod.rs

@@ -34,11 +34,11 @@ pub use client::{
 pub use http_auth::{
     TONIC_RPC_PREFIX, build_auth_headers, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, gen_signature_headers,
     gen_tonic_replay_scope_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
-    set_tonic_mutation_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge,
-    tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_put_file_auth_trailer,
-    verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
-    verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
-    verify_tonic_rpc_signature_with_bootstrap,
+    set_tonic_mutation_body_digest, sign_ns_scanner_capability, sign_put_file_capability, sign_tonic_rpc_response_proof,
+    tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability,
+    verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
+    verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof,
+    verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
 };
 #[cfg(test)]
 pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;

crates/ecstore/src/cluster/rpc/peer_rest_client.rs

@@ -44,9 +44,9 @@ use rustfs_protos::proto_gen::node_service::{
     GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest, GetSysConfigRequest, GetSysErrorsRequest,
     HealControlRequest, LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, LoadPolicyRequest,
     LoadRebalanceMetaRequest, LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest,
-    LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, ScannerActivityRequest,
-    ScannerActivityResponse, ServerInfoRequest, SignalServiceRequest, SignalServiceResponse, StartDecommissionRequest,
-    StartProfilingRequest, StopRebalanceRequest, TierMutationAbortRequest, TierMutationCommitRequest,
+    LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, ReplacementRecoveryStatusRequest,
+    ScannerActivityRequest, ScannerActivityResponse, ServerInfoRequest, SignalServiceRequest, SignalServiceResponse,
+    StartDecommissionRequest, StartProfilingRequest, StopRebalanceRequest, TierMutationAbortRequest, TierMutationCommitRequest,
     TierMutationControlResponse, TierMutationPeerState, TierMutationPrepareRequest, node_service_client::NodeServiceClient,
     tier_mutation_control_service_client::TierMutationControlServiceClient,
 };
@@ -78,6 +78,7 @@ pub const SERVICE_SIGNAL_RELOAD_DYNAMIC: u64 = 2;
 /// reload signal transport.
 pub const KMS_SIGNAL_SUBSYSTEM: &str = "kms";
 const BACKGROUND_HEAL_STATUS_MAX_MESSAGE_SIZE: usize = 64 * 1024;
+const REPLACEMENT_RECOVERY_STATUS_MAX_MESSAGE_SIZE: usize = 64 * 1024;
 const HEAL_CONTROL_FINGERPRINT_MAX_SIZE: usize = 256;
 const HEAL_CONTROL_PAYLOAD_MAX_SIZE: usize = 64 * 1024;
 const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
@@ -1083,6 +1084,38 @@ impl PeerRestClient {
         .await
     }
 
+    pub async fn replacement_recovery_status(&self) -> Result<Option<Vec<u8>>> {
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_client()
+                    .await?
+                    .max_decoding_message_size(REPLACEMENT_RECOVERY_STATUS_MAX_MESSAGE_SIZE);
+                let response = match client
+                    .replacement_recovery_status(Request::new(ReplacementRecoveryStatusRequest::default()))
+                    .await
+                {
+                    Ok(response) => response.into_inner(),
+                    Err(status) if status.code() == tonic::Code::Unimplemented => {
+                        // RUSTFS_COMPAT_TODO(replacement-recovery-status-v1): old peers cannot prove replacement completion during rolling upgrades. Remove after the minimum supported RustFS peer version implements ReplacementRecoveryStatus.
+                        return Ok(None);
+                    }
+                    Err(status) => return Err(status.into()),
+                };
+                if !response.success {
+                    return Err(Error::other(
+                        response
+                            .error_info
+                            .unwrap_or_else(|| "peer replacement recovery status failed without an error".to_string()),
+                    ));
+                }
+                Ok(Some(response.recovery_status.to_vec()))
+            }
+            .await,
+        )
+        .await
+    }
+
     pub async fn prepare_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Result<PeerTierMutationOutcome> {
         self.tier_mutation_control(TierMutationRpcPhase::Prepare, mutation_id, canonical_payload)
             .await

crates/ecstore/src/cluster/rpc/peer_s3_client.rs

@@ -784,7 +784,11 @@ impl PeerS3Client for LocalPeerS3Client {
 
         if opts.force_if_empty && !opts.force {
             for disk in local_disks.iter() {
-                if has_xlmeta_files(&disk.path().join(bucket)).await.map_err(Error::Io)? {
+                let Some(bucket_path) = disk.get_bucket_path_for_io_if_local(bucket) else {
+                    continue;
+                };
+                let bucket_path = bucket_path?;
+                if has_xlmeta_files(&bucket_path).await.map_err(Error::Io)? {
                     return Err(Error::VolumeNotEmpty);
                 }
             }

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -4486,6 +4486,27 @@ mod tests {
         assert_eq!(snapshot.outgoing_requests_total, 0);
     }
 
+    #[tokio::test]
+    #[serial(internode_metrics)]
+    async fn test_remote_disk_create_file_retries_once_on_capability_probe_timeout() {
+        let transport = RetryingOpenWriteInternodeDataTransport::with_steps(vec![
+            OpenWriteTestStep::Error(DiskError::from(rustfs_rio::internode_http_timeout_error(
+                &http::Method::GET,
+                "http://remote-node:9000/rustfs/rpc/put_file_capability",
+            ))),
+            OpenWriteTestStep::Success,
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        crate::cluster::rpc::runtime_sources::reset_internode_metrics_for_test();
+
+        let _created = remote_disk
+            .create_file("orig-bucket", "bucket", "object/part.1", 4096)
+            .await
+            .expect("capability probe timeout should recover on retry");
+
+        assert_eq!(transport.calls().len(), 2, "create_file should retry capability probe timeouts once");
+    }
+
     #[tokio::test]
     async fn test_remote_disk_append_file_does_not_retry_non_retryable_open_write_error() {
         let transport = RetryingOpenWriteInternodeDataTransport::with_steps(vec![OpenWriteTestStep::Error(DiskError::from(

crates/ecstore/src/core/sets.rs

@@ -286,7 +286,7 @@ impl Sets {
         self.get_disks(self.get_hashed_set_index(key))
     }
 
-    fn get_disks_for_heal_object(&self, key: &str, opts: &HealOpts) -> Result<Arc<SetDisks>> {
+    pub(crate) fn get_disks_for_heal_object(&self, key: &str, opts: &HealOpts) -> Result<Arc<SetDisks>> {
         match opts.set {
             Some(set_idx) => self.disk_set.get(set_idx).cloned().ok_or_else(|| {
                 StorageError::InvalidArgument(
@@ -1058,17 +1058,23 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
             for (i, set) in new_format_sets.iter().enumerate() {
                 for (j, fm) in set.iter().enumerate() {
                     if let Some(fm) = fm {
-                        res.after.drives[i * self.set_drive_count + j].uuid = fm.erasure.this.to_string();
-                        res.after.drives[i * self.set_drive_count + j].state = DriveState::Ok.to_string();
                         tmp_new_formats[i * self.set_drive_count + j] = Some(fm.clone());
                     }
                 }
             }
             // Save new formats `format.json` on unformatted disks.
-            for (fm, disk) in tmp_new_formats.iter_mut().zip(disks.iter()) {
-                if fm.is_some() && disk.is_some() && save_format_file(disk, fm).await.is_err() {
-                    let _ = disk.as_ref().unwrap().close().await;
-                    *fm = None;
+            for (index, (fm, disk)) in tmp_new_formats.iter_mut().zip(disks.iter()).enumerate() {
+                if fm.is_some() && disk.is_some() {
+                    if let Err(err) = save_format_file(disk, fm).await {
+                        if let Some(disk) = disk.as_ref() {
+                            let _ = disk.close().await;
+                        }
+                        return Ok((res, Some(err.into())));
+                    }
+                    if let Some(saved_format) = fm.as_ref() {
+                        res.after.drives[index].uuid = saved_format.erasure.this.to_string();
+                        res.after.drives[index].state = DriveState::Ok.to_string();
+                    }
                 }
             }
 
@@ -1215,6 +1221,98 @@ async fn init_storage_disks_with_errors(
     (disks, errs)
 }
 
+#[cfg(test)]
+pub(crate) async fn make_local_two_set_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
+    make_local_two_set_sets_with_ctx(bootstrap_ctx()).await
+}
+
+#[cfg(test)]
+pub(crate) async fn make_local_two_set_sets_with_ctx(ctx: Arc<InstanceContext>) -> (Vec<tempfile::TempDir>, Arc<Sets>) {
+    use crate::layout::endpoint::Endpoint;
+    use rustfs_lock::client::local::LocalClient;
+
+    let format = FormatV3::new(2, 2);
+    let mut temp_dirs = Vec::new();
+    let mut all_endpoints = Vec::new();
+    let mut disk_sets = Vec::new();
+
+    for set_index in 0..2 {
+        let mut endpoints = Vec::new();
+        let mut disks = Vec::new();
+        for disk_index in 0..2 {
+            let temp_dir = tempfile::tempdir().expect("tempdir should be created");
+            let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("tempdir path should be utf8"))
+                .expect("endpoint should parse");
+            endpoint.set_pool_index(0);
+            endpoint.set_set_index(set_index);
+            endpoint.set_disk_index(disk_index);
+            let disk = new_disk(
+                &endpoint,
+                &DiskOption {
+                    cleanup: false,
+                    health_check: false,
+                },
+            )
+            .await
+            .expect("disk should be created");
+            let mut disk_format = format.clone();
+            disk_format.erasure.this = format.erasure.sets[set_index][disk_index];
+            save_format_file(&Some(disk.clone()), &Some(disk_format))
+                .await
+                .expect("format should be saved");
+            temp_dirs.push(temp_dir);
+            all_endpoints.push(endpoint.clone());
+            endpoints.push(endpoint);
+            disks.push(Some(disk));
+        }
+        let lockers = (0..2)
+            .map(|_| {
+                Arc::new(LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::Enabled(Arc::new(
+                    rustfs_lock::FastObjectLockManager::new(),
+                ))))) as Arc<dyn rustfs_lock::LockClient>
+            })
+            .collect();
+        disk_sets.push(
+            SetDisks::new_with_instance_ctx(
+                "test-owner".to_string(),
+                Arc::new(RwLock::new(disks)),
+                2,
+                1,
+                set_index,
+                0,
+                endpoints,
+                format.clone(),
+                lockers,
+                Arc::clone(&ctx),
+            )
+            .await,
+        );
+    }
+
+    let sets = Arc::new(Sets {
+        id: format.id,
+        disk_set: disk_sets,
+        pool_idx: 0,
+        endpoints: PoolEndpoints {
+            legacy: false,
+            set_count: 2,
+            drives_per_set: 2,
+            endpoints: Endpoints::from(all_endpoints),
+            cmd_line: String::new(),
+            platform: String::new(),
+        },
+        format,
+        parity_count: 1,
+        set_count: 2,
+        set_drive_count: 2,
+        default_parity_count: 1,
+        distribution_algo: DistributionAlgoVersion::V1,
+        exit_signal: None,
+        ctx,
+    });
+    (temp_dirs, sets)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1373,84 +1471,9 @@ mod tests {
         assert_eq!(result, (Some(3), Some(1), Some(0)));
     }
 
-    async fn two_set_test_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
-        let format = FormatV3::new(2, 2);
-        let mut temp_dirs = Vec::new();
-        let mut all_endpoints = Vec::new();
-        let mut disk_sets = Vec::new();
-
-        for set_index in 0..2 {
-            let mut endpoints = Vec::new();
-            let mut disks = Vec::new();
-            for disk_index in 0..2 {
-                let temp_dir = tempfile::tempdir().expect("tempdir should be created");
-                let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("tempdir path should be utf8"))
-                    .expect("endpoint should parse");
-                endpoint.set_pool_index(0);
-                endpoint.set_set_index(set_index);
-                endpoint.set_disk_index(disk_index);
-                let disk = new_disk(
-                    &endpoint,
-                    &DiskOption {
-                        cleanup: false,
-                        health_check: false,
-                    },
-                )
-                .await
-                .expect("disk should be created");
-                let mut disk_format = format.clone();
-                disk_format.erasure.this = format.erasure.sets[set_index][disk_index];
-                save_format_file(&Some(disk.clone()), &Some(disk_format))
-                    .await
-                    .expect("format should be saved");
-                temp_dirs.push(temp_dir);
-                all_endpoints.push(endpoint.clone());
-                endpoints.push(endpoint);
-                disks.push(Some(disk));
-            }
-            disk_sets.push(
-                SetDisks::new(
-                    "test-owner".to_string(),
-                    Arc::new(RwLock::new(disks)),
-                    2,
-                    1,
-                    set_index,
-                    0,
-                    endpoints,
-                    format.clone(),
-                    vec![Arc::new(LocalClient::new()), Arc::new(LocalClient::new())],
-                )
-                .await,
-            );
-        }
-
-        let sets = Arc::new(Sets {
-            id: format.id,
-            disk_set: disk_sets,
-            pool_idx: 0,
-            endpoints: PoolEndpoints {
-                legacy: false,
-                set_count: 2,
-                drives_per_set: 2,
-                endpoints: Endpoints::from(all_endpoints),
-                cmd_line: String::new(),
-                platform: String::new(),
-            },
-            format,
-            parity_count: 1,
-            set_count: 2,
-            set_drive_count: 2,
-            default_parity_count: 1,
-            distribution_algo: DistributionAlgoVersion::V1,
-            exit_signal: None,
-            ctx: bootstrap_ctx(),
-        });
-        (temp_dirs, sets)
-    }
-
     #[tokio::test]
     async fn heal_object_uses_explicit_set_scope() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let selected = sets
             .get_disks_for_heal_object(
                 "object",
@@ -1466,7 +1489,7 @@ mod tests {
 
     #[tokio::test]
     async fn heal_object_without_set_scope_keeps_hash_routing() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let object = "object";
         let selected = sets
             .get_disks_for_heal_object(object, &HealOpts::default())
@@ -1477,7 +1500,7 @@ mod tests {
 
     #[tokio::test]
     async fn heal_object_rejects_invalid_set_scope() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let err = sets
             .get_disks_for_heal_object(
                 "object",
@@ -1497,7 +1520,7 @@ mod tests {
 
     #[tokio::test]
     async fn delete_prefix_surfaces_a_hard_error_from_any_set() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let bucket = format!("delete-prefix-{}", Uuid::new_v4().simple());
         sets.make_bucket(&bucket, &MakeBucketOptions::default())
             .await
@@ -1546,7 +1569,7 @@ mod tests {
 
     #[tokio::test]
     async fn delete_prefix_keeps_a_missing_bucket_idempotent_across_sets() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let bucket = format!("delete-prefix-{}", Uuid::new_v4().simple());
         sets.make_bucket(&bucket, &MakeBucketOptions::default())
             .await
@@ -1585,7 +1608,7 @@ mod tests {
 
     #[tokio::test]
     async fn delete_prefix_preserves_a_completely_missing_bucket_error() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let bucket = format!("delete-prefix-missing-{}", Uuid::new_v4().simple());
 
         let err = sets
@@ -1605,7 +1628,7 @@ mod tests {
 
     #[tokio::test]
     async fn delete_prefix_fails_when_one_set_is_entirely_offline() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let bucket = format!("delete-prefix-{}", Uuid::new_v4().simple());
         sets.make_bucket(&bucket, &MakeBucketOptions::default())
             .await
@@ -1652,7 +1675,7 @@ mod tests {
 
     #[tokio::test]
     async fn set_format_heal_accepts_quorum_from_a_nonzero_set() {
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
 
         let (result, err) = sets.disk_set[1]
             .heal_format(false)
@@ -1757,7 +1780,7 @@ mod tests {
     #[serial]
     async fn list_multipart_uploads_merges_all_sets_without_pagination_loss() {
         let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::Erasure).await;
-        let (_temp_dirs, sets) = two_set_test_sets().await;
+        let (_temp_dirs, sets) = make_local_two_set_sets().await;
         let bucket = format!("multipart-list-{}", Uuid::new_v4().simple());
         sets.make_bucket(&bucket, &MakeBucketOptions::default())
             .await
@@ -2189,6 +2212,39 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn replacement_format_only_writes_the_requested_slot() {
+        let (_dirs, _ref_format, sets) = setup_heal_format_sets(1, false).await;
+        let target = sets.endpoints.endpoints.as_ref()[1].to_string();
+        let untouched = sets.endpoints.endpoints.as_ref()[2].to_string();
+        let set = set_level_heal_view(&sets).await;
+
+        let (result, error) = set
+            .heal_replacement_format(false, std::slice::from_ref(&target))
+            .await
+            .expect("target-scoped replacement format should run");
+
+        assert!(error.is_none(), "target format must not report an error: {error:?}");
+        assert!(
+            result
+                .after
+                .drives
+                .iter()
+                .any(|drive| drive.endpoint == target && drive.state == DriveState::Ok.to_string()),
+            "requested replacement slot must be formatted"
+        );
+        let untouched_format = std::path::Path::new(&sets.endpoints.endpoints.as_ref()[2].get_file_path())
+            .join(crate::disk::RUSTFS_META_BUCKET)
+            .join(crate::disk::FORMAT_CONFIG_FILE);
+        assert!(
+            !tokio::fs::try_exists(untouched_format)
+                .await
+                .expect("untouched replacement format path should be inspectable"),
+            "unrequested slot {untouched} must remain unformatted"
+        );
+    }
+
     fn instance_ctx_test_pool_endpoints() -> (FormatV3, PoolEndpoints) {
         let format = FormatV3::new(1, 2);
         let endpoints = vec![

crates/ecstore/src/disk/disk_store.rs

@@ -152,6 +152,7 @@ const DISK_OPERATION_NAMES: &[&str] = &[
     "read_parts",
     "read_multiple",
     "write_all",
+    "compare_and_update_file",
     "read_all",
 ];
 
@@ -1092,6 +1093,18 @@ impl LocalDiskWrapper {
         self.disk.get_object_path(volume, path)
     }
 
+    pub(crate) fn get_object_path_for_io(&self, volume: &str, path: &str) -> crate::disk::error::Result<std::path::PathBuf> {
+        self.disk.get_object_path_for_io(volume, path)
+    }
+
+    pub(crate) fn get_bucket_path_for_io(&self, volume: &str) -> crate::disk::error::Result<std::path::PathBuf> {
+        self.disk.get_bucket_path_for_io(volume)
+    }
+
+    pub fn replacement_mount_lease_root(&self) -> Option<std::path::PathBuf> {
+        self.disk.replacement_mount_lease_root()
+    }
+
     pub fn runtime_state(&self) -> RuntimeDriveHealthState {
         self.health.runtime_state()
     }
@@ -1639,6 +1652,10 @@ impl LocalDiskWrapper {
 
 #[async_trait::async_trait]
 impl DiskAPI for LocalDiskWrapper {
+    fn has_replacement_mount_lease(&self) -> bool {
+        self.disk.has_replacement_mount_lease()
+    }
+
     async fn read_metadata(&self, volume: &str, path: &str) -> Result<Bytes> {
         self.track_disk_health_with_op_and_timeout_action(
             "read_metadata",
@@ -2140,6 +2157,22 @@ impl DiskAPI for LocalDiskWrapper {
         .await
     }
 
+    async fn compare_and_update_file(
+        &self,
+        volume: &str,
+        path: &str,
+        expected: Option<Bytes>,
+        replacement: Option<Bytes>,
+    ) -> Result<crate::disk::ConditionalFileUpdate> {
+        self.track_disk_health_mutation(
+            "compare_and_update_file",
+            DiskMetricMutation::Write,
+            || async { self.disk.compare_and_update_file(volume, path, expected, replacement).await },
+            get_max_timeout_duration(),
+        )
+        .await
+    }
+
     async fn read_all(&self, volume: &str, path: &str) -> Result<Bytes> {
         self.track_disk_health_with_op(
             "read_all",

crates/ecstore/src/disk/mod.rs

@@ -115,6 +115,15 @@ pub enum PartTransactionAction {
     Rollback,
 }
 
+/// Result of an owner-aware file mutation. The disk applies the mutation only
+/// while the current contents match the supplied expected value.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum ConditionalFileUpdate {
+    Updated,
+    Missing,
+    Mismatch,
+}
+
 #[derive(Clone, Copy, Debug)]
 pub struct MmapCopyStageMetrics {
     pub(crate) path: &'static str,
@@ -557,6 +566,26 @@ impl DiskAPI for Disk {
         }
     }
 
+    async fn compare_and_update_file(
+        &self,
+        volume: &str,
+        path: &str,
+        expected: Option<Bytes>,
+        replacement: Option<Bytes>,
+    ) -> Result<ConditionalFileUpdate> {
+        match self {
+            Disk::Local(local_disk) => local_disk.compare_and_update_file(volume, path, expected, replacement).await,
+            Disk::Remote(remote_disk) => remote_disk.compare_and_update_file(volume, path, expected, replacement).await,
+        }
+    }
+
+    fn has_replacement_mount_lease(&self) -> bool {
+        match self {
+            Disk::Local(local_disk) => local_disk.has_replacement_mount_lease(),
+            Disk::Remote(remote_disk) => remote_disk.has_replacement_mount_lease(),
+        }
+    }
+
     #[tracing::instrument(level = "trace", skip_all)]
     async fn read_all(&self, volume: &str, path: &str) -> Result<Bytes> {
         match self {
@@ -695,6 +724,34 @@ impl Disk {
             Disk::Remote(_) => None,
         }
     }
+
+    pub(crate) fn get_object_path_for_io_if_local(
+        &self,
+        volume: &str,
+        path: &str,
+    ) -> Option<crate::disk::error::Result<std::path::PathBuf>> {
+        match self {
+            Disk::Local(w) => Some(w.get_object_path_for_io(volume, path)),
+            Disk::Remote(_) => None,
+        }
+    }
+
+    pub(crate) fn get_bucket_path_for_io_if_local(&self, volume: &str) -> Option<crate::disk::error::Result<std::path::PathBuf>> {
+        match self {
+            Disk::Local(w) => Some(w.get_bucket_path_for_io(volume)),
+            Disk::Remote(_) => None,
+        }
+    }
+
+    /// Return the descriptor-rooted mount path admitted for automatic
+    /// replacement, or `None` when the configured endpoint no longer names
+    /// that held mount instance.
+    pub fn replacement_mount_lease_root(&self) -> Option<PathBuf> {
+        match self {
+            Disk::Local(local_disk) => local_disk.replacement_mount_lease_root(),
+            Disk::Remote(_) => None,
+        }
+    }
 }
 
 pub async fn new_disk(ep: &Endpoint, opt: &DiskOption) -> Result<DiskStore> {
@@ -860,6 +917,24 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
     // CleanAbandonedData
     async fn write_all(&self, volume: &str, path: &str, data: Bytes) -> Result<()>;
     async fn read_all(&self, volume: &str, path: &str) -> Result<Bytes>;
+    /// Atomically replace or remove a small control file only when its current
+    /// contents match `expected`. Implementations that cannot provide this
+    /// cross-process guarantee must fail closed instead of emulating it with a
+    /// read-then-write sequence.
+    async fn compare_and_update_file(
+        &self,
+        _volume: &str,
+        _path: &str,
+        _expected: Option<Bytes>,
+        _replacement: Option<Bytes>,
+    ) -> Result<ConditionalFileUpdate> {
+        Err(DiskError::MethodNotAllowed)
+    }
+    /// Whether local I/O is rooted at a held mount descriptor. Auto-replacement
+    /// refuses destructive work when this is false.
+    fn has_replacement_mount_lease(&self) -> bool {
+        false
+    }
     async fn disk_info(&self, opts: &DiskInfoOptions) -> Result<DiskInfo>;
     fn start_scan(&self) -> ScanGuard;
 }
@@ -1612,6 +1687,7 @@ mod tests {
 
         let endpoint = Endpoint::try_from(test_dir).unwrap();
         let local_disk = LocalDisk::new(&endpoint, false).await.unwrap();
+        let expected_object_path = local_disk.root.join("test-bucket/test-object");
         let disk = Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(local_disk), false)));
 
         // Test basic methods
@@ -1626,6 +1702,19 @@ mod tests {
         // Test path method
         let path = disk.path();
         assert!(path.exists());
+        let object_path = disk
+            .get_object_path_if_local("test-bucket", "test-object")
+            .expect("local disk should expose an object path")
+            .expect("object path should resolve");
+        assert_eq!(object_path, expected_object_path);
+        assert!(!object_path.starts_with("/proc/self/fd/"));
+        #[cfg(target_os = "linux")]
+        assert!(
+            disk.get_object_path_for_io_if_local("test-bucket", "test-object")
+                .expect("local disk should expose an I/O object path")
+                .expect("I/O object path should resolve")
+                .starts_with("/proc/self/fd/")
+        );
 
         // Test disk location
         let location = disk.get_disk_location();

crates/ecstore/src/erasure/coding/decode.rs

@@ -1069,13 +1069,11 @@ where
         }
 
         // Pre-claim per-slot buffers so the `self.readers` borrow below stays
-        // disjoint from `self.buffers`.
-        let participating: Vec<bool> = (0..num_readers)
-            .map(|i| self.engaged[i] && self.readers[i].is_some())
-            .collect();
+        // disjoint from `self.buffers`; `Some(buffer)` also records which slots
+        // participate, avoiding a per-stripe sidecar allocation.
         let mut bufs: Vec<Option<Vec<u8>>> = Vec::with_capacity(num_readers);
-        for (i, participates) in participating.iter().enumerate() {
-            bufs.push(if *participates {
+        for i in 0..num_readers {
+            bufs.push(if self.engaged[i] && self.readers[i].is_some() {
                 Some(self.buffers.take(i, shard_size))
             } else {
                 None
@@ -1085,7 +1083,6 @@ where
         let data_shards = self.data_shards;
         let read_timeout = self.read_timeout;
         let metrics_path = self.metrics_path;
-        let read_costs = self.read_costs.clone();
         let locality_preference_enabled = self.locality_preference_enabled;
         let stripe_read_start = metrics_path.map(|_| Instant::now());
 
@@ -1100,19 +1097,21 @@ where
         // before the retirement pass mutates `self.readers` below.
         {
             let mut sets = FuturesUnordered::new();
-            let reader_iter = ReaderLaunchIter::new(&mut self.readers, &read_costs, locality_preference_enabled);
+            let reader_iter = ReaderLaunchIter::new(&mut self.readers, self.read_costs.as_slice(), locality_preference_enabled);
             for (i, reader) in reader_iter {
-                if reader.is_none() || !participating[i] {
+                if reader.is_none() {
                     continue;
                 }
-                let read_cost = read_costs.get(i).copied().unwrap_or(ShardReadCost::Unknown);
-                let recycled_buf = bufs[i].take();
+                let Some(recycled_buf) = bufs[i].take() else {
+                    continue;
+                };
+                let read_cost = self.read_costs.get(i).copied().unwrap_or(ShardReadCost::Unknown);
                 scheduled += 1;
                 sets.push(read_shard(
                     i,
                     read_cost,
                     reader,
-                    recycled_buf,
+                    Some(recycled_buf),
                     shard_size,
                     data_shards,
                     read_timeout,
@@ -1208,7 +1207,7 @@ where
         // covered by the stripe-aligned parity substitution below.
         if hedged {
             for i in 0..num_readers {
-                if participating[i] && shards[i].is_none() && errs[i].is_none() {
+                if self.engaged[i] && self.readers[i].is_some() && shards[i].is_none() && errs[i].is_none() {
                     errs[i] = Some(Error::from(io::Error::new(ErrorKind::TimedOut, "shard read hedged after a slow shard")));
                     retire_readers.push(i);
                 }
@@ -1237,7 +1236,7 @@ where
             if !self.try_engage_parity(idx, stripe_index) {
                 continue;
             }
-            let read_cost = read_costs.get(idx).copied().unwrap_or(ShardReadCost::Unknown);
+            let read_cost = self.read_costs.get(idx).copied().unwrap_or(ShardReadCost::Unknown);
             let recycled_buf = Some(self.buffers.take(idx, shard_size));
             scheduled += 1;
             let (i, _read_cost, result, _should_retire) = read_shard(

crates/ecstore/src/layout/endpoints.rs

@@ -643,6 +643,48 @@ fn local_host_resolution_timeout_forced(host: &Host<&str>) -> bool {
         .contains(&host)
 }
 
+#[cfg(test)]
+static FORCED_KERNEL_HOSTNAME: LazyLock<Mutex<Option<String>>> = LazyLock::new(|| Mutex::new(None));
+
+#[cfg(test)]
+struct KernelHostnameOverrideGuard;
+
+#[cfg(test)]
+impl Drop for KernelHostnameOverrideGuard {
+    fn drop(&mut self) {
+        *FORCED_KERNEL_HOSTNAME
+            .lock()
+            .expect("kernel-hostname test override mutex poisoned") = None;
+    }
+}
+
+/// Overrides the kernel hostname seen by Kubernetes endpoint-identity
+/// inference so tests stay deterministic on hosts whose kernel hostname is
+/// not a DNS name (e.g. macOS with a DHCP-assigned IP-literal hostname).
+#[cfg(test)]
+fn force_kernel_hostname_for_test(hostname: &str) -> KernelHostnameOverrideGuard {
+    *FORCED_KERNEL_HOSTNAME
+        .lock()
+        .expect("kernel-hostname test override mutex poisoned") = Some(hostname.to_string());
+    KernelHostnameOverrideGuard
+}
+
+fn kernel_hostname_for_endpoint_identity() -> Result<String> {
+    #[cfg(test)]
+    if let Some(hostname) = FORCED_KERNEL_HOSTNAME
+        .lock()
+        .expect("kernel-hostname test override mutex poisoned")
+        .clone()
+    {
+        return Ok(hostname);
+    }
+
+    hostname::get()
+        .map_err(|err| Error::other(format!("failed to read the kernel hostname for Kubernetes endpoint identity: {err}")))?
+        .into_string()
+        .map_err(|_| Error::new(ErrorKind::InvalidData, "kernel hostname is not valid UTF-8"))
+}
+
 fn endpoint_is_local_host(host: Host<&str>, port: u16, local_port: u16) -> Result<bool> {
     #[cfg(test)]
     if local_host_resolution_timeout_forced(&host) {
@@ -1268,12 +1310,7 @@ impl EndpointServerPools {
             && std::env::var_os(ENV_KUBERNETES_SERVICE_HOST).is_some()
             && matches!(wait_mode.as_deref(), None | Some("") | Some("auto") | Some("orchestrated"));
         if infer_kubernetes_host {
-            let kernel_hostname = hostname::get()
-                .map_err(|err| {
-                    Error::other(format!("failed to read the kernel hostname for Kubernetes endpoint identity: {err}"))
-                })?
-                .into_string()
-                .map_err(|_| Error::new(ErrorKind::InvalidData, "kernel hostname is not valid UTF-8"))?;
+            let kernel_hostname = kernel_hostname_for_endpoint_identity()?;
             let local_port = check_local_server_addr(server_addr)?.port();
             match infer_kubernetes_local_endpoint_host(disks_layout, local_port, &kernel_hostname)? {
                 Some(inferred_host) => local_endpoint_host = Some(inferred_host),
@@ -2217,21 +2254,8 @@ mod test {
     #[serial]
     #[tokio::test]
     async fn create_server_endpoints_infers_kubernetes_pod_host_without_peer_dns() {
-        let raw_hostname = hostname::get()
-            .expect("kernel hostname should be available")
-            .into_string()
-            .expect("kernel hostname should be UTF-8");
-        let Host::Domain(kernel_hostname) = Host::parse(raw_hostname.trim()).expect("kernel hostname should be a DNS name")
-        else {
-            panic!("kernel hostname should be a DNS name");
-        };
-        let kernel_hostname =
-            domain_without_optional_trailing_dot(&kernel_hostname).expect("kernel hostname should be canonical");
-        let local_host = if kernel_hostname.contains('.') {
-            kernel_hostname.to_string()
-        } else {
-            format!("{kernel_hostname}.rustfs-headless.ns.svc.cluster.local")
-        };
+        let _kernel_hostname = force_kernel_hostname_for_test("rustfs-0");
+        let local_host = "rustfs-0.rustfs-headless.ns.svc.cluster.local";
 
         async_with_vars(
             [
@@ -2287,6 +2311,7 @@ mod test {
     #[serial]
     #[tokio::test]
     async fn create_server_endpoints_bounds_kubernetes_alias_dns_fallback() {
+        let _kernel_hostname = force_kernel_hostname_for_test("unmatched-test-node");
         let _resolution_timeout =
             force_local_host_resolution_timeout_for_test(&["unrelated-0.example.invalid", "unrelated-1.example.invalid"]);
 
@@ -2319,6 +2344,7 @@ mod test {
     #[serial]
     #[tokio::test]
     async fn create_server_endpoints_preserves_resolvable_kubernetes_aliases() {
+        let _kernel_hostname = force_kernel_hostname_for_test("unmatched-test-node");
         async_with_vars(
             [
                 (ENV_LOCAL_ENDPOINT_HOST, None),

crates/ecstore/src/object_api/readers.rs

@@ -894,6 +894,7 @@ impl GetObjectReader {
             .await?
             .into_reader(reader, oi)
     }
+    #[hotpath::measure(impl_type = "GetObjectReader")]
     pub async fn read_all(&mut self) -> Result<Vec<u8>> {
         let mut data = Vec::new();
         self.stream.read_to_end(&mut data).await?;

crates/ecstore/src/object_api/types.rs

@@ -253,6 +253,11 @@ pub struct ObjectOptions {
     /// fence avoids recursively acquiring the read lock behind a queued writer.
     pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
     pub replication_request: bool,
+    /// Authorized SSE-C replication passthrough: the body is already
+    /// ciphertext, so the write path must not encrypt or compress it and
+    /// stores the restored encryption metadata verbatim. Only the
+    /// replication-authorized options builders may set this.
+    pub preserve_ciphertext: bool,
     pub delete_marker: bool,
     pub synthetic_version_id: bool,
 
@@ -1041,7 +1046,10 @@ impl ObjectInfo {
         if let Some(data) = &self.checksum {
             if self.is_encrypted() {
                 // Object-level encrypted checksum bytes require SSE decrypt material,
-                // so do not expose them as plaintext checksum headers here.
+                // so do not expose them as plaintext checksum headers here. The
+                // `false` multipart flag feeds the response-path COMPOSITE
+                // fallback; callers that need accurate multipart routing must
+                // consult `is_multipart()` instead of this value.
                 return Ok((HashMap::new(), false));
             }
 
@@ -1712,6 +1720,34 @@ mod tests {
         assert!(checksums.is_empty());
     }
 
+    #[test]
+    fn decrypt_checksums_keeps_encrypted_multipart_flag_false_for_response_paths() {
+        let checksum = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::CRC32, b"encrypted-object")
+            .expect("test checksum should be valid");
+        let info = ObjectInfo {
+            checksum: Some(checksum.to_bytes(&[])),
+            // Multipart ETag shape: md5-of-md5s with a part-count suffix.
+            etag: Some("0123456789abcdef0123456789abcdef-3".to_string()),
+            user_defined: Arc::new(HashMap::from([(
+                rustfs_utils::http::headers::AMZ_SERVER_SIDE_ENCRYPTION.to_string(),
+                "AES256".to_string(),
+            )])),
+            ..Default::default()
+        };
+
+        let (checksums, is_multipart) = info
+            .decrypt_checksums(0, &HeaderMap::new())
+            .expect("encrypted checksum should fail closed");
+
+        // The response path infers COMPOSITE from is_multipart=true when the
+        // checksum type is unreadable, so encrypted objects must keep the
+        // flag false here even when the object itself is multipart. Callers
+        // that need routing (replication) consult is_multipart() directly.
+        assert!(checksums.is_empty());
+        assert!(!is_multipart);
+        assert!(info.is_multipart());
+    }
+
     #[test]
     fn decrypt_checksums_keeps_encrypted_part_checksum_metadata() {
         let checksum = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::CRC32, b"encrypted-object")

crates/ecstore/src/runtime/instance.rs

@@ -208,6 +208,11 @@ impl InstanceContext {
         }
     }
 
+    #[cfg(test)]
+    pub(crate) fn with_lock_manager_for_test(lock_manager: Arc<GlobalLockManager>) -> Self {
+        Self::with_lock_manager(lock_manager)
+    }
+
     /// This instance's namespace lock manager.
     pub fn lock_manager(&self) -> Arc<GlobalLockManager> {
         self.lock_manager.clone()

crates/ecstore/src/set_disk/core/io_primitives.rs

@@ -461,6 +461,34 @@ impl MetadataQuorumAccumulator {
         None
     }
 
+    pub(in crate::set_disk) fn can_still_reach_early_stop_with_pending(&self, pending: usize) -> bool {
+        if !self.allow_early_stop {
+            return false;
+        }
+        if self.delete_marker_votes.saturating_add(pending) >= self.default_write_quorum() {
+            return true;
+        }
+        if self.conflicting_metadata
+            || self.delete_marker_seen
+            || self.not_found_responses > 0
+            || self.version_not_found_responses > 0
+            || self.hard_errors > 0
+        {
+            return false;
+        }
+        if !self.requested_version_id.is_empty()
+            && self.matching_version_votes.saturating_add(pending) >= self.read_quorum_for_version()
+        {
+            return true;
+        }
+        match &self.candidate {
+            Some(candidate) => self
+                .candidate_latest_quorum(candidate)
+                .is_some_and(|latest_quorum| self.candidate_votes.saturating_add(pending) >= latest_quorum),
+            None => pending >= self.default_write_quorum(),
+        }
+    }
+
     /// Compute the read quorum threshold for version-aware early-stop.
     /// Uses `total_disks / 2` (like `missing_response_quorum`) when
     /// `default_parity_count` is set, otherwise requires all disks.
@@ -1439,7 +1467,7 @@ async fn try_create_bitrot_readers_via_batch_pread(
         if let Some(disk) = disk_op.as_ref() {
             let data_dir = files[idx].data_dir.unwrap_or_default();
             let path_str = format!("{object}/{data_dir}/part.{part_number}");
-            match disk.get_object_path_if_local(bucket, &path_str) {
+            match disk.get_object_path_for_io_if_local(bucket, &path_str) {
                 Some(Ok(p)) => batch_items.push((idx, p, adj_off, adj_len)),
                 _ => return None,
             }
@@ -1982,7 +2010,7 @@ pub(in crate::set_disk) fn should_allow_metadata_early_stop(
     healing: bool,
     incl_free_versions: bool,
 ) -> bool {
-    if read_data {
+    if read_data && !is_get_metadata_data_read_early_stop_enabled() {
         return false;
     }
 
@@ -2289,23 +2317,39 @@ impl SetDisks {
         let object = Arc::new(object.to_string());
         let version_id = Arc::new(version_id.to_string());
         let mut join_set = JoinSet::new();
+        let bounded_fanout = is_get_metadata_early_stop_bounded_fanout_enabled();
+        let mut next_disk_index = 0usize;
+        let spawn_read_version =
+            |join_set: &mut JoinSet<(usize, disk::error::Result<FileInfo>, Duration)>, index: usize, disk: Option<DiskStore>| {
+                let opts = opts.clone();
+                let org_bucket = org_bucket.clone();
+                let bucket = bucket.clone();
+                let object = object.clone();
+                let version_id = version_id.clone();
+                join_set.spawn(async move {
+                    let response_start = Instant::now();
+                    let result = if let Some(disk) = disk {
+                        Self::record_read_version_call(&object, index);
+                        disk.read_version(&org_bucket, &bucket, &object, &version_id, &opts).await
+                    } else {
+                        Err(DiskError::DiskNotFound)
+                    };
+                    (index, result, response_start.elapsed())
+                });
+            };
 
-        for (index, disk) in disks.iter().cloned().enumerate() {
-            let opts = opts.clone();
-            let org_bucket = org_bucket.clone();
-            let bucket = bucket.clone();
-            let object = object.clone();
-            let version_id = version_id.clone();
-            join_set.spawn(async move {
-                let response_start = Instant::now();
-                let result = if let Some(disk) = disk {
-                    Self::record_read_version_call(&object, index);
-                    disk.read_version(&org_bucket, &bucket, &object, &version_id, &opts).await
-                } else {
-                    Err(DiskError::DiskNotFound)
-                };
-                (index, result, response_start.elapsed())
-            });
+        if bounded_fanout {
+            let initial_target = accumulator.default_write_quorum().min(disks.len());
+            while next_disk_index < initial_target {
+                if let Some(disk) = disks.get(next_disk_index).cloned() {
+                    spawn_read_version(&mut join_set, next_disk_index, disk);
+                }
+                next_disk_index = next_disk_index.saturating_add(1);
+            }
+        } else {
+            for (index, disk) in disks.iter().cloned().enumerate() {
+                spawn_read_version(&mut join_set, index, disk);
+            }
         }
 
         while let Some(result) = join_set.join_next().await {
@@ -2337,7 +2381,11 @@ impl SetDisks {
                 .early_stop_decision()
                 .or_else(|| accumulator.version_early_stop_decision())
             {
-                let saved_responses = join_set.len();
+                let saved_responses = if bounded_fanout {
+                    disks.len().saturating_sub(observations.len())
+                } else {
+                    join_set.len()
+                };
                 join_set.abort_all();
                 rustfs_io_metrics::record_get_object_metadata_early_stop_hit(GET_OBJECT_PATH_LEGACY_DUPLEX, decision.reason);
                 rustfs_io_metrics::record_get_object_metadata_early_stop_saved_responses(
@@ -2348,6 +2396,16 @@ impl SetDisks {
                 let diagnostics = MetadataFanoutDiagnostics::new(fanout_start.elapsed(), observations);
                 return Ok((ress, errors, diagnostics));
             }
+
+            if bounded_fanout
+                && next_disk_index < disks.len()
+                && !accumulator.can_still_reach_early_stop_with_pending(join_set.len())
+            {
+                if let Some(disk) = disks.get(next_disk_index).cloned() {
+                    spawn_read_version(&mut join_set, next_disk_index, disk);
+                }
+                next_disk_index = next_disk_index.saturating_add(1);
+            }
         }
 
         rustfs_io_metrics::record_get_object_metadata_early_stop_miss(
@@ -4651,7 +4709,7 @@ pub(in crate::set_disk) mod cleanup_fault_injection {
 /// unobserved object records nothing, keeping the registry bounded, and each
 /// [`CallCounterScope`] clears only its own object's counts on drop.
 #[cfg(test)]
-pub(in crate::set_disk) mod disk_call_counters {
+pub(crate) mod disk_call_counters {
     use std::collections::{HashMap, HashSet};
     use std::sync::{Mutex, OnceLock};
 
@@ -5254,6 +5312,166 @@ mod tests {
         drop(dirs);
     }
 
+    fn valid_metadata_fanout_fileinfo(
+        bucket: &str,
+        object: &str,
+        version_id: Uuid,
+        data_dir: Uuid,
+        mod_time: OffsetDateTime,
+    ) -> FileInfo {
+        let mut fi = FileInfo::new(object, 2, 2);
+        fi.volume = bucket.to_string();
+        fi.name = object.to_string();
+        fi.size = 1;
+        fi.erasure.index = 1;
+        fi.version_id = Some(version_id);
+        fi.is_latest = true;
+        fi.data_dir = Some(data_dir);
+        fi.mod_time = Some(mod_time);
+        fi.metadata.insert("etag".to_string(), "etag-1".to_string());
+        fi.add_object_part(1, "part-etag".to_string(), 1, fi.mod_time, 1, None, None);
+        fi
+    }
+
+    async fn install_metadata_fanout_fileinfo(
+        disks: &[Option<DiskStore>],
+        bucket: &str,
+        object: &str,
+        missing_part_disk: Option<usize>,
+    ) {
+        let version_id = Uuid::new_v4();
+        let data_dir = Uuid::new_v4();
+        let mod_time = OffsetDateTime::now_utc();
+        for (index, disk) in disks
+            .iter()
+            .enumerate()
+            .filter_map(|(index, disk)| disk.as_ref().map(|disk| (index, disk)))
+        {
+            if missing_part_disk != Some(index) {
+                disk.write_all(bucket, &format!("{object}/{data_dir}/part.1"), Bytes::from_static(b"x"))
+                    .await
+                    .expect("part data should be installed on every disk");
+            }
+            disk.write_metadata(
+                bucket,
+                bucket,
+                object,
+                valid_metadata_fanout_fileinfo(bucket, object, version_id, data_dir, mod_time),
+            )
+            .await
+            .expect("metadata should be installed on every disk");
+        }
+    }
+
+    #[tokio::test]
+    async fn bounded_metadata_early_stop_ab_limits_data_get_read_version_fanout() {
+        const DISKS: usize = 4;
+        let bucket = "bounded-data-get-fanout-bucket";
+        let control_object = "bounded-data-get-control-object";
+        let treatment_object = "bounded-data-get-treatment-object";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+        install_metadata_fanout_fileinfo(&disks, bucket, control_object, None).await;
+        install_metadata_fanout_fileinfo(&disks, bucket, treatment_object, None).await;
+
+        temp_env::async_with_vars(
+            [
+                ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
+                ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", None),
+                ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
+            ],
+            async {
+                let calls = disk_call_counters::observe(control_object);
+                let (_, _, diagnostics) =
+                    SetDisks::read_all_fileinfo_observed(&disks, bucket, bucket, control_object, "", true, false, false, true, 2)
+                        .await
+                        .expect("control metadata should resolve");
+
+                assert_eq!(
+                    calls.total(disk_call_counters::KIND_READ_VERSION),
+                    DISKS as u64,
+                    "control path should keep the default data-read full fanout"
+                );
+                assert_eq!(diagnostics.total_responses(), DISKS);
+            },
+        )
+        .await;
+
+        temp_env::async_with_vars(
+            [
+                ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
+                ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
+                ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
+            ],
+            async {
+                let calls = disk_call_counters::observe(treatment_object);
+                let (parts_metadata, errs, diagnostics) = SetDisks::read_all_fileinfo_observed(
+                    &disks,
+                    bucket,
+                    bucket,
+                    treatment_object,
+                    "",
+                    true,
+                    false,
+                    false,
+                    true,
+                    2,
+                )
+                .await
+                .expect("healthy object metadata should reach early-stop quorum");
+
+                assert_eq!(
+                    calls.total(disk_call_counters::KIND_READ_VERSION),
+                    3,
+                    "treatment path should stop after the 2+2 read/write quorum instead of issuing every disk read"
+                );
+                assert_eq!(diagnostics.total_responses(), 3);
+                assert_eq!(parts_metadata.iter().filter(|fi| fi.name == treatment_object).count(), 3);
+                assert!(errs.iter().all(Option::is_none));
+            },
+        )
+        .await;
+
+        drop(dirs);
+    }
+
+    #[tokio::test]
+    async fn bounded_metadata_early_stop_falls_back_to_full_fanout_on_data_read_error() {
+        const DISKS: usize = 4;
+        let bucket = "bounded-data-get-error-bucket";
+        let object = "bounded-data-get-error-object";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+        install_metadata_fanout_fileinfo(&disks, bucket, object, Some(0)).await;
+
+        temp_env::async_with_vars(
+            [
+                ("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
+                ("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
+                ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
+            ],
+            async {
+                let calls = disk_call_counters::observe(object);
+                let (_, errs, diagnostics) =
+                    SetDisks::read_all_fileinfo_observed(&disks, bucket, bucket, object, "", true, false, false, true, 2)
+                        .await
+                        .expect("metadata fanout should complete after falling back to all disks");
+
+                assert_eq!(
+                    calls.total(disk_call_counters::KIND_READ_VERSION),
+                    DISKS as u64,
+                    "a data-read error must force bounded fanout to schedule every disk before returning"
+                );
+                assert_eq!(diagnostics.total_responses(), DISKS);
+                assert!(
+                    errs.iter()
+                        .any(|err| err.as_ref().is_some_and(|err| matches!(err, DiskError::FileNotFound)))
+                );
+            },
+        )
+        .await;
+
+        drop(dirs);
+    }
+
     /// Bound for the pause handshake. This is a hang-guard, not a timing
     /// dependency: under a working barrier `wait_until_paused` returns via the
     /// `Notify` handshake far below this bound regardless of IO pressure, so the

crates/ecstore/src/set_disk/metadata.rs

@@ -580,7 +580,7 @@ impl SetDisks {
         }
     }
 
-    pub(super) fn file_info_quorum_hash(meta: &FileInfo) -> [u8; 32] {
+    pub(crate) fn file_info_quorum_hash(meta: &FileInfo) -> [u8; 32] {
         let mut hasher = Sha256::new();
         Self::update_file_info_quorum_hash(&mut hasher, meta);
         let digest = hasher.finalize();

crates/ecstore/src/set_disk/mod.rs

@@ -673,9 +673,9 @@ const DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY_THRESHOLD: usize = 128 * 102
 const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ENABLE";
 // Enabled by default (backlog#872): the early-stop path only engages for
 // requests `should_allow_metadata_early_stop` classifies as safe (metadata-only
-// reads without version_id / healing / free-version needs) and still requires
-// a full read-quorum agreement before stopping. Set the env var to `false` to
-// fall back to full-wait metadata fanout.
+// reads by default, without version_id / healing / free-version needs) and
+// still requires a full read-quorum agreement before stopping. Set the env var
+// to `false` to fall back to full-wait metadata fanout.
 const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE: bool = true;
 
 const ENV_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT";
@@ -684,6 +684,12 @@ const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT: u32 = 100;
 const ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE";
 const DEFAULT_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE: bool = false;
 
+const ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE: &str = "RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE";
+const DEFAULT_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE: bool = false;
+
+const ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: &str = "RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT";
+const DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT: bool = false;
+
 // --- Multipart Reader-Setup Prefetch Configuration (backlog#870) ---
 
 const ENV_RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH: &str = "RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH";
@@ -692,6 +698,8 @@ const DEFAULT_RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH: bool = true;
 static OBJECT_LOCK_DIAG_ENABLED: OnceLock<bool> = OnceLock::new();
 
 mod core;
+#[cfg(test)]
+pub(crate) use core::io_primitives::disk_call_counters;
 mod ctx;
 mod metadata;
 mod ops;
@@ -702,8 +710,8 @@ pub(crate) use ops::object::TransitionCleanupStoreBarrier as SetDiskTransitionCl
 pub(crate) use ops::object::body_cache_plaintext_len;
 #[cfg(test)]
 pub(crate) use ops::object::cleanup_rejected_transition_upload_durably;
-#[cfg(test)]
-pub(crate) use ops::object::{PutObjectCommitBarrier, PutObjectCommitPause};
+#[cfg(any(test, feature = "test-util"))]
+pub use ops::object::{PutObjectCommitBarrier, PutObjectCommitPause};
 mod read;
 mod replication;
 pub(crate) mod shard_source;
@@ -731,6 +739,10 @@ impl PreparedGetObjectMetadata {
             .take()
             .expect("prepared GET metadata ObjectInfo must be consumed exactly once")
     }
+
+    pub(crate) fn read_semantics_identity(&self) -> [u8; 32] {
+        SetDisks::file_info_quorum_hash(&self.fi)
+    }
 }
 
 tokio::task_local! {
@@ -1188,6 +1200,46 @@ fn is_version_early_stop_enabled() -> bool {
     }
 }
 
+fn is_get_metadata_data_read_early_stop_enabled() -> bool {
+    #[cfg(test)]
+    {
+        rustfs_utils::get_env_bool(
+            ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE,
+            DEFAULT_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE,
+        )
+    }
+    #[cfg(not(test))]
+    {
+        static CACHED: OnceLock<bool> = OnceLock::new();
+        *CACHED.get_or_init(|| {
+            rustfs_utils::get_env_bool(
+                ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE,
+                DEFAULT_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE,
+            )
+        })
+    }
+}
+
+fn is_get_metadata_early_stop_bounded_fanout_enabled() -> bool {
+    #[cfg(test)]
+    {
+        rustfs_utils::get_env_bool(
+            ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT,
+            DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT,
+        )
+    }
+    #[cfg(not(test))]
+    {
+        static CACHED: OnceLock<bool> = OnceLock::new();
+        *CACHED.get_or_init(|| {
+            rustfs_utils::get_env_bool(
+                ENV_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT,
+                DEFAULT_RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT,
+            )
+        })
+    }
+}
+
 /// Check if multipart reads prefetch the next part's bitrot reader setup
 /// while the current part decodes (backlog#870).
 ///
@@ -2553,6 +2605,53 @@ impl SetDisks {
         ))
     }
 
+    #[cfg(any(test, feature = "test-util"))]
+    async fn acquire_write_lock_diag_with_pending_hook(
+        &self,
+        op: &'static str,
+        bucket: &str,
+        object: &str,
+        on_pending: impl FnOnce(),
+    ) -> Result<ObjectLockDiagGuard> {
+        crate::hp_guard!("SetDisks::acquire_write_lock");
+        let diag_enabled = is_object_lock_diag_enabled();
+        let ns_lock = self.new_ns_lock(bucket, object).await?;
+        let acquire_start = Instant::now();
+        let acquire = ns_lock.get_write_lock(get_lock_acquire_timeout());
+        tokio::pin!(acquire);
+        let mut on_pending = Some(on_pending);
+        let guard = futures::future::poll_fn(|cx| match std::future::Future::poll(acquire.as_mut(), cx) {
+            std::task::Poll::Pending => {
+                if let Some(on_pending) = on_pending.take() {
+                    on_pending();
+                }
+                std::task::Poll::Pending
+            }
+            std::task::Poll::Ready(result) => std::task::Poll::Ready(result),
+        })
+        .await
+        .map_err(|e| self.map_namespace_lock_error(bucket, object, "write", e))?;
+        let owner = diag_enabled.then(|| ns_lock.owner().to_string());
+        self.log_object_lock_acquire_if_slow(
+            op,
+            bucket,
+            object,
+            "write",
+            owner.as_deref(),
+            acquire_start.elapsed(),
+            diag_enabled,
+        );
+        Ok(ObjectLockDiagGuard::new(
+            guard,
+            diag_enabled,
+            op,
+            diag_enabled.then(|| bucket.to_string()),
+            diag_enabled.then(|| object.to_string()),
+            owner,
+            "write",
+        ))
+    }
+
     #[allow(clippy::too_many_arguments)]
     fn log_object_lock_acquire_if_slow(
         &self,

crates/ecstore/src/set_disk/ops/heal.rs

@@ -331,6 +331,85 @@ fn warn_heal_writer_failures(
 }
 
 impl SetDisks {
+    /// Read back one healed version from every explicitly admitted replacement
+    /// target. This is intentionally separate from the normal heal result: a
+    /// successful result describes the transaction attempt, while automatic
+    /// replacement completion needs physical evidence that survives a crash
+    /// before its checkpoint is persisted.
+    pub(crate) async fn replacement_targets_have_version(
+        &self,
+        bucket: &str,
+        object: &str,
+        version_id: &str,
+        targets: &[String],
+    ) -> disk::error::Result<bool> {
+        let disks = self.get_disks_internal().await;
+        let mut target_disks = Vec::with_capacity(targets.len());
+
+        for target in targets {
+            let Some(index) = self.set_endpoints.iter().position(|endpoint| endpoint.to_string() == *target) else {
+                return Ok(false);
+            };
+            let Some(disk) = disks.get(index).and_then(Option::as_ref) else {
+                return Ok(false);
+            };
+            target_disks.push(disk.clone());
+        }
+
+        let read_options = ReadOptions {
+            incl_free_versions: false,
+            read_data: true,
+            healing: true,
+        };
+        let checks = target_disks.into_iter().map(|disk| {
+            let read_options = read_options.clone();
+            async move {
+                let file_info = match disk.read_version("", bucket, object, version_id, &read_options).await {
+                    Ok(file_info) => file_info,
+                    Err(
+                        DiskError::DiskNotFound
+                        | DiskError::VolumeNotFound
+                        | DiskError::FileNotFound
+                        | DiskError::FileVersionNotFound
+                        | DiskError::PathNotFound,
+                    ) => return Ok(false),
+                    Err(err) => return Err(err),
+                };
+                if !file_info_is_valid_for_metadata(&file_info) {
+                    return Ok(false);
+                }
+                if !version_id.is_empty() && file_info.version_id.as_ref().map(ToString::to_string).as_deref() != Some(version_id)
+                {
+                    return Ok(false);
+                }
+                if file_info.is_canonical_delete_marker() || file_info.is_remote() {
+                    return Ok(true);
+                }
+                if (file_info.data.is_some() || file_info.size == 0) && !file_info.parts.is_empty() {
+                    return Ok(true);
+                }
+
+                let check = match disk.check_parts(bucket, object, &file_info).await {
+                    Ok(check) => check,
+                    Err(
+                        DiskError::DiskNotFound
+                        | DiskError::VolumeNotFound
+                        | DiskError::FileNotFound
+                        | DiskError::FileVersionNotFound
+                        | DiskError::PathNotFound,
+                    ) => return Ok(false),
+                    Err(err) => return Err(err),
+                };
+                Ok(!check.results.is_empty() && check.results.iter().all(|result| *result == CHECK_PART_SUCCESS))
+            }
+        });
+
+        Ok(futures::future::try_join_all(checks)
+            .await?
+            .into_iter()
+            .all(|committed| committed))
+    }
+
     #[tracing::instrument(level = "trace", skip(self, opts), fields(bucket = %bucket, object = %object, version_id = %version_id))]
     pub(in crate::set_disk) async fn heal_object(
         &self,
@@ -1711,19 +1790,35 @@ impl SetDisks {
     }
 }
 
-// Heal operation family: the storage-api `HealOperations` contract stays
-// implemented `for SetDisks` (contract bounds unchanged) but now lives beside
-// its inherent helpers in the `set_disk::ops::heal` module. Bodies are moved
-// unchanged; `get_pool_and_set` reads the core through `SetDisksCtx` to keep
-// the Heal family aligned with the borrow pattern from #816.
-#[async_trait::async_trait]
-impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
-    type Error = Error;
-    type HealResultItem = HealResultItem;
-    type HealOptions = HealOpts;
+impl SetDisks {
+    pub(crate) async fn heal_replacement_format(
+        &self,
+        dry_run: bool,
+        targets: &[String],
+    ) -> Result<(HealResultItem, Option<Error>)> {
+        if targets.is_empty() {
+            return Err(Error::other("replacement format requires at least one target"));
+        }
 
-    #[tracing::instrument(skip(self))]
-    async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
+        let mut target_slots = Vec::with_capacity(targets.len());
+        for target in targets {
+            let Some(slot) = self.set_endpoints.iter().position(|endpoint| endpoint.to_string() == *target) else {
+                return Err(Error::other("replacement format target does not belong to the set"));
+            };
+            if target_slots.contains(&slot) {
+                return Err(Error::other("replacement format target is duplicated"));
+            }
+            target_slots.push(slot);
+        }
+
+        self.heal_format_for_slots(dry_run, Some(&target_slots)).await
+    }
+
+    async fn heal_format_for_slots(
+        &self,
+        dry_run: bool,
+        target_slots: Option<&[usize]>,
+    ) -> Result<(HealResultItem, Option<Error>)> {
         let disks = self.disks.read().await.clone();
         let (formats, errs) = load_format_erasure_all(&disks, true).await;
         if errs.iter().any(|err| {
@@ -1785,21 +1880,43 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
 
         if !dry_run {
             for (disk_idx, err) in errs.iter().enumerate() {
-                if !matches!(err, Some(DiskError::UnformattedDisk)) {
+                if !matches!(err, Some(DiskError::UnformattedDisk))
+                    || target_slots.is_some_and(|slots| !slots.contains(&disk_idx))
+                {
                     continue;
                 }
 
                 let mut new_format = ref_format.clone();
                 new_format.erasure.this = ref_format.erasure.sets[self.set_index][disk_idx];
-                if save_format_file(&disks[disk_idx], &Some(new_format.clone())).await.is_ok() {
-                    result.after.drives[disk_idx].uuid = new_format.erasure.this.to_string();
-                    result.after.drives[disk_idx].state = DriveState::Ok.to_string();
+                match save_format_file(&disks[disk_idx], &Some(new_format.clone())).await {
+                    Ok(()) => {
+                        result.after.drives[disk_idx].uuid = new_format.erasure.this.to_string();
+                        result.after.drives[disk_idx].state = DriveState::Ok.to_string();
+                    }
+                    Err(err) => return Ok((result, Some(err.into()))),
                 }
             }
         }
 
         Ok((result, None))
     }
+}
+
+// Heal operation family: the storage-api `HealOperations` contract stays
+// implemented `for SetDisks` (contract bounds unchanged) but now lives beside
+// its inherent helpers in the `set_disk::ops::heal` module. Bodies are moved
+// unchanged; `get_pool_and_set` reads the core through `SetDisksCtx` to keep
+// the Heal family aligned with the borrow pattern from #816.
+#[async_trait::async_trait]
+impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
+    type Error = Error;
+    type HealResultItem = HealResultItem;
+    type HealOptions = HealOpts;
+
+    #[tracing::instrument(skip(self))]
+    async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
+        self.heal_format_for_slots(dry_run, None).await
+    }
 
     #[tracing::instrument(skip(self))]
     async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
@@ -2397,6 +2514,50 @@ mod heal_result_report_tests {
         }
     }
 
+    #[tokio::test]
+    async fn replacement_target_readback_requires_the_committed_shard() {
+        let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
+        let bucket = "replacement-target-readback";
+        let object = "object.bin";
+        for disk in &disks {
+            disk.make_volume(bucket).await.expect("bucket volume should be created");
+        }
+
+        let mut reader = PutObjReader::from_vec(vec![0x5a; 1024 * 1024]);
+        set.put_object(bucket, object, &mut reader, &ObjectOptions::default())
+            .await
+            .expect("source object should be written");
+        let source = disks[2]
+            .read_version("", bucket, object, "", &ReadOptions::default())
+            .await
+            .expect("source metadata should be readable");
+        let data_dir = source.data_dir.expect("non-inline source should have a data directory");
+        let targets = vec![set.set_endpoints[0].to_string(), set.set_endpoints[1].to_string()];
+
+        assert!(
+            set.replacement_targets_have_version(bucket, object, "", &targets)
+                .await
+                .expect("healthy target shards should be readable")
+        );
+
+        tokio::fs::remove_file(
+            temp_dirs[1]
+                .path()
+                .join(bucket)
+                .join(object)
+                .join(data_dir.to_string())
+                .join("part.1"),
+        )
+        .await
+        .expect("target shard should be removed after the initial commit");
+
+        assert!(
+            !set.replacement_targets_have_version(bucket, object, "", &targets)
+                .await
+                .expect("missing target shard should be observable")
+        );
+    }
+
     #[tokio::test]
     async fn format_heal_cached_layout_rejects_a_disk_from_another_slot() {
         let mut _temp_dirs = Vec::new();

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -1897,6 +1897,13 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             }
         }
 
+        // The SSE-C passthrough session marker is upload-scoped; drop it from
+        // the completed object's metadata.
+        rustfs_utils::http::metadata_compat::remove_str(
+            &mut fi.metadata,
+            rustfs_utils::http::SUFFIX_REPLICATION_PRESERVE_CIPHERTEXT,
+        );
+
         if checksum_type.is_set() {
             checksum_type
                 .merge(rustfs_rio::ChecksumType::MULTIPART)
@@ -1919,13 +1926,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         }
 
         // etag
-        let etag = {
-            if let Some(etag) = opts.user_defined.get("etag") {
-                etag.clone()
-            } else {
-                get_complete_multipart_md5(&uploaded_parts)
-            }
-        };
+        let etag = resolve_complete_etag(opts, &uploaded_parts);
 
         fi.metadata.insert("etag".to_owned(), etag);
 
@@ -2167,6 +2168,21 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
     }
 }
 
+/// Final ETag for a completed multipart object. An authorized replication
+/// request preserves the source ETag so the replication HEAD comparison
+/// converges even when the source ETag is not derivable from the uploaded
+/// parts (foreign-origin objects, ciphertext-derived ETags); the internal
+/// metadata override comes next; otherwise the ETag is computed from parts.
+fn resolve_complete_etag(opts: &ObjectOptions, uploaded_parts: &[CompletePart]) -> String {
+    if let Some(etag) = opts.preserve_etag.as_ref().filter(|etag| !etag.is_empty()) {
+        return etag.clone();
+    }
+    if let Some(etag) = opts.user_defined.get("etag") {
+        return etag.clone();
+    }
+    get_complete_multipart_md5(uploaded_parts)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -5190,4 +5206,28 @@ mod tests {
             assert_eq!(body_after, new, "reclaiming the leftover upload must not disturb the committed object");
         }
     }
+
+    #[test]
+    fn resolve_complete_etag_prefers_preserved_source_etag() {
+        // A replication-preserved ETag that no part combination can derive
+        // (foreign-origin object) must win over the computed md5-of-parts.
+        let foreign_etag = "11111111111111111111111111111111-7".to_string();
+        let opts = ObjectOptions {
+            preserve_etag: Some(foreign_etag.clone()),
+            ..Default::default()
+        };
+        assert_eq!(resolve_complete_etag(&opts, &[]), foreign_etag);
+
+        // Empty preserve value degrades to the next source.
+        let opts_empty = ObjectOptions {
+            preserve_etag: Some(String::new()),
+            user_defined: std::collections::HashMap::from([("etag".to_string(), "override-etag".to_string())]),
+            ..Default::default()
+        };
+        assert_eq!(resolve_complete_etag(&opts_empty, &[]), "override-etag");
+
+        // Without either source the ETag is computed from the parts.
+        let computed = resolve_complete_etag(&ObjectOptions::default(), &[]);
+        assert_eq!(computed, get_complete_multipart_md5(&[]));
+    }
 }

crates/ecstore/src/set_disk/ops/object.rs

@@ -1233,6 +1233,16 @@ impl SetDisks {
                 }
             }
 
+            // SSE-C replication carries the source object's sealed checksum
+            // out of band; store it verbatim like the multipart path does.
+            if let Some(cssum) =
+                rustfs_utils::http::get_header_map(&user_defined, rustfs_utils::http::SUFFIX_REPLICATION_SSEC_CRC)
+                && !cssum.is_empty()
+            {
+                fi.checksum = base64_simd::STANDARD.decode_to_vec(&cssum).ok().map(bytes::Bytes::from);
+                rustfs_utils::http::remove_header_map(&mut user_defined, rustfs_utils::http::SUFFIX_REPLICATION_SSEC_CRC);
+            }
+
             if fi.checksum.is_none()
                 && let Some(content_hash) = data.as_hash_reader().content_hash()
             {
@@ -1312,7 +1322,7 @@ impl SetDisks {
             }
 
             if !opts.no_lock && object_lock_guard.is_none() {
-                #[cfg(test)]
+                #[cfg(any(test, feature = "test-util"))]
                 pause_put_object_commit(bucket, object, PutObjectCommitPause::BeforeNamespace).await;
                 if let Some(expected_incarnation_id) = opts.expected_bucket_incarnation_id
                     && opts.bucket_lifecycle_lock_fence.is_none()
@@ -1324,9 +1334,21 @@ impl SetDisks {
                             .await?,
                     );
                 }
-                object_lock_guard = Some(self.acquire_write_lock_diag("put_object_commit", bucket, object).await?);
+                #[cfg(any(test, feature = "test-util"))]
+                {
+                    object_lock_guard = Some(
+                        self.acquire_write_lock_diag_with_pending_hook("put_object_commit", bucket, object, || {
+                            notify_put_object_commit_namespace_pending(bucket, object);
+                        })
+                        .await?,
+                    );
+                }
+                #[cfg(not(any(test, feature = "test-util")))]
+                {
+                    object_lock_guard = Some(self.acquire_write_lock_diag("put_object_commit", bucket, object).await?);
+                }
             }
-            #[cfg(test)]
+            #[cfg(any(test, feature = "test-util"))]
             pause_put_object_commit(bucket, object, PutObjectCommitPause::AfterNamespace).await;
 
             if deferred_data_movement_precondition && let Some(err) = self.check_write_precondition(bucket, object, opts).await {
@@ -2575,41 +2597,43 @@ fn remote_version_state_writer_enabled_for(requested: bool, fleet_confirmed: boo
     requested && fleet_confirmed && fleet_proof_valid
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(crate) enum PutObjectCommitPause {
+pub enum PutObjectCommitPause {
     BeforeNamespace,
     AfterNamespace,
     BeforeMetadata,
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 struct PutObjectCommitBarrierState {
     bucket: String,
     object: String,
     pause: PutObjectCommitPause,
     arrived: tokio::sync::Notify,
     release: tokio::sync::Notify,
+    namespace_pending: tokio::sync::Notify,
 }
 
-#[cfg(test)]
-pub(crate) struct PutObjectCommitBarrier {
+#[cfg(any(test, feature = "test-util"))]
+pub struct PutObjectCommitBarrier {
     state: Arc<PutObjectCommitBarrierState>,
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 static PUT_OBJECT_COMMIT_BARRIER: std::sync::OnceLock<std::sync::Mutex<Vec<Arc<PutObjectCommitBarrierState>>>> =
     std::sync::OnceLock::new();
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 impl PutObjectCommitBarrier {
-    pub(crate) fn install(bucket: &str, object: &str, pause: PutObjectCommitPause) -> Self {
+    pub fn install(bucket: &str, object: &str, pause: PutObjectCommitPause) -> Self {
         let state = Arc::new(PutObjectCommitBarrierState {
             bucket: bucket.to_string(),
             object: object.to_string(),
             pause,
             arrived: tokio::sync::Notify::new(),
             release: tokio::sync::Notify::new(),
+            namespace_pending: tokio::sync::Notify::new(),
         });
         let mut slot = PUT_OBJECT_COMMIT_BARRIER
             .get_or_init(|| std::sync::Mutex::new(Vec::new()))
@@ -2626,18 +2650,27 @@ impl PutObjectCommitBarrier {
         Self { state }
     }
 
-    pub(crate) async fn wait_until_paused(&self) {
+    pub async fn wait_until_paused(&self) {
         tokio::time::timeout(Duration::from_secs(30), self.state.arrived.notified())
             .await
             .expect("put object should reach the deterministic commit barrier");
     }
 
-    pub(crate) fn release(&self) {
+    pub fn release(&self) {
         self.state.release.notify_one();
     }
+
+    pub async fn release_and_wait_until_namespace_pending(&self) {
+        assert_eq!(self.state.pause, PutObjectCommitPause::BeforeNamespace);
+        let namespace_pending = self.state.namespace_pending.notified();
+        self.release();
+        tokio::time::timeout(Duration::from_secs(5), namespace_pending)
+            .await
+            .expect("put object should wait for the namespace lock after leaving the commit barrier");
+    }
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 impl Drop for PutObjectCommitBarrier {
     fn drop(&mut self) {
         self.state.release.notify_one();
@@ -2649,7 +2682,7 @@ impl Drop for PutObjectCommitBarrier {
     }
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 async fn pause_put_object_commit(bucket: &str, object: &str, pause: PutObjectCommitPause) {
     let barrier = PUT_OBJECT_COMMIT_BARRIER
         .get_or_init(|| std::sync::Mutex::new(Vec::new()))
@@ -2664,6 +2697,22 @@ async fn pause_put_object_commit(bucket: &str, object: &str, pause: PutObjectCom
     }
 }
 
+#[cfg(any(test, feature = "test-util"))]
+fn notify_put_object_commit_namespace_pending(bucket: &str, object: &str) {
+    let barrier = PUT_OBJECT_COMMIT_BARRIER
+        .get_or_init(|| std::sync::Mutex::new(Vec::new()))
+        .lock()
+        .expect("put object commit barrier mutex should not poison")
+        .iter()
+        .find(|barrier| {
+            barrier.bucket == bucket && barrier.object == object && barrier.pause == PutObjectCommitPause::BeforeNamespace
+        })
+        .cloned();
+    if let Some(barrier) = barrier {
+        barrier.namespace_pending.notify_one();
+    }
+}
+
 #[cfg(test)]
 struct DeleteObjectCommitBarrierState {
     bucket: String,
@@ -4414,7 +4463,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         self.invalidate_get_object_metadata_cache(bucket, object).await;
 
         // Guard lock for metadata update
-        #[cfg(test)]
+        #[cfg(any(test, feature = "test-util"))]
         pause_put_object_commit(bucket, object, PutObjectCommitPause::BeforeMetadata).await;
         let _lock_guard = if !opts.no_lock {
             Some(self.acquire_write_lock_diag("put_object_metadata", bucket, object).await?)

crates/ecstore/src/set_disk/read.rs

@@ -48,6 +48,7 @@ use metrics::counter;
 use std::{
     collections::{HashMap, VecDeque},
     future::Future,
+    io::IoSlice,
     pin::Pin,
     sync::OnceLock,
     task::{Context, Poll},
@@ -76,6 +77,16 @@ impl<W: AsyncWrite + Unpin> AsyncWrite for GetObjectDownstreamWriter<W> {
             .map(|result| result.map_err(mark_get_object_downstream_closed))
     }
 
+    fn poll_write_vectored(mut self: Pin<&mut Self>, cx: &mut Context<'_>, bufs: &[IoSlice<'_>]) -> Poll<std::io::Result<usize>> {
+        Pin::new(&mut self.inner)
+            .poll_write_vectored(cx, bufs)
+            .map(|result| result.map_err(mark_get_object_downstream_closed))
+    }
+
+    fn is_write_vectored(&self) -> bool {
+        self.inner.is_write_vectored()
+    }
+
     fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
         Pin::new(&mut self.inner)
             .poll_flush(cx)
@@ -3101,7 +3112,7 @@ mod metadata_cache_tests {
 mod tests {
     use super::*;
     use crate::erasure::coding::BitrotWriter;
-    use std::io::{Cursor, ErrorKind};
+    use std::io::{Cursor, ErrorKind, IoSlice};
     use std::sync::{
         Arc,
         atomic::{AtomicUsize, Ordering},
@@ -3128,6 +3139,63 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn downstream_writer_preserves_vectored_write_support() {
+        #[derive(Default)]
+        struct VectoredSink {
+            writes: usize,
+            vectored_writes: usize,
+            bytes: Vec<u8>,
+        }
+
+        impl AsyncWrite for VectoredSink {
+            fn poll_write(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &[u8]) -> Poll<std::io::Result<usize>> {
+                self.writes += 1;
+                self.bytes.extend_from_slice(buf);
+                Poll::Ready(Ok(buf.len()))
+            }
+
+            fn poll_write_vectored(
+                mut self: Pin<&mut Self>,
+                _cx: &mut Context<'_>,
+                bufs: &[IoSlice<'_>],
+            ) -> Poll<std::io::Result<usize>> {
+                self.vectored_writes += 1;
+                let mut written = 0;
+                for buf in bufs {
+                    written += buf.len();
+                    self.bytes.extend_from_slice(buf);
+                }
+                Poll::Ready(Ok(written))
+            }
+
+            fn is_write_vectored(&self) -> bool {
+                true
+            }
+
+            fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+                Poll::Ready(Ok(()))
+            }
+
+            fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+                Poll::Ready(Ok(()))
+            }
+        }
+
+        let mut writer = GetObjectDownstreamWriter::new(VectoredSink::default());
+        assert!(writer.is_write_vectored(), "downstream writer must preserve vectored-write capability");
+
+        let written = writer
+            .write_vectored(&[IoSlice::new(b"hello "), IoSlice::new(b"world")])
+            .await
+            .expect("vectored write through downstream adapter must succeed");
+
+        assert_eq!(written, 11);
+        assert_eq!(writer.inner.vectored_writes, 1);
+        assert_eq!(writer.inner.writes, 0);
+        assert_eq!(writer.inner.bytes, b"hello world");
+    }
+
     async fn local_test_disks(count: usize, bucket: &str) -> (Vec<tempfile::TempDir>, Vec<Option<crate::disk::DiskStore>>) {
         let mut dirs = Vec::with_capacity(count);
         let mut disks = Vec::with_capacity(count);
@@ -3766,18 +3834,19 @@ mod tests {
                 assert!(metadata_early_stop_permitted(true, true, false, "", false, false));
                 // observe=false (non-observed fanout) also disables early-stop.
                 assert!(!metadata_early_stop_permitted(true, false, false, "", false, false));
-                // Data reads are never eligible regardless of caller opt-in.
+                // Data reads require their own explicit rollout gate.
                 assert!(!metadata_early_stop_permitted(true, true, true, "", false, false));
             },
         );
     }
 
     #[test]
-    fn metadata_early_stop_rejects_data_reads() {
+    fn metadata_early_stop_requires_explicit_data_read_opt_in() {
         temp_env::with_vars(
             [
                 (ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE, Some("true")),
                 (ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE, Some("true")),
+                (ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE, None),
             ],
             || {
                 assert!(!should_allow_metadata_early_stop(true, "", false, false));
@@ -3786,6 +3855,17 @@ mod tests {
                 assert!(should_allow_metadata_early_stop(false, "version-id", false, false));
             },
         );
+        temp_env::with_vars(
+            [
+                (ENV_RUSTFS_GET_METADATA_EARLY_STOP_ENABLE, Some("true")),
+                (ENV_RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE, Some("true")),
+                (ENV_RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE, Some("true")),
+            ],
+            || {
+                assert!(should_allow_metadata_early_stop(true, "", false, false));
+                assert!(should_allow_metadata_early_stop(true, "version-id", false, false));
+            },
+        );
     }
 
     #[test]

crates/ecstore/src/storage_api_contracts/mod.rs

@@ -29,9 +29,10 @@ pub(crate) mod internode {
         NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY, NS_SCANNER_SESSION_ID_QUERY,
         NS_SCANNER_SESSION_SEQUENCE_QUERY, NsScannerCapabilityResponse, PUT_FILE_AUTH_QUERY, PUT_FILE_AUTH_TRAILER_DIGEST_LEN,
         PUT_FILE_AUTH_TRAILER_LEN, PUT_FILE_AUTH_TRAILER_MAC_LEN, PUT_FILE_AUTH_TRAILER_MAGIC, PUT_FILE_AUTH_V1,
-        PUT_FILE_NONCE_QUERY, SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION, SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION,
-        SCANNER_ACTIVITY_PROTOCOL_VERSION, WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_STREAM_COMPLETION_QUERY,
-        WALK_DIR_STREAM_COMPLETION_V1,
+        PUT_FILE_CAPABILITY_CHALLENGE_QUERY, PUT_FILE_CAPABILITY_QUERY, PUT_FILE_CAPABILITY_VERSION, PUT_FILE_NONCE_QUERY,
+        PUT_FILE_SERVER_EPOCH_QUERY, PutFileCapabilityResponse, SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION,
+        SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION, SCANNER_ACTIVITY_PROTOCOL_VERSION, WALK_DIR_BODY_SHA256_QUERY,
+        WALK_DIR_STREAM_COMPLETION_QUERY, WALK_DIR_STREAM_COMPLETION_V1,
     };
 }
 

crates/ecstore/src/store/bucket.rs

@@ -52,7 +52,10 @@ fn validate_table_bucket_delete_allowed(
 async fn table_catalog_metadata_exists(ctx: &crate::runtime::instance::InstanceContext, bucket: &str) -> Result<bool> {
     let local_disks = runtime_sources::local_disks_in(ctx).await;
     for disk in local_disks.iter() {
-        let catalog_path = disk.path().join(bucket).join(BUCKET_TABLE_RESERVED_PREFIX);
+        let Some(bucket_path) = disk.get_bucket_path_for_io_if_local(bucket) else {
+            continue;
+        };
+        let catalog_path = bucket_path?.join(BUCKET_TABLE_RESERVED_PREFIX);
         if has_xlmeta_files(&catalog_path).await? {
             return Ok(true);
         }
@@ -727,7 +730,10 @@ impl ECStore {
             if !opts.force {
                 let local_disks = runtime_sources::local_disks_in(&self.ctx).await;
                 for disk in local_disks.iter() {
-                    let bucket_path = disk.path().join(bucket);
+                    let Some(bucket_path) = disk.get_bucket_path_for_io_if_local(bucket) else {
+                        continue;
+                    };
+                    let bucket_path = bucket_path?;
                     if has_xlmeta_files(&bucket_path).await? {
                         return Err(StorageError::BucketNotEmpty(bucket.to_string()));
                     }

crates/ecstore/src/store/heal.rs

@@ -97,6 +97,56 @@ impl ECStore {
         Ok((r, None))
     }
 
+    #[instrument(skip(self, targets), fields(pool_index, set_index, target_count = targets.len()))]
+    pub async fn heal_replacement_format(
+        &self,
+        dry_run: bool,
+        pool_index: usize,
+        set_index: usize,
+        targets: &[String],
+    ) -> Result<(HealResultItem, Option<Error>)> {
+        let pool = self
+            .pools
+            .get(pool_index)
+            .ok_or_else(|| invalid_heal_pool_index(pool_index, self.pools.len()))?;
+        let set = pool.disk_set.get(set_index).cloned().ok_or_else(|| {
+            StorageError::InvalidArgument(
+                "heal".to_string(),
+                "set".to_string(),
+                format!("invalid heal set index {set_index} for pool {pool_index}"),
+            )
+        })?;
+
+        set.heal_replacement_format(dry_run, targets).await
+    }
+
+    #[instrument(skip(self, targets), fields(pool_index, set_index, target_count = targets.len()))]
+    pub async fn replacement_targets_have_version(
+        &self,
+        bucket: &str,
+        object: &str,
+        version_id: &str,
+        pool_index: usize,
+        set_index: usize,
+        targets: &[String],
+    ) -> Result<bool> {
+        let pool = self
+            .pools
+            .get(pool_index)
+            .ok_or_else(|| invalid_heal_pool_index(pool_index, self.pools.len()))?;
+        let set = pool.disk_set.get(set_index).cloned().ok_or_else(|| {
+            StorageError::InvalidArgument(
+                "heal".to_string(),
+                "set".to_string(),
+                format!("invalid heal set index {set_index} for pool {pool_index}"),
+            )
+        })?;
+
+        set.replacement_targets_have_version(bucket, object, version_id, targets)
+            .await
+            .map_err(Into::into)
+    }
+
     #[instrument(skip(self))]
     pub(super) async fn handle_heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
         let res = self.peer_sys.heal_bucket(bucket, opts).await?;
@@ -129,7 +179,30 @@ impl ECStore {
 
         let mut futures = Vec::with_capacity(pools.len());
         for pool in pools.iter() {
-            if self.is_suspended(pool.pool_idx).await {
+            let suspended_complete = {
+                let pool_meta = self.pool_meta.read().await;
+                pool_meta.is_suspended(pool.pool_idx).then(|| {
+                    pool_meta
+                        .pools
+                        .get(pool.pool_idx)
+                        .and_then(|status| status.decommission.as_ref())
+                        .is_some_and(|decommission| decommission.complete)
+                })
+            };
+            if let Some(complete) = suspended_complete {
+                if opts.pool.is_some() {
+                    let _ = pool.get_disks_for_heal_object(&object, opts)?;
+                    let err = if complete {
+                        StorageError::InvalidArgument(
+                            "heal".to_string(),
+                            "pool".to_string(),
+                            format!("heal pool {} has completed decommission", pool.pool_idx),
+                        )
+                    } else {
+                        Error::SlowDown
+                    };
+                    return Ok((HealResultItem::default(), Some(err)));
+                }
                 continue;
             }
             futures.push(pool.heal_object(bucket, &object, version_id, opts));
@@ -196,6 +269,7 @@ impl ECStore {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::core::pools::{PoolDecommissionInfo, PoolStatus};
     use crate::disk::{DiskOption, format::FormatV3, new_disk};
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
     use crate::store::init_format::{load_format_erasure, save_format_file};
@@ -276,6 +350,134 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn scoped_heal_object_defers_when_requested_pool_is_suspended() {
+        let mut store = minimal_heal_store().await;
+        store.pool_meta = RwLock::new(PoolMeta {
+            pools: vec![
+                PoolStatus {
+                    id: 0,
+                    cmd_line: "pool-0".to_string(),
+                    last_update: OffsetDateTime::UNIX_EPOCH,
+                    decommission: None,
+                },
+                PoolStatus {
+                    id: 1,
+                    cmd_line: "pool-1".to_string(),
+                    last_update: OffsetDateTime::UNIX_EPOCH,
+                    decommission: Some(PoolDecommissionInfo {
+                        start_time: Some(OffsetDateTime::UNIX_EPOCH),
+                        ..Default::default()
+                    }),
+                },
+            ],
+            ..Default::default()
+        });
+
+        let (_, err) = store
+            .handle_heal_object(
+                "bucket",
+                "object",
+                "",
+                &HealOpts {
+                    pool: Some(1),
+                    set: Some(0),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("suspended pool should return a deferred heal result");
+
+        assert!(matches!(err, Some(StorageError::SlowDown)));
+
+        let (_, err) = store
+            .handle_heal_object(
+                "bucket",
+                "object",
+                "",
+                &HealOpts {
+                    set: Some(1),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("unscoped heal should return the active pool result");
+
+        assert!(matches!(err, Some(StorageError::InvalidArgument(_, ref field, _)) if field == "set"));
+
+        let err = store
+            .handle_heal_object(
+                "bucket",
+                "object",
+                "",
+                &HealOpts {
+                    pool: Some(1),
+                    set: Some(1),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect_err("invalid set scope should fail before suspended pool deferral");
+
+        assert!(matches!(err, StorageError::InvalidArgument(_, ref field, _) if field == "set"));
+
+        {
+            let mut pool_meta = store.pool_meta.write().await;
+            let decommission = pool_meta.pools[1]
+                .decommission
+                .as_mut()
+                .expect("test pool should have decommission state");
+            decommission.complete = true;
+        }
+        let (_, err) = store
+            .handle_heal_object(
+                "bucket",
+                "object",
+                "",
+                &HealOpts {
+                    pool: Some(1),
+                    set: Some(0),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("completed pool should return a terminal heal result");
+
+        assert!(matches!(
+            err,
+            Some(StorageError::InvalidArgument(_, ref field, ref reason))
+                if field == "pool" && reason.contains("completed decommission")
+        ));
+
+        for canceled in [false, true] {
+            {
+                let mut pool_meta = store.pool_meta.write().await;
+                let decommission = pool_meta.pools[1]
+                    .decommission
+                    .as_mut()
+                    .expect("test pool should have decommission state");
+                decommission.complete = false;
+                decommission.failed = !canceled;
+                decommission.canceled = canceled;
+            }
+            let (_, err) = store
+                .handle_heal_object(
+                    "bucket",
+                    "object",
+                    "",
+                    &HealOpts {
+                        pool: Some(1),
+                        set: Some(0),
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("clearable terminal pool should return a deferred heal result");
+
+            assert!(matches!(err, Some(StorageError::SlowDown)));
+        }
+    }
+
     #[tokio::test]
     async fn handle_heal_format_continues_after_a_pool_error() {
         let canonical_format = FormatV3::new(1, 3);

crates/ecstore/src/store/mod.rs

@@ -152,7 +152,10 @@ mod list;
 pub(crate) mod list_objects;
 mod multipart;
 mod object;
-pub use object::PreparedGetObjectReader;
+pub use object::{
+    PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
+    SnapshotConsistencyError,
+};
 mod peer;
 mod rebalance;
 pub(crate) mod utils;

crates/filemeta/src/fileinfo.rs

@@ -278,7 +278,11 @@ pub struct FileInfo {
 fn is_sensitive_metadata_key(key: &str) -> bool {
     // `is_encryption_metadata_key` covers the x-minio-internal- SSE prefix but not
     // its x-rustfs-internal- twin, which the dual-key invariant writes alongside it.
-    is_encryption_metadata_key(key) || starts_with_ignore_ascii_case(key, "x-rustfs-internal-server-side-encryption-")
+    is_encryption_metadata_key(key)
+        || starts_with_ignore_ascii_case(key, "x-rustfs-internal-server-side-encryption-")
+        || rustfs_utils::http::REPLICATION_SSE_TRANSPORT_PREFIXES
+            .iter()
+            .any(|prefix| starts_with_ignore_ascii_case(key, prefix))
 }
 
 struct RedactedMetadata<'a>(&'a HashMap<String, String>);
@@ -2559,6 +2563,30 @@ mod tests {
         assert!(dump.contains("text/plain"));
     }
 
+    #[test]
+    fn debug_redacts_replication_sse_transport_metadata_values() {
+        let sealed_key = "IAAfANqt7wIJfVSgFAG3f5S6HuC2eyM5DdJlx7RSJKw2ZakSb3d5";
+        let mut fi = FileInfo::default();
+        for key in [
+            "X-Rustfs-Replication-Server-Side-Encryption-Sealed-Key",
+            "X-Rustfs-Replication-Server-Side-Encryption-Iv",
+            "X-Rustfs-Replication-Encryption-Iv",
+            "X-Rustfs-Replication-Ssec-Key-Md5",
+        ] {
+            fi.metadata.insert(key.to_string(), sealed_key.to_string());
+        }
+        fi.metadata.insert("content-type".to_string(), "text/plain".to_string());
+
+        let dump = format!("{fi:?}");
+        assert!(
+            !dump.contains(sealed_key),
+            "replication SSE transport value leaked into Debug output: {dump}"
+        );
+        assert!(dump.contains("X-Rustfs-Replication-Server-Side-Encryption-Sealed-Key"));
+        assert!(dump.contains(&format!("<redacted {} bytes>", sealed_key.len())));
+        assert!(dump.contains("text/plain"));
+    }
+
     #[test]
     fn debug_elides_inline_data_bytes() {
         let fi = FileInfo {

crates/filemeta/src/filemeta.rs

@@ -1016,6 +1016,83 @@ mod test {
     use proptest::collection::vec;
     use proptest::prelude::*;
 
+    /// A restore header meaning "restored copy is on disk until far in the future".
+    /// Format produced by `RestoreStatusOps::to_string` and consumed by
+    /// `parse_restore_obj_status` (fileinfo.rs).
+    const RESTORED_ON_DISK: &str = "ongoing-request=\"false\", expiry-date=\"9999-01-01T00:00:00Z\"";
+
+    /// backlog#1733 (P9-01 §4.3/§7.6, g-key-001): pin the five `s3s::header`
+    /// constants that double as **persisted metadata map keys**. They are not
+    /// just HTTP header names — they are stored inside xl.meta (`meta_user`)
+    /// and read back by fail-open code, so a silent drift produces zero
+    /// HTTP-visible errors while:
+    ///
+    /// 1. **WORM silently dissolves** — `get_object_retention_meta`
+    ///    (ecstore objectlock.rs) returns an empty retention when the lock keys
+    ///    are unreadable, making every compliance-locked object deletable.
+    /// 2. **Live data dirs can be reclaimed** — `MetaObject::uses_data_dir`
+    ///    falls back to `is_restored_object_on_disk`, which returns `false`
+    ///    when `x-amz-restore` is unreadable, so a restored object's data dir
+    ///    is judged unused.
+    ///
+    /// Any migration replacing these constants must keep the literals byte-stable.
+    #[test]
+    fn persisted_metadata_keys_are_byte_stable() {
+        use s3s::header::{
+            X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE,
+            X_AMZ_SERVER_SIDE_ENCRYPTION,
+        };
+        assert_eq!(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str(), "x-amz-object-lock-legal-hold");
+        assert_eq!(X_AMZ_OBJECT_LOCK_MODE.as_str(), "x-amz-object-lock-mode");
+        assert_eq!(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str(), "x-amz-object-lock-retain-until-date");
+        assert_eq!(X_AMZ_RESTORE.as_str(), "x-amz-restore");
+        assert_eq!(X_AMZ_SERVER_SIDE_ENCRYPTION.as_str(), "x-amz-server-side-encryption");
+    }
+
+    /// backlog#1733 g-key-003: a restored-to-local object must keep its data
+    /// dir. The restore marker lives under the pinned `x-amz-restore` key; if
+    /// the key ever drifts this flips to `false` and the data dir becomes
+    /// eligible for reclamation while the restored copy is still being served.
+    #[test]
+    fn restored_object_keeps_using_data_dir() {
+        let mut obj = MetaObject::default();
+        obj.meta_user
+            .insert("x-amz-restore".to_string(), RESTORED_ON_DISK.to_string());
+        assert!(obj.uses_data_dir(), "restored object's data dir must be considered in use");
+
+        // The same fail-open shape the pin protects against: without the marker
+        // the data dir is judged unused — exactly what a key drift would cause.
+        let bare = MetaObject::default();
+        assert!(!bare.uses_data_dir(), "object without restore marker reports data dir unused");
+    }
+
+    /// backlog#1733 g-key-004: a transition-complete object short-circuits to
+    /// `false` even when the restore marker is present — the existing
+    /// precedence must not change.
+    #[test]
+    fn transition_complete_object_does_not_use_data_dir() {
+        use rustfs_utils::http::{SUFFIX_TRANSITION_STATUS, insert_bytes};
+        let mut obj = MetaObject::default();
+        obj.meta_user
+            .insert("x-amz-restore".to_string(), RESTORED_ON_DISK.to_string());
+        insert_bytes(&mut obj.meta_sys, SUFFIX_TRANSITION_STATUS, TRANSITION_COMPLETE.as_bytes().to_vec());
+        assert!(!obj.uses_data_dir(), "transition-complete short-circuit must win over the restore marker");
+    }
+
+    /// The restore-header parser and the pinned key literal must agree: the
+    /// marker written under `x-amz-restore` is only meaningful if the parser
+    /// accepts it.
+    #[test]
+    fn restore_marker_roundtrips_through_parser() {
+        let mut meta = HashMap::new();
+        meta.insert(X_AMZ_RESTORE.as_str().to_string(), RESTORED_ON_DISK.to_string());
+        assert!(crate::is_restored_object_on_disk(&meta));
+
+        // An in-progress restore is not "on disk".
+        meta.insert(X_AMZ_RESTORE.as_str().to_string(), "ongoing-request=\"true\"".to_string());
+        assert!(!crate::is_restored_object_on_disk(&meta));
+    }
+
     /// backlog#580: RustFS parses real MinIO-written object xl.meta (inline,
     /// versioned, and multipart) into equivalent `FileInfo`. Object metadata is
     /// the strong part of MinIO interop; this pins it against real fixtures.

crates/filemeta/src/filemeta/version.rs

@@ -1855,6 +1855,7 @@ impl From<MetaObjectV1ChecksumInfo> for ChecksumInfo {
                 "highwayhash256" => HashAlgorithm::HighwayHash256,
                 "highwayhash256S" => HashAlgorithm::HighwayHash256S,
                 "blake2b" | "blake2b512" => HashAlgorithm::BLAKE2b512,
+                "md5" => HashAlgorithm::Md5,
                 _ => HashAlgorithm::HighwayHash256S,
             },
             hash: Bytes::from(value.hash),

crates/filemeta/src/test_data.rs

@@ -189,7 +189,14 @@ fn encode_legacy_v1_header(version_id: Uuid, mod_time: OffsetDateTime) -> Vec<u8
     wr
 }
 
-fn encode_legacy_v1_body(version_id: Uuid, data_dir: Uuid, mod_time: OffsetDateTime) -> Vec<u8> {
+fn encode_legacy_v1_body(
+    version_id: Uuid,
+    data_dir: Uuid,
+    mod_time: OffsetDateTime,
+    erasure_index: usize,
+    checksum: Option<(&str, &[u8])>,
+    object_size: usize,
+) -> Vec<u8> {
     let mut wr = Vec::new();
 
     rmp::encode::write_map_len(&mut wr, 3).unwrap();
@@ -208,7 +215,7 @@ fn encode_legacy_v1_body(version_id: Uuid, data_dir: Uuid, mod_time: OffsetDateT
     rmp::encode::write_str(&mut wr, "Stat").unwrap();
     rmp::encode::write_map_len(&mut wr, 5).unwrap();
     rmp::encode::write_str(&mut wr, "Size").unwrap();
-    rmp::encode::write_sint(&mut wr, 11).unwrap();
+    rmp::encode::write_sint(&mut wr, object_size as i64).unwrap();
     rmp::encode::write_str(&mut wr, "ModTime").unwrap();
     write_legacy_time(&mut wr, mod_time);
     rmp::encode::write_str(&mut wr, "Name").unwrap();
@@ -229,14 +236,23 @@ fn encode_legacy_v1_body(version_id: Uuid, data_dir: Uuid, mod_time: OffsetDateT
     rmp::encode::write_str(&mut wr, "BlockSize").unwrap();
     rmp::encode::write_sint(&mut wr, 1_048_576).unwrap();
     rmp::encode::write_str(&mut wr, "Index").unwrap();
-    rmp::encode::write_sint(&mut wr, 1).unwrap();
+    rmp::encode::write_sint(&mut wr, erasure_index as i64).unwrap();
     rmp::encode::write_str(&mut wr, "Distribution").unwrap();
     rmp::encode::write_array_len(&mut wr, 6).unwrap();
     for value in 1..=6 {
         rmp::encode::write_sint(&mut wr, value).unwrap();
     }
     rmp::encode::write_str(&mut wr, "Checksums").unwrap();
-    rmp::encode::write_array_len(&mut wr, 0).unwrap();
+    rmp::encode::write_array_len(&mut wr, u32::from(checksum.is_some())).unwrap();
+    if let Some((algorithm, hash)) = checksum {
+        rmp::encode::write_map_len(&mut wr, 3).unwrap();
+        rmp::encode::write_str(&mut wr, "PartNumber").unwrap();
+        rmp::encode::write_sint(&mut wr, 1).unwrap();
+        rmp::encode::write_str(&mut wr, "Algorithm").unwrap();
+        rmp::encode::write_str(&mut wr, algorithm).unwrap();
+        rmp::encode::write_str(&mut wr, "Hash").unwrap();
+        rmp::encode::write_bin(&mut wr, hash).unwrap();
+    }
 
     rmp::encode::write_str(&mut wr, "Meta").unwrap();
     rmp::encode::write_map_len(&mut wr, 1).unwrap();
@@ -251,9 +267,9 @@ fn encode_legacy_v1_body(version_id: Uuid, data_dir: Uuid, mod_time: OffsetDateT
     rmp::encode::write_str(&mut wr, "n").unwrap();
     rmp::encode::write_sint(&mut wr, 1).unwrap();
     rmp::encode::write_str(&mut wr, "s").unwrap();
-    rmp::encode::write_sint(&mut wr, 11).unwrap();
+    rmp::encode::write_sint(&mut wr, object_size as i64).unwrap();
     rmp::encode::write_str(&mut wr, "as").unwrap();
-    rmp::encode::write_sint(&mut wr, 11).unwrap();
+    rmp::encode::write_sint(&mut wr, object_size as i64).unwrap();
     rmp::encode::write_str(&mut wr, "mt").unwrap();
     write_legacy_time(&mut wr, mod_time);
 
@@ -275,8 +291,29 @@ pub fn create_legacy_v1_object_xlmeta() -> Result<Vec<u8>> {
     let mod_time = OffsetDateTime::from_unix_timestamp_nanos(1_705_312_200_123_456_789)?;
 
     let header = encode_legacy_v1_header(version_id, mod_time);
-    let body = encode_legacy_v1_body(version_id, data_dir, mod_time);
+    let body = encode_legacy_v1_body(version_id, data_dir, mod_time, 1, None, 11);
 
+    encode_legacy_v1_xlmeta(header, body)
+}
+
+/// Legacy V1 xl.meta fixture with a per-drive whole-file bitrot checksum.
+pub fn create_legacy_v1_object_xlmeta_with_checksum(
+    erasure_index: usize,
+    algorithm: &str,
+    hash: &[u8],
+    object_size: usize,
+) -> Result<Vec<u8>> {
+    let version_id = Uuid::parse_str("01234567-89ab-cdef-0123-456789abcdef")?;
+    let data_dir = Uuid::parse_str("fedcba98-7654-3210-fedc-ba9876543210")?;
+    let mod_time = OffsetDateTime::from_unix_timestamp_nanos(1_705_312_200_123_456_789)?;
+
+    let header = encode_legacy_v1_header(version_id, mod_time);
+    let body = encode_legacy_v1_body(version_id, data_dir, mod_time, erasure_index, Some((algorithm, hash)), object_size);
+
+    encode_legacy_v1_xlmeta(header, body)
+}
+
+fn encode_legacy_v1_xlmeta(header: Vec<u8>, body: Vec<u8>) -> Result<Vec<u8>> {
     let mut wr = Vec::new();
     wr.extend_from_slice(b"XL2 ");
     wr.extend_from_slice(&1u16.to_le_bytes());

crates/heal/Cargo.toml

@@ -91,6 +91,7 @@ metrics = { workspace = true }
 base64 = { workspace = true }
 
 [dev-dependencies]
+libc = { workspace = true }
 serde_json = { workspace = true, features = ["raw_value"] }
 rustfs-test-utils = { workspace = true }
 serial_test = { workspace = true }

crates/heal/src/heal/erasure_healer.rs

@@ -14,7 +14,10 @@
 
 use crate::heal::{
     progress::HealProgress,
-    resume::{CheckpointManager, ResumeManager, ResumeUtils, compose_key},
+    resume::{
+        CheckpointManager, ReplacementTargetIdentity, ResumeManager, ResumeUtils, compose_key,
+        replacement_target_identities_match,
+    },
     storage::{HealStorageAPI, next_heal_listing_token},
     task::{demote_to_debug_when, is_missing_object_dir_heal_result, take_failure_log_sample},
 };
@@ -22,6 +25,7 @@ use crate::{Error, Result};
 use futures::{StreamExt, stream::FuturesUnordered};
 use metrics::gauge;
 use rustfs_common::heal_channel::{HealOpts, HealRequestSource, HealScanMode};
+use rustfs_madmin::heal_commands::HealResultItem;
 use std::sync::{
     Arc,
     atomic::{AtomicUsize, Ordering},
@@ -85,6 +89,16 @@ pub struct ErasureSetHealer {
     disk: DiskStore,
     heal_opts: HealOpts,
     source: HealRequestSource,
+    target_endpoints: Arc<[String]>,
+    replacement_task_id: Option<String>,
+    replacement_target_identities: Option<Arc<[ReplacementTargetIdentity]>>,
+}
+
+pub(crate) fn target_outcomes_complete(result: &HealResultItem, target_endpoints: &[String]) -> bool {
+    target_endpoints.iter().all(|endpoint| {
+        let mut drives = result.after.drives.iter().filter(|drive| drive.endpoint == *endpoint);
+        matches!(drives.next(), Some(drive) if drive.state == "ok") && drives.next().is_none()
+    })
 }
 
 impl ErasureSetHealer {
@@ -182,9 +196,46 @@ impl ErasureSetHealer {
             disk,
             heal_opts,
             source,
+            target_endpoints: Vec::new().into(),
+            replacement_task_id: None,
+            replacement_target_identities: None,
         }
     }
 
+    pub(crate) fn with_replacement_targets(
+        mut self,
+        mut target_endpoints: Vec<String>,
+        replacement_task_id: Option<String>,
+    ) -> Self {
+        target_endpoints.sort_unstable();
+        target_endpoints.dedup();
+        self.target_endpoints = target_endpoints.into();
+        self.replacement_task_id = replacement_task_id;
+        self
+    }
+
+    pub(crate) fn with_replacement_identity_fence(
+        mut self,
+        replacement_target_identities: Option<Vec<ReplacementTargetIdentity>>,
+    ) -> Self {
+        self.replacement_target_identities = replacement_target_identities.map(Into::into);
+        self
+    }
+
+    async fn verify_replacement_identity_fence(&self, stage: &str) -> Result<()> {
+        let Some(expected_identities) = self.replacement_target_identities.as_ref() else {
+            return Ok(());
+        };
+        let actual_identities = self.storage.replacement_target_identities(&self.target_endpoints).await?;
+        if replacement_target_identities_match(expected_identities, &actual_identities) {
+            return Ok(());
+        }
+
+        Err(Error::TaskExecutionFailed {
+            message: format!("Replacement target changed during {stage}"),
+        })
+    }
+
     /// execute erasure set heal with resume
     #[tracing::instrument(skip(self, buckets), fields(set_disk_id = %set_disk_id, bucket_count = buckets.len()))]
     #[hotpath::measure]
@@ -212,9 +263,15 @@ impl ErasureSetHealer {
             .await;
 
         result?;
+        self.verify_replacement_identity_fence("completion").await?;
+
+        if self.replacement_task_id.is_some() {
+            // A replacement marker must outlive the successful data scan. The
+            // task clears that owner marker before deleting these artifacts.
+            resume_manager.mark_replacement_completed_and_verified().await?;
+            return Ok(());
+        }
 
-        // The healing marker is cleared by the caller only after both cleanup
-        // operations succeed. Cleanup is idempotent, so a retry is safe.
         checkpoint_manager.cleanup().await?;
         resume_manager.cleanup().await?;
         Ok(())
@@ -222,6 +279,21 @@ impl ErasureSetHealer {
 
     /// get or create task id
     async fn get_or_create_task_id(&self, set_disk_id: &str) -> Result<String> {
+        if let Some(task_id) = &self.replacement_task_id {
+            let manager = ResumeManager::load_replacement_intent(self.disk.clone(), task_id).await?;
+            let state = manager.get_state().await;
+            if !state.completed
+                && state.set_disk_id == set_disk_id
+                && state.replacement_targets.as_slice() == self.target_endpoints.as_ref()
+                && state.replacement_generation.as_deref() == Some(task_id.as_str())
+            {
+                return Ok(task_id.clone());
+            }
+            return Err(Error::TaskExecutionFailed {
+                message: format!("Replacement resume intent does not match task {task_id}"),
+            });
+        }
+
         // check if there are resumable tasks
         let resumable_tasks = ResumeUtils::get_resumable_tasks(&self.disk).await?;
 
@@ -231,6 +303,7 @@ impl ErasureSetHealer {
                     let state = manager.get_state().await;
                     if !state.completed
                         && state.set_disk_id == set_disk_id
+                        && state.replacement_targets.as_slice() == self.target_endpoints.as_ref()
                         && ResumeUtils::can_resume_task(&self.disk, &task_id).await
                     {
                         debug!(
@@ -263,7 +336,7 @@ impl ErasureSetHealer {
         }
 
         // create new task id
-        let task_id = format!("{}_{}", set_disk_id, ResumeUtils::generate_task_id());
+        let task_id = ResumeUtils::generate_task_id();
         debug!(
             target: "rustfs::heal::erasure_healer",
             event = EVENT_HEAL_ERASURE_RESUME_STATE,
@@ -285,7 +358,12 @@ impl ErasureSetHealer {
         buckets: &[String],
     ) -> Result<(ResumeManager, CheckpointManager)> {
         // check if resume state exists
-        if ResumeManager::has_resume_state(&self.disk, task_id).await {
+        let has_resume_state = if self.replacement_task_id.is_some() {
+            ResumeManager::has_replacement_intent(&self.disk, task_id).await
+        } else {
+            ResumeManager::has_resume_state(&self.disk, task_id).await
+        };
+        if has_resume_state {
             debug!(
                 target: "rustfs::heal::erasure_healer",
                 event = EVENT_HEAL_ERASURE_RESUME_STATE,
@@ -297,7 +375,11 @@ impl ErasureSetHealer {
                 "Erasure set resume state loading"
             );
 
-            let resume_manager = ResumeManager::load_from_disk(self.disk.clone(), task_id).await?;
+            let resume_manager = if self.replacement_task_id.is_some() {
+                ResumeManager::load_replacement_intent(self.disk.clone(), task_id).await?
+            } else {
+                ResumeManager::load_from_disk(self.disk.clone(), task_id).await?
+            };
             let checkpoint_manager = if CheckpointManager::has_checkpoint(&self.disk, task_id).await {
                 CheckpointManager::load_from_disk(self.disk.clone(), task_id).await?
             } else {
@@ -340,6 +422,9 @@ impl ErasureSetHealer {
                 buckets.to_vec(),
             )
             .await?;
+            resume_manager
+                .set_replacement_targets(self.target_endpoints.as_ref().to_vec())
+                .await?;
 
             let checkpoint_manager = CheckpointManager::new(self.disk.clone(), task_id.to_string()).await?;
 
@@ -485,6 +570,12 @@ impl ErasureSetHealer {
         // later heal cycle via the same bounded-retry mechanism as failures —
         // never hot-retried in place here.
         if failed_objects > 0 || skipped_objects > 0 || failed_buckets > 0 {
+            if self.replacement_task_id.is_some() && resume_manager.schedule_retry().await? {
+                checkpoint_manager.reset_for_retry().await?;
+                return Err(Error::transient_skip(format!(
+                    "Replacement erasure set heal incomplete: {failed_buckets} bucket(s) failed, {failed_objects} object(s) failed, {skipped_objects} object(s) skipped; retry scheduled"
+                )));
+            }
             if resume_manager.schedule_retry().await? {
                 // Both persistence layers must be reset together: schedule_retry
                 // rewinds the resume state (cursor + counters), and the
@@ -508,15 +599,15 @@ impl ErasureSetHealer {
                     state = "retry_scheduled",
                     "Erasure set heal pass finished with unhealed versions; scheduled full re-heal retry"
                 );
-                return Err(Error::other(format!(
+                return Err(Error::transient_skip(format!(
                     "Erasure set heal incomplete: {failed_buckets} bucket(s) failed, {failed_objects} object(s) failed, {skipped_objects} object(s) skipped; retry scheduled"
                 )));
             }
 
-            // Retry budget exhausted: drop the resume/checkpoint state so this
-            // task does not loop, but keep the healing markers (return Err) so a
-            // later heal cycle / the background scanner starts a fresh attempt.
-            // Never silently claim a clean completion while objects are unhealed.
+            // Retry budget exhausted: keep the resume/checkpoint state while
+            // the replacement marker remains. A later repair must retain the
+            // durable evidence of the incomplete generation instead of
+            // starting from an indistinguishable blank state.
             error!(
                 target: "rustfs::heal::erasure_healer",
                 event = EVENT_HEAL_ERASURE_RESUME_STATE,
@@ -529,15 +620,16 @@ impl ErasureSetHealer {
                 state = "failed_after_retries",
                 "Erasure set heal exhausted retries with unrecovered versions"
             );
-            checkpoint_manager.cleanup().await?;
-            resume_manager.cleanup().await?;
             return Err(Error::other(format!(
                 "Erasure set heal exhausted retries with {failed_buckets} bucket(s) failed, {failed_objects} object(s) failed, {skipped_objects} object(s) skipped"
             )));
         }
 
-        // no failures — mark task completed
-        resume_manager.mark_completed().await?;
+        // No failures — ordinary heals are complete now. Replacement heals
+        // atomically transition to Verified after the terminal identity fence.
+        if self.replacement_task_id.is_none() {
+            resume_manager.mark_completed().await?;
+        }
 
         debug!(
             target: "rustfs::heal::erasure_healer",
@@ -628,6 +720,7 @@ impl ErasureSetHealer {
             matches!(self.heal_opts.scan_mode, HealScanMode::Deep) || matches!(self.source, HealRequestSource::AutoHeal);
 
         loop {
+            self.verify_replacement_identity_fence("page scan").await?;
             // Get one page of object versions
             let (objects, next_token, is_truncated) = if use_disk_walk {
                 self.storage
@@ -672,6 +765,8 @@ impl ErasureSetHealer {
                 let set_label = set_disk_id.to_string();
                 let heal_opts = self.heal_opts;
                 let semaphore = semaphore.clone();
+                let target_endpoints = self.target_endpoints.clone();
+                let replacement_commit_evidence_required = self.replacement_task_id.is_some();
 
                 page_tasks.push(async move {
                     let permit = semaphore
@@ -699,6 +794,35 @@ impl ErasureSetHealer {
                             .heal_object(&bucket_name, &object_name, version_id.as_deref(), &heal_opts)
                             .await
                         {
+                            Ok((result, None))
+                                if target_outcomes_complete(&result, &target_endpoints) =>
+                            {
+                                if !replacement_commit_evidence_required {
+                                    Ok(true)
+                                } else {
+                                    match storage
+                                        .replacement_targets_have_version(
+                                            &bucket_name,
+                                            &object_name,
+                                            version_id.as_deref(),
+                                            &heal_opts,
+                                            &target_endpoints,
+                                        )
+                                        .await
+                                    {
+                                        Ok(true) => Ok(true),
+                                        Ok(false) => Err(Error::transient_skip(format!(
+                                            "Skipped heal for {bucket_name}/{object_name} because replacement target readback did not confirm the committed version"
+                                        ))),
+                                        Err(err) => Err(Error::transient_skip(format!(
+                                            "Skipped heal for {bucket_name}/{object_name} because replacement target readback failed: {err}"
+                                        ))),
+                                    }
+                                }
+                            }
+                            Ok((_result, None)) if !target_endpoints.is_empty() => Err(Error::transient_skip(format!(
+                                "Skipped heal for {bucket_name}/{object_name} because a replacement target was not committed"
+                            ))),
                             Ok((_result, None)) => Ok(true),
                             Ok((_, Some(err))) if is_missing_object_dir_heal_result(&object_name, &err) => Ok(false),
                             Ok((_, Some(err))) | Err(err) => match Self::classify_heal_object_error(&err) {
@@ -1011,9 +1135,12 @@ mod resume_loop_tests {
     //! that emits programmable multi-version pages. These exercise the real loop
     //! logic (cursor seeding, per-version dedup, anti-loop guard, absence
     //! handling) — not merely a mock's own output.
-    use super::ErasureSetHealer;
+    use super::{ErasureSetHealer, target_outcomes_complete};
     use crate::heal::progress::HealProgress;
-    use crate::heal::resume::{CheckpointManager, RESUME_CHECKPOINT_FILE, ResumeDeleteFailure, ResumeManager, compose_key};
+    use crate::heal::resume::{
+        CheckpointManager, RESUME_CHECKPOINT_FILE, ReplacementTargetIdentity, ResumeDeleteFailure, ResumeManager, ResumeUtils,
+        compose_key,
+    };
     use crate::heal::storage::{DiskStatus, HealListItem, HealObjectInfo, HealStorageAPI};
     use crate::heal::storage_api::status::BucketInfo;
     use crate::heal::{
@@ -1021,8 +1148,8 @@ mod resume_loop_tests {
     };
     use crate::{Error, Result};
     use rustfs_common::heal_channel::{HealOpts, HealRequestSource};
-    use rustfs_madmin::heal_commands::HealResultItem;
-    use std::collections::HashMap;
+    use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem, Infos};
+    use std::collections::{HashMap, VecDeque};
     use std::sync::atomic::{AtomicBool, Ordering};
     use std::sync::{Arc, Mutex};
     use tempfile::TempDir;
@@ -1037,6 +1164,53 @@ mod resume_loop_tests {
         }
     }
 
+    #[test]
+    fn target_outcomes_require_each_requested_endpoint_once_and_ok() {
+        let result = HealResultItem {
+            after: Infos {
+                drives: vec![
+                    HealDriveInfo {
+                        endpoint: "replacement-a".to_string(),
+                        state: "ok".to_string(),
+                        ..Default::default()
+                    },
+                    HealDriveInfo {
+                        endpoint: "replacement-b".to_string(),
+                        state: "missing".to_string(),
+                        ..Default::default()
+                    },
+                ],
+            },
+            ..Default::default()
+        };
+
+        assert!(target_outcomes_complete(&result, &["replacement-a".to_string()]));
+        assert!(!target_outcomes_complete(
+            &result,
+            &["replacement-a".to_string(), "replacement-b".to_string()]
+        ));
+        assert!(!target_outcomes_complete(&result, &["replacement-c".to_string()]));
+
+        let duplicate = HealResultItem {
+            after: Infos {
+                drives: vec![
+                    HealDriveInfo {
+                        endpoint: "replacement-a".to_string(),
+                        state: "ok".to_string(),
+                        ..Default::default()
+                    },
+                    HealDriveInfo {
+                        endpoint: "replacement-a".to_string(),
+                        state: "missing".to_string(),
+                        ..Default::default()
+                    },
+                ],
+            },
+            ..Default::default()
+        };
+        assert!(!target_outcomes_complete(&duplicate, &["replacement-a".to_string()]));
+    }
+
     #[derive(Clone)]
     struct Page {
         items: Vec<HealListItem>,
@@ -1061,8 +1235,14 @@ mod resume_loop_tests {
         pages: Mutex<HashMap<Option<String>, Page>>,
         /// per-`compose_key` heal outcome; default is `Ok`
         outcomes: Mutex<HashMap<String, HealOutcome>>,
+        /// successful low-level result per `compose_key`; default has no drive outcomes.
+        results: Mutex<HashMap<String, HealResultItem>>,
+        /// Target-specific physical readback evidence per `compose_key`; the
+        /// fake models a healthy backend unless a test explicitly revokes it.
+        replacement_commit_evidence: Mutex<HashMap<String, bool>>,
         /// every heal_object call recorded as (name, version_id)
         heal_calls: Mutex<Vec<(String, Option<String>)>>,
+        replacement_target_identity_sequences: Mutex<VecDeque<Vec<ReplacementTargetIdentity>>>,
         fail_listing: AtomicBool,
     }
 
@@ -1073,6 +1253,15 @@ mod resume_loop_tests {
         fn set_outcome(&self, name: &str, version: Option<&str>, outcome: HealOutcome) {
             self.outcomes.lock().unwrap().insert(compose_key(name, version), outcome);
         }
+        fn set_result(&self, name: &str, version: Option<&str>, result: HealResultItem) {
+            self.results.lock().unwrap().insert(compose_key(name, version), result);
+        }
+        fn set_replacement_commit_evidence(&self, name: &str, version: Option<&str>, committed: bool) {
+            self.replacement_commit_evidence
+                .lock()
+                .unwrap()
+                .insert(compose_key(name, version), committed);
+        }
         fn calls(&self) -> Vec<(String, Option<String>)> {
             self.heal_calls.lock().unwrap().clone()
         }
@@ -1143,7 +1332,7 @@ mod resume_loop_tests {
             let key = compose_key(object, version_id);
             let outcome = self.outcomes.lock().unwrap().get(&key).cloned().unwrap_or(HealOutcome::Ok);
             match outcome {
-                HealOutcome::Ok => Ok((HealResultItem::default(), None)),
+                HealOutcome::Ok => Ok((self.results.lock().unwrap().get(&key).cloned().unwrap_or_default(), None)),
                 HealOutcome::VersionNotFound => {
                     Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::FileVersionNotFound))))
                 }
@@ -1157,6 +1346,21 @@ mod resume_loop_tests {
         async fn heal_format(&self, _dry: bool) -> Result<(HealResultItem, Option<Error>)> {
             Ok((HealResultItem::default(), None))
         }
+        async fn replacement_targets_have_version(
+            &self,
+            _bucket: &str,
+            object: &str,
+            version_id: Option<&str>,
+            _opts: &HealOpts,
+            _targets: &[String],
+        ) -> Result<bool> {
+            Ok(*self
+                .replacement_commit_evidence
+                .lock()
+                .unwrap()
+                .get(&compose_key(object, version_id))
+                .unwrap_or(&true))
+        }
         async fn list_objects_for_heal(&self, _b: &str, _p: &str) -> Result<Vec<HealListItem>> {
             Ok(Vec::new())
         }
@@ -1179,6 +1383,13 @@ mod resume_loop_tests {
         async fn get_disk_for_resume(&self, _id: &str) -> Result<DiskStore> {
             Err(Error::other("not implemented in tests"))
         }
+        async fn replacement_target_identities(&self, _targets: &[String]) -> Result<Vec<ReplacementTargetIdentity>> {
+            self.replacement_target_identity_sequences
+                .lock()
+                .unwrap()
+                .pop_front()
+                .ok_or_else(|| Error::other("replacement identity sequence exhausted"))
+        }
     }
 
     async fn make_disk(temp: &TempDir) -> DiskStore {
@@ -1204,13 +1415,19 @@ mod resume_loop_tests {
         storage: Arc<FakeStorage>,
         resume: ResumeManager,
         checkpoint: CheckpointManager,
+        task_id: String,
         _temp: TempDir,
     }
 
     async fn make_env() -> Env {
+        make_env_with_targets(Vec::new()).await
+    }
+
+    async fn make_env_with_targets(target_endpoints: Vec<String>) -> Env {
         let temp = TempDir::new().unwrap();
         let disk = make_disk(&temp).await;
         let storage = Arc::new(FakeStorage::default());
+        let task_id = ResumeUtils::generate_task_id();
         let healer = ErasureSetHealer::new(
             storage.clone(),
             Arc::new(RwLock::new(HealProgress::new())),
@@ -1218,22 +1435,24 @@ mod resume_loop_tests {
             disk.clone(),
             HealOpts::default(),
             HealRequestSource::Internal,
-        );
+        )
+        .with_replacement_targets(target_endpoints, None);
         let resume = ResumeManager::new(
             disk.clone(),
-            "task".to_string(),
+            task_id.clone(),
             "erasure_set".to_string(),
             "pool_0_set_0".to_string(),
             vec!["b".to_string()],
         )
         .await
         .unwrap();
-        let checkpoint = CheckpointManager::new(disk, "task".to_string()).await.unwrap();
+        let checkpoint = CheckpointManager::new(disk, task_id.clone()).await.unwrap();
         Env {
             healer,
             storage,
             resume,
             checkpoint,
+            task_id,
             _temp: temp,
         }
     }
@@ -1277,6 +1496,112 @@ mod resume_loop_tests {
         assert_eq!(env.resume.resume_cursor().await, None);
     }
 
+    #[tokio::test]
+    async fn replacement_targets_use_a_canonical_order() {
+        let env = make_env_with_targets(vec![
+            "replacement-b".to_string(),
+            "replacement-a".to_string(),
+            "replacement-b".to_string(),
+        ])
+        .await;
+
+        assert_eq!(env.healer.target_endpoints.as_ref(), ["replacement-a", "replacement-b"]);
+    }
+
+    #[tokio::test]
+    async fn replacement_identity_fence_rejects_a_remount_before_page_scan() {
+        let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
+        let expected_identity = ReplacementTargetIdentity {
+            endpoint: "replacement-a".to_string(),
+            canonical_path: "/mnt/replacement-a".to_string(),
+            physical_device_ids: vec!["device-a".to_string()],
+            filesystem_identity: "filesystem-a".to_string(),
+        };
+        let remounted_identity = ReplacementTargetIdentity {
+            physical_device_ids: vec!["device-b".to_string()],
+            filesystem_identity: "filesystem-b".to_string(),
+            ..expected_identity.clone()
+        };
+        env.storage
+            .replacement_target_identity_sequences
+            .lock()
+            .unwrap()
+            .push_back(vec![remounted_identity]);
+        let healer = ErasureSetHealer::new(
+            env.storage.clone(),
+            Arc::new(RwLock::new(HealProgress::new())),
+            CancellationToken::new(),
+            env.healer.disk.clone(),
+            HealOpts::default(),
+            HealRequestSource::AutoHeal,
+        )
+        .with_replacement_targets(vec!["replacement-a".to_string()], Some("generation-a".to_string()))
+        .with_replacement_identity_fence(Some(vec![expected_identity]));
+        let mut current_object_index = 0;
+        let mut processed = 0;
+        let mut successful = 0;
+        let mut failed = 0;
+        let mut skipped = 0;
+
+        let error = healer
+            .heal_bucket_with_resume(
+                "b",
+                "pool_0_set_0",
+                0,
+                &mut current_object_index,
+                &mut processed,
+                &mut successful,
+                &mut failed,
+                &mut skipped,
+                &env.resume,
+                &env.checkpoint,
+            )
+            .await
+            .expect_err("a remounted target must not begin a new page scan");
+
+        assert!(error.to_string().contains("page scan"));
+        assert!(env.storage.calls().is_empty());
+    }
+
+    #[tokio::test]
+    async fn replacement_generation_never_reuses_another_disk_cursor() {
+        let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
+        ResumeManager::new_replacement_intent(
+            env.healer.disk.clone(),
+            ResumeUtils::generate_task_id(),
+            "pool_0_set_0".to_string(),
+            vec!["b".to_string()],
+            vec!["replacement-a".to_string()],
+            vec![crate::heal::resume::ReplacementTargetIdentity {
+                endpoint: "replacement-a".to_string(),
+                canonical_path: "/mnt/replacement-a".to_string(),
+                physical_device_ids: vec!["device-a".to_string()],
+                filesystem_identity: "1:2:3".to_string(),
+            }],
+        )
+        .await
+        .expect("first replacement intent should persist");
+
+        let healer = ErasureSetHealer::new(
+            env.storage.clone(),
+            Arc::new(RwLock::new(HealProgress::new())),
+            CancellationToken::new(),
+            env.healer.disk.clone(),
+            HealOpts::default(),
+            HealRequestSource::AutoHeal,
+        )
+        .with_replacement_targets(vec!["replacement-a".to_string()], Some(ResumeUtils::generate_task_id()));
+
+        let error = healer
+            .get_or_create_task_id("pool_0_set_0")
+            .await
+            .expect_err("a second replacement must not reuse the first replacement cursor");
+        assert!(
+            !error.to_string().contains("generation-a"),
+            "the previous replacement generation must not be selected"
+        );
+    }
+
     #[tokio::test]
     async fn object_timeout_aborts_the_bucket_page_immediately() {
         let env = make_env().await;
@@ -1330,13 +1655,14 @@ mod resume_loop_tests {
             .await
             .expect("new heal should allocate a task id");
 
-        assert_ne!(task_id, "task", "a completed resume state must not suppress a new heal");
+        assert_ne!(task_id, env.task_id, "a completed resume state must not suppress a new heal");
+        assert!(uuid::Uuid::parse_str(&task_id).is_ok(), "new resume task ids must be UUIDs");
     }
 
     #[tokio::test]
     async fn cleanup_failure_keeps_erasure_set_heal_incomplete() {
         let env = make_env().await;
-        let checkpoint_path = format!("{BUCKET_META_PREFIX}/task_{RESUME_CHECKPOINT_FILE}");
+        let checkpoint_path = format!("{BUCKET_META_PREFIX}/{}_{RESUME_CHECKPOINT_FILE}", env.task_id);
         let _failure = ResumeDeleteFailure::install(checkpoint_path, crate::heal::DiskError::DiskAccessDenied);
 
         let error = env
@@ -1346,7 +1672,7 @@ mod resume_loop_tests {
             .expect_err("checkpoint cleanup failure must fail the erasure-set heal");
 
         assert!(matches!(error, Error::Disk(crate::heal::DiskError::DiskAccessDenied)));
-        let state = ResumeManager::load_from_disk(env.healer.disk.clone(), "task")
+        let state = ResumeManager::load_from_disk(env.healer.disk.clone(), &env.task_id)
             .await
             .expect("completed state must remain discoverable after cleanup failure")
             .get_state()
@@ -1354,6 +1680,84 @@ mod resume_loop_tests {
         assert!(state.completed, "successful data heal must be persisted before cleanup is attempted");
     }
 
+    #[tokio::test]
+    async fn replacement_completion_keeps_resume_artifacts_until_marker_cleanup() {
+        let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
+        let replacement_task_id = ResumeUtils::generate_task_id();
+        ResumeManager::new_replacement_intent(
+            env.healer.disk.clone(),
+            replacement_task_id.clone(),
+            "pool_0_set_0".to_string(),
+            vec!["b".to_string()],
+            vec!["replacement-a".to_string()],
+            vec![crate::heal::resume::ReplacementTargetIdentity {
+                endpoint: "replacement-a".to_string(),
+                canonical_path: "/mnt/replacement-a".to_string(),
+                physical_device_ids: vec!["device-a".to_string()],
+                filesystem_identity: "1:2:3".to_string(),
+            }],
+        )
+        .await
+        .expect("replacement intent should persist");
+        let checkpoint = CheckpointManager::new(env.healer.disk.clone(), replacement_task_id.clone())
+            .await
+            .expect("replacement checkpoint should persist");
+        let healer = ErasureSetHealer::new(
+            env.storage.clone(),
+            Arc::new(RwLock::new(HealProgress::new())),
+            CancellationToken::new(),
+            env.healer.disk.clone(),
+            HealOpts::default(),
+            HealRequestSource::AutoHeal,
+        )
+        .with_replacement_targets(vec!["replacement-a".to_string()], Some(replacement_task_id.clone()));
+
+        healer
+            .heal_erasure_set(&["b".to_string()], "pool_0_set_0")
+            .await
+            .expect("replacement data scan should complete");
+
+        let state = ResumeManager::load_replacement_intent(env.healer.disk.clone(), &replacement_task_id)
+            .await
+            .expect("verified replacement state must remain after data scan")
+            .get_state()
+            .await;
+        assert!(state.completed, "the verified state must record a completed data scan");
+        assert_eq!(state.replacement_phase, crate::heal::resume::ReplacementPhase::Verified);
+        assert!(
+            CheckpointManager::has_checkpoint(&env.healer.disk, &replacement_task_id).await,
+            "the checkpoint must survive until the caller clears the healing marker"
+        );
+        drop(checkpoint);
+    }
+
+    #[tokio::test]
+    async fn retry_exhaustion_keeps_resume_artifacts_for_recovery() {
+        let env = make_env().await;
+        for _ in 0..3 {
+            assert!(env.resume.schedule_retry().await.expect("retry state should persist"));
+            env.checkpoint
+                .reset_for_retry()
+                .await
+                .expect("checkpoint reset should persist");
+        }
+        env.storage.fail_listing();
+
+        env.healer
+            .execute_heal_with_resume(&["b".to_string()], "pool_0_set_0", &env.resume, &env.checkpoint)
+            .await
+            .expect_err("exhausted retry state must report the incomplete heal");
+
+        assert!(
+            ResumeManager::has_resume_state(&env.healer.disk, &env.task_id).await,
+            "retry exhaustion must not delete the resumable state while a marker may remain"
+        );
+        assert!(
+            CheckpointManager::has_checkpoint(&env.healer.disk, &env.task_id).await,
+            "retry exhaustion must retain the checkpoint with the resumable state"
+        );
+    }
+
     #[tokio::test]
     async fn retry_resume_repairs_checkpoint_after_crash_between_resets() {
         let env = make_env().await;
@@ -1385,7 +1789,7 @@ mod resume_loop_tests {
 
         let (_, checkpoint) = env
             .healer
-            .initialize_resume_state("task", "pool_0_set_0", &["b".to_string()])
+            .initialize_resume_state(&env.task_id, "pool_0_set_0", &["b".to_string()])
             .await
             .expect("resume initialization should repair a stale checkpoint");
         let checkpoint = checkpoint.get_checkpoint().await;
@@ -1653,4 +2057,132 @@ mod resume_loop_tests {
             "the skipped set must be cleared so the retry re-heals the version"
         );
     }
+
+    #[tokio::test]
+    async fn replacement_target_missing_from_success_result_retries_the_full_pass() {
+        let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
+        env.storage.set_page(
+            None,
+            Page {
+                items: vec![item("object", Some("v1"), false)],
+                next: None,
+                truncated: false,
+            },
+        );
+        env.storage.set_result(
+            "object",
+            Some("v1"),
+            HealResultItem {
+                after: Infos {
+                    drives: vec![HealDriveInfo {
+                        endpoint: "replacement-a".to_string(),
+                        state: "missing".to_string(),
+                        ..Default::default()
+                    }],
+                },
+                ..Default::default()
+            },
+        );
+
+        let result = env
+            .healer
+            .execute_heal_with_resume(&["b".to_string()], "pool_0_set_0", &env.resume, &env.checkpoint)
+            .await;
+
+        result.expect_err("a missing replacement target must not report completion");
+        let state = env.resume.get_state().await;
+        assert!(!state.completed);
+        assert_eq!(state.retry_count, 1);
+        assert_eq!(env.storage.calls(), vec![("object".to_string(), Some("v1".to_string()))]);
+        assert!(env.checkpoint.get_checkpoint().await.processed_objects.is_empty());
+    }
+
+    #[tokio::test]
+    async fn replacement_target_readback_evidence_must_confirm_the_healed_version() {
+        let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
+        let healer = ErasureSetHealer::new(
+            env.storage.clone(),
+            Arc::new(RwLock::new(HealProgress::new())),
+            CancellationToken::new(),
+            env.healer.disk.clone(),
+            HealOpts::default(),
+            HealRequestSource::AutoHeal,
+        )
+        .with_replacement_targets(vec!["replacement-a".to_string()], Some("generation-a".to_string()));
+        env.storage.set_page(
+            None,
+            Page {
+                items: vec![item("object", Some("v1"), false)],
+                next: None,
+                truncated: false,
+            },
+        );
+        env.storage.set_result(
+            "object",
+            Some("v1"),
+            HealResultItem {
+                after: Infos {
+                    drives: vec![HealDriveInfo {
+                        endpoint: "replacement-a".to_string(),
+                        state: "ok".to_string(),
+                        ..Default::default()
+                    }],
+                },
+                ..Default::default()
+            },
+        );
+        env.storage.set_replacement_commit_evidence("object", Some("v1"), false);
+
+        let result = healer
+            .execute_heal_with_resume(&["b".to_string()], "pool_0_set_0", &env.resume, &env.checkpoint)
+            .await;
+
+        result.expect_err("a success result without target readback evidence must retry");
+        assert_eq!(env.resume.get_state().await.retry_count, 1);
+        assert!(env.checkpoint.get_checkpoint().await.processed_objects.is_empty());
+    }
+
+    #[tokio::test]
+    async fn manual_targeted_heal_keeps_existing_best_effort_result_semantics() {
+        let env = make_env_with_targets(vec!["replacement-a".to_string()]).await;
+        let healer = ErasureSetHealer::new(
+            env.storage.clone(),
+            Arc::new(RwLock::new(HealProgress::new())),
+            CancellationToken::new(),
+            env.healer.disk.clone(),
+            HealOpts::default(),
+            HealRequestSource::Admin,
+        )
+        .with_replacement_targets(vec!["replacement-a".to_string()], None);
+        env.storage.set_page(
+            None,
+            Page {
+                items: vec![item("object", Some("v1"), false)],
+                next: None,
+                truncated: false,
+            },
+        );
+        env.storage.set_result(
+            "object",
+            Some("v1"),
+            HealResultItem {
+                after: Infos {
+                    drives: vec![HealDriveInfo {
+                        endpoint: "replacement-a".to_string(),
+                        state: "ok".to_string(),
+                        ..Default::default()
+                    }],
+                },
+                ..Default::default()
+            },
+        );
+        env.storage.set_replacement_commit_evidence("object", Some("v1"), false);
+
+        healer
+            .execute_heal_with_resume(&["b".to_string()], "pool_0_set_0", &env.resume, &env.checkpoint)
+            .await
+            .expect("manual targeted healing must retain its existing success semantics");
+
+        assert_eq!(env.resume.get_state().await.retry_count, 0);
+    }
 }

crates/heal/src/heal/mod.rs

@@ -17,6 +17,7 @@ pub mod erasure_healer;
 pub mod event;
 pub mod manager;
 pub mod progress;
+pub(crate) mod replacement_readiness;
 pub mod resume;
 pub mod storage;
 pub(crate) mod storage_api;
@@ -25,8 +26,8 @@ pub mod utils;
 
 use storage_api::owner::{
     ECSTORE_BUCKET_META_PREFIX, ECSTORE_DATA_USAGE_CACHE_NAME, ECSTORE_HEALING_MARKER_PATH, ECSTORE_RUSTFS_META_BUCKET,
-    EcstoreDeleteOptions, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError, EcstoreDiskResult, EcstoreDiskStore,
-    EcstoreEndpoint, EcstoreErrorType, EcstoreStorageError, EcstoreStore, ObjectIO, ObjectOperations,
+    EcstoreConditionalFileUpdate, EcstoreDeleteOptions, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError, EcstoreDiskResult,
+    EcstoreDiskStore, EcstoreEndpoint, EcstoreErrorType, EcstoreStorageError, EcstoreStore, ObjectIO, ObjectOperations,
     ecstore_local_disk_map_read,
 };
 #[cfg(test)]
@@ -76,56 +77,155 @@ pub(crate) const HEALING_MARKER_PATH: &str = ECSTORE_HEALING_MARKER_PATH;
 
 /// Write the healing marker on the local disks matching `endpoints` so their
 /// `DiskInfo.healing` reports true while the erasure-set heal rebuilds them.
-pub(crate) async fn set_healing_markers(endpoints: &[String], set_disk_id: &str) {
-    apply_healing_markers(endpoints, Some(set_disk_id)).await;
+pub(crate) async fn set_healing_markers(endpoints: &[String], marker: &str) -> crate::Result<()> {
+    apply_healing_markers(endpoints, Some(marker), None, false).await
 }
 
-/// Remove the healing markers written by [`set_healing_markers`].
-pub(crate) async fn clear_healing_markers(endpoints: &[String]) {
-    apply_healing_markers(endpoints, None).await;
+/// Remove an owner marker after the replacement scan's verified state is
+/// durable. A missing marker is idempotent here because a crash may have
+/// happened after the previous terminal clear and before resume cleanup.
+pub(crate) async fn clear_healing_markers_after_verified(endpoints: &[String], marker: &str) -> crate::Result<()> {
+    apply_healing_markers(endpoints, None, Some(marker), true).await
 }
 
-async fn apply_healing_markers(endpoints: &[String], set_disk_id: Option<&str>) {
+#[cfg(test)]
+fn marker_matches(current: &[u8], expected_marker: Option<&str>) -> bool {
+    expected_marker.is_some_and(|expected| current == expected.as_bytes())
+}
+
+async fn apply_healing_markers(
+    endpoints: &[String],
+    marker: Option<&str>,
+    expected_marker: Option<&str>,
+    allow_missing: bool,
+) -> crate::Result<()> {
     if endpoints.is_empty() {
-        return;
+        return Ok(());
     }
-    let local_disk_map = local_disk_map_read().await;
-    for disk in local_disk_map.values().flatten() {
-        let endpoint = EcstoreDiskAPI::endpoint(disk.as_ref()).to_string();
-        if !endpoints.iter().any(|candidate| candidate == &endpoint) {
-            continue;
+    let mut local_disks = std::collections::HashMap::new();
+    {
+        let local_disk_map = local_disk_map_read().await;
+        for disk in local_disk_map.values().flatten() {
+            local_disks.insert(EcstoreDiskAPI::endpoint(disk.as_ref()).to_string(), disk.clone());
         }
-        let result = match set_disk_id {
-            Some(set_disk_id) => {
-                EcstoreDiskAPI::write_all(
+    }
+
+    let mut matched_endpoints = std::collections::HashSet::new();
+    let mut targets = Vec::with_capacity(endpoints.len());
+    for endpoint in endpoints {
+        if !matched_endpoints.insert(endpoint.clone()) {
+            return Err(DiskError::other("healing marker endpoint is duplicated").into());
+        }
+        let Some(disk) = local_disks.remove(endpoint) else {
+            return Err(DiskError::other("healing marker target is unavailable").into());
+        };
+        targets.push(disk);
+    }
+
+    apply_healing_markers_to_targets(targets, marker, expected_marker, allow_missing).await
+}
+
+async fn apply_healing_markers_to_targets(
+    targets: Vec<DiskStore>,
+    marker: Option<&str>,
+    expected_marker: Option<&str>,
+    allow_missing: bool,
+) -> crate::Result<()> {
+    apply_healing_markers_to_targets_with_after_acquire(targets, marker, expected_marker, allow_missing, |_| {}).await
+}
+
+async fn apply_healing_markers_to_targets_with_after_acquire<F>(
+    targets: Vec<DiskStore>,
+    marker: Option<&str>,
+    expected_marker: Option<&str>,
+    allow_missing: bool,
+    mut after_acquire: F,
+) -> crate::Result<()>
+where
+    F: FnMut(&DiskStore),
+{
+    let marker_bytes = marker.map(|marker| EcstoreDiskBytes::copy_from_slice(marker.as_bytes()));
+    let expected_bytes = expected_marker.map(|marker| EcstoreDiskBytes::copy_from_slice(marker.as_bytes()));
+    let mut newly_acquired = Vec::new();
+    for disk in targets {
+        let result = match marker_bytes.as_ref() {
+            Some(marker) => {
+                match EcstoreDiskAPI::compare_and_update_file(
                     disk.as_ref(),
                     RUSTFS_META_BUCKET,
                     HEALING_MARKER_PATH,
-                    EcstoreDiskBytes::copy_from_slice(set_disk_id.as_bytes()),
+                    None,
+                    Some(marker.clone()),
                 )
                 .await
+                {
+                    Ok(EcstoreConditionalFileUpdate::Updated) => {
+                        newly_acquired.push(disk.clone());
+                        after_acquire(&disk);
+                        Ok(())
+                    }
+                    Ok(EcstoreConditionalFileUpdate::Mismatch) => match EcstoreDiskAPI::compare_and_update_file(
+                        disk.as_ref(),
+                        RUSTFS_META_BUCKET,
+                        HEALING_MARKER_PATH,
+                        Some(marker.clone()),
+                        Some(marker.clone()),
+                    )
+                    .await
+                    {
+                        Ok(EcstoreConditionalFileUpdate::Updated) => Ok(()),
+                        Ok(_) => Err(DiskError::other("healing marker ownership changed")),
+                        Err(err) => Err(err),
+                    },
+                    Ok(EcstoreConditionalFileUpdate::Missing) => Err(DiskError::other("healing marker disappeared")),
+                    Err(err) => Err(err),
+                }
+            }
+            None => {
+                match EcstoreDiskAPI::compare_and_update_file(
+                    disk.as_ref(),
+                    RUSTFS_META_BUCKET,
+                    HEALING_MARKER_PATH,
+                    expected_bytes.clone(),
+                    None,
+                )
+                .await
+                {
+                    Ok(EcstoreConditionalFileUpdate::Updated) => Ok(()),
+                    Ok(EcstoreConditionalFileUpdate::Missing) if allow_missing => Ok(()),
+                    Ok(EcstoreConditionalFileUpdate::Missing) => Err(DiskError::other("healing marker is missing")),
+                    Ok(EcstoreConditionalFileUpdate::Mismatch) => Err(DiskError::other("healing marker ownership changed")),
+                    Err(err) => Err(err),
+                }
             }
-            None => match EcstoreDiskAPI::delete(
-                disk.as_ref(),
-                RUSTFS_META_BUCKET,
-                HEALING_MARKER_PATH,
-                EcstoreDeleteOptions::default(),
-            )
-            .await
-            {
-                Err(DiskError::FileNotFound) => Ok(()),
-                other => other,
-            },
         };
         if let Err(err) = result {
-            tracing::warn!(
-                endpoint = %endpoint,
-                action = if set_disk_id.is_some() { "set" } else { "clear" },
-                error = ?err,
-                "failed to update healing marker"
-            );
+            if let Some(marker) = marker_bytes.as_ref() {
+                let mut rollback_error = None;
+                for acquired in newly_acquired.iter().rev() {
+                    if let Err(rollback) = EcstoreDiskAPI::compare_and_update_file(
+                        acquired.as_ref(),
+                        RUSTFS_META_BUCKET,
+                        HEALING_MARKER_PATH,
+                        Some(marker.clone()),
+                        None,
+                    )
+                    .await
+                    {
+                        rollback_error.get_or_insert(rollback);
+                    }
+                }
+                if let Some(rollback) = rollback_error {
+                    return Err(DiskError::other(format!(
+                        "healing marker acquisition failed ({err}) and owner-safe rollback failed ({rollback})"
+                    ))
+                    .into());
+                }
+            }
+            return Err(err.into());
         }
     }
+    Ok(())
 }
 
 pub(crate) type DiskError = EcstoreDiskError;
@@ -203,3 +303,195 @@ where
 pub type HealObjectInfo = <ECStore as ObjectOperations>::ObjectInfo;
 pub type HealObjectOptions = <ECStore as ObjectOperations>::ObjectOptions;
 pub type HealPutObjReader = <ECStore as ObjectIO>::PutObjectReader;
+
+#[cfg(test)]
+mod tests {
+    use super::{
+        DiskError, DiskOption, Endpoint, HEALING_MARKER_PATH, RUSTFS_META_BUCKET, apply_healing_markers_to_targets,
+        apply_healing_markers_to_targets_with_after_acquire, marker_matches, new_disk,
+    };
+    use crate::{
+        Error,
+        heal::storage_api::owner::{EcstoreConditionalFileUpdate, EcstoreDiskAPI, EcstoreDiskBytes},
+    };
+    use tempfile::TempDir;
+
+    async fn make_marker_disk(temp: &TempDir, name: &str) -> super::DiskStore {
+        let path = temp.path().join(name);
+        std::fs::create_dir_all(&path).expect("marker disk directory should be created");
+        let endpoint = Endpoint::try_from(path.to_string_lossy().as_ref()).expect("marker disk endpoint should be valid");
+        let disk = new_disk(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .expect("marker disk should initialize");
+        let metadata_volume = disk.make_volume(RUSTFS_META_BUCKET).await;
+        assert!(
+            matches!(metadata_volume, Ok(()) | Err(DiskError::VolumeExists)),
+            "marker metadata volume should exist: {metadata_volume:?}"
+        );
+        disk
+    }
+
+    #[test]
+    fn marker_clear_requires_the_current_owner_token() {
+        assert!(marker_matches(b"set:task-a", Some("set:task-a")));
+        assert!(!marker_matches(b"set:task-b", Some("set:task-a")));
+        assert!(!marker_matches(b"set:task-a", None));
+    }
+
+    #[tokio::test]
+    async fn marker_acquisition_rolls_back_after_second_disk_ownership_conflict() {
+        let temp = TempDir::new().expect("marker test directory should be created");
+        let first = make_marker_disk(&temp, "first").await;
+        let second = make_marker_disk(&temp, "second").await;
+        let owner_b = EcstoreDiskBytes::from_static(b"owner-b");
+
+        assert_eq!(
+            EcstoreDiskAPI::compare_and_update_file(
+                second.as_ref(),
+                RUSTFS_META_BUCKET,
+                HEALING_MARKER_PATH,
+                None,
+                Some(owner_b.clone()),
+            )
+            .await
+            .expect("second disk owner should acquire marker"),
+            EcstoreConditionalFileUpdate::Updated
+        );
+
+        let err = apply_healing_markers_to_targets(vec![first.clone(), second.clone()], Some("owner-a"), None, false)
+            .await
+            .expect_err("second disk ownership must reject the partial acquisition");
+        assert!(matches!(err, Error::Disk(DiskError::Io(ref io)) if io.to_string() == "healing marker ownership changed"));
+        assert!(matches!(
+            EcstoreDiskAPI::read_all(first.as_ref(), RUSTFS_META_BUCKET, HEALING_MARKER_PATH).await,
+            Err(DiskError::FileNotFound)
+        ));
+        assert_eq!(
+            EcstoreDiskAPI::read_all(second.as_ref(), RUSTFS_META_BUCKET, HEALING_MARKER_PATH)
+                .await
+                .expect("conflicting owner marker must remain"),
+            owner_b
+        );
+    }
+
+    #[tokio::test]
+    async fn marker_acquisition_rolls_back_after_second_disk_io_error() {
+        let temp = TempDir::new().expect("marker test directory should be created");
+        let first = make_marker_disk(&temp, "first").await;
+        let second_path = temp.path().join("second");
+        std::fs::create_dir_all(&second_path).expect("second marker disk directory should be created");
+        let second_endpoint =
+            Endpoint::try_from(second_path.to_string_lossy().as_ref()).expect("second marker endpoint should be valid");
+        let second = new_disk(
+            &second_endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .expect("second marker disk should initialize");
+        std::fs::remove_dir_all(second_path.join(RUSTFS_META_BUCKET))
+            .expect("second marker metadata directory should be removed for the I/O failure fixture");
+        std::fs::write(second_path.join(RUSTFS_META_BUCKET), b"not a directory")
+            .expect("second marker volume should become an I/O failure fixture");
+
+        let err = apply_healing_markers_to_targets(vec![first.clone(), second], Some("owner-a"), None, false)
+            .await
+            .expect_err("second disk I/O failure must reject the partial acquisition");
+        assert!(
+            matches!(err, Error::Disk(DiskError::FileAccessDenied)),
+            "second marker operation must report its mapped filesystem failure: {err:?}"
+        );
+        assert!(matches!(
+            EcstoreDiskAPI::read_all(first.as_ref(), RUSTFS_META_BUCKET, HEALING_MARKER_PATH).await,
+            Err(DiskError::FileNotFound)
+        ));
+    }
+
+    #[tokio::test]
+    async fn marker_acquisition_reports_an_owner_safe_rollback_io_failure() {
+        let temp = TempDir::new().expect("marker test directory should be created");
+        let first = make_marker_disk(&temp, "first").await;
+        let second = make_marker_disk(&temp, "second").await;
+        let owner_b = EcstoreDiskBytes::from_static(b"owner-b");
+        let first_path = EcstoreDiskAPI::path(first.as_ref());
+        let moved_metadata_path = first_path.join("metadata-before-rollback");
+
+        assert_eq!(
+            EcstoreDiskAPI::compare_and_update_file(
+                second.as_ref(),
+                RUSTFS_META_BUCKET,
+                HEALING_MARKER_PATH,
+                None,
+                Some(owner_b),
+            )
+            .await
+            .expect("second disk owner should acquire marker"),
+            EcstoreConditionalFileUpdate::Updated
+        );
+
+        let err =
+            apply_healing_markers_to_targets_with_after_acquire(vec![first, second], Some("owner-a"), None, false, |disk| {
+                let metadata_path = EcstoreDiskAPI::path(disk.as_ref()).join(RUSTFS_META_BUCKET);
+                std::fs::rename(&metadata_path, &moved_metadata_path)
+                    .expect("first marker metadata should move after acquisition");
+                std::fs::write(&metadata_path, b"not a directory")
+                    .expect("first marker metadata should become a rollback I/O failure fixture");
+            })
+            .await
+            .expect_err("rollback I/O failure must remain visible to the caller");
+        let message = err.to_string();
+        assert!(message.contains("healing marker acquisition failed"));
+        assert!(message.contains("owner-safe rollback failed"));
+        assert!(moved_metadata_path.join(HEALING_MARKER_PATH).exists());
+    }
+
+    #[tokio::test]
+    async fn concurrent_marker_acquisition_has_one_owner_on_every_disk() {
+        let temp = TempDir::new().expect("marker test directory should be created");
+        let first = make_marker_disk(&temp, "first").await;
+        let second = make_marker_disk(&temp, "second").await;
+        let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(3));
+
+        let owner_a_barrier = barrier.clone();
+        let owner_a_first = first.clone();
+        let owner_a_second = second.clone();
+        let owner_a = tokio::spawn(async move {
+            owner_a_barrier.wait().await;
+            apply_healing_markers_to_targets(vec![owner_a_first, owner_a_second], Some("owner-a"), None, false).await
+        });
+        let owner_b_barrier = barrier.clone();
+        let owner_b_first = first.clone();
+        let owner_b_second = second.clone();
+        let owner_b = tokio::spawn(async move {
+            owner_b_barrier.wait().await;
+            apply_healing_markers_to_targets(vec![owner_b_first, owner_b_second], Some("owner-b"), None, false).await
+        });
+
+        barrier.wait().await;
+        let owner_a_result = owner_a.await.expect("owner a task should join");
+        let owner_b_result = owner_b.await.expect("owner b task should join");
+        assert_ne!(
+            owner_a_result.is_ok(),
+            owner_b_result.is_ok(),
+            "exactly one owner must acquire both markers"
+        );
+
+        let winning_marker = if owner_a_result.is_ok() { b"owner-a" } else { b"owner-b" };
+        for disk in [&first, &second] {
+            assert_eq!(
+                EcstoreDiskAPI::read_all(disk.as_ref(), RUSTFS_META_BUCKET, HEALING_MARKER_PATH)
+                    .await
+                    .expect("every disk must retain the winning owner marker"),
+                EcstoreDiskBytes::from_static(winning_marker)
+            );
+        }
+    }
+}

crates/heal/src/heal/replacement_readiness.rs

@@ -0,0 +1,381 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use std::{fs, path::Path};
+
+#[cfg(test)]
+use super::Endpoint;
+use super::{DiskStore, HealDiskExt as _, local_disk_map_read, resume::ReplacementTargetIdentity};
+
+pub(crate) async fn auto_replacement_target_ready(disk: &DiskStore, local_disks: &[DiskStore]) -> bool {
+    auto_replacement_target_identity(disk, local_disks).await.is_some()
+}
+
+pub(crate) async fn auto_replacement_target_identity(
+    disk: &DiskStore,
+    local_disks: &[DiskStore],
+) -> Option<ReplacementTargetIdentity> {
+    let lease_root = disk.replacement_mount_lease_root()?;
+    let endpoint = disk.endpoint().to_string();
+    let sibling_lease_roots = local_disks
+        .iter()
+        .filter(|sibling| sibling.endpoint().is_local && sibling.endpoint().to_string() != endpoint)
+        .map(|sibling| sibling.replacement_mount_lease_root())
+        .collect::<Option<Vec<_>>>()?;
+    tokio::task::spawn_blocking(move || {
+        let canonical_path = fs::canonicalize(&lease_root).ok()?;
+        let metadata = fs::metadata(&lease_root).ok()?;
+        let Ok(target_device_ids) = rustfs_utils::os::get_physical_device_ids(lease_root.to_string_lossy().as_ref()) else {
+            return None;
+        };
+        let Ok(root_device_ids) = rustfs_utils::os::get_physical_device_ids("/") else {
+            return None;
+        };
+        if target_device_ids.is_empty()
+            || root_device_ids.is_empty()
+            || target_device_ids.iter().any(|target| root_device_ids.contains(target))
+            || !rustfs_utils::os::is_mount_point(&canonical_path).unwrap_or(false)
+        {
+            return None;
+        }
+
+        if sibling_lease_roots.iter().any(|sibling_lease_root| {
+            rustfs_utils::os::get_physical_device_ids(sibling_lease_root.to_string_lossy().as_ref())
+                .map(|ids| ids.iter().any(|id| target_device_ids.contains(id)))
+                .unwrap_or(true)
+        }) {
+            return None;
+        }
+
+        let filesystem_identity = filesystem_identity(&metadata, &canonical_path)?;
+
+        Some(ReplacementTargetIdentity {
+            endpoint,
+            canonical_path: canonical_path.to_string_lossy().into_owned(),
+            physical_device_ids: target_device_ids,
+            filesystem_identity,
+        })
+    })
+    .await
+    .ok()
+    .flatten()
+}
+
+pub(crate) async fn auto_replacement_targets_ready(targets: &[String]) -> bool {
+    auto_replacement_target_identities(targets).await.is_some()
+}
+
+pub(crate) async fn auto_replacement_target_identities(targets: &[String]) -> Option<Vec<ReplacementTargetIdentity>> {
+    let local_disk_map = local_disk_map_read().await;
+    let local_disks = local_disk_map
+        .values()
+        .flatten()
+        .filter(|disk| disk.endpoint().is_local)
+        .cloned()
+        .collect::<Vec<_>>();
+    drop(local_disk_map);
+
+    let mut identities = Vec::with_capacity(targets.len());
+    for target in targets {
+        let disk = local_disks.iter().find(|disk| disk.endpoint().to_string() == *target)?;
+        identities.push(auto_replacement_target_identity(disk, &local_disks).await?);
+    }
+    identities.sort_by(|left, right| left.endpoint.cmp(&right.endpoint));
+    identities.dedup_by(|left, right| left.endpoint == right.endpoint);
+    (identities.len() == targets.len()).then_some(identities)
+}
+
+#[cfg(target_os = "linux")]
+fn filesystem_identity(metadata: &fs::Metadata, canonical_path: &Path) -> Option<String> {
+    use std::os::unix::fs::MetadataExt as _;
+
+    let escaped_path = canonical_path.to_string_lossy().replace(' ', "\\040");
+    let mountinfo = fs::read_to_string("/proc/self/mountinfo").ok()?;
+    let mount_id = mountinfo.lines().find_map(|line| {
+        let mut fields = line.split_whitespace();
+        let mount_id = fields.next()?;
+        fields.next()?;
+        fields.next()?;
+        fields.next()?;
+        (fields.next()? == escaped_path).then_some(mount_id)
+    })?;
+    Some(format!("{mount_id}:{}:{}", metadata.dev(), metadata.ino()))
+}
+
+#[cfg(all(unix, not(target_os = "linux")))]
+fn filesystem_identity(metadata: &fs::Metadata, _canonical_path: &Path) -> Option<String> {
+    use std::os::unix::fs::MetadataExt as _;
+
+    Some(format!("{}:{}", metadata.dev(), metadata.ino()))
+}
+
+#[cfg(not(unix))]
+fn filesystem_identity(_metadata: &fs::Metadata, _canonical_path: &Path) -> Option<String> {
+    None
+}
+
+#[cfg(test)]
+mod tests {
+    use super::super::{DiskOption, new_disk};
+    use super::*;
+    use tempfile::TempDir;
+
+    #[tokio::test]
+    async fn runtime_environment_cannot_bypass_mount_admission() {
+        temp_env::async_with_vars(
+            [
+                ("RUSTFS_TEST_AUTO_REPLACEMENT_READINESS_BYPASS", Some("1")),
+                ("RUSTFS_E2E_AUTO_REPLACEMENT_READINESS_BYPASS", Some("1")),
+            ],
+            async {
+                let temp = TempDir::new().expect("temporary replacement root should be created");
+                let endpoint =
+                    Endpoint::try_from(temp.path().to_string_lossy().as_ref()).expect("replacement endpoint should parse");
+
+                let disk = new_disk(
+                    &endpoint,
+                    &DiskOption {
+                        cleanup: false,
+                        health_check: false,
+                    },
+                )
+                .await
+                .expect("temporary disk should initialize");
+                assert!(!auto_replacement_target_ready(&disk, std::slice::from_ref(&disk)).await);
+            },
+        )
+        .await;
+    }
+
+    #[cfg(target_os = "linux")]
+    mod linux_privileged_tests {
+        use super::*;
+        use std::error::Error;
+        use std::ffi::CString;
+        use std::os::unix::ffi::OsStrExt;
+        use std::path::Path;
+
+        const ENABLE_ENV: &str = "RUSTFS_PRIVILEGED_MOUNT_READINESS_TESTS";
+
+        struct MountGuard {
+            mounts: Vec<std::path::PathBuf>,
+        }
+
+        impl MountGuard {
+            fn new() -> Result<Self, Box<dyn Error + Send + Sync>> {
+                let rc = unsafe { libc::unshare(libc::CLONE_NEWNS) };
+                if rc != 0 {
+                    return Err(format!("unshare(CLONE_NEWNS) failed: {}", std::io::Error::last_os_error()).into());
+                }
+                make_mounts_private()?;
+                Ok(Self { mounts: Vec::new() })
+            }
+
+            fn mount_tmpfs(&mut self, target: &Path, label: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
+                mount_tmpfs(target, label)?;
+                self.mounts.push(target.to_path_buf());
+                Ok(())
+            }
+
+            fn mount_bind(&mut self, source: &Path, target: &Path) -> Result<(), Box<dyn Error + Send + Sync>> {
+                mount_bind(source, target)?;
+                self.mounts.push(target.to_path_buf());
+                Ok(())
+            }
+        }
+
+        impl Drop for MountGuard {
+            fn drop(&mut self) {
+                for mount in self.mounts.iter().rev() {
+                    if let Ok(target) = c_path(mount) {
+                        let _ = unsafe { libc::umount2(target.as_ptr(), libc::MNT_DETACH) };
+                    }
+                }
+            }
+        }
+
+        fn c_path(path: &Path) -> Result<CString, Box<dyn Error + Send + Sync>> {
+            Ok(CString::new(path.as_os_str().as_bytes())?)
+        }
+
+        fn make_mounts_private() -> Result<(), Box<dyn Error + Send + Sync>> {
+            let root = CString::new("/")?;
+            let rc = unsafe {
+                libc::mount(
+                    std::ptr::null(),
+                    root.as_ptr(),
+                    std::ptr::null(),
+                    (libc::MS_REC | libc::MS_PRIVATE) as libc::c_ulong,
+                    std::ptr::null(),
+                )
+            };
+            if rc != 0 {
+                return Err(format!("making the mount namespace private failed: {}", std::io::Error::last_os_error()).into());
+            }
+            Ok(())
+        }
+
+        fn mount_tmpfs(target: &Path, label: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
+            let source = CString::new(label)?;
+            let target = c_path(target)?;
+            let fstype = CString::new("tmpfs")?;
+            let data = CString::new("size=32m,mode=0700")?;
+            let rc = unsafe {
+                libc::mount(
+                    source.as_ptr(),
+                    target.as_ptr(),
+                    fstype.as_ptr(),
+                    (libc::MS_NOSUID | libc::MS_NODEV) as libc::c_ulong,
+                    data.as_ptr().cast(),
+                )
+            };
+            if rc != 0 {
+                return Err(format!("mount(tmpfs) failed: {}", std::io::Error::last_os_error()).into());
+            }
+            Ok(())
+        }
+
+        fn mount_bind(source: &Path, target: &Path) -> Result<(), Box<dyn Error + Send + Sync>> {
+            let source = c_path(source)?;
+            let target = c_path(target)?;
+            let rc = unsafe {
+                libc::mount(
+                    source.as_ptr(),
+                    target.as_ptr(),
+                    std::ptr::null(),
+                    libc::MS_BIND as libc::c_ulong,
+                    std::ptr::null(),
+                )
+            };
+            if rc != 0 {
+                return Err(format!("mount(MS_BIND) failed: {}", std::io::Error::last_os_error()).into());
+            }
+            Ok(())
+        }
+
+        fn privileged_enabled() -> Result<bool, Box<dyn Error + Send + Sync>> {
+            let enabled = std::env::var(ENABLE_ENV)
+                .ok()
+                .is_some_and(|value| matches!(value.as_str(), "1" | "true" | "TRUE" | "yes" | "YES"));
+            if !enabled {
+                return Ok(false);
+            }
+            if unsafe { libc::geteuid() } != 0 {
+                return Err(format!("{ENABLE_ENV}=1 requires root or CAP_SYS_ADMIN").into());
+            }
+            Ok(true)
+        }
+
+        fn run_privileged_mount_test<F, Fut>(test: F) -> Result<(), Box<dyn Error + Send + Sync>>
+        where
+            F: FnOnce(MountGuard) -> Fut + Send + 'static,
+            Fut: std::future::Future<Output = Result<(), Box<dyn Error + Send + Sync>>> + 'static,
+        {
+            if !privileged_enabled()? {
+                return Ok(());
+            }
+            std::thread::spawn(move || {
+                let guard = MountGuard::new()?;
+                let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build()?;
+                runtime.block_on(test(guard))
+            })
+            .join()
+            .map_err(|_| "privileged mount readiness test thread panicked")?
+        }
+
+        #[test]
+        #[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_MOUNT_READINESS_TESTS=1"]
+        fn auto_replacement_readiness_accepts_an_independent_mount() -> Result<(), Box<dyn Error + Send + Sync>> {
+            run_privileged_mount_test(|mut mounts| async move {
+                let temp = TempDir::new().expect("temporary replacement roots should be created");
+                let target = temp.path().join("target");
+                let sibling = temp.path().join("sibling");
+                std::fs::create_dir(&target).expect("target mountpoint should be created");
+                std::fs::create_dir(&sibling).expect("sibling mountpoint should be created");
+                mounts.mount_tmpfs(&target, "rustfs-readiness-target")?;
+                mounts.mount_tmpfs(&sibling, "rustfs-readiness-sibling")?;
+
+                let target_endpoint = Endpoint::try_from(target.to_string_lossy().as_ref())?;
+                let sibling_endpoint = Endpoint::try_from(sibling.to_string_lossy().as_ref())?;
+                let target_disk = new_disk(
+                    &target_endpoint,
+                    &DiskOption {
+                        cleanup: false,
+                        health_check: false,
+                    },
+                )
+                .await?;
+                let sibling_disk = new_disk(
+                    &sibling_endpoint,
+                    &DiskOption {
+                        cleanup: false,
+                        health_check: false,
+                    },
+                )
+                .await?;
+
+                let identity = auto_replacement_target_identity(&target_disk, &[target_disk.clone(), sibling_disk.clone()]).await;
+                assert!(
+                    identity.is_some(),
+                    "a separately mounted replacement target with no sibling device overlap must be admitted"
+                );
+                Ok(())
+            })
+        }
+
+        #[test]
+        #[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_MOUNT_READINESS_TESTS=1"]
+        fn auto_replacement_readiness_rejects_a_same_device_sibling_bind_mount() -> Result<(), Box<dyn Error + Send + Sync>> {
+            run_privileged_mount_test(|mut mounts| async move {
+                let temp = TempDir::new().expect("temporary replacement roots should be created");
+                let source = temp.path().join("source");
+                let target = temp.path().join("target");
+                let sibling = temp.path().join("sibling");
+                std::fs::create_dir(&source).expect("source mountpoint should be created");
+                std::fs::create_dir(&target).expect("target mountpoint should be created");
+                std::fs::create_dir(&sibling).expect("sibling mountpoint should be created");
+                mounts.mount_tmpfs(&source, "rustfs-readiness-shared-source")?;
+                mounts.mount_bind(&source, &target)?;
+                mounts.mount_bind(&source, &sibling)?;
+
+                let target_endpoint = Endpoint::try_from(target.to_string_lossy().as_ref())?;
+                let sibling_endpoint = Endpoint::try_from(sibling.to_string_lossy().as_ref())?;
+                let target_disk = new_disk(
+                    &target_endpoint,
+                    &DiskOption {
+                        cleanup: false,
+                        health_check: false,
+                    },
+                )
+                .await?;
+                let sibling_disk = new_disk(
+                    &sibling_endpoint,
+                    &DiskOption {
+                        cleanup: false,
+                        health_check: false,
+                    },
+                )
+                .await?;
+
+                assert!(
+                    auto_replacement_target_identity(&target_disk, &[target_disk.clone(), sibling_disk.clone()])
+                        .await
+                        .is_none(),
+                    "replacement readiness must reject a target sharing its physical device with a sibling endpoint"
+                );
+                Ok(())
+            })
+        }
+    }
+}

crates/heal/src/heal/storage.rs

@@ -26,7 +26,7 @@ use super::storage_api::storage::{
     BucketInfo, BucketOperations, DiskSetSelector, HealOperations as _, ListOperations as _, ObjectIO as _,
     ObjectOperations as _, StorageAdminApi,
 };
-use super::{DiskStore, ECStore, Endpoint, StorageError};
+use super::{DiskStore, ECStore, Endpoint, HealDiskExt as _, StorageError, resume::ReplacementTargetIdentity};
 pub use super::{HealObjectInfo, HealObjectOptions, HealPutObjReader};
 
 const LOG_COMPONENT_HEAL: &str = "heal";
@@ -37,6 +37,11 @@ const EVENT_HEAL_STORAGE_OBJECT_VERIFY: &str = "heal_storage_object_verify";
 const EVENT_HEAL_STORAGE_ADMIN_OP: &str = "heal_storage_admin_op";
 const EVENT_HEAL_STORAGE_REPAIR_OP: &str = "heal_storage_repair_op";
 
+pub enum ReplacementResumeDisk {
+    Fresh,
+    Existing(DiskStore),
+}
+
 pub(crate) fn next_heal_listing_token(
     bucket: &str,
     prefix: &str,
@@ -354,6 +359,42 @@ pub trait HealStorageAPI: Send + Sync {
     /// Heal format using ecstore
     async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)>;
 
+    /// Heal only the explicitly admitted replacement targets in one erasure set.
+    ///
+    /// The default is deliberately fail-closed so alternate storage
+    /// implementations cannot accidentally fall back to the global format path.
+    async fn heal_replacement_format(
+        &self,
+        _dry_run: bool,
+        _pool_index: usize,
+        _set_index: usize,
+        _targets: &[String],
+    ) -> Result<(HealResultItem, Option<Error>)> {
+        Err(Error::other("target-scoped replacement format is unsupported"))
+    }
+
+    /// Recheck admitted replacement targets immediately before destructive work.
+    async fn replacement_targets_ready(&self, _targets: &[String]) -> Result<bool> {
+        Ok(false)
+    }
+
+    /// Read target-specific physical evidence for one replacement version.
+    ///
+    /// This is only used by automatic replacement healing after the normal
+    /// transaction returns success. The conservative default prevents an
+    /// alternate backend from turning an unverified replacement into a
+    /// completed generation.
+    async fn replacement_targets_have_version(
+        &self,
+        _bucket: &str,
+        _object: &str,
+        _version_id: Option<&str>,
+        _opts: &HealOpts,
+        _targets: &[String],
+    ) -> Result<bool> {
+        Ok(false)
+    }
+
     /// List object versions for healing (returns all versions, may use significant memory for large buckets)
     ///
     /// WARNING: This method loads all object versions into memory at once. For buckets with many
@@ -390,8 +431,30 @@ pub trait HealStorageAPI: Send + Sync {
         self.list_objects_for_heal_page(bucket, prefix, continuation_token).await
     }
 
-    /// Get disk for resume functionality
+    /// Get disk for resume functionality.
     async fn get_disk_for_resume(&self, set_disk_id: &str) -> Result<DiskStore>;
+
+    /// Get a healthy non-target disk for durable replacement state.
+    async fn get_disk_for_resume_excluding(&self, _set_disk_id: &str, _excluded_targets: &[String]) -> Result<DiskStore> {
+        Err(Error::other("target-excluding resume disk selection is unsupported"))
+    }
+
+    /// Reopen the exact surviving disk that owns an existing replacement
+    /// intent. Falling back to another disk would create a second copy of the
+    /// same generation and split its progress.
+    async fn get_replacement_resume_disk(
+        &self,
+        _set_disk_id: &str,
+        _task_id: &str,
+        _excluded_targets: &[String],
+    ) -> Result<ReplacementResumeDisk> {
+        Err(Error::other("durable replacement resume selection is unsupported"))
+    }
+
+    /// Capture the mounted replacement instance before it is formatted.
+    async fn replacement_target_identities(&self, _targets: &[String]) -> Result<Vec<ReplacementTargetIdentity>> {
+        Err(Error::other("replacement target identity collection is unsupported"))
+    }
 }
 
 /// ECStore Heal storage layer implementation
@@ -1306,6 +1369,44 @@ impl HealStorageAPI for ECStoreHealStorage {
         }
     }
 
+    async fn heal_replacement_format(
+        &self,
+        dry_run: bool,
+        pool_index: usize,
+        set_index: usize,
+        targets: &[String],
+    ) -> Result<(HealResultItem, Option<Error>)> {
+        self.ecstore
+            .heal_replacement_format(dry_run, pool_index, set_index, targets)
+            .await
+            .map(|(result, error)| (result, error.map(Error::Storage)))
+            .map_err(Error::Storage)
+    }
+
+    async fn replacement_targets_ready(&self, targets: &[String]) -> Result<bool> {
+        Ok(super::replacement_readiness::auto_replacement_targets_ready(targets).await)
+    }
+
+    async fn replacement_targets_have_version(
+        &self,
+        bucket: &str,
+        object: &str,
+        version_id: Option<&str>,
+        opts: &HealOpts,
+        targets: &[String],
+    ) -> Result<bool> {
+        let pool_index = opts
+            .pool
+            .ok_or_else(|| Error::other("replacement target readback is missing pool scope"))?;
+        let set_index = opts
+            .set
+            .ok_or_else(|| Error::other("replacement target readback is missing set scope"))?;
+        self.ecstore
+            .replacement_targets_have_version(bucket, object, version_id.unwrap_or(""), pool_index, set_index, targets)
+            .await
+            .map_err(Error::Storage)
+    }
+
     async fn list_objects_for_heal(&self, bucket: &str, prefix: &str) -> Result<Vec<HealListItem>> {
         debug!(
             target: "rustfs::heal::storage",
@@ -1543,6 +1644,10 @@ impl HealStorageAPI for ECStoreHealStorage {
     }
 
     async fn get_disk_for_resume(&self, set_disk_id: &str) -> Result<DiskStore> {
+        self.get_disk_for_resume_excluding(set_disk_id, &[]).await
+    }
+
+    async fn get_disk_for_resume_excluding(&self, set_disk_id: &str, excluded_targets: &[String]) -> Result<DiskStore> {
         debug!(
             target: "rustfs::heal::storage",
             event = EVENT_HEAL_STORAGE_ADMIN_OP,
@@ -1564,8 +1669,18 @@ impl HealStorageAPI for ECStoreHealStorage {
                 message: format!("Failed to get disks for pool {pool_idx} set {set_idx}: {e}"),
             })?;
 
-        // Find the first available disk
-        if let Some(disk_store) = disks.into_iter().flatten().next() {
+        // The replacement target is unformatted before repair and must never
+        // host the intent that authorizes its own formatting.
+        for disk_store in disks.into_iter().flatten() {
+            if !disk_store.endpoint().is_local {
+                continue;
+            }
+            if excluded_targets.contains(&disk_store.endpoint().to_string()) {
+                continue;
+            }
+            if !matches!(disk_store.get_disk_id().await, Ok(Some(id)) if !id.is_nil()) {
+                continue;
+            }
             debug!(
                 target: "rustfs::heal::storage",
                 event = EVENT_HEAL_STORAGE_ADMIN_OP,
@@ -1584,6 +1699,43 @@ impl HealStorageAPI for ECStoreHealStorage {
             message: format!("No available disk found for set_disk_id: {set_disk_id}"),
         })
     }
+
+    async fn get_replacement_resume_disk(
+        &self,
+        set_disk_id: &str,
+        task_id: &str,
+        excluded_targets: &[String],
+    ) -> Result<ReplacementResumeDisk> {
+        let (pool_idx, set_idx) = crate::heal::utils::parse_set_disk_id(set_disk_id)?;
+        let disks = StorageAdminApi::disk_set_inventory(self.ecstore.as_ref(), DiskSetSelector::new(pool_idx, set_idx))
+            .await
+            .map_err(|e| Error::TaskExecutionFailed {
+                message: format!("Failed to get disks for pool {pool_idx} set {set_idx}: {e}"),
+            })?;
+        let mut existing = None;
+        for disk_store in disks.into_iter().flatten() {
+            if !disk_store.endpoint().is_local || excluded_targets.contains(&disk_store.endpoint().to_string()) {
+                continue;
+            }
+            if !matches!(disk_store.get_disk_id().await, Ok(Some(id)) if !id.is_nil()) {
+                continue;
+            }
+            if super::resume::ResumeManager::has_replacement_intent(&disk_store, task_id).await
+                && existing.replace(disk_store).is_some()
+            {
+                return Err(Error::TaskExecutionFailed {
+                    message: format!("Replacement resume intent is duplicated for set_disk_id: {set_disk_id}"),
+                });
+            }
+        }
+        Ok(existing.map_or(ReplacementResumeDisk::Fresh, ReplacementResumeDisk::Existing))
+    }
+
+    async fn replacement_target_identities(&self, targets: &[String]) -> Result<Vec<ReplacementTargetIdentity>> {
+        super::replacement_readiness::auto_replacement_target_identities(targets)
+            .await
+            .ok_or_else(|| Error::other("replacement target is not a stable mounted disk"))
+    }
 }
 
 #[cfg(test)]

crates/heal/src/heal/storage_api.rs

@@ -16,8 +16,9 @@ pub(crate) use rustfs_ecstore::api::data_usage::DATA_USAGE_CACHE_NAME as ECSTORE
 pub(crate) use rustfs_ecstore::api::disk::endpoint::Endpoint as EcstoreEndpoint;
 pub(crate) use rustfs_ecstore::api::disk::error::{DiskError as EcstoreDiskError, Result as EcstoreDiskResult};
 pub(crate) use rustfs_ecstore::api::disk::{
-    BUCKET_META_PREFIX as ECSTORE_BUCKET_META_PREFIX, Bytes as EcstoreDiskBytes, DeleteOptions as EcstoreDeleteOptions,
-    DiskAPI as EcstoreDiskAPI, DiskStore as EcstoreDiskStore, HEALING_MARKER_PATH as ECSTORE_HEALING_MARKER_PATH,
+    BUCKET_META_PREFIX as ECSTORE_BUCKET_META_PREFIX, Bytes as EcstoreDiskBytes,
+    ConditionalFileUpdate as EcstoreConditionalFileUpdate, DeleteOptions as EcstoreDeleteOptions, DiskAPI as EcstoreDiskAPI,
+    DiskStore as EcstoreDiskStore, HEALING_MARKER_PATH as ECSTORE_HEALING_MARKER_PATH,
     RUSTFS_META_BUCKET as ECSTORE_RUSTFS_META_BUCKET,
 };
 #[cfg(test)]
@@ -32,8 +33,9 @@ pub(crate) mod owner {
 
     pub(crate) use super::{
         ECSTORE_BUCKET_META_PREFIX, ECSTORE_DATA_USAGE_CACHE_NAME, ECSTORE_HEALING_MARKER_PATH, ECSTORE_RUSTFS_META_BUCKET,
-        EcstoreDeleteOptions, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError, EcstoreDiskResult, EcstoreDiskStore,
-        EcstoreEndpoint, EcstoreErrorType, EcstoreStorageError, EcstoreStore, ecstore_local_disk_map_read,
+        EcstoreConditionalFileUpdate, EcstoreDeleteOptions, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError,
+        EcstoreDiskResult, EcstoreDiskStore, EcstoreEndpoint, EcstoreErrorType, EcstoreStorageError, EcstoreStore,
+        ecstore_local_disk_map_read,
     };
 
     #[cfg(test)]

crates/heal/src/heal/task.rs

@@ -13,8 +13,12 @@
 // limitations under the License.
 
 use crate::heal::{
-    DiskError, EcstoreError, ErasureSetHealer,
+    DiskError, EcstoreError, ErasureSetHealer, HealDiskExt as _,
+    erasure_healer::target_outcomes_complete,
     progress::HealProgress,
+    resume::{
+        CheckpointManager, ReplacementPhase, ReplacementTargetIdentity, ResumeManager, replacement_target_identities_match,
+    },
     storage::{HealStorageAPI, next_heal_listing_token},
 };
 use crate::{Error, Result};
@@ -350,6 +354,9 @@ pub struct HealTask {
     pub retry_attempts: u32,
     /// Endpoints of the disks being rebuilt (see `HealRequest::heal_endpoints`).
     pub heal_endpoints: Vec<String>,
+    /// Durable resume anchor injected by the manager for an existing automatic
+    /// replacement generation.
+    replacement_resume_endpoint: Option<String>,
     /// Task status
     pub status: Arc<RwLock<HealTaskStatus>>,
     /// Progress tracking
@@ -376,6 +383,24 @@ pub struct HealTask {
 }
 
 impl HealTask {
+    async fn verify_replacement_identity_fence(
+        &self,
+        expected_identities: &[ReplacementTargetIdentity],
+        set_disk_id: &str,
+        stage: &str,
+    ) -> Result<()> {
+        let actual_identities = self
+            .await_with_control(self.storage.replacement_target_identities(&self.heal_endpoints))
+            .await?;
+        if replacement_target_identities_match(expected_identities, &actual_identities) {
+            return Ok(());
+        }
+
+        Err(Error::TaskExecutionFailed {
+            message: format!("Replacement target changed during {stage} for automatic heal {set_disk_id}"),
+        })
+    }
+
     pub fn from_request(request: HealRequest, storage: Arc<dyn HealStorageAPI>) -> Self {
         Self {
             id: request.id,
@@ -385,6 +410,7 @@ impl HealTask {
             source: request.source,
             retry_attempts: request.retry_attempts,
             heal_endpoints: request.heal_endpoints,
+            replacement_resume_endpoint: None,
             status: Arc::new(RwLock::new(HealTaskStatus::Pending)),
             progress: Arc::new(RwLock::new(HealProgress::new())),
             result_items: Arc::new(RwLock::new(Vec::new())),
@@ -416,6 +442,16 @@ impl HealTask {
         }
     }
 
+    pub(crate) fn from_replacement_recovery_request(
+        request: HealRequest,
+        storage: Arc<dyn HealStorageAPI>,
+        replacement_resume_endpoint: Option<String>,
+    ) -> Self {
+        let mut task = Self::from_request(request, storage);
+        task.replacement_resume_endpoint = replacement_resume_endpoint;
+        task
+    }
+
     pub fn metric_type_label(&self) -> &'static str {
         match &self.heal_type {
             HealType::Cluster => "cluster",
@@ -1033,7 +1069,8 @@ impl HealTask {
                     bucket,
                     object,
                     object_size = object_size,
-                    drives_healed = result.after.drives.len(),
+                    drives_healed = result.drives_healed(),
+                    drives_total = result.drives_reported(),
                     result = "ok",
                     "Heal object repaired"
                 );
@@ -1329,7 +1366,8 @@ impl HealTask {
                     subsystem = LOG_SUBSYSTEM_TASK,
                     task_id = %self.id,
                     bucket,
-                    drives_healed = result.after.drives.len(),
+                    drives_healed = result.drives_healed(),
+                    drives_total = result.drives_reported(),
                     recursive = self.options.recursive,
                     result = "ok",
                     "Heal bucket completed"
@@ -1773,7 +1811,8 @@ impl HealTask {
                     task_id = %self.id,
                     bucket,
                     object,
-                    drives_healed = result.after.drives.len(),
+                    drives_healed = result.drives_healed(),
+                    drives_total = result.drives_reported(),
                     result = "ok",
                     "Heal metadata repaired"
                 );
@@ -1904,7 +1943,8 @@ impl HealTask {
                     meta_path,
                     bucket,
                     object = %object,
-                    drives_healed = result.after.drives.len(),
+                    drives_healed = result.drives_healed(),
+                    drives_total = result.drives_reported(),
                     result = "ok",
                     "Heal MRF repaired"
                 );
@@ -2068,7 +2108,8 @@ impl HealTask {
                     bucket,
                     object,
                     object_size,
-                    drives_healed = result.after.drives.len(),
+                    drives_healed = result.drives_healed(),
+                    drives_total = result.drives_reported(),
                     result = "ok",
                     "Heal EC decode repaired"
                 );
@@ -2126,7 +2167,87 @@ impl HealTask {
             progress.update_progress(0, 4, 0, 0);
         }
 
-        let buckets = if buckets.is_empty() {
+        let is_auto_replacement = matches!(self.source, HealRequestSource::AutoHeal) && !self.heal_endpoints.is_empty();
+        let replacement_resume_disk = if is_auto_replacement {
+            let mut requested_targets = self.heal_endpoints.clone();
+            requested_targets.sort_unstable();
+            requested_targets.dedup();
+            let selection = self
+                .await_with_control(
+                    self.storage
+                        .get_replacement_resume_disk(&set_disk_id, &self.id, &self.heal_endpoints),
+                )
+                .await?;
+            let disk = match selection {
+                crate::heal::storage::ReplacementResumeDisk::Existing(disk) => {
+                    if let Some(anchor) = &self.replacement_resume_endpoint
+                        && disk.endpoint().to_string() != *anchor
+                    {
+                        return Err(Error::TaskExecutionFailed {
+                            message: format!("Replacement resume anchor changed for automatic heal {set_disk_id}"),
+                        });
+                    }
+                    Some(disk)
+                }
+                crate::heal::storage::ReplacementResumeDisk::Fresh => {
+                    if self.replacement_resume_endpoint.is_some() {
+                        return Err(Error::TaskExecutionFailed {
+                            message: format!("Replacement resume anchor is unavailable for automatic heal {set_disk_id}"),
+                        });
+                    }
+                    None
+                }
+            };
+            if let Some(disk) = disk.as_ref()
+                && ResumeManager::has_replacement_intent(disk, &self.id).await
+            {
+                let resume_manager = ResumeManager::load_replacement_intent(disk.clone(), &self.id).await?;
+                let state = resume_manager.get_state().await;
+                if state.completed
+                    && matches!(state.replacement_phase, ReplacementPhase::CleanupPending)
+                    && state.set_disk_id == set_disk_id
+                    && state.replacement_targets == requested_targets
+                    && state.replacement_generation.as_deref() == Some(self.id.as_str())
+                {
+                    resume_manager.ensure_replacement_completion_proof().await?;
+                    if CheckpointManager::has_checkpoint(disk, &self.id).await {
+                        CheckpointManager::load_from_disk(disk.clone(), &self.id)
+                            .await?
+                            .cleanup()
+                            .await?;
+                    }
+                    resume_manager.cleanup().await?;
+                    return Ok(());
+                }
+            }
+            disk
+        } else {
+            None
+        };
+
+        if is_auto_replacement
+            && !self
+                .await_with_control(self.storage.replacement_targets_ready(&self.heal_endpoints))
+                .await?
+        {
+            return Err(Error::TaskExecutionFailed {
+                message: format!("Replacement target is no longer ready for automatic heal {set_disk_id}"),
+            });
+        }
+
+        let replacement_resume_disk = if is_auto_replacement {
+            Some(match replacement_resume_disk {
+                Some(disk) => disk,
+                None => {
+                    self.await_with_control(self.storage.get_disk_for_resume_excluding(&set_disk_id, &self.heal_endpoints))
+                        .await?
+                }
+            })
+        } else {
+            None
+        };
+
+        let mut buckets = if buckets.is_empty() {
             debug!(
                 target: "rustfs::heal::task",
                 event = EVENT_HEAL_ERASURE_SET_STAGE,
@@ -2143,6 +2264,50 @@ impl HealTask {
             buckets
         };
 
+        // Persist automatic replacement intent on a surviving disk before the
+        // first target format write. A task retry keeps this id; a newly
+        // admitted blank replacement gets a fresh id and cannot reuse cursor
+        // progress from an older disk at the same endpoint.
+        let replacement_resume = if is_auto_replacement {
+            let identities = self
+                .await_with_control(self.storage.replacement_target_identities(&self.heal_endpoints))
+                .await?;
+            let disk = replacement_resume_disk.clone().ok_or_else(|| Error::TaskExecutionFailed {
+                message: format!("Replacement resume disk is missing for automatic heal {set_disk_id}"),
+            })?;
+            let manager = ResumeManager::new_replacement_intent(
+                disk.clone(),
+                self.id.clone(),
+                set_disk_id.clone(),
+                buckets.clone(),
+                self.heal_endpoints.clone(),
+                identities.clone(),
+            )
+            .await?;
+            buckets = manager.get_state().await.replacement_buckets;
+            Some((disk, manager, identities))
+        } else {
+            None
+        };
+
+        let healing_marker = format!("{set_disk_id}:{}", self.id);
+        if let Some((disk, resume_manager, _)) = replacement_resume.as_ref() {
+            let state = resume_manager.get_state().await;
+            if state.completed && matches!(state.replacement_phase, ReplacementPhase::Verified) {
+                resume_manager.ensure_replacement_completion_proof().await?;
+                super::clear_healing_markers_after_verified(&self.heal_endpoints, &healing_marker).await?;
+                resume_manager.mark_replacement_cleanup_pending().await?;
+                if CheckpointManager::has_checkpoint(disk, &self.id).await {
+                    CheckpointManager::load_from_disk(disk.clone(), &self.id)
+                        .await?
+                        .cleanup()
+                        .await?;
+                }
+                resume_manager.cleanup().await?;
+                return Ok(());
+            }
+        }
+
         // Step 1: Perform disk format heal using ecstore
         debug!(
             target: "rustfs::heal::task",
@@ -2154,7 +2319,32 @@ impl HealTask {
             stage = "heal_format",
             "Heal erasure set stage entered"
         );
-        let format_result = self.await_with_control(self.storage.heal_format(self.options.dry_run)).await;
+        if is_auto_replacement {
+            let Some((_, _, expected_identities)) = replacement_resume.as_ref() else {
+                return Err(Error::TaskExecutionFailed {
+                    message: format!("Replacement intent is missing for automatic heal {set_disk_id}"),
+                });
+            };
+            self.verify_replacement_identity_fence(expected_identities, &set_disk_id, "format")
+                .await?;
+        }
+        let format_result = if is_auto_replacement {
+            let pool_index = self.options.pool_index.ok_or_else(|| Error::TaskExecutionFailed {
+                message: format!("Missing pool scope for automatic replacement heal {set_disk_id}"),
+            })?;
+            let set_index = self.options.set_index.ok_or_else(|| Error::TaskExecutionFailed {
+                message: format!("Missing set scope for automatic replacement heal {set_disk_id}"),
+            })?;
+            self.await_with_control(self.storage.heal_replacement_format(
+                self.options.dry_run,
+                pool_index,
+                set_index,
+                &self.heal_endpoints,
+            ))
+            .await
+        } else {
+            self.await_with_control(self.storage.heal_format(self.options.dry_run)).await
+        };
 
         match format_result {
             Ok((result, error)) => {
@@ -2198,11 +2388,28 @@ impl HealTask {
                         subsystem = LOG_SUBSYSTEM_TASK,
                         task_id = %self.id,
                         set_disk_id,
-                        drives_healed = result.after.drives.len(),
+                        drives_healed = result.drives_healed(),
+                    drives_total = result.drives_reported(),
                         result = "format_ok",
                         "Heal erasure set format repaired"
                     );
                 }
+                if !self.options.dry_run && !target_outcomes_complete(&result, &self.heal_endpoints) {
+                    return Err(Error::TaskExecutionFailed {
+                        message: format!("Failed to verify formatted replacement targets for {set_disk_id}"),
+                    });
+                }
+                if let Some((_, replacement_resume, expected_identities)) = &replacement_resume {
+                    let identities = self
+                        .await_with_control(self.storage.replacement_target_identities(&self.heal_endpoints))
+                        .await?;
+                    if !replacement_target_identities_match(expected_identities, &identities) {
+                        return Err(Error::TaskExecutionFailed {
+                            message: format!("Replacement target changed after format for automatic heal {set_disk_id}"),
+                        });
+                    }
+                    replacement_resume.mark_replacement_rebuilding(identities).await?;
+                }
             }
             Err(Error::TaskCancelled) => return Err(Error::TaskCancelled),
             Err(Error::TaskTimeout) => return Err(Error::TaskTimeout),
@@ -2235,7 +2442,7 @@ impl HealTask {
 
         // The rebuilt disks are formatted now: mark them as healing so
         // DiskInfo.healing reflects the rebuild until it completes.
-        super::set_healing_markers(&self.heal_endpoints, &set_disk_id).await;
+        super::set_healing_markers(&self.heal_endpoints, &healing_marker).await?;
 
         // Step 2: Get disk for resume functionality
         debug!(
@@ -2248,9 +2455,14 @@ impl HealTask {
             stage = "resolve_resume_disk",
             "Heal erasure set stage entered"
         );
-        let disk = self
-            .await_with_control(self.storage.get_disk_for_resume(&set_disk_id))
-            .await?;
+        let replacement_target_identities = replacement_resume.as_ref().map(|(_, _, identities)| identities.clone());
+        let disk = match replacement_resume.as_ref() {
+            Some((disk, _, _)) => disk.clone(),
+            None => {
+                self.await_with_control(self.storage.get_disk_for_resume(&set_disk_id))
+                    .await?
+            }
+        };
 
         {
             let mut progress = self.progress.write().await;
@@ -2274,6 +2486,10 @@ impl HealTask {
         for bucket in buckets.iter() {
             // Check control flags before starting each bucket heal
             self.check_control_flags().await?;
+            if let Some(expected_identities) = replacement_target_identities.as_ref() {
+                self.verify_replacement_identity_fence(expected_identities, &set_disk_id, "bucket prepass")
+                    .await?;
+            }
             let heal_result = self
                 .await_with_control(self.storage.heal_bucket(bucket, &bucket_heal_opts))
                 .await;
@@ -2328,7 +2544,9 @@ impl HealTask {
             disk,
             heal_opts,
             self.source,
-        );
+        )
+        .with_replacement_targets(self.heal_endpoints.clone(), is_auto_replacement.then(|| self.id.clone()))
+        .with_replacement_identity_fence(replacement_target_identities.clone());
 
         {
             let mut progress = self.progress.write().await;
@@ -2352,9 +2570,27 @@ impl HealTask {
 
         // Keep the markers on failure: the resume state also persists, and the
         // next run of this set heal re-marks and eventually clears them.
-        if result.is_ok() {
-            super::clear_healing_markers(&self.heal_endpoints).await;
-        }
+        let result = match result {
+            Ok(()) => {
+                if let Some(expected_identities) = replacement_target_identities.as_ref() {
+                    self.verify_replacement_identity_fence(expected_identities, &set_disk_id, "marker completion")
+                        .await?;
+                }
+                super::clear_healing_markers_after_verified(&self.heal_endpoints, &healing_marker).await?;
+                if let Some((disk, resume_manager, _)) = replacement_resume.as_ref() {
+                    resume_manager.mark_replacement_cleanup_pending().await?;
+                    if CheckpointManager::has_checkpoint(disk, &self.id).await {
+                        CheckpointManager::load_from_disk(disk.clone(), &self.id)
+                            .await?
+                            .cleanup()
+                            .await?;
+                    }
+                    resume_manager.cleanup().await?;
+                }
+                Ok(())
+            }
+            Err(err) => Err(err),
+        };
 
         {
             let mut progress = self.progress.write().await;
@@ -2414,12 +2650,353 @@ mod tests {
     use super::super::{DiskOption, DiskStore, Endpoint, HealDiskExt as _, new_disk};
     use super::*;
     use crate::heal::storage::{DiskStatus, HealListItem, HealObjectInfo};
-    use rustfs_madmin::heal_commands::HealResultItem;
+    use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem, Infos};
     use std::collections::{HashMap, VecDeque};
     use std::sync::Mutex;
     use tempfile::TempDir;
 
     use super::super::storage_api::status::BucketInfo;
+
+    #[test]
+    fn format_result_requires_every_requested_target_to_be_ok() {
+        let result = HealResultItem {
+            after: Infos {
+                drives: vec![
+                    HealDriveInfo {
+                        endpoint: "disk-a".to_string(),
+                        state: "ok".to_string(),
+                        ..Default::default()
+                    },
+                    HealDriveInfo {
+                        endpoint: "disk-b".to_string(),
+                        state: "missing".to_string(),
+                        ..Default::default()
+                    },
+                ],
+            },
+            ..Default::default()
+        };
+
+        assert!(target_outcomes_complete(&result, &["disk-a".to_string()]));
+        assert!(!target_outcomes_complete(&result, &["disk-a".to_string(), "disk-b".to_string()]));
+        assert!(!target_outcomes_complete(&result, &["disk-c".to_string()]));
+    }
+
+    #[tokio::test]
+    async fn automatic_replacement_uses_target_scoped_format() {
+        let temp = TempDir::new().expect("temporary resume disk directory should be created");
+        let disk = make_resume_disk(&temp).await;
+        let storage = Arc::new(MockStorage {
+            replacement_targets_ready: Mutex::new(true),
+            resume_disk: Mutex::new(Some(disk)),
+            ..Default::default()
+        });
+        let mut request = HealRequest::new(
+            HealType::ErasureSet {
+                buckets: Vec::new(),
+                set_disk_id: "pool_0_set_0".to_string(),
+            },
+            HealOptions {
+                pool_index: Some(0),
+                set_index: Some(0),
+                ..Default::default()
+            },
+            HealPriority::Low,
+        );
+        request.source = HealRequestSource::AutoHeal;
+        request.heal_endpoints = vec!["replacement-a".to_string()];
+        let task = HealTask::from_request(request, storage.clone());
+
+        task.execute()
+            .await
+            .expect_err("the mock has no local replacement marker target");
+
+        assert_eq!(
+            *storage.global_format_calls.lock().unwrap(),
+            0,
+            "automatic replacement must not call global format"
+        );
+        assert_eq!(
+            storage.replacement_format_calls.lock().unwrap().as_slice(),
+            &[(0, 0, vec!["replacement-a".to_string()])],
+            "automatic replacement must pass the exact pool, set, and target"
+        );
+    }
+
+    #[tokio::test]
+    async fn automatic_replacement_persists_intent_before_format() {
+        let storage = Arc::new(MockStorage {
+            replacement_targets_ready: Mutex::new(true),
+            ..Default::default()
+        });
+        let mut request = HealRequest::new(
+            HealType::ErasureSet {
+                buckets: vec!["bucket-a".to_string()],
+                set_disk_id: "pool_0_set_0".to_string(),
+            },
+            HealOptions {
+                pool_index: Some(0),
+                set_index: Some(0),
+                ..HealOptions::default()
+            },
+            HealPriority::Low,
+        );
+        request.source = HealRequestSource::AutoHeal;
+        request.heal_endpoints = vec!["replacement-a".to_string()];
+
+        HealTask::from_request(request, storage.clone())
+            .execute()
+            .await
+            .expect_err("intent persistence needs a healthy non-target disk");
+
+        assert!(
+            storage.replacement_format_calls.lock().unwrap().is_empty(),
+            "format must not start before the durable replacement intent exists"
+        );
+    }
+
+    #[tokio::test]
+    async fn recovered_replacement_never_uses_a_fresh_resume_disk() {
+        let storage = Arc::new(MockStorage {
+            replacement_targets_ready: Mutex::new(true),
+            ..Default::default()
+        });
+        let mut request = HealRequest::new(
+            HealType::ErasureSet {
+                buckets: vec!["bucket-a".to_string()],
+                set_disk_id: "pool_0_set_0".to_string(),
+            },
+            HealOptions {
+                pool_index: Some(0),
+                set_index: Some(0),
+                ..HealOptions::default()
+            },
+            HealPriority::Low,
+        );
+        request.source = HealRequestSource::AutoHeal;
+        request.heal_endpoints = vec!["replacement-a".to_string()];
+
+        let error = HealTask::from_replacement_recovery_request(request, storage.clone(), Some("survivor-a".to_string()))
+            .execute()
+            .await
+            .expect_err("a durable recovery must not fall back to another resume disk");
+
+        assert!(error.to_string().contains("resume anchor is unavailable"));
+        assert!(
+            storage.replacement_format_calls.lock().unwrap().is_empty(),
+            "an unavailable durable anchor must block formatting before any write"
+        );
+        assert!(!*storage.listed.lock().unwrap(), "an unavailable durable anchor must not list buckets");
+    }
+
+    #[tokio::test]
+    async fn automatic_replacement_rejects_a_new_identity_after_format() {
+        let temp = TempDir::new().expect("temporary resume disk directory should be created");
+        let disk = make_resume_disk(&temp).await;
+        let first_identity = replacement_identity("replacement-a", "device-a", "filesystem-a");
+        let second_identity = replacement_identity("replacement-a", "device-b", "filesystem-b");
+        let storage = Arc::new(MockStorage {
+            replacement_targets_ready: Mutex::new(true),
+            replacement_target_identity_sequences: Mutex::new(VecDeque::from([
+                vec![first_identity.clone()],
+                vec![first_identity.clone()],
+                vec![second_identity],
+            ])),
+            resume_disk: Mutex::new(Some(disk.clone())),
+            ..Default::default()
+        });
+        let mut request = HealRequest::new(
+            HealType::ErasureSet {
+                buckets: vec!["bucket-a".to_string()],
+                set_disk_id: "pool_0_set_0".to_string(),
+            },
+            HealOptions {
+                pool_index: Some(0),
+                set_index: Some(0),
+                ..HealOptions::default()
+            },
+            HealPriority::Low,
+        );
+        request.source = HealRequestSource::AutoHeal;
+        request.heal_endpoints = vec!["replacement-a".to_string()];
+        let task = HealTask::from_request(request, storage.clone());
+
+        let error = task
+            .execute()
+            .await
+            .expect_err("a remounted target after format must fail closed");
+
+        assert!(error.to_string().contains("changed after format"));
+        assert_eq!(storage.replacement_format_calls.lock().unwrap().len(), 1);
+        assert!(storage.bucket_heal_calls.lock().unwrap().is_empty());
+        assert!(storage.heal_object_calls.lock().unwrap().is_empty());
+
+        let state = ResumeManager::load_replacement_intent(disk, &task.id)
+            .await
+            .expect("durable replacement intent should remain available")
+            .get_state()
+            .await;
+        assert_eq!(state.replacement_phase, crate::heal::resume::ReplacementPhase::Intent);
+        assert_eq!(state.replacement_target_identities, vec![first_identity]);
+    }
+
+    #[tokio::test]
+    async fn automatic_replacement_reuses_an_existing_non_target_resume_anchor() {
+        let temp = TempDir::new().expect("temporary resume disk directory should be created");
+        let anchor = make_resume_disk(&temp).await;
+        let task_id = crate::heal::resume::ResumeUtils::generate_task_id();
+        let identity = ReplacementTargetIdentity {
+            endpoint: "replacement-a".to_string(),
+            canonical_path: "/replacement/replacement-a".to_string(),
+            physical_device_ids: vec!["replacement-a".to_string()],
+            filesystem_identity: "identity-replacement-a".to_string(),
+        };
+        ResumeManager::new_replacement_intent(
+            anchor.clone(),
+            task_id.clone(),
+            "pool_0_set_0".to_string(),
+            vec!["bucket-a".to_string()],
+            vec!["replacement-a".to_string()],
+            vec![identity],
+        )
+        .await
+        .expect("existing intent should be stored on the non-target anchor");
+        let storage = Arc::new(MockStorage {
+            replacement_targets_ready: Mutex::new(true),
+            replacement_resume_disk: Mutex::new(Some(anchor.clone())),
+            ..Default::default()
+        });
+        let mut request = HealRequest::new(
+            HealType::ErasureSet {
+                buckets: vec!["bucket-a".to_string()],
+                set_disk_id: "pool_0_set_0".to_string(),
+            },
+            HealOptions {
+                pool_index: Some(0),
+                set_index: Some(0),
+                ..HealOptions::default()
+            },
+            HealPriority::Low,
+        );
+        request.id = task_id.clone();
+        request.source = HealRequestSource::AutoHeal;
+        request.heal_endpoints = vec!["replacement-a".to_string()];
+
+        HealTask::from_request(request, storage.clone())
+            .execute()
+            .await
+            .expect_err("the test has no mounted marker target after format");
+
+        assert_eq!(
+            storage.replacement_format_calls.lock().unwrap().len(),
+            1,
+            "an existing non-target anchor must be reused instead of falling back to a fresh anchor"
+        );
+        assert!(
+            storage.resume_disk.lock().unwrap().is_none(),
+            "the fresh resume-anchor fallback must remain unused"
+        );
+        let state = ResumeManager::load_replacement_intent(anchor, &task_id)
+            .await
+            .expect("the existing non-target anchor should retain the generation")
+            .get_state()
+            .await;
+        assert_eq!(state.replacement_phase, ReplacementPhase::Rebuilding);
+    }
+
+    #[tokio::test]
+    async fn automatic_replacement_defers_before_bucket_listing_when_target_is_unready() {
+        let storage = Arc::new(MockStorage::default());
+        let mut request = HealRequest::new(
+            HealType::ErasureSet {
+                buckets: Vec::new(),
+                set_disk_id: "pool_0_set_0".to_string(),
+            },
+            HealOptions {
+                pool_index: Some(0),
+                set_index: Some(0),
+                ..Default::default()
+            },
+            HealPriority::Low,
+        );
+        request.source = HealRequestSource::AutoHeal;
+        request.heal_endpoints = vec!["replacement-a".to_string()];
+
+        HealTask::from_request(request, storage.clone())
+            .execute()
+            .await
+            .expect_err("an unsafe replacement must defer before any scan work");
+
+        assert!(!*storage.listed.lock().unwrap(), "unsafe targets must not list buckets");
+        assert!(
+            storage.replacement_format_calls.lock().unwrap().is_empty(),
+            "unsafe targets must not format"
+        );
+    }
+
+    #[tokio::test]
+    async fn cleanup_pending_recovery_skips_target_readiness_and_format() {
+        let temp = TempDir::new().expect("temporary resume disk directory should be created");
+        let anchor = make_resume_disk(&temp).await;
+        let task_id = crate::heal::resume::ResumeUtils::generate_task_id();
+        let identity = replacement_identity("replacement-a", "device-a", "filesystem-a");
+        let resume_manager = ResumeManager::new_replacement_intent(
+            anchor.clone(),
+            task_id.clone(),
+            "pool_0_set_0".to_string(),
+            vec!["bucket-a".to_string()],
+            vec!["replacement-a".to_string()],
+            vec![identity],
+        )
+        .await
+        .expect("terminal replacement state should persist on the survivor anchor");
+        resume_manager
+            .mark_replacement_completed_and_verified()
+            .await
+            .expect("terminal replacement proof should persist before cleanup");
+        resume_manager
+            .mark_replacement_cleanup_pending()
+            .await
+            .expect("failed cleanup must retain a cleanup-pending state");
+
+        let storage = Arc::new(MockStorage {
+            replacement_resume_disk: Mutex::new(Some(anchor.clone())),
+            ..Default::default()
+        });
+        let mut request = HealRequest::new(
+            HealType::ErasureSet {
+                buckets: vec!["bucket-a".to_string()],
+                set_disk_id: "pool_0_set_0".to_string(),
+            },
+            HealOptions {
+                pool_index: Some(0),
+                set_index: Some(0),
+                ..HealOptions::default()
+            },
+            HealPriority::Low,
+        );
+        request.id = task_id.clone();
+        request.source = HealRequestSource::AutoHeal;
+        request.heal_endpoints = vec!["replacement-a".to_string()];
+
+        HealTask::from_replacement_recovery_request(request, storage.clone(), Some(anchor.endpoint().to_string()))
+            .execute()
+            .await
+            .expect("cleanup-pending recovery must not require a mounted replacement target");
+
+        assert!(
+            !ResumeManager::has_resume_state(&anchor, &task_id).await,
+            "terminal cleanup must remove the retained resume state"
+        );
+        assert_eq!(*storage.global_format_calls.lock().unwrap(), 0);
+        assert!(
+            storage.replacement_format_calls.lock().unwrap().is_empty(),
+            "terminal cleanup must not format replacement targets"
+        );
+        assert!(storage.bucket_heal_calls.lock().unwrap().is_empty());
+        assert!(!*storage.listed.lock().unwrap());
+    }
+
     #[derive(Default)]
     struct MockStorage {
         listed: Mutex<bool>,
@@ -2434,6 +3011,10 @@ mod tests {
         heal_object_outcomes: Mutex<HashMap<String, VecDeque<MockHealObjectOutcome>>>,
         deleted_objects: Mutex<Vec<String>>,
         format_no_heal_required: Mutex<bool>,
+        global_format_calls: Mutex<u32>,
+        replacement_format_calls: Mutex<Vec<(usize, usize, Vec<String>)>>,
+        replacement_targets_ready: Mutex<bool>,
+        replacement_target_identity_sequences: Mutex<VecDeque<Vec<crate::heal::resume::ReplacementTargetIdentity>>>,
         listed_prefixes: Mutex<Vec<String>>,
         truncate_without_token: Mutex<bool>,
         include_object_dir_candidate: Mutex<bool>,
@@ -2442,6 +3023,7 @@ mod tests {
         bucket_heal_calls: Mutex<Vec<String>>,
         block_heal_object: Mutex<bool>,
         resume_disk: Mutex<Option<DiskStore>>,
+        replacement_resume_disk: Mutex<Option<DiskStore>>,
     }
 
     #[test]
@@ -2513,10 +3095,24 @@ mod tests {
         }
     }
 
+    fn replacement_identity(
+        endpoint: &str,
+        physical_device_id: &str,
+        filesystem_identity: &str,
+    ) -> crate::heal::resume::ReplacementTargetIdentity {
+        crate::heal::resume::ReplacementTargetIdentity {
+            endpoint: endpoint.to_string(),
+            canonical_path: format!("/replacement/{endpoint}"),
+            physical_device_ids: vec![physical_device_id.to_string()],
+            filesystem_identity: filesystem_identity.to_string(),
+        }
+    }
+
     enum MockHealObjectOutcome {
         OkWithOtherError(&'static str),
         ErrOther(&'static str),
         RetryableReadQuorum,
+        RetryableSlowDown,
         PermanentOther(&'static str),
     }
 
@@ -2650,6 +3246,9 @@ mod tests {
                         bucket.to_string(),
                         object.to_string(),
                     ))),
+                    MockHealObjectOutcome::RetryableSlowDown => {
+                        Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::SlowDown))))
+                    }
                     MockHealObjectOutcome::PermanentOther(message) => Err(Error::other(message)),
                     MockHealObjectOutcome::OkWithOtherError(message) => {
                         Ok((HealResultItem::default(), Some(Error::other(message))))
@@ -2669,6 +3268,9 @@ mod tests {
                         bucket.to_string(),
                         object.to_string(),
                     ))),
+                    MockHealObjectOutcome::RetryableSlowDown => {
+                        Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::SlowDown))))
+                    }
                 };
             }
             if bucket == RUSTFS_META_BUCKET && object == format!("{BUCKET_META_PREFIX}/{DATA_USAGE_CACHE_NAME}") {
@@ -2708,6 +3310,7 @@ mod tests {
         }
 
         async fn heal_format(&self, _dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
+            *self.global_format_calls.lock().unwrap() += 1;
             let no_heal_required = *self.format_no_heal_required.lock().unwrap();
             if no_heal_required {
                 Ok((HealResultItem::default(), Some(Error::Storage(EcstoreError::NoHealRequired))))
@@ -2716,6 +3319,39 @@ mod tests {
             }
         }
 
+        async fn heal_replacement_format(
+            &self,
+            _dry_run: bool,
+            pool_index: usize,
+            set_index: usize,
+            targets: &[String],
+        ) -> Result<(HealResultItem, Option<Error>)> {
+            self.replacement_format_calls
+                .lock()
+                .unwrap()
+                .push((pool_index, set_index, targets.to_vec()));
+            Ok((
+                HealResultItem {
+                    after: Infos {
+                        drives: targets
+                            .iter()
+                            .map(|endpoint| HealDriveInfo {
+                                endpoint: endpoint.clone(),
+                                state: "ok".to_string(),
+                                ..Default::default()
+                            })
+                            .collect(),
+                    },
+                    ..Default::default()
+                },
+                None,
+            ))
+        }
+
+        async fn replacement_targets_ready(&self, _targets: &[String]) -> Result<bool> {
+            Ok(*self.replacement_targets_ready.lock().unwrap())
+        }
+
         async fn list_objects_for_heal(&self, _bucket: &str, _prefix: &str) -> Result<Vec<HealListItem>> {
             Ok(vec![heal_item("object-a"), heal_item("object-b")])
         }
@@ -2759,6 +3395,79 @@ mod tests {
                 .clone()
                 .ok_or_else(|| Error::other("not implemented in tests"))
         }
+
+        async fn get_disk_for_resume_excluding(&self, set_disk_id: &str, _excluded_targets: &[String]) -> Result<DiskStore> {
+            self.get_disk_for_resume(set_disk_id).await
+        }
+
+        async fn get_replacement_resume_disk(
+            &self,
+            _set_disk_id: &str,
+            _task_id: &str,
+            _excluded_targets: &[String],
+        ) -> Result<crate::heal::storage::ReplacementResumeDisk> {
+            if let Some(disk) = self.replacement_resume_disk.lock().unwrap().clone() {
+                return Ok(crate::heal::storage::ReplacementResumeDisk::Existing(disk));
+            }
+            Ok(crate::heal::storage::ReplacementResumeDisk::Fresh)
+        }
+
+        async fn replacement_target_identities(
+            &self,
+            targets: &[String],
+        ) -> Result<Vec<crate::heal::resume::ReplacementTargetIdentity>> {
+            if !*self.replacement_targets_ready.lock().unwrap() {
+                return Err(Error::other("replacement target is not ready"));
+            }
+            if let Some(identities) = self.replacement_target_identity_sequences.lock().unwrap().pop_front() {
+                return Ok(identities);
+            }
+            Ok(targets
+                .iter()
+                .map(|endpoint| crate::heal::resume::ReplacementTargetIdentity {
+                    endpoint: endpoint.clone(),
+                    canonical_path: format!("/replacement/{endpoint}"),
+                    physical_device_ids: vec![endpoint.clone()],
+                    filesystem_identity: format!("identity-{endpoint}"),
+                })
+                .collect())
+        }
+    }
+
+    #[tokio::test]
+    async fn scoped_object_heal_slowdown_is_not_treated_as_deleted() {
+        let storage = Arc::new(MockStorage {
+            object_exists: Mutex::new(Some(true)),
+            heal_object_outcome: Mutex::new(Some(MockHealObjectOutcome::RetryableSlowDown)),
+            ..Default::default()
+        });
+        let task = HealTask::from_request(
+            HealRequest::new(
+                HealType::Object {
+                    bucket: "bucket".to_string(),
+                    object: "object".to_string(),
+                    version_id: None,
+                },
+                HealOptions {
+                    pool_index: Some(0),
+                    set_index: Some(1),
+                    ..Default::default()
+                },
+                HealPriority::Normal,
+            ),
+            storage.clone(),
+        );
+
+        let err = task.execute().await.expect_err("SlowDown must fail the current heal attempt");
+
+        assert!(matches!(err, Error::Storage(EcstoreError::SlowDown)));
+        assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. }));
+        let opts = storage
+            .object_heal_opts
+            .lock()
+            .expect("heal options lock should be available");
+        assert_eq!(opts[0].pool, Some(0));
+        assert_eq!(opts[0].set, Some(1));
     }
 
     async fn make_resume_disk(temp: &TempDir) -> DiskStore {

crates/heal/src/lib.rs

@@ -18,9 +18,12 @@ pub mod heal;
 pub use error::{Error, Result};
 pub use heal::{
     HealManager, HealOperationsSnapshot, HealOptions, HealPriority, HealPriorityCounts, HealRequest, HealSourceCounts, HealType,
-    channel::HealChannelProcessor, progress::HealProgress,
+    channel::HealChannelProcessor,
+    progress::HealProgress,
+    resume::{ReplacementRecoveryRecord, ReplacementRecoveryState, ResumeUtils},
 };
 use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
+use std::collections::BTreeMap;
 use std::future::Future;
 use std::sync::atomic::{AtomicU64, Ordering};
 use std::sync::{Arc, OnceLock};
@@ -73,6 +76,17 @@ static GLOBAL_HEAL_RUNTIME_INIT: Mutex<()> = Mutex::const_new(());
 static GLOBAL_HEAL_ACTIVE_TASKS: AtomicU64 = AtomicU64::new(0);
 static GLOBAL_HEAL_QUEUE_LENGTH: AtomicU64 = AtomicU64::new(0);
 
+/// Local view of durable replacement recovery state. `definitive` only covers
+/// the local survivor-disk records; a distributed caller must additionally
+/// establish that every peer returned a compatible snapshot.
+#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct ReplacementRecoverySnapshot {
+    pub records: Vec<ReplacementRecoveryRecord>,
+    pub definitive: bool,
+    pub reason: Option<String>,
+}
+
 #[cfg(test)]
 #[derive(Default)]
 struct HealRuntimeInitTestHook {
@@ -243,6 +257,81 @@ pub async fn current_heal_progress_snapshot() -> Option<HealProgress> {
     }
 }
 
+/// Read all local survivor-disk replacement records without conflating an I/O
+/// failure or conflicting copies with successful completion.
+pub async fn current_replacement_recovery_snapshot() -> ReplacementRecoverySnapshot {
+    if !heal_runtime_initialized() {
+        return ReplacementRecoverySnapshot {
+            records: Vec::new(),
+            definitive: false,
+            reason: Some("heal runtime is not initialized".to_string()),
+        };
+    }
+
+    let disks = {
+        let local_disk_map = heal::local_disk_map_read().await;
+        local_disk_map.values().flatten().cloned().collect::<Vec<_>>()
+    };
+    if disks.is_empty() {
+        return ReplacementRecoverySnapshot {
+            records: Vec::new(),
+            definitive: false,
+            reason: Some("no local survivor disks are available".to_string()),
+        };
+    }
+
+    let mut records = BTreeMap::<String, ReplacementRecoveryRecord>::new();
+    let mut reason = None;
+    for disk in disks {
+        match ResumeUtils::get_replacement_recovery_records(&disk).await {
+            Ok(disk_records) => {
+                for record in disk_records {
+                    let task_id = record.task_id.clone();
+                    if matches!(record.state, ReplacementRecoveryState::Unknown) {
+                        reason.get_or_insert_with(|| "invalid durable replacement record".to_string());
+                    }
+                    match records.entry(task_id.clone()) {
+                        std::collections::btree_map::Entry::Vacant(entry) => {
+                            entry.insert(record);
+                        }
+                        std::collections::btree_map::Entry::Occupied(entry) if entry.get() == &record => {}
+                        std::collections::btree_map::Entry::Occupied(entry)
+                            if matches!(entry.get().state, ReplacementRecoveryState::CleanupPending)
+                                && matches!(record.state, ReplacementRecoveryState::Completed) => {}
+                        std::collections::btree_map::Entry::Occupied(mut entry)
+                            if matches!(entry.get().state, ReplacementRecoveryState::Completed)
+                                && matches!(record.state, ReplacementRecoveryState::CleanupPending) =>
+                        {
+                            entry.insert(record);
+                        }
+                        std::collections::btree_map::Entry::Occupied(mut entry) => {
+                            entry.insert(ReplacementRecoveryRecord {
+                                task_id,
+                                state: ReplacementRecoveryState::Unknown,
+                                generation: None,
+                                set_disk_id: None,
+                                target_slots: Vec::new(),
+                                reason: Some("conflicting durable replacement records across survivor disks".to_string()),
+                                verified_at: None,
+                            });
+                            reason.get_or_insert_with(|| "conflicting durable replacement records".to_string());
+                        }
+                    }
+                }
+            }
+            Err(error) => {
+                reason.get_or_insert_with(|| format!("failed to read local replacement recovery records: {error}"));
+            }
+        }
+    }
+
+    ReplacementRecoverySnapshot {
+        records: records.into_values().collect(),
+        definitive: reason.is_none(),
+        reason,
+    }
+}
+
 fn usize_to_u64_saturated(value: usize) -> u64 {
     u64::try_from(value).unwrap_or(u64::MAX)
 }

crates/io-metrics/src/internode_metrics.rs

@@ -22,6 +22,7 @@ use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
 
 pub const INTERNODE_OPERATION_READ_FILE_STREAM: &str = "read_file_stream";
 pub const INTERNODE_OPERATION_PUT_FILE_STREAM: &str = "put_file_stream";
+pub const INTERNODE_OPERATION_PUT_FILE_CAPABILITY: &str = "put_file_capability";
 pub const INTERNODE_OPERATION_WALK_DIR: &str = "walk_dir";
 pub const INTERNODE_OPERATION_NS_SCANNER: &str = "ns_scanner";
 pub const INTERNODE_OPERATION_GRPC_READ_ALL: &str = "grpc_read_all";
@@ -999,6 +1000,7 @@ mod tests {
     fn operation_metric_names_and_low_cardinality_values_are_stable() {
         assert_eq!(INTERNODE_OPERATION_READ_FILE_STREAM, "read_file_stream");
         assert_eq!(INTERNODE_OPERATION_PUT_FILE_STREAM, "put_file_stream");
+        assert_eq!(INTERNODE_OPERATION_PUT_FILE_CAPABILITY, "put_file_capability");
         assert_eq!(INTERNODE_OPERATION_WALK_DIR, "walk_dir");
         assert_eq!(INTERNODE_OPERATION_GRPC_READ_ALL, "grpc_read_all");
         assert_eq!(INTERNODE_OPERATION_GRPC_WRITE_ALL, "grpc_write_all");

crates/madmin/src/heal_commands.rs

@@ -29,6 +29,11 @@ pub struct Infos {
     pub drives: Vec<HealDriveInfo>,
 }
 
+/// String form of `DriveState::Ok` as recorded in `HealDriveInfo::state`
+/// (this crate stores drive states as strings and does not depend on the
+/// enum's crate).
+const DRIVE_STATE_OK: &str = "ok";
+
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct HealResultItem {
     #[serde(rename = "resultId")]
@@ -58,3 +63,81 @@ pub struct HealResultItem {
     #[serde(rename = "objectSize")]
     pub object_size: usize,
 }
+
+impl HealResultItem {
+    /// Number of drives this heal repaired: pairwise `before`/`after` state
+    /// transitions to ok (issue #5863). `None` when the result carries no
+    /// aligned drive data (e.g. remote bucket results) — not the same as zero.
+    pub fn drives_healed(&self) -> Option<usize> {
+        if self.after.drives.is_empty() || self.before.drives.len() != self.after.drives.len() {
+            return None;
+        }
+        Some(
+            self.before
+                .drives
+                .iter()
+                .zip(&self.after.drives)
+                .filter(|(before, after)| before.state != after.state && after.state == DRIVE_STATE_OK)
+                .count(),
+        )
+    }
+
+    /// Drives consulted, or `None` when the result has no drive entries.
+    pub fn drives_reported(&self) -> Option<usize> {
+        if self.after.drives.is_empty() {
+            None
+        } else {
+            Some(self.after.drives.len())
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn drive(state: &str) -> HealDriveInfo {
+        HealDriveInfo {
+            uuid: String::new(),
+            endpoint: String::new(),
+            state: state.to_string(),
+        }
+    }
+
+    #[test]
+    fn drives_healed_counts_transitions_to_ok_not_consulted_drives() {
+        let mut item = HealResultItem::default();
+        item.before.drives = vec![drive("ok"), drive("missing"), drive("corrupt"), drive("offline")];
+        item.after.drives = vec![drive("ok"), drive("ok"), drive("ok"), drive("offline")];
+        // 4 drives consulted, 2 repaired (missing->ok, corrupt->ok); the
+        // already-ok drive and the still-offline drive are not repairs.
+        assert_eq!(item.drives_healed(), Some(2));
+        assert_eq!(item.drives_reported(), Some(4));
+
+        let mut noop = HealResultItem::default();
+        noop.before.drives = vec![drive("ok"); 12];
+        noop.after.drives = vec![drive("ok"); 12];
+        assert_eq!(noop.drives_healed(), Some(0));
+    }
+
+    #[test]
+    fn drives_healed_reports_unknown_not_zero_without_drive_data() {
+        // Empty successful remote result (RemotePeerS3Client::heal_bucket
+        // default) is "unknown", never a definitive zero.
+        let remote = HealResultItem::default();
+        assert_eq!(remote.drives_healed(), None);
+        assert_eq!(remote.drives_reported(), None);
+
+        // A local missing -> ok result keeps its real count.
+        let mut local = HealResultItem::default();
+        local.before.drives = vec![drive("ok"), drive("missing")];
+        local.after.drives = vec![drive("ok"), drive("ok")];
+        assert_eq!(local.drives_healed(), Some(1));
+
+        // Misaligned arrays cannot be paired: also unknown.
+        let mut misaligned = HealResultItem::default();
+        misaligned.before.drives = vec![drive("missing")];
+        misaligned.after.drives = vec![drive("ok"), drive("ok")];
+        assert_eq!(misaligned.drives_healed(), None);
+    }
+}

crates/policy/src/policy/function/key_name.rs

@@ -198,6 +198,9 @@ pub enum S3KeyName {
     #[strum(serialize = "s3:object-lock-retain-until-date")]
     S3ObjectLockRetainUntilDate,
 
+    #[strum(serialize = "s3:object-lock-mode")]
+    S3ObjectLockMode,
+
     #[strum(serialize = "s3:max-keys")]
     S3MaxKeys,
 
@@ -385,6 +388,7 @@ mod tests {
     #[test_case("s3:x-amz-copy-source", KeyName::S3(S3KeyName::S3XAmzCopySource))]
     #[test_case("s3:VersionId", KeyName::S3(S3KeyName::S3VersionId) ; "aws_version_id")]
     #[test_case("s3:versionid", KeyName::S3(S3KeyName::S3VersionId) ; "minio_version_id")]
+    #[test_case("s3:object-lock-mode", KeyName::S3(S3KeyName::S3ObjectLockMode))]
     #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))]
     #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))]
     #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))]
@@ -407,6 +411,7 @@ mod tests {
     #[test_case("s3:x-amz-copy-source", KeyName::S3(S3KeyName::S3XAmzCopySource))]
     #[test_case("s3:VersionId", KeyName::S3(S3KeyName::S3VersionId) ; "aws_version_id")]
     #[test_case("s3:versionid", KeyName::S3(S3KeyName::S3VersionId) ; "minio_version_id")]
+    #[test_case("s3:object-lock-mode", KeyName::S3(S3KeyName::S3ObjectLockMode))]
     #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))]
     #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))]
     #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))]
@@ -425,6 +430,7 @@ mod tests {
 
     #[test_case("s3:x-amz-copy-source", KeyName::S3(S3KeyName::S3XAmzCopySource))]
     #[test_case("s3:versionid", KeyName::S3(S3KeyName::S3VersionId))]
+    #[test_case("s3:object-lock-mode", KeyName::S3(S3KeyName::S3ObjectLockMode))]
     #[test_case("aws:SecureTransport", KeyName::Aws(AwsKeyName::AWSSecureTransport))]
     #[test_case("jwt:sub", KeyName::Jwt(JwtKeyName::JWTSub))]
     #[test_case("ldap:user", KeyName::Ldap(LdapKeyName::User))]

crates/policy/src/policy/function/string.rs

@@ -287,7 +287,7 @@ mod tests {
     };
     use std::collections::HashMap;
 
-    use crate::policy::function::key_name::S3KeyName::S3LocationConstraint;
+    use crate::policy::function::key_name::S3KeyName::{S3LocationConstraint, S3ObjectLockMode};
     use test_case::test_case;
 
     fn new_func(name: KeyName, variable: Option<String>, values: Vec<&str>) -> StringFunc {
@@ -308,6 +308,7 @@ mod tests {
     ))]
     #[test_case(r#"{"aws:username/value": ["johndoe", "aaa"]}"#, new_func(Aws(AWSUsername), Some("value".into()), vec!["johndoe", "aaa"]
     ))]
+    #[test_case(r#"{"s3:object-lock-mode": "COMPLIANCE"}"#, new_func(S3(S3ObjectLockMode), None, vec!["COMPLIANCE"]))]
     fn test_deser(input: &str, expect: StringFunc) -> Result<(), serde_json::Error> {
         let v: StringFunc = serde_json::from_str(input)?;
         assert_eq!(v, expect);
@@ -410,6 +411,7 @@ mod tests {
     #[test_case(new_fkv("s3:ExistingObjectTag/security", vec!["public"]), false, vec![("ExistingObjectTag/project", vec!["webapp"])] => false ; "21")]
     #[test_case(new_fkv("s3:VersionId", vec!["version-1"]), false, vec![("versionid", vec!["version-1"])] => true ; "aws_version_id")]
     #[test_case(new_fkv("s3:versionid", vec!["version-1"]), false, vec![("versionid", vec!["version-1"])] => true ; "minio_version_id")]
+    #[test_case(new_fkv("s3:object-lock-mode", vec!["COMPLIANCE"]), false, vec![("object-lock-mode", vec!["COMPLIANCE"])] => true ; "object_lock_mode")]
     fn test_string_equals(s: FuncKeyValue<StringFuncValue>, for_all: bool, values: Vec<(&str, Vec<&str>)>) -> bool {
         test_eval(s, for_all, false, false, values)
     }

crates/protos/src/generated/proto_gen/node_service.rs

@@ -1221,6 +1221,17 @@ pub struct BackgroundHealStatusResponse {
     #[prost(string, optional, tag = "3")]
     pub error_info: ::core::option::Option<::prost::alloc::string::String>,
 }
+#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
+pub struct ReplacementRecoveryStatusRequest {}
+#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
+pub struct ReplacementRecoveryStatusResponse {
+    #[prost(bool, tag = "1")]
+    pub success: bool,
+    #[prost(bytes = "bytes", tag = "2")]
+    pub recovery_status: ::prost::bytes::Bytes,
+    #[prost(string, optional, tag = "3")]
+    pub error_info: ::core::option::Option<::prost::alloc::string::String>,
+}
 #[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
 pub struct HealControlRequest {
     #[prost(uint32, tag = "1")]
@@ -2692,6 +2703,21 @@ pub mod node_service_client {
                 .insert(GrpcMethod::new("node_service.NodeService", "BackgroundHealStatus"));
             self.inner.unary(req, path, codec).await
         }
+        pub async fn replacement_recovery_status(
+            &mut self,
+            request: impl tonic::IntoRequest<super::ReplacementRecoveryStatusRequest>,
+        ) -> std::result::Result<tonic::Response<super::ReplacementRecoveryStatusResponse>, tonic::Status> {
+            self.inner
+                .ready()
+                .await
+                .map_err(|e| tonic::Status::unknown(format!("Service was not ready: {}", e.into())))?;
+            let codec = tonic_prost::ProstCodec::default();
+            let path = http::uri::PathAndQuery::from_static("/node_service.NodeService/ReplacementRecoveryStatus");
+            let mut req = request.into_request();
+            req.extensions_mut()
+                .insert(GrpcMethod::new("node_service.NodeService", "ReplacementRecoveryStatus"));
+            self.inner.unary(req, path, codec).await
+        }
         pub async fn get_metacache_listing(
             &mut self,
             request: impl tonic::IntoRequest<super::GetMetacacheListingRequest>,
@@ -3179,6 +3205,10 @@ pub mod node_service_server {
             &self,
             request: tonic::Request<super::BackgroundHealStatusRequest>,
         ) -> std::result::Result<tonic::Response<super::BackgroundHealStatusResponse>, tonic::Status>;
+        async fn replacement_recovery_status(
+            &self,
+            request: tonic::Request<super::ReplacementRecoveryStatusRequest>,
+        ) -> std::result::Result<tonic::Response<super::ReplacementRecoveryStatusResponse>, tonic::Status>;
         async fn get_metacache_listing(
             &self,
             request: tonic::Request<super::GetMetacacheListingRequest>,
@@ -5479,6 +5509,34 @@ pub mod node_service_server {
                     };
                     Box::pin(fut)
                 }
+                "/node_service.NodeService/ReplacementRecoveryStatus" => {
+                    #[allow(non_camel_case_types)]
+                    struct ReplacementRecoveryStatusSvc<T: NodeService>(pub Arc<T>);
+                    impl<T: NodeService> tonic::server::UnaryService<super::ReplacementRecoveryStatusRequest> for ReplacementRecoveryStatusSvc<T> {
+                        type Response = super::ReplacementRecoveryStatusResponse;
+                        type Future = BoxFuture<tonic::Response<Self::Response>, tonic::Status>;
+                        fn call(&mut self, request: tonic::Request<super::ReplacementRecoveryStatusRequest>) -> Self::Future {
+                            let inner = Arc::clone(&self.0);
+                            let fut = async move { <T as NodeService>::replacement_recovery_status(&inner, request).await };
+                            Box::pin(fut)
+                        }
+                    }
+                    let accept_compression_encodings = self.accept_compression_encodings;
+                    let send_compression_encodings = self.send_compression_encodings;
+                    let max_decoding_message_size = self.max_decoding_message_size;
+                    let max_encoding_message_size = self.max_encoding_message_size;
+                    let inner = self.inner.clone();
+                    let fut = async move {
+                        let method = ReplacementRecoveryStatusSvc(inner);
+                        let codec = tonic_prost::ProstCodec::default();
+                        let mut grpc = tonic::server::Grpc::new(codec)
+                            .apply_compression_config(accept_compression_encodings, send_compression_encodings)
+                            .apply_max_message_size_config(max_decoding_message_size, max_encoding_message_size);
+                        let res = grpc.unary(method, req).await;
+                        Ok(res)
+                    };
+                    Box::pin(fut)
+                }
                 "/node_service.NodeService/GetMetacacheListing" => {
                     #[allow(non_camel_case_types)]
                     struct GetMetacacheListingSvc<T: NodeService>(pub Arc<T>);

crates/protos/src/node.proto

@@ -851,6 +851,14 @@ message BackgroundHealStatusResponse {
   optional string error_info = 3;
 }
 
+message ReplacementRecoveryStatusRequest {}
+
+message ReplacementRecoveryStatusResponse {
+  bool success = 1;
+  bytes recovery_status = 2;
+  optional string error_info = 3;
+}
+
 message HealControlRequest {
   uint32 version = 1;
   string topology_fingerprint = 2;
@@ -1084,6 +1092,7 @@ service NodeService {
   rpc SignalService(SignalServiceRequest) returns (SignalServiceResponse) {}; // auth-policy: body-bound
   rpc ScannerActivity(ScannerActivityRequest) returns (ScannerActivityResponse) {}; // auth-policy: body-bound
   rpc BackgroundHealStatus(BackgroundHealStatusRequest) returns (BackgroundHealStatusResponse) {}; // auth-policy: read-only
+  rpc ReplacementRecoveryStatus(ReplacementRecoveryStatusRequest) returns (ReplacementRecoveryStatusResponse) {}; // auth-policy: read-only
   rpc GetMetacacheListing(GetMetacacheListingRequest) returns (GetMetacacheListingResponse) {}; // auth-policy: unimplemented
   rpc UpdateMetacacheListing(UpdateMetacacheListingRequest) returns (UpdateMetacacheListingResponse) {}; // auth-policy: unimplemented
   rpc ReloadPoolMeta(ReloadPoolMetaRequest) returns (ReloadPoolMetaResponse) {}; // auth-policy: body-bound

crates/replication/src/config.rs

@@ -794,6 +794,33 @@ mod tests {
         );
     }
 
+    #[test]
+    fn structure_validation_counts_rule_id_limit_in_bytes() {
+        let mut within_byte_limit = replication_rule(&"\u{00e9}".repeat(127), "arn:target:a");
+        within_byte_limit.priority = Some(1);
+        assert_eq!(
+            within_byte_limit.id.as_ref().expect("rule id should be present").len(),
+            REPLICATION_CONFIG_MAX_RULE_ID_LEN - 1
+        );
+        assert_eq!(validate_replication_config_structure(&structure_config(vec![within_byte_limit])), Ok(()));
+
+        let mut over_byte_limit = replication_rule(&"\u{00e9}".repeat(128), "arn:target:a");
+        over_byte_limit.priority = Some(1);
+        assert!(
+            over_byte_limit
+                .id
+                .as_ref()
+                .expect("rule id should be present")
+                .chars()
+                .count()
+                < REPLICATION_CONFIG_MAX_RULE_ID_LEN
+        );
+        assert_eq!(
+            validate_replication_config_structure(&structure_config(vec![over_byte_limit])),
+            Err(ReplicationConfigStructureError::RuleIdTooLong)
+        );
+    }
+
     #[test]
     fn structure_validation_rejects_filter_with_both_prefix_and_tag() {
         let mut rule = replication_rule("rule-1", "arn:target:a");

crates/rio/src/http_reader.rs

@@ -19,8 +19,8 @@ use http::{HeaderMap, Version};
 use pin_project_lite::pin_project;
 use reqwest::{Certificate, Client, Identity, Method, RequestBuilder};
 use rustfs_io_metrics::internode_metrics::{
-    INTERNODE_OPERATION_NS_SCANNER, INTERNODE_OPERATION_PUT_FILE_STREAM, INTERNODE_OPERATION_READ_FILE_STREAM,
-    INTERNODE_OPERATION_WALK_DIR,
+    INTERNODE_OPERATION_NS_SCANNER, INTERNODE_OPERATION_PUT_FILE_CAPABILITY, INTERNODE_OPERATION_PUT_FILE_STREAM,
+    INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_OPERATION_WALK_DIR,
 };
 use rustfs_tls_runtime::load_cert_bundle_der_bytes;
 use rustfs_utils::{get_env_bool, get_env_opt_str, get_env_opt_u64, get_env_opt_usize};
@@ -43,6 +43,8 @@ use tracing::{error, warn};
 
 const READ_FILE_STREAM_PATH: &str = "/rustfs/rpc/read_file_stream";
 const PUT_FILE_STREAM_PATH: &str = "/rustfs/rpc/put_file_stream";
+const PUT_FILE_AUTH_STREAM_PATH: &str = "/rustfs/rpc/put_file_stream_v1";
+const PUT_FILE_CAPABILITY_PATH: &str = "/rustfs/rpc/put_file_capability";
 const WALK_DIR_PATH: &str = "/rustfs/rpc/walk_dir";
 const NS_SCANNER_PATH: &str = "/rustfs/rpc/ns_scanner";
 const HTTP_VERSION_09_LABEL: &str = "http/0.9";
@@ -261,6 +263,31 @@ pub fn new_test_internode_http_io_error(kind: InternodeHttpErrorKind) -> io::Err
     InternodeHttpError::new_for_test(kind).into_io_error()
 }
 
+/// Build a retryable internode timeout error with the request's operation context.
+#[doc(hidden)]
+pub fn internode_http_timeout_error(method: &Method, url: &str) -> io::Error {
+    internode_kind_error(method, url, internode_rpc_operation(url), InternodeHttpErrorKind::ConnectTimeout)
+}
+
+/// Clone an internode HTTP I/O error while retaining its structured classification.
+///
+/// The underlying transport source is intentionally omitted because it is not
+/// cloneable. The request context and remote disk marker remain available to
+/// retry and error-mapping code.
+#[doc(hidden)]
+pub fn clone_internode_http_io_error(error: &io::Error) -> Option<io::Error> {
+    let source = error.get_ref()?.downcast_ref::<InternodeHttpError>()?;
+    Some(
+        InternodeHttpError {
+            kind: source.kind,
+            context: source.context.clone(),
+            remote_disk_error: source.remote_disk_error,
+            source: None,
+        }
+        .into_io_error(),
+    )
+}
+
 #[doc(hidden)]
 pub fn new_test_remote_file_not_found_http_io_error() -> io::Error {
     InternodeHttpError::with_remote_disk_error(
@@ -1223,7 +1250,8 @@ fn internode_rpc_operation(url: &str) -> Option<&'static str> {
     let url = reqwest::Url::parse(url).ok()?;
     match url.path() {
         READ_FILE_STREAM_PATH => Some(INTERNODE_OPERATION_READ_FILE_STREAM),
-        PUT_FILE_STREAM_PATH => Some(INTERNODE_OPERATION_PUT_FILE_STREAM),
+        PUT_FILE_STREAM_PATH | PUT_FILE_AUTH_STREAM_PATH => Some(INTERNODE_OPERATION_PUT_FILE_STREAM),
+        PUT_FILE_CAPABILITY_PATH => Some(INTERNODE_OPERATION_PUT_FILE_CAPABILITY),
         WALK_DIR_PATH => Some(INTERNODE_OPERATION_WALK_DIR),
         NS_SCANNER_PATH => Some(INTERNODE_OPERATION_NS_SCANNER),
         _ => None,
@@ -1920,6 +1948,14 @@ mod tests {
             internode_rpc_operation(&format!("http://node:9000{PUT_FILE_STREAM_PATH}?disk=d")),
             Some(INTERNODE_OPERATION_PUT_FILE_STREAM)
         );
+        assert_eq!(
+            internode_rpc_operation(&format!("http://node:9000{PUT_FILE_AUTH_STREAM_PATH}?disk=d")),
+            Some(INTERNODE_OPERATION_PUT_FILE_STREAM)
+        );
+        assert_eq!(
+            internode_rpc_operation(&format!("http://node:9000{PUT_FILE_CAPABILITY_PATH}?put_file_capability=1")),
+            Some(INTERNODE_OPERATION_PUT_FILE_CAPABILITY)
+        );
         assert_eq!(
             internode_rpc_operation(&format!("http://node:9000{WALK_DIR_PATH}?disk=d")),
             Some(INTERNODE_OPERATION_WALK_DIR)
@@ -1935,6 +1971,21 @@ mod tests {
         );
     }
 
+    #[test]
+    fn internode_http_timeout_error_retains_operation_context() {
+        let error =
+            internode_http_timeout_error(&Method::GET, "http://node:9000/rustfs/rpc/put_file_capability?put_file_capability=1");
+        let source = error
+            .get_ref()
+            .and_then(|source| source.downcast_ref::<InternodeHttpError>())
+            .expect("timeout should retain internode classification");
+
+        assert_eq!(source.kind(), InternodeHttpErrorKind::ConnectTimeout);
+        assert_eq!(source.context().method(), "GET");
+        assert_eq!(source.context().target(), PUT_FILE_CAPABILITY_PATH);
+        assert_eq!(source.context().operation(), Some(INTERNODE_OPERATION_PUT_FILE_CAPABILITY));
+    }
+
     #[test]
     fn http_version_metrics_labels_are_low_cardinality() {
         assert_eq!(http_version_metric_label(Version::HTTP_09), HTTP_VERSION_09_LABEL);
@@ -2385,6 +2436,42 @@ mod tests {
         assert!(source.context().target().contains(PUT_FILE_STREAM_PATH));
     }
 
+    #[test]
+    fn cloned_internode_http_error_retains_classification_and_context() {
+        let original = internode_status_error(
+            &Method::GET,
+            "http://node:9000/rustfs/rpc/put_file_capability?put_file_capability=1",
+            Some(INTERNODE_OPERATION_PUT_FILE_CAPABILITY),
+            reqwest::StatusCode::SERVICE_UNAVAILABLE,
+        );
+        let cloned = clone_internode_http_io_error(&original).expect("internode error should clone");
+        let source = cloned
+            .get_ref()
+            .and_then(|source| source.downcast_ref::<InternodeHttpError>())
+            .expect("clone should retain internode source");
+
+        assert_eq!(
+            source.kind(),
+            InternodeHttpErrorKind::HttpStatus(reqwest::StatusCode::SERVICE_UNAVAILABLE)
+        );
+        assert!(source.kind().is_retryable());
+        assert_eq!(source.context().method(), "GET");
+        assert_eq!(source.context().target(), PUT_FILE_CAPABILITY_PATH);
+        assert_eq!(source.context().operation(), Some(INTERNODE_OPERATION_PUT_FILE_CAPABILITY));
+    }
+
+    #[test]
+    fn cloned_internode_http_error_retains_remote_disk_marker() {
+        let original = new_test_remote_file_not_found_http_io_error();
+        let cloned = clone_internode_http_io_error(&original).expect("internode error should clone");
+        let source = cloned
+            .get_ref()
+            .and_then(|source| source.downcast_ref::<InternodeHttpError>())
+            .expect("clone should retain internode source");
+
+        assert!(source.is_remote_file_not_found());
+    }
+
     #[test]
     fn loopback_urls_bypass_proxy_selection() {
         assert!(should_bypass_proxy_for_url("http://127.0.0.1:9000/stream"));

crates/s3select-api/Cargo.toml

@@ -77,11 +77,12 @@ parking_lot.workspace = true
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tokio-util = { workspace = true, features = ["io", "compat"] }
 tracing.workspace = true
+uuid.workspace = true
 transform-stream.workspace = true
 url.workspace = true
 
 [dev-dependencies]
-rustfs-test-utils.workspace = true
+rustfs-test-utils = { workspace = true, features = ["put-object-commit-barrier"] }
 serial_test.workspace = true
 
 [lib]

crates/s3select-api/src/lib.rs

@@ -15,22 +15,23 @@
 #![recursion_limit = "256"]
 
 use datafusion::{common::DataFusionError, sql::sqlparser::parser::ParserError};
-use std::fmt::Display;
+use std::{error::Error as StdError, fmt::Display};
 use thiserror::Error;
 
 pub mod object_store;
 pub mod query;
 pub mod server;
 mod storage_api;
+pub use storage_api::SelectObjectSnapshot;
 
 #[cfg(test)]
 mod test;
 
 pub type QueryResult<T> = Result<T, QueryError>;
 pub(crate) use storage_api::crate_boundary::{
-    SELECT_DEFAULT_READ_BUFFER_SIZE, SelectGetObjectReader, SelectObjectInfo, SelectObjectOptions, SelectStorageError,
-    SelectStore, resolve_select_object_store_handle, select_is_err_bucket_not_found, select_is_err_object_not_found,
-    select_is_err_version_not_found,
+    PrepareSelectObjectSnapshotError, SELECT_DEFAULT_READ_BUFFER_SIZE, SelectGetObjectReader, SelectObjectOptions,
+    SelectObjectSnapshotReadError, SelectStorageError, SelectStore, SnapshotConsistencyError, resolve_select_object_store_handle,
+    select_is_err_bucket_not_found, select_is_err_object_not_found, select_is_err_version_not_found,
 };
 
 #[derive(Debug, Error)]
@@ -79,24 +80,24 @@ pub enum S3SelectPolicyError {
     QueryTimeout { seconds: u64 },
 }
 
-impl S3SelectPolicyError {
-    fn from_error<'a>(mut err: &'a (dyn std::error::Error + 'static)) -> Option<&'a Self> {
+impl QueryError {
+    fn source_error<T: StdError + 'static>(&self) -> Option<&T> {
+        let mut err: &(dyn StdError + 'static) = self;
         for _ in 0..16 {
-            if let Some(policy_error) = err.downcast_ref::<Self>() {
-                return Some(policy_error);
+            if let Some(source) = err.downcast_ref::<T>() {
+                return Some(source);
             }
             err = err.source()?;
         }
         None
     }
-}
 
-impl QueryError {
+    pub fn is_snapshot_consistency_error(&self) -> bool {
+        self.source_error::<SnapshotConsistencyError>().is_some()
+    }
+
     pub fn s3_select_policy_error(&self) -> Option<&S3SelectPolicyError> {
-        match self {
-            Self::Datafusion { source } => S3SelectPolicyError::from_error(source.as_ref()),
-            _ => None,
-        }
+        self.source_error()
     }
 }
 
@@ -230,6 +231,17 @@ mod tests {
         ));
     }
 
+    #[test]
+    fn snapshot_consistency_error_is_recoverable_without_string_matching() {
+        let err = QueryError::Datafusion {
+            source: Box::new(DataFusionError::External(Box::new(SelectObjectSnapshotReadError::Consistency(
+                SnapshotConsistencyError::LockLost,
+            )))),
+        };
+
+        assert!(err.is_snapshot_consistency_error());
+    }
+
     #[test]
     fn test_query_error_from_parser_error() {
         let parser_error = ParserError::ParserError("syntax error".to_string());

crates/s3select-api/src/query/dispatcher.rs

@@ -22,6 +22,7 @@ use super::{
     Query,
     execution::{Output, QueryStateMachine},
     logical_planner::Plan,
+    session::QueryAdmission,
 };
 
 #[async_trait]
@@ -32,6 +33,14 @@ pub trait QueryDispatcher: Send + Sync {
 
     async fn execute_query(&self, query: &Query) -> QueryResult<Output>;
 
+    fn try_reserve_query(&self) -> QueryResult<QueryAdmission> {
+        Ok(QueryAdmission::unmanaged())
+    }
+
+    async fn execute_query_admitted(&self, query: &Query, _admission: QueryAdmission) -> QueryResult<Output> {
+        self.execute_query(query).await
+    }
+
     async fn build_logical_plan(&self, query_state_machine: Arc<QueryStateMachine>) -> QueryResult<Option<Plan>>;
 
     async fn execute_logical_plan(&self, logical_plan: Plan, query_state_machine: Arc<QueryStateMachine>) -> QueryResult<Output>;

crates/s3select-api/src/query/mod.rs

@@ -15,6 +15,8 @@
 use s3s::dto::SelectObjectContentInput;
 use std::sync::Arc;
 
+use crate::SelectObjectSnapshot;
+
 pub mod analyzer;
 pub mod ast;
 pub mod dispatcher;
@@ -37,12 +39,26 @@ pub struct Context {
 pub struct Query {
     context: Context,
     content: String,
+    snapshot: Option<Arc<SelectObjectSnapshot>>,
 }
 
 impl Query {
     #[inline(always)]
     pub fn new(context: Context, content: String) -> Self {
-        Self { context, content }
+        Self {
+            context,
+            content,
+            snapshot: None,
+        }
+    }
+
+    #[inline(always)]
+    pub fn new_with_snapshot(context: Context, content: String, snapshot: Arc<SelectObjectSnapshot>) -> Self {
+        Self {
+            context,
+            content,
+            snapshot: Some(snapshot),
+        }
     }
 
     pub fn context(&self) -> &Context {
@@ -52,4 +68,8 @@ impl Query {
     pub fn content(&self) -> &str {
         self.content.as_str()
     }
+
+    pub fn snapshot(&self) -> Option<&Arc<SelectObjectSnapshot>> {
+        self.snapshot.as_ref()
+    }
 }

crates/s3select-api/src/query/session.rs

@@ -12,14 +12,16 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use crate::SelectObjectSnapshot;
 use crate::query::Context;
-use crate::{QueryError, QueryResult, SelectStore, object_store::EcObjectStore};
+use crate::{QueryError, QueryResult, object_store::EcObjectStore};
 use datafusion::{
     arrow::{
         array::{Int32Array, StringArray},
         datatypes::{DataType, Field, Schema},
         record_batch::RecordBatch,
     },
+    common::DataFusionError,
     execution::{SessionStateBuilder, config::SessionConfig, context::SessionState, runtime_env::RuntimeEnvBuilder},
     object_store::{ObjectStore, ObjectStoreExt, memory::InMemory, path::Path},
     parquet::arrow::ArrowWriter,
@@ -39,6 +41,28 @@ use tracing::error;
 
 pub type QueryExecutionGuard = Arc<OwnedSemaphorePermit>;
 
+/// A one-shot query admission reservation handed from the request boundary to
+/// the dispatcher that owns the corresponding concurrency semaphore.
+pub struct QueryAdmission {
+    query_guard: Option<QueryExecutionGuard>,
+}
+
+impl QueryAdmission {
+    pub fn new(query_guard: QueryExecutionGuard) -> Self {
+        Self {
+            query_guard: Some(query_guard),
+        }
+    }
+
+    pub fn into_query_guard(mut self) -> Option<QueryExecutionGuard> {
+        self.query_guard.take()
+    }
+
+    pub(crate) fn unmanaged() -> Self {
+        Self { query_guard: None }
+    }
+}
+
 #[derive(Clone, Default)]
 pub struct QueryExecutionOwner {
     identity: Arc<()>,
@@ -300,30 +324,30 @@ impl SessionCtxFactory {
         query_tracker: QueryExecutionTracker,
         memory_limit_bytes: usize,
     ) -> QueryResult<SessionCtx> {
-        self.create_session_ctx_inner(context, Some(query_tracker), None, memory_limit_bytes)
+        self.create_session_ctx_inner(context, None, Some(query_tracker), memory_limit_bytes)
             .await
     }
 
-    #[cfg(test)]
-    async fn create_session_ctx_with_tracker_and_store(
+    pub async fn create_session_ctx_with_snapshot_and_tracker_and_memory_limit(
         &self,
         context: &Context,
+        snapshot: Arc<SelectObjectSnapshot>,
         query_tracker: QueryExecutionTracker,
-        store: Arc<SelectStore>,
+        memory_limit_bytes: usize,
     ) -> QueryResult<SessionCtx> {
-        self.create_session_ctx_inner(context, Some(query_tracker), Some(store), DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES)
+        self.create_session_ctx_inner(context, Some(snapshot), Some(query_tracker), memory_limit_bytes)
             .await
     }
 
     async fn create_session_ctx_inner(
         &self,
         context: &Context,
+        snapshot: Option<Arc<SelectObjectSnapshot>>,
         query_tracker: Option<QueryExecutionTracker>,
-        store: Option<Arc<SelectStore>>,
         memory_limit_bytes: usize,
     ) -> QueryResult<SessionCtx> {
         let df_session_ctx = self
-            .build_df_session_context(context, query_tracker.clone(), store, memory_limit_bytes)
+            .build_df_session_context(context, snapshot, query_tracker.clone(), memory_limit_bytes)
             .await?;
 
         Ok(SessionCtx {
@@ -336,8 +360,8 @@ impl SessionCtxFactory {
     async fn build_df_session_context(
         &self,
         context: &Context,
+        snapshot: Option<Arc<SelectObjectSnapshot>>,
         query_tracker: Option<QueryExecutionTracker>,
-        store: Option<Arc<SelectStore>>,
         memory_limit_bytes: usize,
     ) -> QueryResult<SessionContext> {
         let path = format!("s3://{}", context.input.bucket);
@@ -416,11 +440,13 @@ impl SessionCtxFactory {
         } else {
             let store: EcObjectStore = match query_tracker {
                 Some(query_tracker) => {
-                    EcObjectStore::new_with_query_tracker(context.input.clone(), memory_pool, query_tracker, store)
+                    EcObjectStore::new_with_query_tracker(context.input.clone(), memory_pool, query_tracker, snapshot)
                 }
-                None => EcObjectStore::new_with_memory_pool(context.input.clone(), memory_pool),
+                None => EcObjectStore::new_with_memory_pool(context.input.clone(), memory_pool, snapshot),
             }
-            .map_err(|_| QueryError::NotImplemented { err: String::new() })?;
+            .map_err(|err| QueryError::Datafusion {
+                source: Box::new(DataFusionError::External(Box::new(err))),
+            })?;
             df_session_state.with_object_store(&store_url, Arc::new(store)).build()
         };
 
@@ -498,6 +524,7 @@ mod tests {
         },
         execution::memory_pool::MemoryLimit,
     };
+    use http::HeaderMap;
     use s3s::dto::{
         CSVInput, CSVOutput, ExpressionType, InputSerialization, JSONInput, OutputSerialization, ParquetInput, ScanRange,
         SelectObjectContentInput, SelectObjectContentRequest,
@@ -531,6 +558,16 @@ mod tests {
         }
     }
 
+    async fn prepare_test_snapshot(context: &Context) -> Arc<SelectObjectSnapshot> {
+        let env = crate::storage_api::select_test_ecstore_env().await;
+        Arc::new(
+            env.ecstore
+                .prepare_select_object_snapshot(&context.input.bucket, &context.input.key, &HeaderMap::new(), &Default::default())
+                .await
+                .expect("prepare SelectObjectContent snapshot"),
+        )
+    }
+
     #[test]
     fn session_factory_fields_remain_source_compatible() {
         let factory = SessionCtxFactory {
@@ -679,10 +716,57 @@ mod tests {
         ));
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn production_session_preserves_lazy_snapshot_entry() {
+        let _env = crate::storage_api::select_test_ecstore_env().await;
+        let session = SessionCtxFactory::new(false)
+            .create_session_ctx(&test_context())
+            .await
+            .expect("legacy production session should install a lazy object store");
+
+        assert_eq!(session.inner().config().target_partitions(), SessionConfig::new().target_partitions());
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn legacy_tracked_production_session_preserves_lazy_snapshot_entry() {
+        let _env = crate::storage_api::select_test_ecstore_env().await;
+        let permit = Arc::new(tokio::sync::Semaphore::new(1))
+            .acquire_owned()
+            .await
+            .expect("query permit should be available");
+        let tracker = QueryExecutionTracker::new(
+            &QueryExecutionOwner::new(),
+            Arc::new(permit),
+            Instant::now() + std::time::Duration::from_secs(300),
+            300,
+        );
+        let session = SessionCtxFactory::new(false)
+            .create_session_ctx_with_tracker_and_memory_limit(
+                &test_context(),
+                tracker.clone(),
+                DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES,
+            )
+            .await
+            .expect("legacy tracked session should install a lazy object store");
+
+        assert!(session.is_bound_to(&tracker));
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     #[serial_test::serial]
     async fn session_factory_propagates_query_guard_to_ec_store() {
         let env = crate::storage_api::select_test_ecstore_env().await;
+        let mut context = test_context();
+        Arc::make_mut(&mut context.input).bucket = "s3select-query-guard-snapshot".to_string();
+        env.make_bucket(&context.input.bucket, false).await;
+        let mut reader = SelectPutObjReader::from_vec(b"id,name\n1,Alice\n".to_vec());
+        env.ecstore
+            .put_object(&context.input.bucket, &context.input.key, &mut reader, &Default::default())
+            .await
+            .expect("put query guard fixture");
+        let snapshot = prepare_test_snapshot(&context).await;
 
         let admission = Arc::new(tokio::sync::Semaphore::new(1));
         let permit = Arc::clone(&admission)
@@ -697,7 +781,12 @@ mod tests {
             300,
         );
         let session = SessionCtxFactory::new(false)
-            .create_session_ctx_with_tracker_and_store(&test_context(), query_tracker, Arc::clone(&env.ecstore))
+            .create_session_ctx_with_snapshot_and_tracker_and_memory_limit(
+                &context,
+                snapshot,
+                query_tracker,
+                DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES,
+            )
             .await
             .expect("production session should be created with the query guard");
 
@@ -706,6 +795,51 @@ mod tests {
         assert_eq!(Arc::strong_count(&query_guard), 1);
     }
 
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    #[serial_test::serial]
+    async fn session_factory_preserves_snapshot_binding_error_source() {
+        let env = crate::storage_api::select_test_ecstore_env().await;
+        let mut source_context = test_context();
+        Arc::make_mut(&mut source_context.input).bucket = "s3select-session-snapshot-identity".to_string();
+        Arc::make_mut(&mut source_context.input).key = "source.csv".to_string();
+        env.make_bucket(&source_context.input.bucket, false).await;
+        let mut reader = SelectPutObjReader::from_vec(b"source-marker\n".to_vec());
+        env.ecstore
+            .put_object(&source_context.input.bucket, &source_context.input.key, &mut reader, &Default::default())
+            .await
+            .expect("put snapshot identity fixture");
+        let snapshot = prepare_test_snapshot(&source_context).await;
+
+        let mut target_context = source_context.clone();
+        Arc::make_mut(&mut target_context.input).key = "different.csv".to_string();
+        let permit = Arc::new(tokio::sync::Semaphore::new(1))
+            .acquire_owned()
+            .await
+            .expect("query permit should be available");
+        let tracker = QueryExecutionTracker::new(
+            &QueryExecutionOwner::new(),
+            Arc::new(permit),
+            Instant::now() + std::time::Duration::from_secs(300),
+            300,
+        );
+
+        let error = match SessionCtxFactory::new(false)
+            .create_session_ctx_with_snapshot_and_tracker_and_memory_limit(
+                &target_context,
+                snapshot,
+                tracker,
+                DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES,
+            )
+            .await
+        {
+            Ok(_) => panic!("a session must reject a snapshot for a different object"),
+            Err(error) => error,
+        };
+
+        assert!(error.is_snapshot_consistency_error());
+        assert!(error.to_string().contains("snapshot consistency failure"));
+    }
+
     #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
     #[serial_test::serial]
     async fn scan_range_is_preserved_across_large_csv_partition_boundary() {
@@ -721,6 +855,7 @@ mod tests {
         assert!(data.len() > 1024 * 1024);
 
         let mut context = test_context();
+        Arc::make_mut(&mut context.input).bucket = "s3select-scan-range-partition-snapshot".to_string();
         let selected_start = i64::try_from(SELECTED_ROW * ROW_WIDTH).expect("selected row offset should fit in i64");
         Arc::make_mut(&mut context.input).request.scan_range = Some(ScanRange {
             start: Some(selected_start),
@@ -732,6 +867,7 @@ mod tests {
             .put_object(&context.input.bucket, &context.input.key, &mut reader, &Default::default())
             .await
             .expect("put large ScanRange CSV fixture");
+        let snapshot = prepare_test_snapshot(&context).await;
 
         let admission = Arc::new(tokio::sync::Semaphore::new(1));
         let permit = Arc::clone(&admission)
@@ -746,7 +882,12 @@ mod tests {
         );
         let session = SessionCtxFactory::new(false)
             .with_target_partitions(2)
-            .create_session_ctx_with_tracker_and_store(&context, query_tracker, Arc::clone(&env.ecstore))
+            .create_session_ctx_with_snapshot_and_tracker_and_memory_limit(
+                &context,
+                snapshot,
+                query_tracker,
+                DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES,
+            )
             .await
             .expect("create production ScanRange session");
         assert!(!session.inner().config().options().optimizer.repartition_file_scans);

crates/s3select-api/src/server/dbms.rs

@@ -20,6 +20,7 @@ use crate::{
         Query,
         execution::{Output, QueryStateMachineRef},
         logical_planner::Plan,
+        session::QueryAdmission,
     },
 };
 
@@ -44,7 +45,16 @@ impl QueryHandle {
 
 #[async_trait]
 pub trait DatabaseManagerSystem {
+    fn try_reserve_query(&self) -> QueryResult<QueryAdmission> {
+        Ok(QueryAdmission::unmanaged())
+    }
+
     async fn execute(&self, query: &Query) -> QueryResult<QueryHandle>;
+
+    async fn execute_admitted(&self, query: &Query, _admission: QueryAdmission) -> QueryResult<QueryHandle> {
+        self.execute(query).await
+    }
+
     async fn build_query_state_machine(&self, query: Query) -> QueryResult<QueryStateMachineRef>;
     async fn build_logical_plan(&self, query_state_machine: QueryStateMachineRef) -> QueryResult<Option<Plan>>;
     async fn execute_logical_plan(

crates/s3select-api/src/storage_api.rs

@@ -22,35 +22,51 @@ use rustfs_ecstore::api::error::{
 };
 #[cfg(test)]
 pub(crate) use rustfs_ecstore::api::object::PutObjReader as SelectPutObjReader;
+pub use rustfs_ecstore::api::object::SelectObjectSnapshot;
+pub(crate) use rustfs_ecstore::api::object::{
+    PrepareSelectObjectSnapshotError, SelectObjectSnapshotReadError, SnapshotConsistencyError,
+};
 use rustfs_ecstore::api::runtime::object_store_handle as resolve_select_object_store_handle_from_backend;
 pub(crate) use rustfs_ecstore::api::set_disk::DEFAULT_READ_BUFFER_SIZE as SELECT_DEFAULT_READ_BUFFER_SIZE;
 pub(crate) use rustfs_ecstore::api::storage::ECStore as SelectStore;
 use rustfs_storage_api as storage_contracts;
 
+#[cfg(test)]
+static SELECT_TEST_OBJECT_STORE: std::sync::OnceLock<Arc<SelectStore>> = std::sync::OnceLock::new();
+
 pub(crate) mod object_store {
-    pub(crate) use super::storage_contracts::{HTTPRangeSpec, ObjectIO, ObjectOperations};
+    pub(crate) use super::storage_contracts::HTTPRangeSpec;
+    #[cfg(test)]
+    pub(crate) use super::storage_contracts::ObjectIO;
 }
 
 pub(crate) mod crate_boundary {
     pub(crate) use super::{
-        SELECT_DEFAULT_READ_BUFFER_SIZE, SelectGetObjectReader, SelectObjectInfo, SelectObjectOptions, SelectStorageError,
-        SelectStore, resolve_select_object_store_handle, select_is_err_bucket_not_found, select_is_err_object_not_found,
+        PrepareSelectObjectSnapshotError, SELECT_DEFAULT_READ_BUFFER_SIZE, SelectGetObjectReader, SelectObjectOptions,
+        SelectObjectSnapshotReadError, SelectStorageError, SelectStore, SnapshotConsistencyError,
+        resolve_select_object_store_handle, select_is_err_bucket_not_found, select_is_err_object_not_found,
         select_is_err_version_not_found,
     };
 }
 
 pub(crate) type SelectGetObjectReader = <SelectStore as storage_contracts::ObjectIO>::GetObjectReader;
-pub(crate) type SelectObjectInfo = <SelectStore as storage_contracts::ObjectOperations>::ObjectInfo;
 pub(crate) type SelectObjectOptions = <SelectStore as storage_contracts::ObjectOperations>::ObjectOptions;
 
 #[cfg(test)]
 pub(crate) async fn select_test_ecstore_env() -> &'static rustfs_test_utils::TestECStoreEnv {
     static ENV: tokio::sync::OnceCell<rustfs_test_utils::TestECStoreEnv> = tokio::sync::OnceCell::const_new();
-    ENV.get_or_init(|| async { rustfs_test_utils::TestECStoreEnv::builder().build().await })
-        .await
+    let env = ENV
+        .get_or_init(|| async { rustfs_test_utils::TestECStoreEnv::builder().build().await })
+        .await;
+    let _ = SELECT_TEST_OBJECT_STORE.set(Arc::clone(&env.ecstore));
+    env
 }
 
 pub(crate) fn resolve_select_object_store_handle() -> Option<Arc<SelectStore>> {
+    #[cfg(test)]
+    if let Some(store) = SELECT_TEST_OBJECT_STORE.get() {
+        return Some(Arc::clone(store));
+    }
     resolve_select_object_store_handle_from_backend()
 }
 

crates/s3select-query/Cargo.toml

@@ -54,5 +54,8 @@ s3s = { workspace = true, features = ["minio"] }
 tokio = { workspace = true, features = ["fs", "rt-multi-thread", "sync", "time"] }
 tracing = { workspace = true }
 
+[dev-dependencies]
+rustfs-test-utils = { workspace = true, features = ["put-object-commit-barrier"] }
+
 [lib]
 doctest = false

crates/s3select-query/src/dispatcher/manager.rs

@@ -48,8 +48,8 @@ use rustfs_s3select_api::{
         logical_planner::{LogicalPlanner, Plan},
         parser::Parser,
         session::{
-            DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES, QueryExecutionOwner, QueryExecutionStatus, QueryExecutionTracker, SessionCtx,
-            SessionCtxFactory,
+            DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES, QueryAdmission, QueryExecutionOwner, QueryExecutionStatus,
+            QueryExecutionTracker, SessionCtx, SessionCtxFactory,
         },
     },
 };
@@ -120,13 +120,20 @@ impl Drop for QueryPhaseGuard<'_> {
 #[async_trait]
 impl QueryDispatcher for SimpleQueryDispatcher {
     async fn execute_query(&self, query: &Query) -> QueryResult<Output> {
-        let query_state_machine = self.build_query_state_machine(query.clone()).await?;
-        let logical_plan = self.build_logical_plan(Arc::clone(&query_state_machine)).await?;
-        let Some(logical_plan) = logical_plan else {
-            return Ok(Output::Nil(()));
-        };
+        self.execute_query_inner(query, None).await
+    }
 
-        self.execute_logical_plan(logical_plan, query_state_machine).await
+    fn try_reserve_query(&self) -> QueryResult<QueryAdmission> {
+        let permit = self
+            .query_admission
+            .clone()
+            .try_acquire_owned()
+            .map_err(|_| QueryError::from(S3SelectPolicyError::QueryConcurrencyLimit))?;
+        Ok(QueryAdmission::new(Arc::new(permit)))
+    }
+
+    async fn execute_query_admitted(&self, query: &Query, admission: QueryAdmission) -> QueryResult<Output> {
+        self.execute_query_inner(query, Some(admission)).await
     }
 
     async fn build_logical_plan(&self, query_state_machine: Arc<QueryStateMachine>) -> QueryResult<Option<Plan>> {
@@ -205,20 +212,64 @@ impl QueryDispatcher for SimpleQueryDispatcher {
     }
 
     async fn build_query_state_machine(&self, query: Query) -> QueryResult<Arc<QueryStateMachine>> {
-        let permit = self
-            .query_admission
-            .clone()
-            .try_acquire_owned()
-            .map_err(|_| QueryError::from(S3SelectPolicyError::QueryConcurrencyLimit))?;
+        self.build_query_state_machine_inner(query, None).await
+    }
+}
+
+impl SimpleQueryDispatcher {
+    async fn execute_query_inner(&self, query: &Query, admission: Option<QueryAdmission>) -> QueryResult<Output> {
+        let query_state_machine = self.build_query_state_machine_inner(query.clone(), admission).await?;
+        let logical_plan = self.build_logical_plan(Arc::clone(&query_state_machine)).await?;
+        let Some(logical_plan) = logical_plan else {
+            return Ok(Output::Nil(()));
+        };
+
+        self.execute_logical_plan(logical_plan, query_state_machine).await
+    }
+
+    async fn build_query_state_machine_inner(
+        &self,
+        query: Query,
+        admission: Option<QueryAdmission>,
+    ) -> QueryResult<Arc<QueryStateMachine>> {
+        let query_guard = match admission {
+            Some(admission) => {
+                let query_guard = admission.into_query_guard().ok_or(QueryError::Cancel)?;
+                if !Arc::ptr_eq(query_guard.semaphore(), &self.query_admission) {
+                    return Err(QueryError::Cancel);
+                }
+                query_guard
+            }
+            None => {
+                let permit = self
+                    .query_admission
+                    .clone()
+                    .try_acquire_owned()
+                    .map_err(|_| QueryError::from(S3SelectPolicyError::QueryConcurrencyLimit))?;
+                Arc::new(permit)
+            }
+        };
         let query_tracker = QueryExecutionTracker::new(
             &self.query_execution_owner,
-            Arc::new(permit),
+            query_guard,
             Instant::now() + self.query_timeout,
             self.query_timeout.as_secs(),
         );
         let phase_guard = QueryPhaseGuard::new(&query_tracker, &self.query_execution_owner);
-        let session = self
-            .run_with_query_deadline(
+        let session = if let Some(snapshot) = query.snapshot().cloned() {
+            self.run_with_query_deadline(
+                &query_tracker,
+                self.session_factory
+                    .create_session_ctx_with_snapshot_and_tracker_and_memory_limit(
+                        query.context(),
+                        snapshot,
+                        query_tracker.clone(),
+                        self.memory_limit_bytes,
+                    ),
+            )
+            .await?
+        } else {
+            self.run_with_query_deadline(
                 &query_tracker,
                 self.session_factory.create_session_ctx_with_tracker_and_memory_limit(
                     query.context(),
@@ -226,7 +277,8 @@ impl QueryDispatcher for SimpleQueryDispatcher {
                     self.memory_limit_bytes,
                 ),
             )
-            .await?;
+            .await?
+        };
         if !query_tracker.mark_admitted(&self.query_execution_owner) {
             drop(session);
             return Err(self.query_tracker_error(&query_tracker));
@@ -234,9 +286,6 @@ impl QueryDispatcher for SimpleQueryDispatcher {
         phase_guard.disarm();
         Ok(Arc::new(QueryStateMachine::begin_tracked(query, session, query_tracker)?))
     }
-}
-
-impl SimpleQueryDispatcher {
     async fn run_with_query_deadline<T>(
         &self,
         query_tracker: &QueryExecutionTracker,
@@ -730,15 +779,17 @@ mod tests {
     use async_trait::async_trait;
     use datafusion::{
         arrow::{
-            datatypes::{Schema, SchemaRef},
+            array::{Int32Array, StringArray},
+            datatypes::{DataType, Field, Schema, SchemaRef},
             record_batch::RecordBatch,
         },
         common::DataFusionError,
         execution::object_store::ObjectStoreUrl,
         object_store::{ObjectStoreExt, path::Path},
+        parquet::arrow::ArrowWriter,
         physical_plan::{RecordBatchStream, stream::RecordBatchStreamAdapter},
     };
-    use futures::{StreamExt, stream};
+    use futures::{StreamExt, TryStreamExt, stream};
     use rustfs_s3select_api::{
         QueryError, QueryResult, S3SelectPolicyError,
         query::{
@@ -749,14 +800,15 @@ mod tests {
             },
             logical_planner::Plan,
             session::{
-                DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES, QueryExecutionOwner, QueryExecutionStatus, QueryExecutionTracker,
-                SessionCtxFactory,
+                DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES, QueryAdmission, QueryExecutionOwner, QueryExecutionStatus,
+                QueryExecutionTracker, SessionCtxFactory,
             },
         },
     };
+    use rustfs_test_utils::{PutObjectCommitBarrier, TestECStoreEnv};
     use s3s::dto::{
-        CSVInput, CSVOutput, ExpressionType, FileHeaderInfo, InputSerialization, OutputSerialization, SelectObjectContentInput,
-        SelectObjectContentRequest,
+        CSVInput, CSVOutput, ExpressionType, FileHeaderInfo, InputSerialization, JSONInput, JSONOutput, JSONType,
+        OutputSerialization, ParquetInput, SelectObjectContentInput, SelectObjectContentRequest,
     };
     use std::{
         pin::Pin,
@@ -998,6 +1050,260 @@ mod tests {
         (dispatcher, input)
     }
 
+    async fn snapshot_test_env() -> &'static TestECStoreEnv {
+        static ENV: tokio::sync::OnceCell<TestECStoreEnv> = tokio::sync::OnceCell::const_new();
+        ENV.get_or_init(|| async { TestECStoreEnv::builder().prefix("s3select_query_snapshot").build().await })
+            .await
+    }
+
+    fn production_dispatcher(input: Arc<SelectObjectContentInput>) -> Arc<SimpleQueryDispatcher> {
+        let optimizer = Arc::new(CascadeOptimizerBuilder::default().build());
+        let scheduler = Arc::new(LocalScheduler {});
+        SimpleQueryDispatcherBuilder::default()
+            .with_input(input)
+            .with_default_table_provider(Arc::new(BaseTableProvider::default()))
+            .with_session_factory(Arc::new(SessionCtxFactory::new(false)))
+            .with_parser(Arc::new(DefaultParser::default()))
+            .with_query_execution_factory(Arc::new(SqlQueryExecutionFactory::new(optimizer, scheduler)))
+            .with_func_manager(Arc::new(SimpleFunctionMetadataManager::default()))
+            .build()
+            .expect("production query dispatcher should build")
+    }
+
+    async fn collect_utf8_output(output: Output) -> Vec<String> {
+        let Output::StreamData(stream) = output else {
+            panic!("snapshot query should return rows");
+        };
+        stream
+            .try_collect::<Vec<_>>()
+            .await
+            .expect("collect snapshot query output")
+            .iter()
+            .flat_map(|batch| {
+                batch
+                    .column(0)
+                    .as_any()
+                    .downcast_ref::<StringArray>()
+                    .expect("snapshot marker column should be Utf8")
+                    .iter()
+                    .map(|value| value.expect("snapshot marker should not be null").to_string())
+                    .collect::<Vec<_>>()
+            })
+            .collect()
+    }
+
+    async fn run_snapshot_generation_race(
+        input: Arc<SelectObjectContentInput>,
+        old_generation: Vec<u8>,
+        new_generation: Vec<u8>,
+        expected_old_markers: &[&str],
+    ) {
+        let env = snapshot_test_env().await;
+        env.make_bucket(&input.bucket, false).await;
+        env.put_object_bytes(&input.bucket, &input.key, old_generation).await;
+
+        let snapshot = env.prepare_select_object_snapshot(&input.bucket, &input.key).await;
+        let dispatcher = production_dispatcher(Arc::clone(&input));
+        let query = Query::new_with_snapshot(
+            QueryContext {
+                input: Arc::clone(&input),
+            },
+            input.request.expression.clone(),
+            snapshot,
+        );
+        let query_state_machine = dispatcher
+            .build_query_state_machine(query.clone())
+            .await
+            .expect("build production snapshot session");
+        let logical_plan = dispatcher
+            .build_logical_plan(Arc::clone(&query_state_machine))
+            .await
+            .expect("infer old-generation schema")
+            .expect("SELECT should produce a logical plan");
+
+        let commit_barrier = PutObjectCommitBarrier::before_namespace(&input.bucket, &input.key);
+        let writer_env = env;
+        let writer_bucket = input.bucket.clone();
+        let writer_object = input.key.clone();
+        let writer = tokio::spawn(async move {
+            writer_env
+                .put_object_bytes(&writer_bucket, &writer_object, new_generation)
+                .await;
+        });
+        commit_barrier.wait_until_paused().await;
+        commit_barrier.release_and_wait_until_namespace_pending().await;
+        assert!(!writer.is_finished(), "overwrite must wait for the SelectObjectContent snapshot");
+
+        let output = dispatcher
+            .execute_logical_plan(logical_plan, query_state_machine)
+            .await
+            .expect("scan old-generation rows");
+        let values = collect_utf8_output(output).await;
+        assert_eq!(values, expected_old_markers);
+        assert!(!writer.is_finished(), "overwrite must remain blocked while Query owns the snapshot");
+
+        drop(query);
+        tokio::time::timeout(Duration::from_secs(5), writer)
+            .await
+            .expect("overwrite should finish after snapshot release")
+            .expect("overwrite task should join");
+    }
+
+    fn json_snapshot_input() -> Arc<SelectObjectContentInput> {
+        Arc::new(SelectObjectContentInput {
+            bucket: "s3select-json-snapshot-race".to_string(),
+            expected_bucket_owner: None,
+            key: "input.jsonl".to_string(),
+            sse_customer_algorithm: None,
+            sse_customer_key: None,
+            sse_customer_key_md5: None,
+            request: SelectObjectContentRequest {
+                expression: "SELECT old_marker FROM S3Object".to_string(),
+                expression_type: ExpressionType::from_static(ExpressionType::SQL),
+                input_serialization: InputSerialization {
+                    json: Some(JSONInput {
+                        type_: Some(JSONType::from_static(JSONType::LINES)),
+                    }),
+                    ..Default::default()
+                },
+                output_serialization: OutputSerialization {
+                    json: Some(JSONOutput::default()),
+                    ..Default::default()
+                },
+                request_progress: None,
+                scan_range: None,
+            },
+        })
+    }
+
+    fn parquet_snapshot_input() -> Arc<SelectObjectContentInput> {
+        Arc::new(SelectObjectContentInput {
+            bucket: "s3select-parquet-snapshot-race".to_string(),
+            expected_bucket_owner: None,
+            key: "input.parquet".to_string(),
+            sse_customer_algorithm: None,
+            sse_customer_key: None,
+            sse_customer_key_md5: None,
+            request: SelectObjectContentRequest {
+                expression: "SELECT old_marker FROM S3Object".to_string(),
+                expression_type: ExpressionType::from_static(ExpressionType::SQL),
+                input_serialization: InputSerialization {
+                    parquet: Some(ParquetInput {}),
+                    ..Default::default()
+                },
+                output_serialization: OutputSerialization {
+                    json: Some(JSONOutput::default()),
+                    ..Default::default()
+                },
+                request_progress: None,
+                scan_range: None,
+            },
+        })
+    }
+
+    fn old_parquet_generation() -> Vec<u8> {
+        let schema = Arc::new(Schema::new(vec![Field::new("old_marker", DataType::Utf8, false)]));
+        let first = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(StringArray::from(vec!["parquet-old-footer"]))])
+            .expect("build first old-generation parquet row group");
+        let second = RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(StringArray::from(vec!["parquet-old-row-group"]))])
+            .expect("build second old-generation parquet row group");
+        let mut bytes = Vec::new();
+        let mut writer = ArrowWriter::try_new(&mut bytes, schema, None).expect("create old-generation parquet writer");
+        writer.write(&first).expect("write first old-generation parquet row group");
+        writer.flush().expect("flush first old-generation parquet row group");
+        writer.write(&second).expect("write second old-generation parquet row group");
+        writer.close().expect("close old-generation parquet writer");
+        bytes
+    }
+
+    fn new_parquet_generation() -> Vec<u8> {
+        let schema = Arc::new(Schema::new(vec![Field::new("new_schema_poison", DataType::Int32, false)]));
+        let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(Int32Array::from(vec![1629]))])
+            .expect("build new-generation parquet poison row group");
+        let mut bytes = Vec::new();
+        let mut writer = ArrowWriter::try_new(&mut bytes, schema, None).expect("create new-generation parquet writer");
+        writer.write(&batch).expect("write new-generation parquet poison row group");
+        writer.close().expect("close new-generation parquet writer");
+        bytes
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn json_schema_inference_and_scan_use_one_snapshot_generation() {
+        run_snapshot_generation_race(
+            json_snapshot_input(),
+            b"{\"old_marker\":\"json-old-schema\"}\n{\"old_marker\":\"json-old-scan\"}\n".to_vec(),
+            b"{\"new_schema_poison\":1629}\n".to_vec(),
+            &["json-old-schema", "json-old-scan"],
+        )
+        .await;
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn parquet_footer_and_row_group_reads_use_one_snapshot_generation() {
+        run_snapshot_generation_race(
+            parquet_snapshot_input(),
+            old_parquet_generation(),
+            new_parquet_generation(),
+            &["parquet-old-footer", "parquet-old-row-group"],
+        )
+        .await;
+    }
+
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn planner_failure_drops_snapshot_and_unblocks_overwrite() {
+        let mut input = json_snapshot_input();
+        let input_mut = Arc::make_mut(&mut input);
+        input_mut.bucket = "s3select-planner-failure-snapshot".to_string();
+        input_mut.request.expression = "SELECT missing_binding FROM S3Object".to_string();
+
+        let env = snapshot_test_env().await;
+        env.make_bucket(&input.bucket, false).await;
+        env.put_object_bytes(&input.bucket, &input.key, b"{\"old_marker\":\"old\"}\n".to_vec())
+            .await;
+        let snapshot = env.prepare_select_object_snapshot(&input.bucket, &input.key).await;
+        let dispatcher = production_dispatcher(Arc::clone(&input));
+        let query = Query::new_with_snapshot(
+            QueryContext {
+                input: Arc::clone(&input),
+            },
+            input.request.expression.clone(),
+            snapshot,
+        );
+        let query_state_machine = dispatcher
+            .build_query_state_machine(query.clone())
+            .await
+            .expect("build production snapshot session");
+
+        let commit_barrier = PutObjectCommitBarrier::before_namespace(&input.bucket, &input.key);
+        let writer_env = env;
+        let writer_bucket = input.bucket.clone();
+        let writer_object = input.key.clone();
+        let writer = tokio::spawn(async move {
+            writer_env
+                .put_object_bytes(&writer_bucket, &writer_object, b"{\"new_schema_poison\":1629}\n".to_vec())
+                .await;
+        });
+        commit_barrier.wait_until_paused().await;
+        commit_barrier.release_and_wait_until_namespace_pending().await;
+        assert!(!writer.is_finished(), "overwrite must wait for the planner's snapshot");
+
+        let Err(error) = dispatcher.build_logical_plan(Arc::clone(&query_state_machine)).await else {
+            panic!("missing schema binding must fail planning");
+        };
+        assert!(error.to_string().contains("missing_binding"));
+        assert!(
+            !writer.is_finished(),
+            "planner failure must not release snapshots still owned by Query/qsm"
+        );
+
+        drop(query_state_machine);
+        drop(query);
+        tokio::time::timeout(Duration::from_secs(5), writer)
+            .await
+            .expect("overwrite should finish after failed-plan snapshot release")
+            .expect("overwrite task should join");
+    }
+
     fn test_query_tracker(
         permit: tokio::sync::OwnedSemaphorePermit,
         deadline: Instant,
@@ -1118,6 +1424,42 @@ mod tests {
         ));
     }
 
+    #[tokio::test]
+    async fn reserved_admission_is_handed_to_tracker_without_reacquiring() {
+        let admission = Arc::new(Semaphore::new(1));
+        let (dispatcher, input) = test_dispatcher(Arc::clone(&admission), Duration::from_secs(300));
+        let reservation = dispatcher.try_reserve_query().expect("query reservation should succeed");
+        assert_eq!(admission.available_permits(), 0);
+
+        let query = Query::new(QueryContext { input }, "SELECT * FROM S3Object".to_string());
+        let query_state_machine = dispatcher
+            .build_query_state_machine_inner(query, Some(reservation))
+            .await
+            .expect("reserved query should not acquire a second permit");
+
+        assert_eq!(admission.available_permits(), 0);
+        drop(query_state_machine);
+        assert_eq!(admission.available_permits(), 1);
+    }
+
+    #[tokio::test]
+    async fn reserved_admission_rejects_a_foreign_semaphore() {
+        let admission = Arc::new(Semaphore::new(1));
+        let foreign_admission = Arc::new(Semaphore::new(1));
+        let foreign_permit = Arc::clone(&foreign_admission)
+            .try_acquire_owned()
+            .expect("foreign permit should be available");
+        let reservation = QueryAdmission::new(Arc::new(foreign_permit));
+        let (dispatcher, input) = test_dispatcher(Arc::clone(&admission), Duration::from_secs(300));
+        let query = Query::new(QueryContext { input }, "SELECT * FROM S3Object".to_string());
+
+        let result = dispatcher.build_query_state_machine_inner(query, Some(reservation)).await;
+
+        assert!(matches!(result, Err(QueryError::Cancel)));
+        assert_eq!(admission.available_permits(), 1);
+        assert_eq!(foreign_admission.available_permits(), 1);
+    }
+
     #[tokio::test]
     async fn staged_query_rejects_when_admission_is_saturated() {
         let admission = Arc::new(Semaphore::new(1));

crates/s3select-query/src/instance.rs

@@ -26,7 +26,7 @@ use rustfs_s3select_api::{
         dispatcher::QueryDispatcher,
         execution::QueryStateMachineRef,
         logical_planner::Plan,
-        session::{DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES as DEFAULT_MEMORY_LIMIT_BYTES, SessionCtxFactory},
+        session::{DEFAULT_S3SELECT_MEMORY_LIMIT_BYTES as DEFAULT_MEMORY_LIMIT_BYTES, QueryAdmission, SessionCtxFactory},
     },
     server::dbms::{DatabaseManagerSystem, QueryHandle},
 };
@@ -64,12 +64,22 @@ impl<D> DatabaseManagerSystem for RustFSms<D>
 where
     D: QueryDispatcher,
 {
+    fn try_reserve_query(&self) -> QueryResult<QueryAdmission> {
+        self.query_dispatcher.try_reserve_query()
+    }
+
     async fn execute(&self, query: &Query) -> QueryResult<QueryHandle> {
         let result = self.query_dispatcher.execute_query(query).await?;
 
         Ok(QueryHandle::new(query.clone(), result))
     }
 
+    async fn execute_admitted(&self, query: &Query, admission: QueryAdmission) -> QueryResult<QueryHandle> {
+        let result = self.query_dispatcher.execute_query_admitted(query, admission).await?;
+
+        Ok(QueryHandle::new(query.clone(), result))
+    }
+
     async fn build_query_state_machine(&self, query: Query) -> QueryResult<QueryStateMachineRef> {
         let query_state_machine = self.query_dispatcher.build_query_state_machine(query).await?;
 

crates/storage-api/src/lib.rs

@@ -20,11 +20,23 @@ pub const WALK_DIR_BODY_SHA256_QUERY: &str = "walk_dir_body_sha256";
 pub const PUT_FILE_AUTH_QUERY: &str = "put_file_auth";
 pub const PUT_FILE_AUTH_V1: &str = "digest-trailer-v1";
 pub const PUT_FILE_NONCE_QUERY: &str = "put_file_nonce";
+pub const PUT_FILE_CAPABILITY_QUERY: &str = "put_file_capability";
+pub const PUT_FILE_CAPABILITY_CHALLENGE_QUERY: &str = "put_file_challenge";
+pub const PUT_FILE_CAPABILITY_VERSION: u16 = 1;
+pub const PUT_FILE_SERVER_EPOCH_QUERY: &str = "put_file_server_epoch";
 pub const PUT_FILE_AUTH_TRAILER_MAGIC: &[u8; 16] = b"RFS-PUT-AUTH-V1\0";
 pub const PUT_FILE_AUTH_TRAILER_DIGEST_LEN: usize = 64;
 pub const PUT_FILE_AUTH_TRAILER_MAC_LEN: usize = 32;
 pub const PUT_FILE_AUTH_TRAILER_LEN: usize =
     PUT_FILE_AUTH_TRAILER_MAGIC.len() + PUT_FILE_AUTH_TRAILER_DIGEST_LEN + PUT_FILE_AUTH_TRAILER_MAC_LEN;
+
+#[derive(Debug, serde::Deserialize, serde::Serialize)]
+#[serde(deny_unknown_fields)]
+pub struct PutFileCapabilityResponse {
+    pub version: u16,
+    pub server_epoch: uuid::Uuid,
+    pub proof: Vec<u8>,
+}
 pub const NS_SCANNER_BODY_SHA256_QUERY: &str = "ns_scanner_body_sha256";
 pub const NS_SCANNER_CAPABILITY_CHALLENGE_QUERY: &str = "ns_scanner_challenge";
 pub const NS_SCANNER_CYCLE_QUERY: &str = "ns_scanner_cycle";

crates/targets/Cargo.toml

@@ -59,6 +59,10 @@ hyper = { workspace = true, features = ["http2", "http1", "server"] }
 hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "tls12", "logging", "aws-lc-rs"] }
 lapin = { workspace = true, default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 libc = { workspace = true }
+# pulsar regenerates its protobuf bindings from `PulsarApi.proto` in build.rs on every
+# build, which needs `protoc`. Keep this dependency lean and install a system
+# protobuf compiler on platforms where it is not provided by the base image.
+# See https://github.com/rustfs/rustfs/issues/3195.
 pulsar = { workspace = true, default-features = false, features = ["tokio-rustls-runtime", "telemetry"] }
 regex = { workspace = true }
 rayon = { workspace = true }

crates/test-utils/Cargo.toml

@@ -27,6 +27,7 @@ documentation = "https://docs.rs/rustfs-test-utils/latest/rustfs_test_utils/"
 
 [features]
 default = []
+put-object-commit-barrier = ["rustfs-ecstore/test-util"]
 hotpath = [
     "hotpath/hotpath",
     "hotpath/tokio",
@@ -51,7 +52,7 @@ hotpath-cpu = [
 
 [dependencies]
 hotpath.workspace = true
-rustfs-ecstore = { workspace = true }
+rustfs-ecstore.workspace = true
 rustfs-storage-api = { workspace = true }
 tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tokio-util = { workspace = true, features = ["io", "compat"] }