fix(security): redact OIDC secrets from logs and errors (#5147)

Stop the OIDC subsystem from writing credential-grade secrets into logs and returned errors.

- crates/iam/src/oidc.rs: redact sensitive header values (authorization, proxy-authorization, cookie, set-cookie) in format_http_headers, emitting only name and length; drop the raw request/response body from the DEBUG events (keep byte length); stop logging and stop splicing the raw token response body into the error returned on token_response_parse_failed (the TokenResponseBodyShape summary and length are retained); remove the now-unused format_http_body helper.
- rustfs/src/admin/handlers/oidc.rs: stop logging the raw authorization code and state on the code-exchange error path (code_len/state_len are kept).

This is the OIDC log/error redaction pre-work (batch 0B) from the OIDC review in rustfs/backlog#1437. It changes diagnostic content only; HTTP/STS status codes and legitimate request results are unchanged.

.agents/skills/adversarial-validation/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: adversarial-validation
-description: Execute the Adversarial Validation policy from the root AGENTS.md — run the six reviewer roles (correctness, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
 ---
 
 # Adversarial Validation Playbooks
@@ -53,14 +53,22 @@ shipped bug or rule that earns each probe its place.
 - Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
   - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
   - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
-- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
-  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
-  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Constant and String Usage'; Adversarial Validation section names the smaller-diff clause as a correctness-adversary finding.
 - For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
   - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
   - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
 
-Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, mid-stream reconstruct error propagation, and a minimal-diff rewrite — no break found; diff is already the minimal in-place edit."
+Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
+
+### Simplicity adversary
+
+- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
+- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
+  - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
+
+Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
 
 ### Security reviewer
 
@@ -243,7 +251,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 - If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
   - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
   - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
-- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, and remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent). Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
+- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
   - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
   - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
 - For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.

.agents/skills/code-change-verification/SKILL.md

@@ -43,16 +43,7 @@ Use this skill to review code changes consistently before merge, before release,
 
 #### Rust-specific checks (apply to all Rust changes)
 
-- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
-- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
-- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
-- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
-- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
-- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
-- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
-- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
-- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
-- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
+Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
 
 ### 4) Findings-first output
 - Order findings by severity:

.agents/skills/rust-code-quality/SKILL.md

@@ -36,6 +36,9 @@ rg -n 'println!\|eprintln!' <changed-files> | grep -v test
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
+
+# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
+rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 ```
 
 ## Manual Review Checklist
@@ -55,8 +58,8 @@ For every Rust code change, verify:
 - [ ] No `f64 as usize` without prior clamping
 
 ### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used
-- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
+- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
+- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
 - [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
@@ -84,12 +87,19 @@ For every Rust code change, verify:
 - [ ] No camelCase statics or Hungarian notation
 - [ ] New string literals don't duplicate existing constants
 
+### Reuse and Necessity
+- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
+- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
+- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
+- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
+- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
+
 ## Severity Classification
 
 - **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
-- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
-- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
-- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
 
 ## Output Template
 

.agents/skills/rustfs-release-publish/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: rustfs-release-publish
+description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
+---
+# RustFS Release Publish (preview-validated pipeline)
+
+This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
+
+Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
+
+Pipeline shape:
+
+```
+bump version files to <target> (final version, ONE commit) -> merge
+  -> tag <preview-tag> at that commit -> CI green
+  -> verify release artifacts -> run binary locally + console checks
+  -> validate with latest rc client
+  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
+```
+
+On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
+
+## Required inputs
+
+- Final target version, for example `1.0.0-beta.10`.
+- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
+
+If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
+
+## Semver gate — confirm the target version before touching anything
+
+Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
+
+```
+1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
+```
+
+Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
+
+Rules:
+
+- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
+- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
+- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
+
+## Preview tag naming
+
+- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
+- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
+
+## Hard rules
+
+- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
+- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
+- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
+- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
+- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
+- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
+
+## Phase 0 — Preflight
+
+- `git status --short` clean; `git fetch origin main --tags`.
+- `gh auth status` works; confirm you can view `gh release list -L 3`.
+- Confirm the exact final target version with the user if not explicit.
+
+## Phase 1 — Version bump to the final target (once)
+
+- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
+- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
+- Get the PR merged into main. Record the resulting main commit:
+
+```bash
+git fetch origin main
+PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
+```
+
+`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
+
+## Phase 2 — Publish the preview tag
+
+```bash
+git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
+git push origin "<preview-tag>"
+```
+
+Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
+
+On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
+
+## Phase 3 — CI and artifact verification
+
+- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
+- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
+  - `isPrerelease` must be `true`.
+  - Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
+- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
+- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
+
+## Phase 4 — Run the artifact locally, verify the console
+
+Work inside the session scratchpad directory; never leave stray data dirs.
+
+```bash
+gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
+cd "$SCRATCH" && unzip -o rustfs-*.zip
+./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
+mkdir -p data
+RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
+```
+
+Defaults: S3 endpoint `:9000`, embedded console `:9001`.
+
+Checks (all must pass):
+
+- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
+- `curl -fsS http://localhost:9000/health/ready` returns ready.
+- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
+- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
+
+## Phase 5 — Validate with the latest rc client
+
+`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
+
+- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
+- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
+
+```bash
+rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
+rc ls preview/
+rc mb preview/rel-check
+rc cp <local-file> preview/rel-check/
+rc stat preview/rel-check/<file>
+rc cat preview/rel-check/<file>          # matches source
+rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
+rc cp -r <local-dir>/ preview/rel-check/dir/
+rc find preview/rel-check --name "*"
+rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
+rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
+rc rb preview/rel-check
+rc admin user list preview/
+rc admin user add preview/ relcheckuser relchecksecret12
+rc admin user remove preview/ relcheckuser
+rc alias remove preview
+```
+
+- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
+
+## Phase 6 — Publish the final tag on the validated commit
+
+No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
+
+```bash
+git fetch origin --tags
+git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
+git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
+git push origin "<target>"
+```
+
+- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
+- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
+- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
+
+## Output contract
+
+Always report:
+
+- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
+- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
+- Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -18,6 +18,8 @@ Validated baseline: release pattern used in PR `#2957`.
 
 If target version is missing or ambiguous, stop and ask before editing.
 
+Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+
 ## Read before editing
 
 - `AGENTS.md` (root and nearest path-specific files).

.agents/skills/security-advisory-lessons/SKILL.md

@@ -60,12 +60,14 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 ### IAM and service accounts
 - Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
 - For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
+- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
 - Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
 - Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
 
 ### STS, OIDC, and federation flows
 - Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
 - JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
+- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
 - Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
 - OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
 
@@ -137,7 +139,9 @@ Use these prompts while reviewing a diff:
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
 - Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
+- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
 - Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
+- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
 - Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
 - Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -27,14 +27,15 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### IAM import, service accounts, and privilege boundaries
 
 - `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
 - `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
 - `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
 - `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
 
 ### STS, OIDC, and federation flows
 
-- `GHSA-5qfg-mf7r-jp3w`: `AssumeRoleWithWebIdentity` was reachable without the required request authentication and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption.
-- `GHSA-ccrv-v8v9-ch9q`: service-account-controlled material could self-sign JWT session tokens with forged policy claims. Lesson: session tokens must be signed by a trusted issuer/key path and validation must reject self-signed or principal-controlled tokens.
+- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
+- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
@@ -58,7 +59,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, and `GHSA-9gf3-jx4p-4xxf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -122,6 +123,7 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/lint-fmt.mak

@@ -60,6 +60,11 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
+.PHONY: log-analyzer-rules-check
+log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
+	@echo "🩺 Checking log-analyzer rule anchors..."
+	./scripts/check_log_analyzer_rules.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -23,7 +23,7 @@ pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-gua
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check

.config/make/tests.mak

@@ -26,6 +26,9 @@ script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
 	./scripts/test_entrypoint_credentials.sh
+	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
+	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
+	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
 
 .PHONY: test
 test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)

.config/nextest.toml

@@ -42,7 +42,7 @@ e2e-reliability = { max-threads = 1 }
 
 # --- default profile (local): serialize the flaky groups, never retry --------
 [[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/))'
+filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
 
 # Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
@@ -99,7 +99,7 @@ retries = 2
 # QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
 # make_bucket into InsufficientWriteQuorum via shared global state under load.
 [[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/)'
+filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
 test-group = 'ecstore-serial-flaky'
 retries = 2
 
@@ -138,6 +138,10 @@ test-group = 'ecstore-serial-flaky'
 # Each e2e test spawns its own rustfs server on a random port with an isolated
 # temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
 #
+# Replication failure harness (backlog#1147 repl-8): the first clause admits
+# its four in-process fake-target self-tests. They bind random loopback ports,
+# use no external service, and finish in under a second.
+#
 # Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
 # FAST bucket-replication tests from replication_extension_test — the
 # target-registration / replication-check / list / remove / delete admin paths
@@ -188,7 +192,7 @@ test-group = 'ecstore-serial-flaky'
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud)_test::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools)_test::|^fake_s3_target::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
@@ -259,8 +263,8 @@ path = "junit.xml"
 # The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
 #   * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
 #     ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
-#   * the 6 cluster suites that spin up a RustFSTestClusterEnvironment
-#     (cluster_concurrency, stale_multipart_cleanup_cluster,
+#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
+#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
 #     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
 #     object_lambda) — too heavy for the merge budget; they run in ci-7's
 #     nightly 4-node lane.
@@ -282,14 +286,8 @@ path = "junit.xml"
 # ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
 # OPEN issue, and the fixing PR MUST delete the exclusion. The passing
 # negative-path siblings of each family stay in as regression guards.
-#   * rustfs#4842 — extract/snowball expand pipeline 500s (mtime=0
-#     OffsetDateTime deserialization + same-path failures).
 #   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
 #     archive even under ignore-errors semantics.
-#   * rustfs#4844 — anonymous POST-object with SSE-S3 / bucket-default SSE
-#     returns 500.
-#   * rustfs#4845 — 403 on allowed anonymous POST object-lock fields and on
-#     the list metadata=true extension.
 #   * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
 #     under parallel load (multi-node in-process clusters; natural home is
 #     ci-7's nightly cluster lane).
@@ -297,15 +295,10 @@ path = "junit.xml"
 default-filter = """
     package(e2e_test)
     & !test(/^protocols::/)
-    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
+    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
     & !test(/^replication_extension_test::/)
-    & !test(/^multipart_auth_test::test_signed_put_object_extract_(accepts_compat_header|expands_tar_entries_with_prefix_headers|expands_tar_gz_archive|expands_tbz2_archive|expands_tgz_archive|expands_txz_archive|expands_tzst_archive|normalizes_prefix_header_value|preserves_directory_markers_by_default|preserves_object_lock_legal_hold|preserves_object_lock_retention|preserves_pax_metadata_and_version_id|preserves_request_metadata_on_extracted_objects|preserves_sse_c|preserves_sse_s3_and_redirect|preserves_storage_class|uses_bucket_default_sse_s3)$/)
-    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(prefers_exact_minio_prefix_over_suffix_fallback|supports_minio_prefix_and_directory_markers)$/)
     & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
     & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
-    & !test(/^multipart_auth_test::test_anonymous_post_object_(accepts_sse_s3|rejects_sse_s3_missing_from_policy_conditions|uses_bucket_default_sse_kms|uses_bucket_default_sse_s3)$/)
-    & !test(/^multipart_auth_test::test_anonymous_post_object_(accepts_object_lock_legal_hold_field|accepts_object_lock_retention_fields)$/)
-    & !test(/^list_object(s_v2|_versions)_metadata_extension_test::/)
     & !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
 """
 fail-fast = false

.docker/observability/grafana/dashboards/rustfs.json

@@ -1744,7 +1744,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "sum by (bucket) (rustfs_bucket_api_objects_total{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1844,7 +1844,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "sum by (bucket) (rustfs_bucket_api_usage_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -11583,7 +11583,7 @@
           "text": "All",
           "value": "$__all"
         },
-        "definition": "label_values(rustfs_bucket_api_objects_total,bucket)",
+        "definition": "label_values(rustfs_cluster_usage_buckets_objects_count,bucket)",
         "includeAll": true,
         "label": "Bucket",
         "multi": true,
@@ -11591,7 +11591,7 @@
         "options": [],
         "query": {
           "qryType": 1,
-          "query": "label_values(rustfs_bucket_api_objects_total,bucket)",
+          "query": "label_values(rustfs_cluster_usage_buckets_objects_count,bucket)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
         "refresh": 2,

.docker/test/site-replication/run-object-flow-check.sh

@@ -18,6 +18,7 @@ set -eu
 ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
 SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
 BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
+DELETE_BUCKET="${RUSTFS_SITE_REPL_DELETE_BUCKET:-site-repl-delete-$(date +%Y%m%d-%H%M%S)-$$}"
 PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
 WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
 WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
@@ -85,17 +86,39 @@ wait_for_object() {
 
 wait_for_bucket() {
   site="$1"
+  bucket="${2:-$BUCKET}"
   attempt=1
 
   while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
-    if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
+    if mc stat "$site/$bucket" >/dev/null 2>&1; then
       return 0
     fi
     sleep "$WAIT_SLEEP_SECONDS"
     attempt=$((attempt + 1))
   done
 
-  echo "bucket was not replicated in time: $site/$BUCKET" >&2
+  echo "bucket was not replicated in time: $site/$bucket" >&2
+  return 1
+}
+
+wait_for_bucket_delete() {
+  site="$1"
+  bucket="$2"
+  attempt=1
+
+  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+    if result="$(mc stat --json "$site/$bucket" 2>&1)"; then
+      :
+    else
+      case "$result" in
+        *NoSuchBucket*) return 0 ;;
+      esac
+    fi
+    sleep "$WAIT_SLEEP_SECONDS"
+    attempt=$((attempt + 1))
+  done
+
+  echo "bucket deletion was not replicated in time: $site/$bucket" >&2
   return 1
 }
 
@@ -186,6 +209,20 @@ EOF
   echo "verified replicated downloads for $object_name"
 done
 
+echo "creating empty bucket for replicated delete check: $DELETE_BUCKET"
+mc mb "site1/$DELETE_BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket "$site" "$DELETE_BUCKET"
+done
+
+echo "deleting empty bucket on site1: $DELETE_BUCKET"
+mc rb "site1/$DELETE_BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket_delete "$site" "$DELETE_BUCKET"
+done
+
 echo "site replication object flow check passed"
 echo "bucket: $BUCKET"
 echo "prefix: $PREFIX"

.github/cla.yml

@@ -15,8 +15,8 @@
 enabled: true
 
 document:
-  version: v1
-  url: https://github.com/rustfs/cla/blob/main/cla/v1.md
+  version: v2
+  url: https://github.com/rustfs/cla/blob/main/cla/v2.md
 
 signing:
   mode: comment

.github/pull_request_template.md

@@ -33,4 +33,4 @@ documentation impact. Use N/A when there is no expected impact.
 
 ---
 
-Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
+Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v2.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.

.github/workflows/build.yml

@@ -289,6 +289,7 @@ jobs:
             local console_url
             local console_sha256
             local curl_auth_args=()
+            console_tag=""
 
             if [[ "${{ matrix.platform }}" == "windows" ]]; then
               curl_bin="curl.exe"
@@ -311,7 +312,7 @@ jobs:
             "$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
               -o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
 
-            read -r console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
+            read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
           import json
           import re
           import sys
@@ -319,6 +320,8 @@ jobs:
           with open(sys.argv[1], encoding="utf-8") as handle:
               release = json.load(handle)
 
+          tag = release.get("tag_name", "") or "unknown"
+
           for asset in release.get("assets", []):
               name = asset.get("name", "")
               digest = asset.get("digest", "")
@@ -327,7 +330,7 @@ jobs:
                   sha256 = digest.split(":", 1)[1]
                   if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
                       raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
-                  sys.stdout.buffer.write(f"{url} {sha256}\n".encode("utf-8"))
+                  sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
                   break
           else:
               raise SystemExit("no console zip asset with sha256 digest found")
@@ -339,6 +342,8 @@ jobs:
               return 1
             fi
 
+            echo "Console release: ${console_tag}"
+            echo "Downloading console asset: ${console_url}"
             "$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
             verify_sha256 "$console_sha256" console.zip || return 2
             unzip -o console.zip -d ./rustfs/static || return 2
@@ -351,12 +356,19 @@ jobs:
             rm -f console.zip console-release.json
             if [[ "$status" -eq 2 ]]; then
               echo "Console asset integrity verification failed" >&2
-              exit 1
             fi
-            echo "Warning: Failed to download verified console assets, continuing without them"
-            echo "// Static assets not available" > ./rustfs/static/empty.txt
+            echo "Failed to download verified console assets" >&2
+            exit 1
           fi
 
+          if [[ ! -s ./rustfs/static/index.html ]]; then
+            echo "Console asset archive is missing static/index.html" >&2
+            exit 1
+          fi
+
+          asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
+          echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
+
       - name: Build RustFS
         shell: bash
         run: |
@@ -558,6 +570,60 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
+      - name: Verify packaged console
+        if: matrix.target == 'x86_64-unknown-linux-gnu'
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          find_free_port() {
+            python3 - <<'PY'
+          import socket
+
+          with socket.socket() as sock:
+              sock.bind(("127.0.0.1", 0))
+              print(sock.getsockname()[1])
+          PY
+          }
+
+          package_dir="$(mktemp -d)"
+          data_dir="$(mktemp -d)"
+          log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
+          server_pid=""
+          trap '
+            if [[ -n "${server_pid:-}" ]]; then
+              kill "$server_pid" 2>/dev/null || true
+              wait "$server_pid" 2>/dev/null || true
+            fi
+            rm -rf "$package_dir" "$data_dir"
+          ' EXIT
+
+          unzip -q "${{ steps.package.outputs.package_file }}" -d "$package_dir"
+          api_port="$(find_free_port)"
+          console_port="$(find_free_port)"
+          console_url="http://127.0.0.1:${console_port}/rustfs/console/"
+
+          "$package_dir/rustfs" server \
+            --address "127.0.0.1:${api_port}" \
+            --console-enable \
+            --console-address "127.0.0.1:${console_port}" \
+            --access-key console-smoke \
+            --secret-key console-smoke-secret \
+            "$data_dir" >"$log_file" 2>&1 &
+          server_pid=$!
+
+          for _ in {1..40}; do
+            response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
+            if [[ "$response" == 200\ text/html* ]]; then
+              exit 0
+            fi
+            sleep 0.25
+          done
+
+          echo "Console endpoint did not return 200 text/html: $response" >&2
+          cat "$log_file"
+          exit 1
+
       - name: Upload to GitHub artifacts
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:

.github/workflows/ci.yml

@@ -167,6 +167,13 @@ jobs:
           cargo nextest run --profile ci --all --exclude e2e_test
           cargo test --all --doc
 
+      # rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
+      # verbatim in the source tree (log message drifted without updating the
+      # rule). Placed here where the workspace — including the la-dump-anchors
+      # bin — is already built by the clippy/test steps above.
+      - name: Check log-analyzer rule anchors
+        run: ./scripts/check_log_analyzer_rules.sh
+
       - name: Upload test junit report
         if: always()
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -221,29 +228,23 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      # The #4877 restore self-deadlock is fixed in this PR, which re-enabled
-      # test_multipart_restore_preserves_parts_and_etag. The remaining exclusions
-      # each hit a DIFFERENT, independent issue (all tracked under
-      # rustfs/backlog#1148; they keep #[ignore] with a backlog reference):
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract). The remaining exclusions each hit a
+      # DIFFERENT, independent issue (all tracked under rustfs/backlog#1148;
+      # they keep #[ignore] with a backlog reference):
       #   - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
       #     noncurrent transition/expiry after an immediate compensation transition.
-      #   - test_transition_and_restore_flows: transition metadata is missing on
-      #     one drive (assert_transition_meta_consistent: "missing xl.meta ... on
-      #     disk2") - an EC metadata-distribution issue, not the restore lock.
-      #   - test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore:
-      #     DeleteRestoredAction sets opts.transition.expire_restored, but no
-      #     delete path reads that flag, so cleanup deletes the whole object
-      #     instead of only the local restored copy (ObjectNotFound afterwards).
-      #     The expire_restored delete semantics are unimplemented.
-      #   - restore_object_usecase_reports_ongoing_conflict_and_completion: asserts
-      #     a concurrent get_object_info observes ongoing-request=true mid-restore,
-      #     which #4877's read-vs-restore serialization rules out (see backlog#1148
-      #     ilm-8 criterion 1 - an API-semantics decision, not a bug).
       - name: Run ignored ILM integration tests serially
         run: |
           cargo nextest run -j1 --run-ignored ignored-only \
             -p rustfs-scanner -p rustfs \
-            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition) or test(restore_object_usecase_reports_ongoing_conflict_and_completion) or test(test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore))'
+            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
 
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)

.github/workflows/star-history.yml

@@ -0,0 +1,26 @@
+name: Star History
+
+on:
+  schedule:
+    - cron: "17 3 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: star-history
+  cancel-in-progress: false
+
+jobs:
+  update:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
+        with:
+          github-token: ${{ github.token }}
+          output-branch: star-history
+          output-path: .
+          chart-style: gradient
+          animate: "true"
+          contributors: "true"

AGENTS.md

@@ -14,7 +14,7 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 
 ## Execution Discipline
 
-- Read the relevant existing code, tests, and local guidance before changing behavior.
+- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write).
 - State assumptions when they affect the implementation or verification path.
 - If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
 - For multi-step work, keep the plan minimal and tied to verifiable outcomes.
@@ -41,14 +41,27 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - Do not refactor existing code only to make it easier to unit test.
 - Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
 - Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
 - Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
-## Constant and String Usage
+## Reuse Before You Write
 
-- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
-- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
-- If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
-- When changing existing behavior, keep naming and format consistency by aligning with established project constants.
+Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
+
+- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
+- **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
+- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
+- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
+
+## Necessary Code Only
+
+Net-new code — files, types, branches, comments — is cost to justify, not progress:
+
+- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
+- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
+- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
+- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success.
+- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value.
 
 ## Sources of Truth
 
@@ -141,7 +154,7 @@ Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 - **Exempt:** docs/comments/instruction-only changes, formatting, typos with
   no runtime surface. Skip this section.
 - **Mechanical:** pure renames, file moves, test-only or tooling changes —
-  correctness adversary only.
+  correctness and simplicity adversaries only.
 - **Standard (the default):** any change that affects behavior.
 - **High risk:** touches locking, erasure coding, quorum/heal, replication,
   multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`),
@@ -161,9 +174,8 @@ encode this repo's shipped bugs.
 
 - **Correctness adversary** — construct a concrete input/state/interleaving
   that yields wrong output, data loss, or a crash. Probe error paths and edge
-  values (empty, nil UUID, zero-length, quorum−1, missing version). For code
-  diffs, a materially smaller or more idiomatic diff achieving the same
-  behavior is also a finding (see Change Style for Existing Logic).
+  values (empty, nil UUID, zero-length, quorum−1, missing version).
+- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
 - **Security reviewer** — authn/authz bypass, injection, secret leakage,
   untrusted deserialization (see Serde Safety), path traversal, timing leaks.
 - **Concurrency/durability reviewer** — lock ordering, races, cancellation,
@@ -179,11 +191,12 @@ encode this repo's shipped bugs.
   wrong while all tests stay green — if one exists, coverage is insufficient.
   A missing test is a finding, not a note.
 
-Standard tier: correctness adversary + test-coverage skeptic, plus every
-role whose domain the diff touches (async or shared-state code →
-concurrency; parsing of untrusted input → security; public crate API shape
-→ compatibility; per-request or per-object hot paths → performance).
-High risk: all six roles.
+Standard tier: correctness adversary + simplicity adversary + test-coverage
+skeptic, plus every role whose domain the diff touches (async or
+shared-state code → concurrency; parsing of untrusted input → security;
+public crate API shape → compatibility; per-request or per-object hot paths
+→ performance).
+High risk: all seven roles.
 
 ### Protocol
 

ARCHITECTURE.md

@@ -41,7 +41,7 @@ The repository is a Cargo workspace with a flat `crates/` layout:
 
 ```
 rustfs/                      # Workspace root (virtual manifest)
-├── rustfs/                  # Main binary + library crate (75K lines)
+├── rustfs/                  # Main binary + library crate
 │   └── src/
 │       ├── main.rs          # Entry point, startup sequence
 │       ├── lib.rs           # Module tree root
@@ -53,7 +53,7 @@ rustfs/                      # Workspace root (virtual manifest)
 │       ├── config/          # CLI args, config parsing, workload profiles
 │       └── ...
 ├── crates/                  # library crates (authoritative list: Cargo.toml [workspace].members)
-│   ├── ecstore/             # Erasure-coded storage engine (⚠️ 87K lines)
+│   ├── ecstore/             # Erasure-coded storage engine
 │   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
 │   ├── io-core/             # Zero-copy I/O, scheduling, buffer pool
 │   ├── io-metrics/          # I/O metrics collection
@@ -83,124 +83,25 @@ A request flows **downward** through the layers. No layer should reach upward
 
 ### Crate Reference
 
-> Depth levels, line counts, and crate counts in this section are a
-> point-in-time snapshot and drift with refactors. Treat them as orders of
-> magnitude; `Cargo.toml` and `cargo tree` are the source of truth.
-
-Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
-
-```
-Depth 0 — LEAF (no internal deps):
-  appauth, checksums, config, credentials, crypto, io-metrics,
-  madmin, s3-common, workers, zip
-
-Depth 1:
-  io-core (→ io-metrics)
-  policy (→ config, credentials, crypto)
-  utils                                   (historical → config edge removed; now effectively leaf)
-
-Depth 2:
-  concurrency, filemeta, keystone, kms, lock, obs,
-  signer, targets, trusted-proxies
-
-Depth 3:
-  common                                  (historical → filemeta/madmin edges removed; now effectively leaf)
-
-Depth 4:
-  object-capacity, protos, rio
-
-Depth 5 — CORE:
-  ecstore (16 internal deps, 11 dependents — the architectural heart)
-
-Depth 6:
-  audit, heal, iam, metrics, notify, s3select-api, scanner
-
-Depth 7:
-  object-io, protocols, s3select-query
-
-Depth 8 — TOP:
-  rustfs (35 internal deps — the binary, depends on almost everything)
-```
+`Cargo.toml` is the authoritative workspace membership and `cargo tree` is the
+authoritative dependency graph. This overview deliberately avoids line-count
+and dependency-depth snapshots because both quickly become stale during
+refactors.
 
 #### By Domain
 
-**Core Infrastructure:**
+| Domain | Current workspace crates | Responsibility |
+|--------|--------------------------|----------------|
+| Foundation | `checksums`, `common`, `config`, `data-usage`, `utils` | Shared configuration, data-usage models, utilities, and checksums. |
+| I/O and storage | `concurrency`, `ecstore`, `filemeta`, `heal`, `io-core`, `io-metrics`, `lifecycle`, `lock`, `object-capacity`, `object-data-cache`, `replication`, `rio`, `rio-v2`, `scanner`, `storage-api` | Erasure-coded object storage, metadata, recovery, lifecycle, replication, locking, cache, and I/O pipelines. |
+| Security and identity | `credentials`, `crypto`, `iam`, `keystone`, `kms`, `policy`, `security-governance`, `signer`, `tls-runtime`, `trusted-proxies` | Credentials, authentication, authorization, encryption, key management, TLS, and security contracts. |
+| Protocols and contracts | `extension-schema`, `madmin`, `protos`, `protocols`, `s3-ops`, `s3-types`, `s3select-api`, `s3select-query` | Admin, inter-node, S3, S3 Select, and optional protocol contracts. |
+| Operations and integration | `audit`, `notify`, `obs`, `targets`, `zip` | Auditing, observability, event delivery, notification targets, and archive support. |
+| Test support | `e2e_test`, `test-utils` | End-to-end validation and shared test bootstrap utilities. |
 
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `config` | 3.3K | Configuration types and environment parsing |
-| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
-| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
-| `madmin` | 5.5K | Admin API request/response types |
-
-**I/O Pipeline:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
-| `io-metrics` | 4.5K | I/O operation metrics and counters |
-| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
-| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
-| `concurrency` | 0.8K | Shared concurrency contract types: workload admission snapshots, worker-slot pool, policy types (runtime control lives in `rustfs/src/storage`) |
-
-**Storage Engine:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
-| `filemeta` | 10K | File/object metadata types and versioning |
-| `checksums` | 732 | Checksum computation |
-| `lock` | 7.1K | Distributed lock manager |
-| `heal` | 5.9K | Data healing / bitrot repair |
-| `scanner` | 5.4K | Background data usage scanner |
-| `object-capacity` | 2.5K | Capacity tracking and management |
-
-**Security & Auth:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `crypto` | 1.6K | Encryption primitives |
-| `credentials` | 713 | Credential types (access key / secret key) |
-| `signer` | 1.4K | S3 v4 request signing |
-| `iam` | 9.0K | Identity and access management |
-| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
-| `kms` | 8.1K | Key management service integration |
-| `keystone` | 1.9K | OpenStack Keystone auth |
-| `appauth` | 143 | Application-level auth tokens |
-
-**Protocol & API:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
-| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
-| `s3-common` | 738 | Shared S3 types |
-| `s3select-api` | 1.9K | S3 Select interface |
-| `s3select-query` | 3.6K | S3 Select query engine |
-
-**Observability:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `metrics` | 8.4K | Prometheus metric collectors |
-| `io-metrics` | 4.5K | I/O-specific metrics |
-| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
-| `audit` | 2.4K | Audit logging |
-
-**Events:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `notify` | 5.5K | Event notification system |
-| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
-
-**Other:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
-| `zip` | 986 | ZIP archive support for bulk downloads |
-| `workers` | 136 | Simple worker abstraction |
+The `rustfs` binary crate composes these libraries into the running server.
+`ecstore` remains the storage engine at the architectural center; its internal
+module split is tracked under `docs/architecture/`.
 
 ## Architecture Invariants
 
@@ -212,7 +113,7 @@ Depth 8 — TOP:
    No upward imports.
 
 2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
-   `io-metrics`, `madmin`, `s3-common` should depend only on external crates.
+   `io-metrics`, and `madmin` should depend only on external crates.
    - ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin`
      edges were removed; do not reintroduce them (see Known Structural Issues).
 

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
+- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
 
 ### Added
 - **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
@@ -38,6 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
+- **Storage-class validation on startup (upgrade note)**: A persisted explicit storage class (`RUSTFS_STORAGE_CLASS_STANDARD` / `RUSTFS_STORAGE_CLASS_RRS`, for example `EC:2`) is now validated against the actual per-pool drive counts at startup and rejected when a pool cannot satisfy it. This is fail-closed and correct, but a cluster that persisted a storage class larger than a small or heterogeneous pool can hold (for example `EC:2` alongside a 2-drive pool), which earlier releases accepted and silently resolved to an invalid layout, will now refuse to start after upgrade. To recover, unset `RUSTFS_STORAGE_CLASS_STANDARD` so the server derives a valid per-pool default automatically, or set it to a value every pool can satisfy.
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
 - **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.

Cargo.toml

@@ -31,6 +31,7 @@ members = [
     "crates/lifecycle", # Lifecycle rule evaluation contracts
     "crates/kms", # Key Management Service
     "crates/lock", # Distributed locking implementation
+    "crates/log-analyzer", # Offline log fault-analysis core (rustfs diagnose)
     "crates/madmin", # Management dashboard and admin API interface
     "crates/notify", # Notification system for events
     "crates/obs", # Observability utilities
@@ -68,7 +69,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.96.0"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.11"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -85,51 +86,52 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.10-preview.1" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.10-preview.1" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.10-preview.1" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.10-preview.1" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.10-preview.1" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.10-preview.1" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.10-preview.1" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.10-preview.1" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.10-preview.1" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.10-preview.1" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.10-preview.1" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.10-preview.1" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.10-preview.1" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.10-preview.1" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.10-preview.1" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.10-preview.1" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.10-preview.1" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.10-preview.1" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.10-preview.1" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.10-preview.1" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.10-preview.1" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.10-preview.1" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.10-preview.1" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.10-preview.1" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.10-preview.1" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.10-preview.1" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.10-preview.1" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.10-preview.1" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.10-preview.1" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.10-preview.1" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.10-preview.1" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.10-preview.1" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.10-preview.1" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.10-preview.1" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.10-preview.1" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.10-preview.1" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.10-preview.1" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.10-preview.1" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.10-preview.1" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.10-preview.1" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.10-preview.1" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.11" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.11" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.11" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.11" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.11" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.11" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.11" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.11" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.11" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.11" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.11" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.11" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.11" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.11" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.11" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.11" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.11" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.11" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.11" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.11" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.11" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.11" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.11" }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.11" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.11" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.11" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.11" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.11" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.11" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.11" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.11" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.11" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.11" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.11" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.11" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.11" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.11" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.11" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.11" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.11" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.11" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.11" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.11" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.11" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.11" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.11" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -137,31 +139,31 @@ async_zip = { default-features = false, version = "0.0.18" }
 mysql_async = { default-features = false, version = "0.37" }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
-async-trait = "0.1.89"
-async-nats = "0.49.1"
+async-trait = "0.1.91"
+async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
-futures = "0.3.32"
-futures-core = "0.3.32"
+futures = "0.3.33"
+futures-core = "0.3.33"
 futures-lite = "2.6.1"
-futures-util = "0.3.32"
+futures-util = "0.3.33"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.8.0" }
 lapin = { default-features = false, version = "4.10.0" }
-hyper = { version = "1.10.1" }
+hyper = { version = "1.11.0" }
 hyper-rustls = { default-features = false, version = "0.27.9" }
 hyper-util = { version = "0.1.20" }
 http = "1.4.2"
 http-body = "1.1.0"
 http-body-util = "0.1.4"
 minlz = "1.2.3"
-reqwest = { default-features = false, version = "0.13.4" }
+reqwest = "0.13.4"
 rustfs-kafka-async = { version = "1.2.0" }
 socket2 = { version = "0.6.5" }
-tokio = { version = "1.52.3" }
+tokio = { version = "1.53.1" }
 tokio-rustls = { default-features = false, version = "0.26.4" }
-tokio-stream = { version = "0.1.18" }
+tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.5"
-tokio-util = { version = "0.7.18" }
+tokio-util = { version = "0.7.19" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
@@ -179,8 +181,8 @@ prost = "0.14.4"
 quick-xml = "0.41.0"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
-serde = { version = "1.0.228" }
-serde_json = { version = "1.0.150" }
+serde = { version = "1.0.229" }
+serde_json = { version = "1.0.151" }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -209,8 +211,8 @@ zeroize = { version = "1.9.0" }
 # Time and Date
 chrono = { version = "0.4.45" }
 humantime = "2.4.0"
-jiff = { version = "0.2.32" }
-time = { version = "0.3.53" }
+jiff = { version = "0.2.34" }
+time = { version = "0.3.54" }
 
 # Database
 deadpool-postgres = { version = "0.14" }
@@ -218,21 +220,21 @@ tokio-postgres = { default-features = false, version = "0.7.18" }
 tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
-anyhow = "1.0.103"
+anyhow = "1.0.104"
 arc-swap = "1.9.2"
-astral-tokio-tar = "0.6.3"
+astral-tokio-tar = "0.6.4"
 atoi = "3.1.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.9.0" }
+aws-config = { version = "1.10.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-s3 = { default-features = false, version = "1.138.0" }
+aws-sdk-s3 = { default-features = false, version = "1.139.0" }
 aws-smithy-http-client = { default-features = false, version = "1.2.0" }
 aws-smithy-runtime-api = { version = "1.13.0" }
 aws-smithy-types = { version = "1.6.1" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.2" }
+clap = { version = "4.6.4" }
 const-str = { version = "1.1.0" }
 convert_case = "0.11.0"
 criterion = { version = "0.8" }
@@ -242,10 +244,10 @@ crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
 datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
 derive_builder = "0.20.2"
-enumset = "1.1.13"
+enumset = "1.1.14"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
-glob = "0.3.3"
+glob = "0.3.4"
 google-cloud-storage = "1.16.0"
 google-cloud-auth = "1.14.0"
 hashbrown = { version = "0.17.1" }
@@ -254,7 +256,7 @@ hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 ipnetwork = { version = "0.21.1" }
 lazy_static = "1.5.0"
-libc = "0.2.186"
+libc = "0.2.189"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
 memmap2 = "0.9.11"
@@ -277,15 +279,14 @@ rand = { version = "0.10.2" }
 ratelimit = "0.10.1"
 rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.0" }
-#reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
-rumqttc = { package = "rumqttc-next", version = "0.33.2" }
-redis = { version = "1.4.0" }
+rumqttc = { package = "rumqttc-next", version = "0.33.3" }
+redis = { version = "1.4.1" }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "ce69c3f10824535c7c24b2f71cdb2aaa4dffb5e0" }
+s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "a5471625975f5014f7b28eee7e4d801f1b32f529" }
 serial_test = "3.5.0"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
@@ -298,7 +299,7 @@ sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.18"
+thiserror = "2.0.19"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
@@ -309,9 +310,10 @@ url = "2.5.8"
 urlencoding = "2.1.3"
 uuid = { version = "1.24.0" }
 vaultrs = { version = "0.8.0" }
+tar = "0.4.46"
 walkdir = "2.5.0"
 windows = { version = "0.62.2" }
-xxhash-rust = { version = "0.8.17" }
+xxhash-rust = { version = "0.8.18" }
 zip = "8.6.0"
 zstd = "0.13.3"
 
@@ -324,22 +326,22 @@ opentelemetry-otlp = { version = "0.32.0" }
 opentelemetry_sdk = { version = "0.32.1" }
 opentelemetry-semantic-conventions = { version = "0.32.1" }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.1.0" }
+pyroscope = { version = "2.1.1" }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
 suppaftp = { version = "10.0.1" }
-rcgen = "0.14.8"
-russh = { version = "0.62.2" }
+rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
+russh = { version = "0.62.3" }
 russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = "0.1"
-hotpath = "0.21"
+mimalloc = "0.1.52"
+hotpath = "0.21.5"
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 
@@ -368,21 +370,3 @@ inherits = "release"
 inherits = "release"
 debug = true
 strip = "none"
-
-# Pin hyper to a revision that carries the HTTP/1 "flush buffered data before
-# shutdown" fix (hyperium/hyper#4018, commit 72046cc7). This lands as a
-# `[patch.crates-io]` entry — not on the `hyper` workspace dependency — so that
-# every consumer in the tree, including the transitive `hyper-util` server path
-# (`conn::auto` / `GracefulShutdown`) that actually drives our connections,
-# resolves to the fixed hyper rather than the buggy crates.io copy.
-#
-# hyper <= 1.10.1 can call `poll_shutdown()` on the socket while response bytes
-# are still buffered (a prior `poll_flush()` returned `Poll::Pending` and the
-# result was discarded). A backpressured / slow-reading peer then receives a
-# graceful FIN before the full Content-Length body is flushed, which standard S3
-# clients (minio-go / warp) report as `unexpected EOF` on large-object GET under
-# load. The fix is not in any crates.io release yet as of hyper 1.10.1; drop
-# this patch once a released version (> 1.10.1) contains commit 72046cc7.
-# See rustfs/backlog#1232.
-[patch.crates-io]
-hyper = { git = "https://github.com/hyperium/hyper.git", rev = "ccc1e850dc0cda3e71b0acd11f60ca3d48d09034" }

Dockerfile

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM alpine:3.23.4 AS build
+FROM alpine:3.24.1 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -70,7 +70,7 @@ RUN set -eux; \
     rm -rf rustfs.zip /build/.tmp || true
 
 
-FROM alpine:3.23.4
+FROM alpine:3.24.1
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -88,7 +88,9 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-RUN apk update && \
+# Upgrade base-image packages so published images pick up security fixes
+# (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
+RUN apk upgrade --no-cache && \
     apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

Dockerfile.glibc

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:24.04 AS build
+FROM ubuntu:26.04 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -76,7 +76,7 @@ RUN set -eux; \
     chmod +x /build/rustfs; \
     rm -rf rustfs.zip /build/.tmp || true
 
-FROM ubuntu:24.04
+FROM ubuntu:26.04
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -93,7 +93,10 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-RUN apt-get update && apt-get install -y --no-install-recommends \
+# Upgrade base-image packages so published images pick up security fixes
+# (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
+RUN apt-get update && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
     && rm -rf /var/lib/apt/lists/*

Dockerfile.source

@@ -208,7 +208,7 @@ CMD ["cargo", "run", "--bin", "rustfs", "--"]
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
-FROM ubuntu:24.04
+FROM ubuntu:26.04
 
 ARG BUILD_DATE
 ARG VCS_REF
@@ -223,6 +223,7 @@ LABEL name="RustFS (dev-local)" \
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \
     apt-get update; \
+    apt-get upgrade -y; \
     apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10-preview.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -218,7 +218,10 @@ For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status,
 and single-node single-disk idle CPU tuning, see
 [Scanner Runtime Controls](docs/operations/scanner-runtime-controls.md). For
 repeatable scanner-pressure validation, see
-[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md).
+[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md). For
+drive timeout knobs on slow storage — including the walk stall budget that
+governs `ListObjects` on large prefixes — see
+[Drive Timeout Tuning](docs/operations/drive-timeout-tuning.md).
 
 ### 5\. Nix Flake (Option 5)
 
@@ -335,12 +338,18 @@ If you have any questions or need assistance:
 RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS contributors">
+</picture>
 </a>
 
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS star history chart">
+</picture>
 
 ## License
 

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10-preview.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
@@ -247,12 +247,18 @@ rustfs --help
 RustFS 是一个社区驱动的项目，我们感谢所有的贡献。请查看 [贡献者](https://github.com/rustfs/rustfs/graphs/contributors) 页面，看看那些让 RustFS 变得更好的了不起的人们。
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS 贡献者">
+</picture>
 </a>
 
 ## Star 历史
 
-[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS Star 历史图表">
+</picture>
 
 ## 许可证
 

crates/audit/src/pipeline.rs

@@ -292,8 +292,8 @@ impl AuditPipeline {
     }
 
     pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
-        let registry = self.registry.lock().await;
-        registry.runtime_manager().health_snapshots().await
+        let targets = self.registry.lock().await.list_target_values();
+        rustfs_targets::health_snapshots_for_targets(targets).await
     }
 }
 
@@ -570,7 +570,7 @@ mod tests {
     use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
     use rustfs_targets::{StoreError, Target, TargetError};
     use std::sync::Arc;
-    use tokio::sync::Mutex;
+    use tokio::sync::{Mutex, Notify};
 
     /// Mock target whose `save()` outcome is fixed at construction so tests can
     /// force full-success / full-failure / partial-failure fan-outs.
@@ -578,6 +578,7 @@ mod tests {
     struct MockTarget {
         id: TargetID,
         fail: bool,
+        health_gate: Option<(Arc<Notify>, Arc<Notify>)>,
     }
 
     impl MockTarget {
@@ -585,8 +586,14 @@ mod tests {
             Self {
                 id: TargetID::new(id.to_string(), "webhook".to_string()),
                 fail,
+                health_gate: None,
             }
         }
+
+        fn with_health_gate(mut self, started: Arc<Notify>, release: Arc<Notify>) -> Self {
+            self.health_gate = Some((started, release));
+            self
+        }
     }
 
     #[async_trait]
@@ -599,6 +606,10 @@ mod tests {
         }
 
         async fn is_active(&self) -> Result<bool, TargetError> {
+            if let Some((started, release)) = &self.health_gate {
+                started.notify_one();
+                release.notified().await;
+            }
             Ok(true)
         }
 
@@ -673,6 +684,24 @@ mod tests {
         pipeline.dispatch(entry()).await.expect("no targets should return Ok");
     }
 
+    #[tokio::test]
+    async fn health_probe_does_not_hold_the_registry_lock() {
+        let started = Arc::new(Notify::new());
+        let release = Arc::new(Notify::new());
+        let pipeline = pipeline_with(vec![MockTarget::new("blocked", false).with_health_gate(started.clone(), release.clone())]);
+        let registry = Arc::clone(&pipeline.registry);
+        let snapshot_task = tokio::spawn(async move { pipeline.snapshot_target_health().await });
+        started.notified().await;
+
+        let guard = tokio::time::timeout(std::time::Duration::from_secs(1), registry.lock())
+            .await
+            .expect("network health probe must not retain the audit registry lock");
+        drop(guard);
+        release.notify_one();
+
+        assert_eq!(snapshot_task.await.expect("snapshot task should finish").len(), 1);
+    }
+
     // backlog#962: dispatch_batch must mirror dispatch and propagate a
     // whole-batch loss instead of returning Ok.
     #[tokio::test]

crates/common/src/globals.rs

@@ -54,6 +54,15 @@ pub async fn get_global_local_node_name() -> String {
     GLOBAL_LOCAL_NODE_NAME.read().await.clone()
 }
 
+/// Read the local node name without waiting for initialization or a writer.
+pub fn try_get_global_local_node_name() -> Option<String> {
+    GLOBAL_LOCAL_NODE_NAME
+        .try_read()
+        .ok()
+        .map(|name| name.clone())
+        .filter(|name| !name.is_empty())
+}
+
 /// Set the global RustFS initialization time to the current UTC time.
 pub async fn set_global_init_time_now() {
     let now = Utc::now();

crates/common/src/heal_channel.rs

@@ -243,6 +243,19 @@ pub enum HealAdmissionResult {
     Dropped(HealAdmissionDropReason),
 }
 
+/// Admission decision together with the canonical task identifier.
+///
+/// A merged request must return the identifier of the task that already owns
+/// the work instead of exposing the discarded request identifier as a new
+/// client token.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct HealAdmissionReceipt {
+    /// Admission decision for the submitted request.
+    pub result: HealAdmissionResult,
+    /// Canonical identifier of the accepted or merged task.
+    pub task_id: String,
+}
+
 impl HealAdmissionResult {
     pub fn result_label(self) -> &'static str {
         match self {
@@ -382,8 +395,25 @@ pub type HealChannelSender = mpsc::UnboundedSender<HealChannelCommand>;
 /// Heal channel receiver
 pub type HealChannelReceiver = mpsc::UnboundedReceiver<HealChannelCommand>;
 
+/// Canonical-receipt start command kept separate from the legacy public enum.
+#[derive(Debug)]
+pub struct HealReceiptCommand {
+    /// Heal request to admit.
+    pub request: HealChannelRequest,
+    /// Completion channel for the admission receipt.
+    pub response_tx: oneshot::Sender<Result<HealAdmissionReceipt, String>>,
+}
+
+/// Canonical-receipt command receiver.
+pub type HealReceiptReceiver = mpsc::UnboundedReceiver<HealReceiptCommand>;
+
+struct HealChannelSenders {
+    command: HealChannelSender,
+    receipt: mpsc::UnboundedSender<HealReceiptCommand>,
+}
+
 /// Global heal channel sender
-static GLOBAL_HEAL_CHANNEL_SENDER: OnceLock<HealChannelSender> = OnceLock::new();
+static GLOBAL_HEAL_CHANNEL_SENDERS: OnceLock<HealChannelSenders> = OnceLock::new();
 
 type HealResponseSender = broadcast::Sender<HealChannelResponse>;
 
@@ -392,17 +422,24 @@ static GLOBAL_HEAL_RESPONSE_SENDER: OnceLock<HealResponseSender> = OnceLock::new
 
 /// Initialize global heal channel
 pub fn init_heal_channel() -> Result<HealChannelReceiver, &'static str> {
-    let (tx, rx) = mpsc::unbounded_channel();
-    if GLOBAL_HEAL_CHANNEL_SENDER.set(tx).is_ok() {
-        Ok(rx)
-    } else {
-        Err("Heal channel sender already initialized")
-    }
+    let (receiver, receipt_receiver) = init_heal_channels()?;
+    drop(receipt_receiver);
+    Ok(receiver)
+}
+
+/// Initialize the legacy command and canonical-receipt channels atomically.
+pub fn init_heal_channels() -> Result<(HealChannelReceiver, HealReceiptReceiver), &'static str> {
+    let (command, command_receiver) = mpsc::unbounded_channel();
+    let (receipt, receipt_receiver) = mpsc::unbounded_channel();
+    GLOBAL_HEAL_CHANNEL_SENDERS
+        .set(HealChannelSenders { command, receipt })
+        .map_err(|_| "Heal channel sender already initialized")?;
+    Ok((command_receiver, receipt_receiver))
 }
 
 /// Get global heal channel sender
 pub fn get_heal_channel_sender() -> Option<&'static HealChannelSender> {
-    GLOBAL_HEAL_CHANNEL_SENDER.get()
+    GLOBAL_HEAL_CHANNEL_SENDERS.get().map(|senders| &senders.command)
 }
 
 /// Send heal command through global channel
@@ -436,6 +473,21 @@ pub fn subscribe_heal_responses() -> broadcast::Receiver<HealChannelResponse> {
     heal_response_sender().subscribe()
 }
 
+/// Send heal start request and wait for structured admission feedback.
+pub async fn send_heal_request_with_receipt(request: HealChannelRequest) -> Result<HealAdmissionReceipt, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    let senders = GLOBAL_HEAL_CHANNEL_SENDERS
+        .get()
+        .ok_or_else(|| "Heal channel not initialized".to_string())?;
+    senders
+        .receipt
+        .send(HealReceiptCommand { request, response_tx })
+        .map_err(|err| format!("Failed to send heal receipt command: {err}"))?;
+    response_rx
+        .await
+        .map_err(|e| format!("Failed to receive heal admission response: {e}"))?
+}
+
 /// Send heal start request and wait for structured admission feedback.
 pub async fn send_heal_request_with_admission(request: HealChannelRequest) -> Result<HealAdmissionResult, String> {
     let (response_tx, response_rx) = oneshot::channel();

crates/common/src/metrics.rs

@@ -2434,6 +2434,18 @@ impl Metrics {
         self.current_scan_cycle_work_active.store(false, Ordering::Relaxed);
     }
 
+    pub fn current_scan_cycle_has_unresolved_heal_work(&self) -> bool {
+        if !self.current_scan_cycle_work_active.load(Ordering::Relaxed) {
+            return false;
+        }
+
+        let source_work = self.scanner_source_work_since(&self.current_scan_cycle_source_work_start_values());
+        [ScannerWorkSource::Heal, ScannerWorkSource::Bitrot]
+            .into_iter()
+            .filter_map(|source| source_work.get(source.index()))
+            .any(|work| work.queued > 0 || work.skipped > 0 || work.failed > 0 || work.missed > 0)
+    }
+
     fn scan_cycle_work_snapshot(&self) -> ScanCycleWorkSnapshot {
         ScanCycleWorkSnapshot {
             objects_scanned: self.lifetime(Metric::ScanObject),
@@ -3361,6 +3373,40 @@ mod tests {
         metrics.finish_scan_cycle_work(start);
     }
 
+    #[test]
+    fn unresolved_heal_work_only_reflects_the_active_cycle() {
+        let metrics = Metrics::new();
+        assert!(!metrics.current_scan_cycle_has_unresolved_heal_work());
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_missed(ScannerWorkSource::Heal, 1);
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+
+        metrics.finish_scan_cycle_work(start);
+        assert!(!metrics.current_scan_cycle_has_unresolved_heal_work());
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_queued(ScannerWorkSource::Heal, 1);
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+        metrics.finish_scan_cycle_work(start);
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_work(
+            ScannerWorkSource::Bitrot,
+            ScannerSourceWorkUpdate {
+                skipped: 1,
+                ..Default::default()
+            },
+        );
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+        metrics.finish_scan_cycle_work(start);
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_failed(ScannerWorkSource::Bitrot, 1);
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+        metrics.finish_scan_cycle_work(start);
+    }
+
     #[tokio::test]
     async fn report_marks_transition_failures_as_blocked_lifecycle_control() {
         let metrics = Metrics::new();

crates/config/src/constants/api.rs

@@ -50,3 +50,43 @@ pub const ENV_API_RATE_LIMIT_BURST: &str = "RUSTFS_API_RATE_LIMIT_BURST";
 
 /// Default for `RUSTFS_API_RATE_LIMIT_BURST` (`0` = same as RPM).
 pub const DEFAULT_API_RATE_LIMIT_BURST: u32 = 0;
+
+/// Sustained S3 API request budget per addressed bucket, in requests per
+/// minute — a collective ceiling shared by all clients of that bucket.
+///
+/// Complements the per-client-IP dimension: it protects the server from one
+/// hot bucket regardless of how many client IPs the traffic comes from. `0`
+/// disables the bucket dimension. Requires `RUSTFS_API_RATE_LIMIT_ENABLE`.
+/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_RPM
+/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_RPM=60000
+pub const ENV_API_RATE_LIMIT_BUCKET_RPM: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_RPM";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_RPM` (`0` = dimension disabled).
+pub const DEFAULT_API_RATE_LIMIT_BUCKET_RPM: u32 = 0;
+
+/// Burst capacity per bucket (maximum tokens in the bucket-dimension bucket).
+///
+/// `0` means "same as `RUSTFS_API_RATE_LIMIT_BUCKET_RPM`".
+/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_BURST
+/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_BURST=2000
+pub const ENV_API_RATE_LIMIT_BUCKET_BURST: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_BURST";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_BURST` (`0` = same as bucket RPM).
+pub const DEFAULT_API_RATE_LIMIT_BUCKET_BURST: u32 = 0;
+
+/// Maximum concurrently served connections on the main API listener.
+///
+/// `0` (the default) means unlimited. When set, the accept loop stops
+/// accepting once the cap is reached and lets the kernel backlog absorb
+/// bursts, releasing capacity as connections close. This bounds file
+/// descriptor and memory usage under a connection flood.
+///
+/// The cap covers everything on the main listener — S3, admin, console,
+/// and internode gRPC — so size it well above peer-node count plus the
+/// expected client concurrency.
+/// Environment variable: RUSTFS_API_MAX_CONNECTIONS
+/// Example: RUSTFS_API_MAX_CONNECTIONS=10000
+pub const ENV_API_MAX_CONNECTIONS: &str = "RUSTFS_API_MAX_CONNECTIONS";
+
+/// Default for `RUSTFS_API_MAX_CONNECTIONS` (`0` = unlimited).
+pub const DEFAULT_API_MAX_CONNECTIONS: usize = 0;

crates/config/src/constants/drive.rs

@@ -39,6 +39,11 @@ pub const DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS: u64 = 5;
 pub const ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS";
 pub const DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: u64 = 5;
 
+/// Maximum time the metacache merge consumer waits for the next visible
+/// `walk_dir()` entry from a reader before detaching it from the merge.
+pub const ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: u64 = 10;
+
 /// Interval in seconds between active health probes for local and remote drives.
 pub const ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_INTERVAL_SECS";
 pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;

crates/config/src/constants/object.rs

@@ -73,15 +73,36 @@ pub const DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS: usize = 64;
 /// - Example: `export RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=5`
 pub const ENV_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT";
 
-/// Maximum time a GET request waits for a disk read permit (seconds).
+/// Maximum time a GET request waits for a primary disk read permit (seconds).
 ///
 /// Permits are held for the whole response body transfer, so slow clients can
 /// occupy all of them while the disks sit idle. Instead of stalling until the
-/// request-level timeout fires, a GET that waits longer than this proceeds
-/// without a permit (degraded pass-through) and the bypass is counted in
-/// metrics/logs. Set to 0 to wait indefinitely (previous behavior).
+/// request-level timeout fires, a GET that waits longer than this falls through
+/// to a bounded degraded admission lane; if that lane is also full the request
+/// is rejected with `SlowDown`/503 rather than proceeding without any permit.
+/// Set to 0 to wait on the primary lane indefinitely (never degrade or reject).
 pub const DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: u64 = 5;
 
+/// Environment variable for the bounded degraded disk-read admission lane size.
+/// - Purpose: Cap how many GETs may proceed after the primary disk-read permit
+///   pool is saturated, giving a hard upper bound on concurrent disk-active
+///   reads (primary cap + degraded cap) instead of an unbounded pass-through.
+/// - Unit: request count (usize). `0` means "mirror the primary cap", so the
+///   absolute hard cap defaults to twice the primary disk-read cap.
+/// - Example: `export RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP=16`
+pub const ENV_OBJECT_DISK_DEGRADED_READ_CAP: &str = "RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP";
+
+/// Size of the bounded degraded disk-read admission lane.
+///
+/// When the primary disk-read permit pool is saturated and a GET exceeds
+/// [`DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT`], it may take one permit from this
+/// bounded overflow lane instead of reading without any admission token. The
+/// total number of GETs performing disk-active reads is therefore hard-capped at
+/// `primary_cap + degraded_cap`; beyond that a GET is rejected with `SlowDown`.
+/// The default `0` mirrors the primary cap, so the hard cap is twice the primary
+/// disk-read concurrency.
+pub const DEFAULT_OBJECT_DISK_DEGRADED_READ_CAP: usize = 0;
+
 /// Skip bitrot hash verification on GetObject reads.
 ///
 /// When enabled, GetObject reads skip the per-shard hash
@@ -128,6 +149,39 @@ pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT"
 /// Default disk read timeout in seconds.
 pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
 
+/// Environment variable for the per-shard erasure write stall timeout (seconds).
+///
+/// A single shard write (or shard-writer shutdown) that makes no forward
+/// progress for longer than this budget is failed and its disk is dropped
+/// before commit, so a black-hole peer that accepts the connection but never
+/// drains the body cannot pin an otherwise-healthy write quorum forever
+/// (see `MultiWriter` in `erasure/coding/encode.rs`). The budget is re-armed on
+/// every shard write, so it bounds a *stall* rather than the total transfer
+/// time of a large object.
+///
+/// Unit: seconds (u64). `0` disables the stall deadline (previous behavior:
+/// wait indefinitely). Default: 30 seconds.
+pub const ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT";
+
+/// Default per-shard erasure write stall timeout in seconds.
+pub const DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT: u64 = 30;
+
+/// Environment variable for the absolute per-object erasure write cap (seconds).
+///
+/// Optional administrator backstop against a "slow-drip" peer that produces
+/// just enough forward progress to reset the per-shard stall timeout on every
+/// block while never converging. When set, the shard writers for one object are
+/// engaged for at most this long in aggregate before a stalled writer is failed
+/// and dropped. It is disabled by default because a legitimate large upload
+/// over a slow-but-honest link must not be killed on total time alone; the
+/// per-shard stall timeout is the primary guarantee.
+///
+/// Unit: seconds (u64). `0` (default) disables the absolute cap.
+pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP";
+
+/// Default absolute per-object erasure write cap in seconds (`0` = disabled).
+pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
+
 /// Environment variable for minimum GetObject timeout in seconds.
 ///
 /// When dynamic timeout calculation is enabled, this is the minimum timeout

crates/config/src/constants/zero_copy.rs

@@ -56,3 +56,33 @@ pub const DEFAULT_OBJECT_MMAP_READ_ENABLE: bool = true;
 ///
 /// Prefer [`DEFAULT_OBJECT_MMAP_READ_ENABLE`].
 pub const DEFAULT_OBJECT_ZERO_COPY_ENABLE: bool = DEFAULT_OBJECT_MMAP_READ_ENABLE;
+
+/// Environment variable capping the byte length a single mmap-copy read may
+/// materialize in memory.
+///
+/// The mmap-copy read path returns the whole requested range as one owned
+/// allocation before the first byte is served. GET/heal shard reads request
+/// the entire part span in one call, so for a large single-part object
+/// (e.g. a multi-gigabyte non-multipart upload) an uncapped mmap-copy read
+/// allocates the whole shard in memory — stalling first-byte latency past the
+/// disk-read timeout and OOM-killing memory-limited deployments
+/// (<https://github.com/rustfs/rustfs/issues/5123>). Reads longer than this
+/// cap fall back to the bounded streaming reader instead.
+///
+/// - Purpose: Bound per-shard-read memory for mmap-based reads
+/// - Acceptable values: byte count as an unsigned integer; `0` disables
+///   mmap-copy for all non-empty reads (every read streams)
+/// - Example: `export RUSTFS_OBJECT_MMAP_READ_MAX_LENGTH=8388608`
+pub const ENV_OBJECT_MMAP_READ_MAX_LENGTH: &str = "RUSTFS_OBJECT_MMAP_READ_MAX_LENGTH";
+
+/// Default mmap-copy read length cap: 32 MiB per shard read.
+///
+/// Large enough that typical multipart part shards (parts up to a few hundred
+/// megabytes across the erasure set) keep the mmap fast path, small enough
+/// that whole-part reads of huge single-part objects stream instead of
+/// materializing gigabytes per shard.
+///
+/// The cap bounds memory per shard reader, so a single part read can still
+/// materialize up to `data_shards x cap` bytes; raising the cap raises that
+/// per-request bound proportionally.
+pub const DEFAULT_OBJECT_MMAP_READ_MAX_LENGTH: usize = 32 * 1024 * 1024;

crates/credentials/src/credentials.rs

@@ -260,13 +260,10 @@ fn resolve_rpc_secret(env_secret: Option<&str>, global_access: Option<&str>, glo
 
     match (global_access, global_secret) {
         (Some(access_key), Some(secret_key)) => {
-            // Fail closed: never derive the RPC secret while the default secret
-            // key is in effect. The derivation uses `secret_key` as the HMAC key,
-            // so a public default secret yields a publicly computable RPC secret
-            // that any network peer can use to forge internode RPC signatures.
-            // Operators running with default credentials must configure
-            // RUSTFS_RPC_SECRET (or set a non-default RUSTFS_SECRET_KEY) instead.
-            if secret_key.trim() == DEFAULT_SECRET_KEY {
+            // Fail closed when either half of the active credential pair still
+            // uses the public default. Operators must configure both custom
+            // credentials or provide RUSTFS_RPC_SECRET explicitly.
+            if access_key.trim() == DEFAULT_ACCESS_KEY || secret_key.trim() == DEFAULT_SECRET_KEY {
                 return None;
             }
             derive_rpc_secret(access_key, secret_key)
@@ -589,18 +586,11 @@ mod tests {
     fn test_resolve_rpc_secret_rejects_default_credentials_for_derivation() {
         assert!(resolve_rpc_secret(None, None, None).is_none());
 
-        // Fail closed: the default secret key must not yield a derivable RPC
-        // secret, otherwise the derived value is publicly computable and any
-        // network peer can forge internode RPC signatures.
+        // Fail closed when either half of the credential pair uses the public
+        // default.
         assert!(resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).is_none());
-
-        // A default access key paired with a non-default secret key is still
-        // safe to derive: the HMAC key (the secret key) is not public.
-        let expected = derive_rpc_secret(DEFAULT_ACCESS_KEY, "custom-global-secret").expect("secret should derive");
-        assert_eq!(
-            resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some("custom-global-secret")).as_deref(),
-            Some(expected.as_str())
-        );
+        assert!(resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some("custom-global-secret")).is_none());
+        assert!(resolve_rpc_secret(None, Some("custom-access"), Some(DEFAULT_SECRET_KEY)).is_none());
 
         assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-access"), Some("custom-global-secret")).is_none());
     }
@@ -635,6 +625,10 @@ mod tests {
             resolve_rpc_secret(Some("custom-rpc-secret"), None, None).as_deref(),
             Some("custom-rpc-secret")
         );
+        assert_eq!(
+            resolve_rpc_secret(Some("custom-rpc-secret"), Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).as_deref(),
+            Some("custom-rpc-secret")
+        );
         let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
         assert_eq!(
             resolve_rpc_secret(None, Some("custom-access"), Some("custom-global-secret")).as_deref(),

crates/data-usage/src/data_usage.rs

@@ -18,9 +18,27 @@ use std::{
     collections::{HashMap, HashSet},
     hash::{DefaultHasher, Hash, Hasher},
     path::Path,
-    time::SystemTime,
+    time::{Duration, SystemTime},
 };
 
+/// Maximum amount a persisted `last_update` may lead the local wall clock before the
+/// persisted timestamp is treated as untrustworthy.
+///
+/// Invariant: the "skip stale usage update" monotonicity check (incoming `last_update`
+/// <= existing `last_update` => skip persisting) is only valid while the existing
+/// timestamp could plausibly have been produced by a healthy clock. If the on-disk
+/// snapshot is future-dated beyond this tolerance (NTP step-back, or scanner
+/// leadership moving to a node with a slower clock), the comparison would skip every
+/// save forever and freeze admin usage stats; callers must bypass the skip instead.
+pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 * 60);
+
+/// Returns true when `existing_last_update` is ahead of `now` by more than
+/// [`USAGE_LAST_UPDATE_FUTURE_TOLERANCE`], i.e. the persisted timestamp cannot be
+/// trusted for staleness comparisons and a fresh snapshot save must be allowed.
+pub fn usage_last_update_is_untrusted_future(existing_last_update: SystemTime, now: SystemTime) -> bool {
+    existing_last_update > now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE
+}
+
 #[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq)]
 pub struct TierStats {
     pub total_size: u64,
@@ -1297,6 +1315,22 @@ pub struct CompressionTotalInfo {
 mod tests {
     use super::*;
 
+    #[test]
+    fn test_usage_last_update_future_tolerance_boundary() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+
+        // Within tolerance (including the exact boundary) the timestamp is trusted.
+        assert!(!usage_last_update_is_untrusted_future(now, now));
+        assert!(!usage_last_update_is_untrusted_future(now - Duration::from_secs(60), now));
+        assert!(!usage_last_update_is_untrusted_future(now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE, now));
+
+        // Beyond tolerance the persisted timestamp is untrustworthy.
+        assert!(usage_last_update_is_untrusted_future(
+            now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE + Duration::from_secs(1),
+            now
+        ));
+    }
+
     #[test]
     fn test_data_usage_info_creation() {
         let mut info = DataUsageInfo::new();

crates/e2e_test/Cargo.toml

@@ -33,6 +33,7 @@ rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
 rustfs-data-usage.workspace = true
 rustfs-rio.workspace = true
+rustfs-utils = { workspace = true, features = ["egress"] }
 flatbuffers.workspace = true
 futures.workspace = true
 rustfs-lock.workspace = true
@@ -57,7 +58,7 @@ http.workspace = true
 http-body-util.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
-reqwest = { workspace = true, default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "multipart"] }
+reqwest = { workspace = true, features = ["json", "multipart", "stream"] }
 rustfs-signer.workspace = true
 tracing = { workspace = true }
 tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }

crates/e2e_test/src/admin_pools_test.rs

@@ -0,0 +1,79 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use http::header::HOST;
+use reqwest::StatusCode;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serde::Deserialize;
+use std::error::Error;
+
+#[derive(Debug, Deserialize)]
+struct PoolListItem {
+    id: usize,
+    cmdline: String,
+    status: String,
+}
+
+async fn signed_admin_get(env: &RustFSTestEnvironment, path: &str) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}{path}", env.url);
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
+    let request = http::Request::builder()
+        .method(http::Method::GET)
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+        .body(Body::empty())?;
+    let signed = sign_v4(request, 0, &env.access_key, &env.secret_key, "", "us-east-1");
+
+    let mut request = local_http_client().get(&url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    Ok(request.send().await?)
+}
+
+#[tokio::test]
+async fn single_drive_pools_list_succeeds_without_enabling_decommission_status() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let response = signed_admin_get(&env, "/rustfs/admin/v3/pools/list").await?;
+    let status = response.status();
+    let body = response.bytes().await?;
+
+    assert_eq!(status, StatusCode::OK, "pools list failed: {}", String::from_utf8_lossy(&body));
+    let pools: Vec<PoolListItem> = serde_json::from_slice(&body)?;
+    assert_eq!(pools.len(), 1);
+    assert_eq!(pools[0].id, 0);
+    assert_eq!(pools[0].cmdline, env.temp_dir);
+    assert_eq!(pools[0].status, "active");
+
+    let response = signed_admin_get(&env, "/rustfs/admin/v3/decommission/status").await?;
+    let status = response.status();
+    let body = response.text().await?;
+    assert_eq!(
+        status,
+        StatusCode::NOT_IMPLEMENTED,
+        "decommission status changed for a single pool: {body}"
+    );
+    assert!(body.contains("NotImplemented"), "unexpected decommission error body: {body}");
+
+    Ok(())
+}

crates/e2e_test/src/api_rate_limit_test.rs

@@ -86,6 +86,52 @@ async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResu
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn api_rate_limit_bucket_dimension_throttles_per_bucket() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    // Bucket dimension only: the readiness-poll ListBuckets calls hit "/"
+    // (no bucket) and therefore do not consume any budget.
+    env.start_rustfs_server_with_env(
+        vec![],
+        &[
+            ("RUSTFS_API_RATE_LIMIT_ENABLE", "true"),
+            ("RUSTFS_API_RATE_LIMIT_BUCKET_RPM", "60"),
+            ("RUSTFS_API_RATE_LIMIT_BUCKET_BURST", "5"),
+        ],
+    )
+    .await?;
+
+    let client = local_http_client();
+
+    // Unauthenticated GETs are still counted arrivals (403, not 429, while
+    // within budget); the sixth rapid hit on the same bucket must throttle.
+    let mut throttled = false;
+    for i in 0..6 {
+        let response = client.get(format!("{}/hot-bucket/object-{i}", env.url)).send().await?;
+        if response.status() == reqwest::StatusCode::TOO_MANY_REQUESTS {
+            throttled = true;
+            assert!(
+                response.headers().contains_key(reqwest::header::RETRY_AFTER),
+                "bucket-dimension 429 must carry Retry-After"
+            );
+            break;
+        }
+    }
+    assert!(throttled, "6 rapid requests against burst 5 must trip the bucket dimension");
+
+    // A different bucket has its own budget.
+    let other = client.get(format!("{}/cold-bucket/object", env.url)).send().await?;
+    assert_ne!(
+        other.status(),
+        reqwest::StatusCode::TOO_MANY_REQUESTS,
+        "an unrelated bucket must not be throttled"
+    );
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn api_rate_limit_stays_inert_by_default() -> TestResult {

crates/e2e_test/src/cluster_multidrive_pool_test.rs

@@ -0,0 +1,108 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Smoke tests for the multi-drive / multi-pool cluster harness.
+//!
+//! These belong to the nightly 4-node cluster lane (they spin up real RustFS
+//! processes, so they are excluded from the merge-gate `e2e-full` profile in
+//! `.config/nextest.toml`). They assert that:
+//!
+//! * `drivesPerNode > 1` produces a bootable single-pool cluster whose
+//!   `RUSTFS_VOLUMES` string enumerates every drive, and that a PUT/GET
+//!   round-trips through the multi-drive erasure layout.
+//! * A two-pool topology (one node per pool, `drivesPerNode` drives each) boots
+//!   with the ellipses `RUSTFS_VOLUMES` form and round-trips a PUT/GET.
+//!
+//! Readiness is established by the harness's `start()` handshake (TCP reachability
+//! plus an S3 `ListBuckets` poll) — there are no fixed sleeps.
+//!
+//! Out of scope for this block (tracked separately): network fault injection
+//! (toxiproxy / socket proxy) and 5GiB large-object budgets.
+
+use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use serial_test::serial;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+const BUCKET: &str = "cluster-multidrive-pool-bucket";
+
+/// PUT an object then GET it back and assert the bytes round-trip.
+async fn put_get_roundtrip(cluster: &RustFSTestClusterEnvironment, key: &str, payload: &[u8]) -> TestResult {
+    let writer = cluster.create_s3_client(0)?;
+    writer
+        .put_object()
+        .bucket(BUCKET)
+        .key(key)
+        .body(bytes::Bytes::copy_from_slice(payload).into())
+        .send()
+        .await?;
+
+    // Read back from the last node to exercise the cross-node/cross-pool path.
+    let reader = cluster.create_s3_client(cluster.nodes.len() - 1)?;
+    let got = reader.get_object().bucket(BUCKET).key(key).send().await?;
+    let body = got.body.collect().await?.into_bytes();
+    assert_eq!(body.as_ref(), payload, "round-tripped object bytes must match");
+    Ok(())
+}
+
+/// 4 nodes x 2 drives, single pool: the multi-drive layout boots and round-trips.
+#[tokio::test]
+#[serial]
+async fn cluster_multidrive_single_pool_smoke() -> TestResult {
+    crate::common::init_logging();
+
+    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 2)).await?;
+
+    // The single-pool multi-drive layout must list every (node, drive) endpoint
+    // explicitly (8 endpoints, no ellipses) so the server keeps one legacy pool.
+    let volumes = cluster.rustfs_volumes_arg();
+    assert_eq!(volumes.split(' ').count(), 8, "expected 8 explicit endpoints, got: {volumes}");
+    assert!(!volumes.contains('{'), "single-pool layout must not use ellipses: {volumes}");
+
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let payload = vec![0xA5u8; 512 * 1024];
+    put_get_roundtrip(&cluster, "multidrive/object", &payload).await?;
+    Ok(())
+}
+
+/// Two single-node pools, 2 drives each: the multi-pool layout boots and
+/// round-trips. Every pool is a distinct erasure pool (`pool_idx` 0 and 1).
+#[tokio::test]
+#[serial]
+async fn cluster_two_pool_smoke() -> TestResult {
+    crate::common::init_logging();
+
+    let mut cluster =
+        RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]])).await?;
+
+    // The two-pool layout must emit one ellipses argument per pool.
+    let volumes = cluster.rustfs_volumes_arg();
+    let args: Vec<&str> = volumes.split(' ').collect();
+    assert_eq!(args.len(), 2, "expected two pool arguments, got: {volumes}");
+    assert!(
+        args.iter().all(|a| a.contains("/drive{0...1}")),
+        "each pool arg must use the drive ellipses form: {volumes}"
+    );
+    assert_eq!(cluster.nodes[0].pool_idx, 0);
+    assert_eq!(cluster.nodes[1].pool_idx, 1);
+
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let payload = vec![0x5Au8; 256 * 1024];
+    put_get_roundtrip(&cluster, "twopool/object", &payload).await?;
+    Ok(())
+}

crates/e2e_test/src/common.rs

@@ -717,15 +717,166 @@ pub async fn awscurl_delete(
     execute_awscurl(url, "DELETE", None, access_key, secret_key).await
 }
 
+/// Cluster topology declaration for a `RustFSTestClusterEnvironment`.
+///
+/// Describes how many nodes to launch, how many data drives each node exposes,
+/// and how those nodes are grouped into erasure pools. The topology drives both
+/// on-disk directory creation and the `RUSTFS_VOLUMES` string assembled for the
+/// node processes.
+///
+/// # Single-host expressibility constraints
+///
+/// The harness runs every node on `127.0.0.1` with a distinct port, so the
+/// `RUSTFS_VOLUMES` string can only use the two forms the server parser accepts
+/// on a single machine (verified against `ecstore` `DisksLayout::from_volumes`):
+///
+/// * **Single pool** — every `(node, drive)` endpoint listed explicitly, no
+///   ellipses. This is the legacy form and yields exactly one pool spanning all
+///   nodes and drives (`DistErasure`). Any `drives_per_node >= 1` works.
+/// * **Multiple pools** — one ellipses arg per pool. A pool argument is a single
+///   URL template, so it can enumerate several drives on *one* host but cannot
+///   enumerate multiple distinct-port hosts. A pool that spanned two localhost
+///   nodes would need a host ellipses (`127.0.0.1:900{0...1}`), which forces the
+///   *same* on-disk path across those ports and collides physically. Therefore
+///   every pool in a multi-pool topology owns exactly one node. In addition, the
+///   parser rejects a single-drive ellipses pool (`drive{0...0}`), so a
+///   multi-pool topology requires `drives_per_node >= 2`.
+///
+/// Genuine multi-node pools (a pool striped across several hosts) require real
+/// multi-host infrastructure and are deferred to the nightly cluster CI lane
+/// (backlog #1313 / #1314); they are intentionally not expressible here.
+#[derive(Clone, Debug)]
+pub struct ClusterTopology {
+    /// Number of node processes to launch.
+    pub node_count: usize,
+    /// Number of independent data drives (directories) each node exposes.
+    pub drives_per_node: usize,
+    /// Pool membership as a list of node-index groups. An empty vector means a
+    /// single pool spanning every node.
+    pub pools: Vec<Vec<usize>>,
+}
+
+impl ClusterTopology {
+    /// Single pool, one drive per node — identical to the historical
+    /// `RustFSTestClusterEnvironment::new` layout.
+    pub fn single_pool(node_count: usize) -> Self {
+        Self {
+            node_count,
+            drives_per_node: 1,
+            pools: Vec::new(),
+        }
+    }
+
+    /// Single pool with `drives_per_node` drives on every node. The pool spans
+    /// all nodes and all drives (one `DistErasure` pool).
+    pub fn single_pool_multidrive(node_count: usize, drives_per_node: usize) -> Self {
+        Self {
+            node_count,
+            drives_per_node,
+            pools: Vec::new(),
+        }
+    }
+
+    /// Multi-pool topology where every pool owns exactly one node. `pools` lists
+    /// the node index backing each pool (e.g. `vec![vec![0], vec![1]]` for a
+    /// two-pool cluster). Requires `drives_per_node >= 2` (see the type-level
+    /// note on single-host expressibility).
+    pub fn per_node_pools(drives_per_node: usize, pools: Vec<Vec<usize>>) -> Self {
+        let node_count = pools.iter().flatten().copied().max().map(|m| m + 1).unwrap_or(0);
+        Self {
+            node_count,
+            drives_per_node,
+            pools,
+        }
+    }
+
+    /// Normalized pool membership: an empty `pools` becomes a single pool over
+    /// all node indices.
+    fn normalized_pools(&self) -> Vec<Vec<usize>> {
+        if self.pools.is_empty() {
+            vec![(0..self.node_count).collect()]
+        } else {
+            self.pools.clone()
+        }
+    }
+
+    /// Validate that the topology is expressible on a single localhost machine.
+    fn validate(&self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        if self.node_count == 0 {
+            return Err("Node count must be greater than zero".into());
+        }
+        if self.drives_per_node == 0 {
+            return Err("drives_per_node must be greater than zero".into());
+        }
+
+        let pools = self.normalized_pools();
+
+        // Every referenced node index must be in range.
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            if nodes.is_empty() {
+                return Err(format!("pool {pool_idx} has no nodes").into());
+            }
+            for &n in nodes {
+                if n >= self.node_count {
+                    return Err(format!("pool {pool_idx} references node {n} but node_count is {}", self.node_count).into());
+                }
+            }
+        }
+
+        // Each node must belong to exactly one pool.
+        let mut seen = vec![0usize; self.node_count];
+        for nodes in &pools {
+            for &n in nodes {
+                seen[n] += 1;
+            }
+        }
+        for (n, count) in seen.iter().enumerate() {
+            match count {
+                0 => return Err(format!("node {n} is not assigned to any pool").into()),
+                1 => {}
+                _ => return Err(format!("node {n} is assigned to more than one pool").into()),
+            }
+        }
+
+        // Multi-pool constraints imposed by the single-host RUSTFS_VOLUMES syntax.
+        if pools.len() > 1 {
+            if self.drives_per_node < 2 {
+                return Err(
+                    "multi-pool topology requires drives_per_node >= 2 (the server parser rejects a single-drive ellipses pool)"
+                        .into(),
+                );
+            }
+            for (pool_idx, nodes) in pools.iter().enumerate() {
+                if nodes.len() != 1 {
+                    return Err(format!(
+                        "pool {pool_idx} spans {} nodes; a pool striped across multiple localhost nodes is not expressible (needs host ellipses, which collides on shared paths). Use one node per pool, or the nightly multi-host lane (backlog #1313/#1314).",
+                        nodes.len()
+                    )
+                    .into());
+                }
+            }
+        }
+
+        Ok(())
+    }
+}
+
 /// Represents a single RustFS server instance in a test cluster.
 ///
 /// Each `ClusterNode` tracks the node's network address, base URL for
-/// S3-compatible requests, on-disk data directory, and the underlying
+/// S3-compatible requests, on-disk data directories, and the underlying
 /// child process handle when the node is running.
 pub struct ClusterNode {
     pub address: String,
     pub url: String,
+    /// Primary data directory for the node. For single-drive nodes this is the
+    /// node's only drive; for multi-drive nodes it is the first drive. Kept as a
+    /// stable field so existing single-drive tests continue to compile.
     pub data_dir: String,
+    /// All data drives exposed by this node (`data_dirs[0] == data_dir`).
+    pub data_dirs: Vec<String>,
+    /// Index of the pool this node belongs to.
+    pub pool_idx: usize,
     pub process: Option<Child>,
 }
 
@@ -741,6 +892,7 @@ pub struct RustFSTestClusterEnvironment {
     pub access_key: String,
     pub secret_key: String,
     pub extra_env: Vec<(String, String)>,
+    pub topology: ClusterTopology,
 }
 
 impl RustFSTestClusterEnvironment {
@@ -762,34 +914,83 @@ impl RustFSTestClusterEnvironment {
     /// * `Err(Box<dyn Error + Send + Sync>)` - An error if any step fails, such as temporary
     ///   directory creation failure or available port lookup failure.
     pub async fn new(node_count: usize) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
-        if node_count == 0 {
-            return Err("Node count must be greater than zero".into());
-        }
+        Self::with_topology(ClusterTopology::single_pool(node_count)).await
+    }
+
+    /// Create a RustFS test cluster environment from an explicit topology.
+    ///
+    /// Allocates a unique temporary root directory, an available TCP port per
+    /// node, and one data directory per drive. The topology is validated up front
+    /// (node/pool assignment, single-host multi-pool constraints); node processes
+    /// are not started at this stage.
+    ///
+    /// When the topology exposes more than one local drive on a node (multi-drive
+    /// or multi-pool layouts), `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` is added to
+    /// every node's environment: those drives live on the same temp filesystem, so
+    /// the server's distinct-physical-disk safety check would otherwise reject
+    /// startup. Single-drive single-pool clusters keep the historical environment
+    /// unchanged.
+    pub async fn with_topology(topology: ClusterTopology) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
+        topology.validate()?;
+
         let temp_dir = format!("/tmp/rustfs_cluster_test_{}", Uuid::new_v4());
         fs::create_dir_all(&temp_dir).await?;
 
-        let mut nodes = Vec::with_capacity(node_count);
-        for i in 0..node_count {
+        // Map every node to its owning pool index.
+        let pools = topology.normalized_pools();
+        let mut pool_of_node = vec![0usize; topology.node_count];
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            for &n in nodes {
+                pool_of_node[n] = pool_idx;
+            }
+        }
+
+        let multidrive = topology.drives_per_node > 1;
+
+        let mut nodes = Vec::with_capacity(topology.node_count);
+        for (i, &pool_idx) in pool_of_node.iter().enumerate() {
             let port = RustFSTestEnvironment::find_available_port().await?;
             let address = format!("127.0.0.1:{}", port);
             let url = format!("http://{}", address);
-            let data_dir = format!("{}/node{}", temp_dir, i);
-            fs::create_dir_all(&data_dir).await?;
+
+            // Single-drive nodes keep the historical `{temp}/node{i}` path so
+            // existing tests (and their on-disk assertions) are unaffected.
+            // Multi-drive nodes nest one `drive{d}` directory per drive so the
+            // ellipses form `drive{0...N-1}` can address them.
+            let data_dirs: Vec<String> = if multidrive {
+                (0..topology.drives_per_node)
+                    .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
+                    .collect()
+            } else {
+                vec![format!("{}/node{}", temp_dir, i)]
+            };
+            for dir in &data_dirs {
+                fs::create_dir_all(dir).await?;
+            }
 
             nodes.push(ClusterNode {
                 address,
                 url,
-                data_dir,
+                data_dir: data_dirs[0].clone(),
+                data_dirs,
+                pool_idx,
                 process: None,
             });
         }
 
+        let mut extra_env = Vec::new();
+        extra_env.push(("RUSTFS_RPC_SECRET".to_string(), String::new()));
+        if multidrive {
+            extra_env.push(("RUSTFS_UNSAFE_BYPASS_DISK_CHECK".to_string(), "true".to_string()));
+        }
+
         Ok(Self {
             nodes,
             temp_dir,
-            access_key: DEFAULT_ACCESS_KEY.to_string(),
-            secret_key: DEFAULT_SECRET_KEY.to_string(),
-            extra_env: Vec::new(),
+            access_key: "rustfs-cluster-test-access".to_string(),
+            secret_key: "rustfs-cluster-test-secret".to_string(),
+            extra_env,
+            topology,
         })
     }
 
@@ -809,14 +1010,51 @@ impl RustFSTestClusterEnvironment {
         Ok(())
     }
 
-    /// Build the volumes argument string for RustFS binary (internal helper method).
+    /// Build the `RUSTFS_VOLUMES` argument string for the cluster's topology.
     ///
-    /// Concatenates the address and data directory of all cluster nodes into a single string
-    /// used as the `RUSTFS_VOLUMES` environment variable for RustFS node processes.
+    /// * **Single pool** — every `(node, drive)` endpoint is listed explicitly and
+    ///   space-joined. With no ellipses the server parser collapses them into one
+    ///   legacy pool spanning all nodes and drives.
+    /// * **Multiple pools** — each pool (exactly one node) contributes one ellipses
+    ///   argument `http://<addr><node-base>/drive{0...N-1}`; the space-separated
+    ///   arguments become one pool each.
+    ///
+    /// The server splits `RUSTFS_VOLUMES` on spaces (`value_delimiter = ' '`), which
+    /// this assembly matches, and the resulting layout is verified against the
+    /// `ecstore` parser in the unit tests below.
+    /// Public view of the assembled `RUSTFS_VOLUMES` string, so tests can assert
+    /// the pool/drive layout without starting node processes.
+    pub fn rustfs_volumes_arg(&self) -> String {
+        self.build_volumes_arg()
+    }
+
     fn build_volumes_arg(&self) -> String {
-        self.nodes
+        let pools = self.topology.normalized_pools();
+
+        if pools.len() <= 1 {
+            // Single pool: explicit enumeration of every drive on every node.
+            return self
+                .nodes
+                .iter()
+                .flat_map(|n| n.data_dirs.iter().map(move |dir| format!("http://{}{}", n.address, dir)))
+                .collect::<Vec<_>>()
+                .join(" ");
+        }
+
+        // Multi-pool: one ellipses argument per single-node pool. The drive
+        // directories are `<temp>/node{i}/drive{0..N-1}`, so the ellipses base is
+        // the shared parent of the node's drives.
+        pools
             .iter()
-            .map(|n| format!("http://{}{}", n.address, n.data_dir))
+            .map(|nodes| {
+                let node = &self.nodes[nodes[0]];
+                let base = node
+                    .data_dirs
+                    .first()
+                    .and_then(|d| d.rsplit_once('/').map(|(parent, _)| parent))
+                    .unwrap_or(&node.data_dir);
+                format!("http://{}{}/drive{{0...{}}}", node.address, base, self.topology.drives_per_node - 1)
+            })
             .collect::<Vec<_>>()
             .join(" ")
     }
@@ -1091,4 +1329,130 @@ mod tests {
 
         stdfs::remove_file(stamp_path).ok();
     }
+
+    /// Build a cluster environment struct in-memory (no ports, no processes) so
+    /// that `build_volumes_arg` can be exercised as a pure string builder. Node
+    /// directories mirror what `with_topology` would create for the topology.
+    fn fake_cluster(topology: ClusterTopology) -> RustFSTestClusterEnvironment {
+        let temp_dir = "/tmp/rustfs_cluster_test_FAKE".to_string();
+        let pools = topology.normalized_pools();
+        let mut pool_of_node = vec![0usize; topology.node_count];
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            for &n in nodes {
+                pool_of_node[n] = pool_idx;
+            }
+        }
+        let multidrive = topology.drives_per_node > 1;
+
+        let nodes = (0..topology.node_count)
+            .map(|i| {
+                let address = format!("127.0.0.1:{}", 9000 + i);
+                let data_dirs: Vec<String> = if multidrive {
+                    (0..topology.drives_per_node)
+                        .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
+                        .collect()
+                } else {
+                    vec![format!("{}/node{}", temp_dir, i)]
+                };
+                ClusterNode {
+                    url: format!("http://{}", address),
+                    address,
+                    data_dir: data_dirs[0].clone(),
+                    data_dirs,
+                    pool_idx: pool_of_node[i],
+                    process: None,
+                }
+            })
+            .collect();
+
+        RustFSTestClusterEnvironment {
+            nodes,
+            temp_dir,
+            access_key: DEFAULT_ACCESS_KEY.to_string(),
+            secret_key: DEFAULT_SECRET_KEY.to_string(),
+            extra_env: Vec::new(),
+            topology,
+        }
+    }
+
+    #[test]
+    fn volumes_single_pool_single_drive_is_backward_compatible() {
+        // The historical layout: one explicit endpoint per node, space-joined,
+        // no ellipses, no `drive` sub-directory.
+        let env = fake_cluster(ClusterTopology::single_pool(4));
+        assert_eq!(
+            env.build_volumes_arg(),
+            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0 \
+             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1 \
+             http://127.0.0.1:9002/tmp/rustfs_cluster_test_FAKE/node2 \
+             http://127.0.0.1:9003/tmp/rustfs_cluster_test_FAKE/node3"
+        );
+    }
+
+    #[test]
+    fn volumes_single_pool_multidrive_enumerates_every_drive() {
+        // 4 nodes x 2 drives -> 8 explicit endpoints, one legacy pool. No
+        // ellipses, so the server parser keeps this as a single DistErasure pool.
+        let env = fake_cluster(ClusterTopology::single_pool_multidrive(4, 2));
+        let expected = (0..4)
+            .flat_map(|i| {
+                (0..2).map(move |d| format!("http://127.0.0.1:{}/tmp/rustfs_cluster_test_FAKE/node{}/drive{}", 9000 + i, i, d))
+            })
+            .collect::<Vec<_>>()
+            .join(" ");
+        assert_eq!(env.build_volumes_arg(), expected);
+        assert_eq!(env.build_volumes_arg().split(' ').count(), 8);
+    }
+
+    #[test]
+    fn volumes_two_pool_uses_one_ellipses_arg_per_pool() {
+        // Two single-node pools, 2 drives each -> two ellipses arguments. The
+        // server parser treats each space-separated ellipses arg as its own pool.
+        let env = fake_cluster(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]));
+        assert_eq!(
+            env.build_volumes_arg(),
+            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0/drive{0...1} \
+             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1/drive{0...1}"
+        );
+    }
+
+    #[test]
+    fn topology_rejects_multi_node_pool() {
+        // A pool striped across two localhost nodes is not expressible.
+        let err = ClusterTopology::per_node_pools(2, vec![vec![0, 1], vec![2, 3]])
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("not expressible"), "unexpected error: {err}");
+    }
+
+    #[test]
+    fn topology_rejects_single_drive_multi_pool() {
+        // The server parser rejects a single-drive ellipses pool (`drive{0...0}`).
+        let err = ClusterTopology::per_node_pools(1, vec![vec![0], vec![1]])
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("drives_per_node >= 2"), "unexpected error: {err}");
+    }
+
+    #[test]
+    fn topology_rejects_unassigned_and_duplicated_nodes() {
+        // Node 2 is never assigned to a pool.
+        let mut t = ClusterTopology::single_pool_multidrive(3, 2);
+        t.pools = vec![vec![0], vec![1]];
+        assert!(t.validate().unwrap_err().to_string().contains("not assigned"));
+
+        // Node 0 assigned twice.
+        let mut t = ClusterTopology::single_pool_multidrive(2, 2);
+        t.pools = vec![vec![0], vec![0]];
+        assert!(t.validate().unwrap_err().to_string().contains("more than one pool"));
+    }
+
+    #[test]
+    fn topology_single_pool_accepts_any_drive_count() {
+        assert!(ClusterTopology::single_pool(4).validate().is_ok());
+        assert!(ClusterTopology::single_pool_multidrive(4, 4).validate().is_ok());
+        assert!(ClusterTopology::single_pool_multidrive(1, 1).validate().is_ok());
+    }
 }

crates/e2e_test/src/connection_cap_test.rs

@@ -0,0 +1,127 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! E2E coverage for the opt-in global connection cap on the main API listener
+//! (backlog#1191 follow-up, `RUSTFS_API_MAX_CONNECTIONS`): permits must be
+//! released when connections close (no leak), and the cap must actually bound
+//! concurrency — a queued connection is served only after a held one closes.
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use serial_test::serial;
+use std::time::Duration;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::TcpStream;
+use tokio::time::timeout;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+/// Open a TCP connection and write one unauthenticated `GET /` (any response,
+/// e.g. 403, proves the connection was accepted and served).
+async fn open_and_request(addr: &str, connection: &str) -> std::io::Result<TcpStream> {
+    let mut stream = TcpStream::connect(addr).await?;
+    let request = format!("GET / HTTP/1.1\r\nHost: {addr}\r\nConnection: {connection}\r\n\r\n");
+    stream.write_all(request.as_bytes()).await?;
+    Ok(stream)
+}
+
+/// Read until the response head is complete, or `None` on timeout/close —
+/// a `None` on an open socket means the connection sits unaccepted in the
+/// kernel backlog behind the cap.
+async fn read_response_head(stream: &mut TcpStream, dur: Duration) -> Option<String> {
+    let deadline = tokio::time::Instant::now() + dur;
+    let mut buf = vec![0u8; 4096];
+    let mut collected = String::new();
+    loop {
+        let remaining = deadline.checked_duration_since(tokio::time::Instant::now())?;
+        match timeout(remaining, stream.read(&mut buf)).await {
+            Ok(Ok(0)) | Ok(Err(_)) | Err(_) => return None,
+            Ok(Ok(n)) => {
+                collected.push_str(&String::from_utf8_lossy(&buf[..n]));
+                if collected.contains("\r\n\r\n") {
+                    return Some(collected);
+                }
+            }
+        }
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn connection_cap_releases_permits_on_close() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_API_MAX_CONNECTIONS", "2")])
+        .await?;
+
+    // Ten sequential connections against cap 2: if permits leaked, the third
+    // request would already hang in the backlog and time out.
+    for i in 0..10 {
+        let mut stream = open_and_request(&env.address, "close").await?;
+        let head = read_response_head(&mut stream, Duration::from_secs(10))
+            .await
+            .unwrap_or_else(|| panic!("request {i} got no response — a connection permit leaked"));
+        assert!(head.starts_with("HTTP/1.1"), "request {i} unexpected response: {head}");
+    }
+
+    Ok(())
+}
+
+/// Open a TCP connection and send an INCOMPLETE request head. Once accepted
+/// it pins a connection permit: hyper waits for the rest of the head (75s
+/// default header timeout) until we close the socket.
+async fn open_and_stall(addr: &str) -> std::io::Result<TcpStream> {
+    let mut stream = TcpStream::connect(addr).await?;
+    stream
+        .write_all(format!("GET / HTTP/1.1\r\nHost: {addr}\r\n").as_bytes())
+        .await?;
+    Ok(stream)
+}
+
+#[tokio::test]
+#[serial]
+async fn connection_cap_blocks_excess_connections_until_permits_free() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_API_MAX_CONNECTIONS", "2")])
+        .await?;
+    // Let the readiness poller's pooled connection close and free its permit.
+    tokio::time::sleep(Duration::from_secs(1)).await;
+
+    // Two stalled connections saturate cap 2 (a served-and-closed connection
+    // would release its permit immediately, so stalling is what makes the
+    // occupancy deterministic).
+    let stalled_a = open_and_stall(&env.address).await?;
+    let stalled_b = open_and_stall(&env.address).await?;
+    tokio::time::sleep(Duration::from_millis(300)).await;
+
+    // A complete request now sits in the kernel backlog: connect() succeeds
+    // but no permit is available, so no response arrives.
+    let mut blocked = open_and_request(&env.address, "close").await?;
+    assert!(
+        read_response_head(&mut blocked, Duration::from_secs(3)).await.is_none(),
+        "cap 2 with two stalled connections must leave the third unserved"
+    );
+
+    // Dropping the stalled connections releases their permits (hyper sees
+    // EOF while reading the head); the queued request's bytes already sit in
+    // the socket buffer, so it must now be accepted and served.
+    drop(stalled_a);
+    drop(stalled_b);
+    let head = read_response_head(&mut blocked, Duration::from_secs(10))
+        .await
+        .expect("queued connection must be served after permits are released");
+    assert!(head.starts_with("HTTP/1.1"), "unexpected response: {head}");
+
+    Ok(())
+}

crates/e2e_test/src/copy_object_checksum_test.rs

@@ -0,0 +1,307 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression test for Issue #4996: CopyObject must return the destination object's
+//! checksum in `CopyObjectResult` and persist it so a later checksum-mode HEAD/GET
+//! returns the same value. Covers both the requested-algorithm case (compute fresh)
+//! and the no-algorithm case (preserve the source object's existing checksum).
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::{BucketVersioningStatus, ChecksumAlgorithm, ChecksumMode, VersioningConfiguration};
+    use base64::Engine as _;
+    use base64::engine::general_purpose::STANDARD as BASE64;
+    use serial_test::serial;
+    use sha2::{Digest, Sha256};
+    use tracing::info;
+
+    async fn create_versioned_bucket(client: &aws_sdk_s3::Client, bucket: &str) {
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("Failed to create bucket");
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await
+            .expect("Failed to enable versioning");
+    }
+
+    /// Requested algorithm: a CopyObject asking for SHA256 must compute it over the copied
+    /// bytes, return it in `CopyObjectResult.ChecksumSHA256`, and persist it so a checksum-mode
+    /// HEAD on the destination returns the identical value.
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_with_checksum_algorithm_returns_and_persists_sha256() {
+        init_logging();
+        info!("Issue #4996: CopyObject with ChecksumAlgorithm=SHA256 must return and persist the checksum");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-checksum-req-src";
+        let dst_bucket = "copy-checksum-req-dst";
+        let src_key = "objects/source.bin";
+        let dst_key = "objects/dest.bin";
+
+        create_versioned_bucket(&client, src_bucket).await;
+        create_versioned_bucket(&client, dst_bucket).await;
+
+        let content = b"deterministic synthetic payload for copy-object checksum #4996";
+        let expected_sha256 = BASE64.encode(Sha256::digest(content));
+
+        client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT source failed");
+
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}"))
+            .checksum_algorithm(ChecksumAlgorithm::Sha256)
+            .send()
+            .await
+            .expect("CopyObject with ChecksumAlgorithm must succeed");
+
+        // (3) The response must carry the freshly computed SHA-256 of the copied bytes.
+        let result = copy_out
+            .copy_object_result()
+            .expect("issue #4996: CopyObject must return a CopyObjectResult");
+        assert_eq!(
+            result.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: CopyObjectResult.ChecksumSHA256 must equal the SHA-256 of the copied bytes"
+        );
+
+        // (4) A checksum-mode HEAD on the destination must return the same SHA-256.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .checksum_mode(ChecksumMode::Enabled)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: destination checksum-mode HEAD must return the same SHA-256 the copy reported"
+        );
+
+        env.stop_server();
+    }
+
+    /// No algorithm requested: when the source object already carries a checksum, the copy must
+    /// preserve it on the destination (AWS default), visible via a checksum-mode HEAD.
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_without_algorithm_preserves_source_checksum() {
+        init_logging();
+        info!("Issue #4996: CopyObject without ChecksumAlgorithm must preserve the source object's checksum");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-checksum-preserve-src";
+        let dst_bucket = "copy-checksum-preserve-dst";
+        let src_key = "objects/source.bin";
+        let dst_key = "objects/dest.bin";
+
+        create_versioned_bucket(&client, src_bucket).await;
+        create_versioned_bucket(&client, dst_bucket).await;
+
+        let content = b"another deterministic payload whose source checksum must survive the copy";
+        let expected_sha256 = BASE64.encode(Sha256::digest(content));
+
+        // Store the source WITH a SHA-256 checksum so it has one to preserve.
+        let put_src = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .checksum_algorithm(ChecksumAlgorithm::Sha256)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT source with checksum failed");
+        assert_eq!(
+            put_src.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "source PUT must report the SHA-256 it stored"
+        );
+
+        // Copy WITHOUT specifying a checksum algorithm.
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}"))
+            .send()
+            .await
+            .expect("CopyObject without ChecksumAlgorithm must succeed");
+
+        // The response should echo the preserved source checksum.
+        let result = copy_out
+            .copy_object_result()
+            .expect("issue #4996: CopyObject must return a CopyObjectResult");
+        assert_eq!(
+            result.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: a no-algorithm copy must preserve and report the source object's SHA-256"
+        );
+
+        // And a checksum-mode HEAD on the destination must return that same preserved SHA-256.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .checksum_mode(ChecksumMode::Enabled)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: destination checksum-mode HEAD must return the preserved source SHA-256"
+        );
+
+        env.stop_server();
+    }
+
+    /// Requested algorithm differs from the source's: a source stored with SHA256, copied while
+    /// requesting CRC32, must return/persist the freshly computed CRC32 and must NOT carry the
+    /// source's SHA256 through. Guards the request-over-source precedence and the destination's
+    /// checksum-not-inherited path, and exercises the CRC32 code path (a different branch of
+    /// ChecksumType::from_string than SHA256).
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_requested_algorithm_overrides_source_checksum() {
+        init_logging();
+        info!("Issue #4996: a requested CopyObject checksum algorithm must override the source object's algorithm");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-checksum-override-src";
+        let dst_bucket = "copy-checksum-override-dst";
+        let src_key = "objects/source.bin";
+        let ref_key = "objects/reference-crc32.bin";
+        let dst_key = "objects/dest.bin";
+
+        create_versioned_bucket(&client, src_bucket).await;
+        create_versioned_bucket(&client, dst_bucket).await;
+
+        let content = b"payload whose copy must be re-checksummed with a different algorithm";
+        let expected_sha256 = BASE64.encode(Sha256::digest(content));
+
+        // Source is stored WITH a SHA-256 checksum.
+        client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .checksum_algorithm(ChecksumAlgorithm::Sha256)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT source with SHA256 failed");
+
+        // Establish the canonical CRC32 the server computes for this content via a reference PUT,
+        // so the copy's CRC32 can be asserted against an exact server-computed value.
+        let ref_put = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(ref_key)
+            .checksum_algorithm(ChecksumAlgorithm::Crc32)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("reference PUT with CRC32 failed");
+        let expected_crc32 = ref_put
+            .checksum_crc32()
+            .expect("reference PUT must report a CRC32")
+            .to_string();
+
+        // Copy the SHA256 source while requesting CRC32.
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}"))
+            .checksum_algorithm(ChecksumAlgorithm::Crc32)
+            .send()
+            .await
+            .expect("CopyObject requesting a different algorithm must succeed");
+
+        let result = copy_out
+            .copy_object_result()
+            .expect("issue #4996: CopyObject must return a CopyObjectResult");
+        // The requested CRC32 must be computed and returned.
+        assert_eq!(
+            result.checksum_crc32(),
+            Some(expected_crc32.as_str()),
+            "issue #4996: a requested CRC32 must be computed fresh over the copied bytes"
+        );
+        // The source's SHA256 must NOT leak through — the requested algorithm wins.
+        assert_eq!(
+            result.checksum_sha256(),
+            None,
+            "issue #4996: the source object's SHA256 must not be inherited when a different algorithm is requested"
+        );
+        assert_ne!(
+            result.checksum_crc32(),
+            Some(expected_sha256.as_str()),
+            "sanity: CRC32 field must not carry the SHA256 value"
+        );
+
+        // The destination must persist CRC32 (and only CRC32) for a checksum-mode HEAD.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .checksum_mode(ChecksumMode::Enabled)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.checksum_crc32(),
+            Some(expected_crc32.as_str()),
+            "issue #4996: destination checksum-mode HEAD must return the requested CRC32"
+        );
+        assert_eq!(
+            head.checksum_sha256(),
+            None,
+            "issue #4996: destination must not report the source's SHA256 after an override copy"
+        );
+
+        env.stop_server();
+    }
+}

crates/e2e_test/src/copy_object_version_restore_test.rs

@@ -160,4 +160,146 @@ mod tests {
 
         env.stop_server();
     }
+
+    /// Regression test for Issue #4976: a versioned-source CopyObject must echo the exact source
+    /// version copied via `x-amz-copy-source-version-id` (SDK `CopySourceVersionId`), kept distinct
+    /// from the newly created destination `x-amz-version-id`.
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_of_non_latest_source_version_returns_copy_source_version_id() {
+        init_logging();
+        info!("Issue #4976: versioned CopyObject must return x-amz-copy-source-version-id for the exact source version");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-source-version-header-src";
+        let dst_bucket = "copy-source-version-header-dst";
+        let src_key = "reports/quarantined.bin";
+        let dst_key = "reports/promoted.bin";
+
+        for bucket in [src_bucket, dst_bucket] {
+            client
+                .create_bucket()
+                .bucket(bucket)
+                .send()
+                .await
+                .expect("Failed to create bucket");
+            client
+                .put_bucket_versioning()
+                .bucket(bucket)
+                .versioning_configuration(
+                    VersioningConfiguration::builder()
+                        .status(BucketVersioningStatus::Enabled)
+                        .build(),
+                )
+                .send()
+                .await
+                .expect("Failed to enable versioning");
+        }
+
+        // Source version 1: the exact (non-latest) version we will copy.
+        let v1_content = b"quarantined payload -- source version one (target of the copy)";
+        let put_v1 = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .body(ByteStream::from_static(v1_content))
+            .send()
+            .await
+            .expect("PUT source v1 failed");
+        let v1_id = put_v1.version_id().expect("source v1 must have a version id").to_string();
+
+        // Source version 2: becomes the latest, so v1 is deliberately NOT the current version.
+        let v2_content = b"quarantined payload -- source version two (now current, must be ignored)";
+        let put_v2 = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .body(ByteStream::from_static(v2_content))
+            .send()
+            .await
+            .expect("PUT source v2 failed");
+        let v2_id = put_v2.version_id().expect("source v2 must have a version id").to_string();
+        assert_ne!(v1_id, v2_id, "the two source puts must produce distinct versions");
+
+        // Copy the exact NON-LATEST source version into the destination bucket.
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}?versionId={v1_id}"))
+            .send()
+            .await
+            .expect("CopyObject of a versioned source must succeed");
+
+        // (3) The response must echo the exact source version copied.
+        let copy_source_version_id = copy_out
+            .copy_source_version_id()
+            .expect("issue #4976: response must include x-amz-copy-source-version-id for a versioned source");
+        assert_eq!(
+            copy_source_version_id, v1_id,
+            "x-amz-copy-source-version-id must equal the exact source version requested, not the latest source version"
+        );
+        assert_ne!(
+            copy_source_version_id, v2_id,
+            "x-amz-copy-source-version-id must not be the latest source version"
+        );
+
+        // (4) The destination header must identify a distinct, newly created version.
+        let dst_version_id = copy_out
+            .version_id()
+            .expect("destination copy must create a new version id")
+            .to_string();
+        assert!(!dst_version_id.is_empty(), "destination version id must be present");
+        assert_ne!(dst_version_id, v1_id, "destination version must be distinct from the source version");
+        assert_ne!(
+            dst_version_id, v2_id,
+            "destination version must be distinct from the source latest version"
+        );
+
+        // (5) The destination must hold the exact bytes/size of the copied (v1) source version.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.content_length(),
+            Some(v1_content.len() as i64),
+            "destination size must equal the copied source version (v1)"
+        );
+
+        let get_dst = client
+            .get_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .version_id(&dst_version_id)
+            .send()
+            .await
+            .expect("GET destination failed");
+        let dst_body = get_dst.body.collect().await.expect("collect destination body").into_bytes();
+        assert_eq!(
+            dst_body.as_ref(),
+            v1_content,
+            "destination bytes must exactly equal the copied source version (v1), not the latest (v2)"
+        );
+
+        // (6) The source version copied from must remain present and independently readable.
+        let get_src_v1 = client
+            .get_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .version_id(&v1_id)
+            .send()
+            .await
+            .expect("GET source v1 failed after copy");
+        let src_v1_body = get_src_v1.body.collect().await.expect("collect source v1 body").into_bytes();
+        assert_eq!(src_v1_body.as_ref(), v1_content, "source v1 must remain intact after the copy");
+
+        env.stop_server();
+    }
 }

crates/e2e_test/src/fake_s3_target/README.md

@@ -0,0 +1,11 @@
+# Programmable fake S3 target
+
+This module is the shared failure-injection boundary for replication end-to-end tests. It runs an in-process, path-style S3 endpoint backed by `s3s`; no production crate depends on it.
+
+`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
+
+Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
+
+Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
+
+The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.

crates/e2e_test/src/fault_proxy.rs

@@ -0,0 +1,498 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! In-process, socket-level network fault-injection proxy for black-box
+//! cluster E2E tests (backlog#1325 network fault-injection block).
+//!
+//! `FaultProxy` binds a TCP listener on a random local port, forwards every
+//! accepted connection to a fixed target address, and lets a test flip the
+//! forwarding behaviour at runtime. It is the black-box counterpart to the
+//! in-process white-box hooks (rename barrier, disk call counters): those
+//! `#[cfg(test)]` primitives cannot reach across the process boundary into a
+//! spawned RustFS server, whereas this proxy sits on the wire between two
+//! nodes and needs no cooperation from the peer.
+//!
+//! Service targets:
+//! - **#1312 / #1319** lock-plane one-way partition and accept-then-blackhole
+//!   peer: run a node's lock/RPC port through the proxy, then
+//!   [`FaultMode::Partition`] one direction (data plane stays reachable via a
+//!   separate direct port) or [`FaultMode::Blackhole`] the whole connection.
+//! - **#1327** cross-process replay / tamper harness: the proxy is the wire
+//!   seam a replay/tamper filter can later hook into.
+//!
+//! Supported fault modes:
+//! - [`FaultMode::Pass`]: transparent byte forwarding (round-trip identical).
+//! - [`FaultMode::Latency`]: delay every forwarded chunk by a fixed duration.
+//! - [`FaultMode::Blackhole`]: accept the connection but forward nothing in
+//!   either direction and never respond (models an accepted-then-dead peer);
+//!   the client observes a read timeout, the proxy never panics.
+//! - [`FaultMode::Partition`]: block exactly one direction while the other
+//!   keeps flowing (models a lock-RPC one-way partition with a live data
+//!   plane).
+//!
+//! Wiring this into the cluster harness ("expose node port N through a proxy")
+//! is intentionally left as a follow-up: the harness multi-drive / 2-pool
+//! changes land separately (rustfs#4937), so this block ships the proxy tool
+//! plus its local self-tests against a loopback echo server and stays
+//! independent of that harness work.
+//!
+//! # Example
+//!
+//! ```ignore
+//! let proxy = FaultProxy::start(target_addr).await?;
+//! let via = proxy.local_addr(); // point the client here instead of `target_addr`
+//! proxy.set_mode(FaultMode::Latency(Duration::from_millis(50)));
+//! // ... exercise the client ...
+//! proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
+//! // ... assert the blocked direction is dead ...
+//! proxy.shutdown().await; // releases the listener port
+//! ```
+
+use std::io;
+use std::net::{Ipv4Addr, SocketAddr};
+use std::time::Duration;
+
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::{TcpListener, TcpStream};
+use tokio::sync::watch;
+use tokio::task::JoinHandle;
+
+/// A single logical direction of a proxied connection.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Direction {
+    /// Bytes travelling from the connecting client towards the target server
+    /// (the forward / request path).
+    ClientToServer,
+    /// Bytes travelling from the target server back to the client (the return
+    /// / response path).
+    ServerToClient,
+}
+
+/// Runtime-switchable forwarding behaviour of a [`FaultProxy`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FaultMode {
+    /// Forward bytes transparently in both directions.
+    Pass,
+    /// Forward bytes in both directions, but delay every chunk by this
+    /// duration before it is written onward.
+    Latency(Duration),
+    /// Accept connections but forward nothing in either direction and never
+    /// respond. Bytes read from either side are drained and discarded so the
+    /// proxy never blocks or panics; the peer simply never hears back.
+    Blackhole,
+    /// Block exactly one direction (drop its bytes) while the other direction
+    /// keeps forwarding normally.
+    Partition(Direction),
+}
+
+/// An async TCP proxy that forwards a random local port to a fixed target and
+/// can inject network faults at runtime.
+///
+/// The proxy owns a background accept loop; each accepted connection is
+/// handled by its own task pair (one per direction). [`FaultProxy::shutdown`]
+/// stops the accept loop and drops the listener, releasing the bound port.
+pub struct FaultProxy {
+    listen_addr: SocketAddr,
+    target: SocketAddr,
+    mode_tx: watch::Sender<FaultMode>,
+    shutdown_tx: watch::Sender<bool>,
+    accept_task: JoinHandle<()>,
+}
+
+impl FaultProxy {
+    /// Bind a listener on `127.0.0.1:0` and start forwarding accepted
+    /// connections to `target`. Starts in [`FaultMode::Pass`].
+    pub async fn start(target: SocketAddr) -> io::Result<Self> {
+        let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
+        let listen_addr = listener.local_addr()?;
+
+        let (mode_tx, mode_rx) = watch::channel(FaultMode::Pass);
+        let (shutdown_tx, shutdown_rx) = watch::channel(false);
+
+        let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx));
+
+        Ok(Self {
+            listen_addr,
+            target,
+            mode_tx,
+            shutdown_tx,
+            accept_task,
+        })
+    }
+
+    /// Local address the proxy is listening on. Point clients here instead of
+    /// the real target to route their traffic through the proxy.
+    pub fn local_addr(&self) -> SocketAddr {
+        self.listen_addr
+    }
+
+    /// The fixed target address every connection is forwarded to.
+    pub fn target_addr(&self) -> SocketAddr {
+        self.target
+    }
+
+    /// The mode currently in effect.
+    pub fn mode(&self) -> FaultMode {
+        *self.mode_tx.borrow()
+    }
+
+    /// Switch the fault mode at runtime. Takes effect on the next chunk read by
+    /// each direction of every live and future connection.
+    pub fn set_mode(&self, mode: FaultMode) {
+        // A send only fails if every receiver has dropped, i.e. the accept loop
+        // and all connections have already ended; there is nothing to steer.
+        let _ = self.mode_tx.send(mode);
+    }
+
+    /// Stop the accept loop, signal live connections to wind down, and drop the
+    /// listener so the bound port is released. Awaits the accept loop so the
+    /// port is free once this returns.
+    pub async fn shutdown(self) {
+        let _ = self.shutdown_tx.send(true);
+        let _ = self.accept_task.await;
+    }
+}
+
+/// Accept loop: takes new connections until shutdown is signalled, then returns
+/// (dropping `listener`, which releases the port).
+async fn accept_loop(
+    listener: TcpListener,
+    target: SocketAddr,
+    mode_rx: watch::Receiver<FaultMode>,
+    mut shutdown_rx: watch::Receiver<bool>,
+) {
+    loop {
+        tokio::select! {
+            biased;
+            _ = shutdown_rx.changed() => {
+                if *shutdown_rx.borrow() {
+                    break;
+                }
+            }
+            accepted = listener.accept() => {
+                match accepted {
+                    Ok((client, _peer)) => {
+                        tokio::spawn(handle_connection(
+                            client,
+                            target,
+                            mode_rx.clone(),
+                            shutdown_rx.clone(),
+                        ));
+                    }
+                    // A transient accept error should not tear the proxy down.
+                    Err(_) => continue,
+                }
+            }
+        }
+    }
+}
+
+/// Bridge one accepted client connection to a fresh connection to the target,
+/// running an independent pump per direction.
+async fn handle_connection(
+    client: TcpStream,
+    target: SocketAddr,
+    mode_rx: watch::Receiver<FaultMode>,
+    shutdown_rx: watch::Receiver<bool>,
+) {
+    let server = match TcpStream::connect(target).await {
+        Ok(s) => s,
+        // Target unreachable: nothing to bridge; drop the client.
+        Err(_) => return,
+    };
+
+    let (client_rd, client_wr) = client.into_split();
+    let (server_rd, server_wr) = server.into_split();
+
+    let up = tokio::spawn(pump(
+        Direction::ClientToServer,
+        client_rd,
+        server_wr,
+        mode_rx.clone(),
+        shutdown_rx.clone(),
+    ));
+    let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx));
+
+    let _ = up.await;
+    let _ = down.await;
+}
+
+/// Copy bytes from `from` to `to` for a single direction, applying the current
+/// [`FaultMode`]. Returns when the source hits EOF/error, a write fails, or
+/// shutdown is signalled.
+async fn pump<R, W>(
+    dir: Direction,
+    mut from: R,
+    mut to: W,
+    mut mode_rx: watch::Receiver<FaultMode>,
+    mut shutdown_rx: watch::Receiver<bool>,
+) where
+    R: AsyncReadExt + Unpin,
+    W: AsyncWriteExt + Unpin,
+{
+    let mut buf = vec![0u8; 16 * 1024];
+    loop {
+        let n = tokio::select! {
+            biased;
+            _ = shutdown_rx.changed() => break,
+            read = from.read(&mut buf) => match read {
+                Ok(0) => break,        // clean EOF
+                Ok(n) => n,
+                Err(_) => break,       // reset / error: end this direction
+            },
+        };
+
+        // Snapshot the mode without holding the borrow across an await.
+        let mode = *mode_rx.borrow_and_update();
+        match mode {
+            // Whole connection is a black hole: drain and drop.
+            FaultMode::Blackhole => continue,
+            // This direction is partitioned off: drop its bytes; the peer keeps
+            // reading nothing while the other direction may still flow.
+            FaultMode::Partition(blocked) if blocked == dir => continue,
+            FaultMode::Latency(delay) => {
+                tokio::time::sleep(delay).await;
+                if to.write_all(&buf[..n]).await.is_err() {
+                    break;
+                }
+                if to.flush().await.is_err() {
+                    break;
+                }
+            }
+            FaultMode::Pass | FaultMode::Partition(_) => {
+                if to.write_all(&buf[..n]).await.is_err() {
+                    break;
+                }
+                if to.flush().await.is_err() {
+                    break;
+                }
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::Arc;
+    use std::time::Instant;
+    use tokio::sync::Mutex;
+
+    /// A loopback echo server that records every byte it receives, so tests can
+    /// distinguish "the server never got it" (client->server blocked) from "the
+    /// server got it but the reply never came back" (server->client blocked).
+    struct EchoServer {
+        addr: SocketAddr,
+        received: Arc<Mutex<Vec<u8>>>,
+        _task: JoinHandle<()>,
+    }
+
+    impl EchoServer {
+        async fn start() -> io::Result<Self> {
+            let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
+            let addr = listener.local_addr()?;
+            let received = Arc::new(Mutex::new(Vec::new()));
+            let received_task = received.clone();
+            let task = tokio::spawn(async move {
+                loop {
+                    let Ok((mut sock, _)) = listener.accept().await else {
+                        break;
+                    };
+                    let received_conn = received_task.clone();
+                    tokio::spawn(async move {
+                        let mut buf = vec![0u8; 16 * 1024];
+                        loop {
+                            match sock.read(&mut buf).await {
+                                Ok(0) | Err(_) => break,
+                                Ok(n) => {
+                                    received_conn.lock().await.extend_from_slice(&buf[..n]);
+                                    if sock.write_all(&buf[..n]).await.is_err() {
+                                        break;
+                                    }
+                                    let _ = sock.flush().await;
+                                }
+                            }
+                        }
+                    });
+                }
+            });
+            Ok(Self {
+                addr,
+                received,
+                _task: task,
+            })
+        }
+
+        async fn received(&self) -> Vec<u8> {
+            self.received.lock().await.clone()
+        }
+    }
+
+    /// Write `payload`, then try to read up to `want` bytes with a deadline.
+    /// Returns the bytes actually received before the timeout (possibly empty).
+    async fn write_then_read(stream: &mut TcpStream, payload: &[u8], want: usize, timeout: Duration) -> Vec<u8> {
+        stream.write_all(payload).await.expect("client write");
+        stream.flush().await.expect("client flush");
+
+        let mut out = Vec::new();
+        let mut buf = vec![0u8; want.max(1)];
+        let deadline = tokio::time::Instant::now() + timeout;
+        while out.len() < want {
+            let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
+            if remaining.is_zero() {
+                break;
+            }
+            match tokio::time::timeout(remaining, stream.read(&mut buf)).await {
+                Ok(Ok(0)) => break, // peer closed
+                Ok(Ok(n)) => out.extend_from_slice(&buf[..n]),
+                Ok(Err(_)) => break, // connection error
+                Err(_) => break,     // read timed out
+            }
+        }
+        out
+    }
+
+    #[tokio::test]
+    async fn pass_mode_round_trips_bytes() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        assert_eq!(proxy.mode(), FaultMode::Pass);
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"hello-fault-proxy";
+        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
+
+        assert_eq!(got, payload, "pass mode must forward bytes unchanged");
+        assert_eq!(echo.received().await, payload, "server must have received the bytes");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn latency_mode_delays_forwarding() {
+        // Loose lower bound only: sleep() guarantees *at least* the delay, so
+        // the assertion cannot flake on a slow machine, and we never assert an
+        // upper bound.
+        let delay = Duration::from_millis(200);
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Latency(delay));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"delayed";
+        let start = Instant::now();
+        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
+        let elapsed = start.elapsed();
+
+        assert_eq!(got, payload, "latency mode must still deliver the bytes");
+        // One delay on the request path plus one on the echo return path: the
+        // round trip must exceed at least a single configured delay.
+        assert!(elapsed >= delay, "round trip {elapsed:?} shorter than configured delay {delay:?}");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn blackhole_mode_yields_no_response_and_no_panic() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Blackhole);
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let got = write_then_read(&mut client, b"into-the-void", 1, Duration::from_millis(300)).await;
+
+        assert!(got.is_empty(), "blackhole mode must not return any bytes, got {got:?}");
+        assert!(echo.received().await.is_empty(), "blackhole must not forward to the server");
+
+        // Proxy is still alive and steerable after a blackholed connection.
+        proxy.set_mode(FaultMode::Pass);
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn partition_client_to_server_blocks_request_path() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let got = write_then_read(&mut client, b"blocked-request", 1, Duration::from_millis(300)).await;
+
+        assert!(got.is_empty(), "client->server partition must yield no echo");
+        assert!(
+            echo.received().await.is_empty(),
+            "client->server partition must stop bytes from reaching the server"
+        );
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn partition_server_to_client_blocks_only_return_path() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Partition(Direction::ServerToClient));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"one-way";
+        let got = write_then_read(&mut client, payload, 1, Duration::from_millis(300)).await;
+
+        // Client hears nothing back...
+        assert!(got.is_empty(), "server->client partition must block the reply");
+        // ...but the request path is live, so the server did receive the bytes.
+        // Poll briefly to avoid racing the forward direction.
+        let mut server_saw = Vec::new();
+        for _ in 0..30 {
+            server_saw = echo.received().await;
+            if !server_saw.is_empty() {
+                break;
+            }
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
+        assert_eq!(server_saw, payload, "server->client partition must leave the request path intact");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn set_mode_switches_behaviour_at_runtime() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+
+        // Start passing: first exchange round-trips.
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let first = write_then_read(&mut client, b"first", 5, Duration::from_secs(5)).await;
+        assert_eq!(first, b"first", "initial pass exchange must round-trip");
+
+        // Flip to blackhole on the same live connection: the next write gets no reply.
+        proxy.set_mode(FaultMode::Blackhole);
+        let second = write_then_read(&mut client, b"second", 1, Duration::from_millis(300)).await;
+        assert!(second.is_empty(), "after switching to blackhole the live connection must go silent");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn shutdown_releases_the_listener_port() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        let addr = proxy.local_addr();
+
+        proxy.shutdown().await;
+
+        // The port must be free to bind again once shutdown returns.
+        let rebind = TcpListener::bind(addr).await;
+        assert!(rebind.is_ok(), "shutdown must release the listener port {addr}");
+    }
+}

crates/e2e_test/src/lib.rs

@@ -23,6 +23,18 @@ pub mod common;
 #[cfg(test)]
 pub mod chaos;
 
+// Programmable S3 target for replication failure-path tests (backlog#1147 repl-8).
+#[cfg(test)]
+pub mod fake_s3_target;
+
+// Socket-level network fault-injection proxy for black-box cluster tests
+// (backlog#1325 network fault-injection block): latency / blackhole / one-way
+// partition on the wire between nodes. Serves #1312/#1319 (lock-plane one-way
+// partition, accept-then-blackhole peer) and #1327 (cross-process replay/tamper
+// seam); cluster-harness wiring is a follow-up (rustfs#4937).
+#[cfg(test)]
+pub mod fault_proxy;
+
 // Reliability tests built on the fault-injection harness
 #[cfg(test)]
 mod reliability_disk_fault_test;
@@ -71,6 +83,10 @@ mod security_boundary_test;
 #[cfg(test)]
 mod api_rate_limit_test;
 
+// Opt-in global connection cap on the main listener (backlog#1191 follow-up)
+#[cfg(test)]
+mod connection_cap_test;
+
 // Admin authorization gate: non-admin denial + root-credential lifecycle (backlog#1151 sec-4)
 #[cfg(test)]
 mod admin_auth_test;
@@ -147,6 +163,10 @@ mod object_lock;
 #[cfg(test)]
 mod cluster_concurrency_test;
 
+// Multi-drive (drivesPerNode) and 2-pool cluster harness smoke tests
+#[cfg(test)]
+mod cluster_multidrive_pool_test;
+
 // PutObject / MultipartUpload with checksum (Content-MD5, x-amz-checksum-*)
 #[cfg(test)]
 mod checksum_upload_test;
@@ -170,6 +190,9 @@ mod copy_object_metadata_test;
 #[cfg(test)]
 mod copy_object_version_restore_test;
 
+#[cfg(test)]
+mod copy_object_checksum_test;
+
 // S3 dummy-compat bucket API tests
 #[cfg(test)]
 mod bucket_logging_test;
@@ -219,6 +242,9 @@ mod console_smoke_test;
 #[cfg(test)]
 mod admin_iam_crud_test;
 
+#[cfg(test)]
+mod admin_pools_test;
+
 // Replication extension end-to-end regression tests
 #[cfg(test)]
 mod replication_extension_test;

crates/e2e_test/src/multipart_auth_test.rs

@@ -53,6 +53,17 @@ fn sse_customer_key_md5_base64(key: &str) -> String {
     base64::engine::general_purpose::STANDARD.encode(md5::compute(key).0)
 }
 
+/// Env var consumed by the local SSE-S3 DEK provider when KMS is not configured.
+///
+/// Since rustfs#3564 the server fails closed on managed SSE (SSE-S3 or
+/// bucket-default encryption) unless KMS is configured or this master key is
+/// provided, so tests exercising managed SSE on a bare server must seed it.
+const LOCAL_SSE_MASTER_KEY_ENV: &str = "RUSTFS_SSE_S3_MASTER_KEY";
+
+fn local_sse_master_key_value() -> String {
+    base64::engine::general_purpose::STANDARD.encode([0x42u8; 32])
+}
+
 async fn make_tar(files: &[(&str, &[u8])], dirs: &[&str]) -> Vec<u8> {
     let buf = Cursor::new(Vec::new());
     let mut builder = tokio_tar::Builder::new(buf);
@@ -105,7 +116,11 @@ async fn make_tar_with_pax_entry(path: &str, data: &[u8], mtime: Option<u64>, pa
             pax_payload.extend(build_pax_record(key, value));
         }
 
-        let mut pax_header = tokio_tar::Header::new_gnu();
+        // Pax extension entries must carry a POSIX ustar header — this is what real
+        // tar writers emit, and the server-side reader rejects an XHeader typeflag on
+        // GNU-format headers ("extension typeflag is not permitted on an unrecognized
+        // header").
+        let mut pax_header = tokio_tar::Header::new_ustar();
         pax_header.set_entry_type(tokio_tar::EntryType::XHeader);
         pax_header.set_size(pax_payload.len() as u64);
         pax_header.set_mode(0o644);
@@ -926,7 +941,9 @@ async fn test_anonymous_post_object_accepts_sse_s3() -> Result<(), Box<dyn std::
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-sse-s3";
     let object_key = "post-sse-s3-object.txt";
@@ -982,7 +999,9 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_s3() -> Result<(), B
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-default-sse-s3";
     let object_key = "post-default-sse-s3-object.txt";
@@ -1053,7 +1072,9 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_kms() -> Result<(),
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-default-sse-kms";
     let object_key = "post-default-sse-kms-object.txt";
@@ -1172,15 +1193,24 @@ async fn test_anonymous_post_object_rejects_sse_s3_policy_mismatch() -> Result<(
 
 #[tokio::test]
 #[serial]
-async fn test_anonymous_post_object_rejects_sse_s3_missing_from_policy_conditions()
+async fn test_anonymous_post_object_accepts_sse_s3_missing_from_policy_conditions()
 -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
 
+    // MinIO-compatible POST-policy validation (s3s-project/s3s#608) exempts the
+    // x-amz-server-side-encryption* form fields from the "every form field must
+    // appear in the policy conditions" rule, so an SSE-S3 field that the policy
+    // does not mention is accepted and encryption is applied. When the policy
+    // does cover the field, a value mismatch is still rejected — see
+    // test_anonymous_post_object_rejects_sse_s3_policy_mismatch.
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-sse-s3-missing";
     let object_key = "post-sse-s3-missing-object.txt";
+    let expected_body = b"post-sse-s3-missing".to_vec();
 
     let admin_client = env.create_s3_client();
     admin_client.create_bucket().bucket(bucket).send().await?;
@@ -1198,7 +1228,7 @@ async fn test_anonymous_post_object_rejects_sse_s3_missing_from_policy_condition
         .text("x-amz-server-side-encryption", "AES256")
         .part(
             "file",
-            reqwest::multipart::Part::bytes(b"post-sse-s3-missing".to_vec())
+            reqwest::multipart::Part::bytes(expected_body.clone())
                 .file_name("upload.txt")
                 .mime_str("text/plain")?,
         );
@@ -1212,11 +1242,15 @@ async fn test_anonymous_post_object_rejects_sse_s3_missing_from_policy_condition
     let status = post_resp.status();
     let response_body = post_resp.text().await?;
 
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(
-        response_body.contains("<Code>AccessDenied</Code>"),
-        "response should contain AccessDenied code, got: {response_body}"
-    );
+    assert_eq!(status, reqwest::StatusCode::NO_CONTENT);
+    assert!(response_body.is_empty(), "204 response should not contain a body, got: {response_body}");
+
+    let head = admin_client.head_object().bucket(bucket).key(object_key).send().await?;
+    assert_eq!(head.server_side_encryption().map(|value| value.as_str()), Some("AES256"));
+
+    let uploaded = admin_client.get_object().bucket(bucket).key(object_key).send().await?;
+    let uploaded = uploaded.body.collect().await?.into_bytes();
+    assert_eq!(uploaded.as_ref(), expected_body.as_slice());
 
     Ok(())
 }
@@ -5041,7 +5075,9 @@ async fn test_signed_put_object_extract_preserves_sse_s3_and_redirect() -> Resul
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let sse_master_key = base64::engine::general_purpose::STANDARD.encode([0x42u8; 32]);
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_SSE_S3_MASTER_KEY", sse_master_key.as_str())])
+        .await?;
 
     let bucket = "signed-extract-sse-s3-redirect";
     let archive_key = "encrypted-metadata.tar";
@@ -5318,7 +5354,9 @@ async fn test_signed_put_object_extract_uses_bucket_default_sse_s3() -> Result<(
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let sse_master_key = base64::engine::general_purpose::STANDARD.encode([0x42u8; 32]);
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_SSE_S3_MASTER_KEY", sse_master_key.as_str())])
+        .await?;
 
     let bucket = "signed-extract-default-sse-s3";
     let archive_key = "default-encryption.tar";

crates/e2e_test/src/notification_webhook_test.rs

@@ -39,10 +39,17 @@ use local_ip_address::local_ip;
 use reqwest::StatusCode;
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
+use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
 use s3s::Body;
 use serde_json::Value;
 use serial_test::serial;
 use std::error::Error;
+use std::path::Path;
+use std::sync::{
+    Arc, Once,
+    atomic::{AtomicBool, Ordering},
+};
+use std::thread;
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::TcpListener;
 use tokio::sync::mpsc;
@@ -64,6 +71,11 @@ fn target_arn(target_name: &str) -> String {
     format!("arn:rustfs:sqs:{NOTIFY_REGION}:{target_name}:webhook")
 }
 
+fn endpoint_origin(endpoint: &str) -> Result<String, BoxError> {
+    let parsed = reqwest::Url::parse(endpoint)?;
+    Ok(parsed.origin().ascii_serialization())
+}
+
 // ---------------------------------------------------------------------------
 // In-test HTTP event receiver
 // ---------------------------------------------------------------------------
@@ -147,7 +159,175 @@ async fn spawn_event_collector() -> Result<(String, mpsc::UnboundedReceiver<Valu
     let endpoint_ip = local_ip()?;
     let (tx, rx) = mpsc::unbounded_channel();
     let handle = serve_event_collector(listener, tx);
-    Ok((format!("http://{endpoint_ip}.nip.io:{port}/events"), rx, handle))
+    Ok((format!("http://{}/events", std::net::SocketAddr::new(endpoint_ip, port)), rx, handle))
+}
+
+struct HttpsEventCollector {
+    endpoint: String,
+    running: Arc<AtomicBool>,
+    handle: Option<thread::JoinHandle<()>>,
+    events: mpsc::UnboundedReceiver<Value>,
+}
+
+impl HttpsEventCollector {
+    fn endpoint(&self) -> &str {
+        &self.endpoint
+    }
+
+    fn events_mut(&mut self) -> &mut mpsc::UnboundedReceiver<Value> {
+        &mut self.events
+    }
+
+    fn shutdown(&mut self) -> TestResult {
+        self.running.store(false, Ordering::Relaxed);
+        if let Ok(parsed) = self.endpoint.parse::<reqwest::Url>()
+            && let Some(port) = parsed.port()
+        {
+            let _ = std::net::TcpStream::connect(("127.0.0.1", port));
+        }
+        if let Some(handle) = self.handle.take() {
+            handle.join().map_err(|_| "https event collector thread panicked")?;
+        }
+        Ok(())
+    }
+}
+
+impl Drop for HttpsEventCollector {
+    fn drop(&mut self) {
+        let _ = self.shutdown();
+    }
+}
+
+fn spawn_https_event_collector(ca_path: &Path) -> Result<HttpsEventCollector, BoxError> {
+    use rustls::{
+        ServerConfig,
+        pki_types::{PrivateKeyDer, PrivatePkcs8KeyDer},
+    };
+    use std::io::ErrorKind;
+    use std::net::TcpListener as StdTcpListener;
+
+    static INSTALL_CRYPTO_PROVIDER: Once = Once::new();
+    INSTALL_CRYPTO_PROVIDER.call_once(|| {
+        let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+    });
+
+    let listener = StdTcpListener::bind("0.0.0.0:0")?;
+    listener.set_nonblocking(true)?;
+    let addr = listener.local_addr()?;
+    let endpoint_ip = local_ip()?;
+    let endpoint_host = endpoint_ip.to_string();
+
+    let rcgen::CertifiedKey { cert, signing_key } = rcgen::generate_simple_self_signed(vec![endpoint_host])?;
+    std::fs::write(ca_path, cert.pem())?;
+
+    let cert_chain = vec![cert.der().clone()];
+    let key_der = PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(signing_key.serialize_der()));
+    let server_config = Arc::new(
+        ServerConfig::builder()
+            .with_no_client_auth()
+            .with_single_cert(cert_chain, key_der)?,
+    );
+
+    let running = Arc::new(AtomicBool::new(true));
+    let server_running = Arc::clone(&running);
+    let (tx, events) = mpsc::unbounded_channel();
+    let handle = thread::spawn(move || {
+        let mut connections = Vec::new();
+        while server_running.load(Ordering::Relaxed) {
+            match listener.accept() {
+                Ok((stream, _)) => {
+                    let config = Arc::clone(&server_config);
+                    let tx = tx.clone();
+                    connections.push(thread::spawn(move || {
+                        let _ = handle_https_request(stream, config, tx);
+                    }));
+                }
+                Err(err) if err.kind() == ErrorKind::WouldBlock => thread::sleep(Duration::from_millis(20)),
+                Err(_) => break,
+            }
+        }
+        for connection in connections {
+            let _ = connection.join();
+        }
+    });
+
+    Ok(HttpsEventCollector {
+        endpoint: format!("https://{}/events", std::net::SocketAddr::new(endpoint_ip, addr.port())),
+        running,
+        handle: Some(handle),
+        events,
+    })
+}
+
+fn read_sync_http_message<R: std::io::Read>(stream: &mut R) -> Result<(String, Vec<u8>), BoxError> {
+    let mut buffer = Vec::new();
+    let mut chunk = [0_u8; 4096];
+    let header_end = loop {
+        let read = stream.read(&mut chunk)?;
+        if read == 0 {
+            return Err("connection closed before request headers were complete".into());
+        }
+        buffer.extend_from_slice(&chunk[..read]);
+        if let Some(pos) = buffer.windows(4).position(|window| window == b"\r\n\r\n") {
+            break pos;
+        }
+    };
+
+    let header_text = std::str::from_utf8(&buffer[..header_end])?;
+    let mut lines = header_text.split("\r\n");
+    let method = lines
+        .next()
+        .and_then(|line| line.split_whitespace().next())
+        .ok_or("missing request method")?
+        .to_string();
+    let mut content_length = 0usize;
+    for line in lines {
+        if let Some((name, value)) = line.split_once(':')
+            && name.trim().eq_ignore_ascii_case("content-length")
+        {
+            content_length = value.trim().parse()?;
+        }
+    }
+
+    let body_offset = header_end + 4;
+    while buffer.len().saturating_sub(body_offset) < content_length {
+        let read = stream.read(&mut chunk)?;
+        if read == 0 {
+            return Err("connection closed before request body was complete".into());
+        }
+        buffer.extend_from_slice(&chunk[..read]);
+    }
+    Ok((method, buffer[body_offset..body_offset + content_length].to_vec()))
+}
+
+fn handle_https_request(
+    stream: std::net::TcpStream,
+    server_config: Arc<rustls::ServerConfig>,
+    tx: mpsc::UnboundedSender<Value>,
+) -> Result<(), BoxError> {
+    use std::io::Write;
+
+    stream.set_nonblocking(false)?;
+    stream.set_read_timeout(Some(Duration::from_secs(5)))?;
+    stream.set_write_timeout(Some(Duration::from_secs(5)))?;
+    let connection = rustls::ServerConnection::new(server_config)?;
+    let mut tls_stream = rustls::StreamOwned::new(connection, stream);
+    let (method, body) = read_sync_http_message(&mut tls_stream)?;
+    let response = "HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n";
+    tls_stream.write_all(response.as_bytes())?;
+    tls_stream.flush()?;
+    tls_stream.conn.send_close_notify();
+    while tls_stream.conn.wants_write() {
+        tls_stream.conn.write_tls(&mut tls_stream.sock)?;
+    }
+    let _ = tls_stream.sock.shutdown(std::net::Shutdown::Write);
+    if method == "POST"
+        && !body.is_empty()
+        && let Ok(event) = serde_json::from_slice(&body)
+    {
+        let _ = tx.send(event);
+    }
+    Ok(())
 }
 
 /// Decoded object key of the first record in an event envelope.
@@ -284,13 +464,24 @@ async fn enable_notify_module(env: &RustFSTestEnvironment) -> TestResult {
 /// Registers a webhook notification target with a persistent queue directory, so
 /// delivery goes through the durable store-and-forward path.
 async fn configure_webhook_target(env: &RustFSTestEnvironment, target_name: &str, endpoint: &str) -> TestResult {
+    configure_webhook_target_with_key_values(env, target_name, vec![("endpoint", endpoint.to_string())]).await
+}
+
+async fn configure_webhook_target_with_key_values(
+    env: &RustFSTestEnvironment,
+    target_name: &str,
+    mut key_values: Vec<(&str, String)>,
+) -> TestResult {
     let queue_dir = format!("{}/notify-queue-{target_name}", env.temp_dir);
     tokio::fs::create_dir_all(&queue_dir).await?;
+    if !key_values.iter().any(|(key, _)| *key == "queue_dir") {
+        key_values.push(("queue_dir", queue_dir));
+    }
     let payload = serde_json::json!({
-        "key_values": [
-            { "key": "endpoint", "value": endpoint },
-            { "key": "queue_dir", "value": queue_dir },
-        ]
+        "key_values": key_values
+            .into_iter()
+            .map(|(key, value)| serde_json::json!({ "key": key, "value": value }))
+            .collect::<Vec<_>>(),
     });
     let url = format!("{}/rustfs/admin/v3/target/notify_webhook/{target_name}", env.url);
     let response = signed_admin_request(env, http::Method::PUT, &url, Some(payload.to_string().into_bytes())).await?;
@@ -321,6 +512,31 @@ async fn wait_for_target_registered(env: &RustFSTestEnvironment, target_name: &s
     Err(format!("target {target_name} was not registered in admin ARNs").into())
 }
 
+async fn wait_for_target_online(env: &RustFSTestEnvironment, target_name: &str) -> TestResult {
+    let url = format!("{}/rustfs/admin/v3/target/list", env.url);
+    for _ in 0..40 {
+        let response = signed_admin_request(env, http::Method::GET, &url, None).await?;
+        if response.status() == StatusCode::OK {
+            let body: Value = serde_json::from_slice(&response.bytes().await?)?;
+            if body["notify_enabled"].as_bool() != Some(true) {
+                return Err(format!("admin target list did not report notify_enabled=true: {body}").into());
+            }
+            let listed = body["notification_endpoints"].as_array().is_some_and(|endpoints| {
+                endpoints.iter().any(|endpoint| {
+                    endpoint["account_id"].as_str() == Some(target_name)
+                        && endpoint["service"].as_str() == Some("webhook")
+                        && endpoint["status"].as_str() == Some("online")
+                })
+            });
+            if listed {
+                return Ok(());
+            }
+        }
+        tokio::time::sleep(Duration::from_millis(250)).await;
+    }
+    Err(format!("target {target_name} did not become online in admin targets").into())
+}
+
 /// Binds a bucket to a webhook target for ObjectCreated:*/ObjectRemoved:* events,
 /// filtered to `prefix` + `suffix`.
 async fn put_notification_config(client: &Client, bucket: &str, target_name: &str, prefix: &str, suffix: &str) -> TestResult {
@@ -367,6 +583,62 @@ fn trimmed_etag(value: Option<&str>) -> Option<String> {
 // Tests
 // ---------------------------------------------------------------------------
 
+/// Regression for rustfs#5052: with the notify module enabled through
+/// RUSTFS_NOTIFY_ENABLE, an HTTPS webhook using a configured CA must become
+/// online and receive a real S3 event POST.
+#[tokio::test]
+#[serial]
+async fn test_https_webhook_target_delivers_event_with_notify_env_enabled() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    let ca_path = Path::new(&env.temp_dir).join("https-webhook-ca.pem");
+    let mut collector = spawn_https_event_collector(&ca_path)?;
+    let allowed_origin = endpoint_origin(collector.endpoint())?;
+    env.start_rustfs_server_with_env(
+        vec![],
+        &[
+            ("RUSTFS_NOTIFY_ENABLE", "true"),
+            (ENV_OUTBOUND_ALLOW_ORIGINS, allowed_origin.as_str()),
+        ],
+    )
+    .await?;
+    let target = "peri1https";
+    let bucket = "peri1-https-events";
+    let key = "uploads/https.dat";
+    let client = env.create_s3_client();
+    client.create_bucket().bucket(bucket).send().await?;
+
+    configure_webhook_target_with_key_values(
+        &env,
+        target,
+        vec![
+            ("endpoint", collector.endpoint().to_string()),
+            ("client_ca", ca_path.to_string_lossy().into_owned()),
+        ],
+    )
+    .await?;
+    wait_for_target_online(&env, target).await?;
+    wait_for_target_registered(&env, target).await?;
+    put_notification_config(&client, bucket, target, "uploads/", ".dat").await?;
+
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from_static(b"https webhook event body"))
+        .send()
+        .await?;
+    let event = wait_for_event(collector.events_mut(), key, "s3:ObjectCreated:", Duration::from_secs(20)).await?;
+    assert_eq!(event["EventName"].as_str(), Some("s3:ObjectCreated:Put"));
+    assert_eq!(event["Records"][0]["s3"]["bucket"]["name"].as_str(), Some(bucket));
+    assert_eq!(event_key(&event).as_deref(), Some(key));
+
+    env.stop_server();
+    collector.shutdown()?;
+    Ok(())
+}
+
 /// PUT / multipart-complete / DELETE each deliver one event with correct fields,
 /// and the prefix/suffix filter drops non-matching keys.
 #[tokio::test]
@@ -375,9 +647,11 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
     init_logging();
 
     let (endpoint, mut rx, handle) = spawn_event_collector().await?;
+    let allowed_origin = endpoint_origin(&endpoint)?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    env.start_rustfs_server_with_env(vec![], &[(ENV_OUTBOUND_ALLOW_ORIGINS, allowed_origin.as_str())])
+        .await?;
     enable_notify_module(&env).await?;
 
     let bucket = "peri1-events";
@@ -529,15 +803,6 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
 async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
     init_logging();
 
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-    enable_notify_module(&env).await?;
-
-    let bucket = "peri1-redeliver";
-    let target = "peri1redeliver";
-    let client = env.create_s3_client();
-    client.create_bucket().bucket(bucket).send().await?;
-
     // Configure the target while its endpoint is reachable so activation and
     // ARN registration complete deterministically. Registering against a dead
     // endpoint stalls behind the reachability probe's timeout and flakes the
@@ -546,10 +811,21 @@ async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
     let listener = TcpListener::bind("0.0.0.0:0").await?;
     let port = listener.local_addr()?.port();
     let endpoint_ip = local_ip()?;
-    let endpoint = format!("http://{endpoint_ip}.nip.io:{port}/events");
+    let endpoint = format!("http://{}/events", std::net::SocketAddr::new(endpoint_ip, port));
+    let allowed_origin = endpoint_origin(&endpoint)?;
     let (setup_tx, _setup_rx) = mpsc::unbounded_channel();
     let setup_handle = serve_event_collector(listener, setup_tx);
 
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[(ENV_OUTBOUND_ALLOW_ORIGINS, allowed_origin.as_str())])
+        .await?;
+    enable_notify_module(&env).await?;
+
+    let bucket = "peri1-redeliver";
+    let target = "peri1redeliver";
+    let client = env.create_s3_client();
+    client.create_bucket().bucket(bucket).send().await?;
+
     configure_webhook_target(&env, target, &endpoint).await?;
     wait_for_target_registered(&env, target).await?;
     put_notification_config(&client, bucket, target, "uploads/", ".dat").await?;

crates/e2e_test/src/object_lambda_test.rs

@@ -18,6 +18,7 @@ use http::header::{CONTENT_TYPE, HOST};
 use reqwest::StatusCode;
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::{pre_sign_v4, sign_v4};
+use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
 use s3s::Body;
 use serial_test::serial;
 use std::collections::HashMap;
@@ -49,7 +50,7 @@ fn find_header_terminator(buf: &[u8]) -> Option<usize> {
 
 async fn read_http_request(
     stream: &mut tokio::net::TcpStream,
-) -> Result<(HashMap<String, String>, Vec<u8>), Box<dyn Error + Send + Sync>> {
+) -> Result<(String, HashMap<String, String>, Vec<u8>), Box<dyn Error + Send + Sync>> {
     let mut buffer = Vec::new();
     let mut chunk = [0_u8; 4096];
 
@@ -67,7 +68,7 @@ async fn read_http_request(
     let header_bytes = &buffer[..header_end];
     let header_text = std::str::from_utf8(header_bytes)?;
     let mut lines = header_text.split("\r\n");
-    let _request_line = lines.next().ok_or("missing request line")?;
+    let request_line = lines.next().ok_or("missing request line")?.to_string();
     let mut headers = HashMap::new();
     for line in lines {
         if line.is_empty() {
@@ -79,8 +80,9 @@ async fn read_http_request(
 
     let content_length = headers
         .get("content-length")
-        .ok_or("missing content-length header")?
-        .parse::<usize>()?;
+        .map(|value| value.parse::<usize>())
+        .transpose()?
+        .unwrap_or_default();
     let body_offset = header_end + 4;
     while buffer.len().saturating_sub(body_offset) < content_length {
         let read = stream.read(&mut chunk).await?;
@@ -90,7 +92,7 @@ async fn read_http_request(
         buffer.extend_from_slice(&chunk[..read]);
     }
 
-    Ok((headers, buffer[body_offset..body_offset + content_length].to_vec()))
+    Ok((request_line, headers, buffer[body_offset..body_offset + content_length].to_vec()))
 }
 
 async fn spawn_object_lambda_webhook_server() -> Result<
@@ -130,9 +132,17 @@ async fn spawn_object_lambda_webhook_server_with_response(
     let handle = tokio::spawn(async move {
         loop {
             let (mut stream, _) = listener.accept().await?;
-            let Ok(Ok((headers, body))) = timeout(Duration::from_secs(2), read_http_request(&mut stream)).await else {
+            let Ok(Ok((request_line, headers, body))) = timeout(Duration::from_secs(2), read_http_request(&mut stream)).await
+            else {
                 continue;
             };
+            if request_line == "HEAD / HTTP/1.1" {
+                stream
+                    .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
+                    .await?;
+                stream.shutdown().await?;
+                continue;
+            }
             let payload: serde_json::Value = serde_json::from_slice(&body)?;
 
             let output_route = payload["getObjectContext"]["outputRoute"]
@@ -412,9 +422,33 @@ async fn wait_for_target_absence(
     Err(format!("target {target_name} remained visible in admin APIs; targets={last_targets}, arns={last_arns:?}").into())
 }
 
-async fn restart_rustfs_server(env: &mut RustFSTestEnvironment) -> Result<(), Box<dyn Error + Send + Sync>> {
+fn endpoint_origin(endpoint: &str) -> Result<String, Box<dyn Error + Send + Sync>> {
+    Ok(reqwest::Url::parse(endpoint)?.origin().ascii_serialization())
+}
+
+async fn start_rustfs_server_for_endpoint(
+    env: &mut RustFSTestEnvironment,
+    endpoint: &str,
+) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let origin = endpoint_origin(endpoint)?;
+    env.start_rustfs_server_with_env(
+        vec![],
+        &[
+            (ENV_OUTBOUND_ALLOW_ORIGINS, origin.as_str()),
+            ("RUSTFS_NOTIFY_ENABLE", "true"),
+        ],
+    )
+    .await
+}
+
+async fn restart_rustfs_server(env: &mut RustFSTestEnvironment, endpoint: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let origin = endpoint_origin(endpoint)?;
     env.stop_server();
-    env.start_rustfs_server_without_cleanup(vec![]).await
+    env.start_rustfs_server_without_cleanup_with_env(&[
+        (ENV_OUTBOUND_ALLOW_ORIGINS, origin.as_str()),
+        ("RUSTFS_NOTIFY_ENABLE", "true"),
+    ])
+    .await
 }
 
 async fn spawn_http_origin_probe_server() -> Result<
@@ -521,7 +555,7 @@ async fn test_notification_target_persists_across_restart_and_delete() -> Result
     let (webhook_url, _request_rx, webhook_handle) = spawn_object_lambda_webhook_server().await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let target_name = "restart-target";
     configure_webhook_target(&env, target_name, &webhook_url, "secret-token").await?;
@@ -535,7 +569,7 @@ async fn test_notification_target_persists_across_restart_and_delete() -> Result
         "target ARN missing after initial configure: {visible_arns:?}"
     );
 
-    restart_rustfs_server(&mut env).await?;
+    restart_rustfs_server(&mut env, &webhook_url).await?;
 
     let (targets_after_restart, arns_after_restart) = wait_for_target_visibility(&env, target_name).await?;
     assert!(notification_target_is_listed(&targets_after_restart, target_name));
@@ -556,7 +590,7 @@ async fn test_notification_target_persists_across_restart_and_delete() -> Result
         "target ARN still visible after delete: {arns_after_delete:?}"
     );
 
-    restart_rustfs_server(&mut env).await?;
+    restart_rustfs_server(&mut env, &webhook_url).await?;
 
     let (targets_after_delete_restart, arns_after_delete_restart) = wait_for_target_absence(&env, target_name).await?;
     assert!(!notification_target_is_listed(&targets_after_delete_restart, target_name));
@@ -581,8 +615,7 @@ async fn test_notification_target_with_path_is_online_via_transport_probe() -> R
     let (webhook_url, mut probe_rx, probe_handle) = spawn_http_origin_probe_server().await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "true")])
-        .await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let target_name = "path-probe";
     configure_webhook_target(&env, target_name, &webhook_url, "secret-token").await?;
@@ -615,7 +648,7 @@ async fn test_get_object_lambda_accepts_presigned_requests() -> Result<(), Box<d
     let (webhook_url, request_rx, webhook_handle) = spawn_object_lambda_webhook_server().await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let bucket = "object-lambda-e2e-presigned";
     let key = "input.txt";
@@ -656,7 +689,7 @@ async fn test_get_object_lambda_accepts_named_webhook_target_arn() -> Result<(),
     let (webhook_url, request_rx, webhook_handle) = spawn_object_lambda_webhook_server().await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let bucket = "object-lambda-e2e-named-target";
     let key = "input.txt";
@@ -696,7 +729,7 @@ async fn test_get_object_lambda_invokes_runtime_webhook_target() -> Result<(), B
     let (webhook_url, request_rx, webhook_handle) = spawn_object_lambda_webhook_server().await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let bucket = "object-lambda-e2e";
     let key = "input.txt";
@@ -777,7 +810,7 @@ async fn test_get_object_lambda_passthroughs_non_success_webhook_response() -> R
     .await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let bucket = "object-lambda-e2e-failure";
     let key = "input.txt";
@@ -832,7 +865,7 @@ async fn test_get_object_lambda_rejects_success_response_without_auth_headers()
     .await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let bucket = "object-lambda-e2e-missing-auth";
     let key = "input.txt";
@@ -879,7 +912,7 @@ async fn test_get_object_lambda_rejects_success_response_with_mismatched_auth_he
     .await?;
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    start_rustfs_server_for_endpoint(&mut env, &webhook_url).await?;
 
     let bucket = "object-lambda-e2e-mismatched-auth";
     let key = "input.txt";

crates/e2e_test/src/quota_test.rs

@@ -277,6 +277,64 @@ mod integration_tests {
         Ok(())
     }
 
+    /// backlog#1336 regression: a PUT that merely declares `Content-Encoding: aws-chunked`
+    /// (no SigV4 streaming payload, so no `x-amz-decoded-content-length`) carries an unframed
+    /// body whose wire Content-Length is the real object size. Quota admission must use that
+    /// length — with and without a hard quota configured — instead of rejecting the request
+    /// with 400 UnexpectedContent, and an over-quota aws-chunked PUT must still get the quota
+    /// rejection.
+    #[tokio::test]
+    #[serial]
+    async fn test_quota_admission_aws_chunked_declared_encoding() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        init_logging();
+        if skip_without_awscurl() {
+            return Ok(());
+        }
+        let env = QuotaTestEnv::new().await?;
+        env.create_bucket().await?;
+
+        let put_aws_chunked = |key: &'static str, size_bytes: usize| {
+            env.client
+                .put_object()
+                .bucket(&env.bucket_name)
+                .key(key)
+                .content_encoding("aws-chunked")
+                .body(aws_sdk_s3::primitives::ByteStream::from(vec![0u8; size_bytes]))
+                .send()
+        };
+
+        // No quota configured: the declared aws-chunked PUT must be admitted.
+        put_aws_chunked("no-quota.bin", 512)
+            .await
+            .expect("declared aws-chunked PUT without quota must succeed");
+        assert!(env.object_exists("no-quota.bin").await?);
+
+        // Hard quota configured: a within-quota declared aws-chunked PUT is admitted
+        // against its wire Content-Length.
+        env.set_bucket_quota(4 * 1024).await?;
+        put_aws_chunked("within-quota.bin", 1024)
+            .await
+            .expect("declared aws-chunked PUT within quota must succeed");
+        assert!(env.object_exists("within-quota.bin").await?);
+
+        // An over-quota declared aws-chunked PUT is rejected by quota admission —
+        // not with UnexpectedContent.
+        let err = put_aws_chunked("over-quota.bin", 16 * 1024)
+            .await
+            .expect_err("declared aws-chunked PUT over quota must be rejected");
+        let err_debug = format!("{err:?}");
+        assert!(
+            !err_debug.contains("UnexpectedContent"),
+            "over-quota rejection must be the quota error, not UnexpectedContent: {err_debug}"
+        );
+        assert!(!env.object_exists("over-quota.bin").await?);
+
+        env.clear_bucket_quota().await?;
+        env.cleanup_bucket().await?;
+
+        Ok(())
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_quota_update_and_clear() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -798,6 +798,13 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
+    async fn scanner_activity(
+        &self,
+        _request: Request<rustfs_protos::proto_gen::node_service::ScannerActivityRequest>,
+    ) -> Result<Response<rustfs_protos::proto_gen::node_service::ScannerActivityResponse>, Status> {
+        Err(Status::unimplemented("lock-only test server"))
+    }
+
     async fn background_heal_status(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::BackgroundHealStatusRequest>,

crates/e2e_test/src/reliant/tiering.rs

@@ -64,7 +64,7 @@ const TIER_NAME: &str = "COLDTIER";
 const TIER_BUCKET: &str = "ilm7-cold-tier";
 const TIER_PREFIX: &str = "tiered";
 const SOURCE_BUCKET: &str = "ilm7-hot";
-const OBJECT_KEY: &str = "tier/report.bin";
+const OBJECT_KEY: &str = "tier/鲁A12345/report.bin";
 const CONTENT_TYPE: &str = "application/x-ilm7";
 const USER_META_KEY: &str = "ilm7-origin";
 const USER_META_VAL: &str = "hermetic-transition";

crates/e2e_test/src/replication_extension_test.rs

@@ -1331,6 +1331,42 @@ async fn assert_managed_sse_replication_fails_explicitly(label: &str, kms: bool)
     Ok(())
 }
 
+async fn wait_for_source_delete_marker_replication_failed(
+    env: &RustFSTestEnvironment,
+    bucket: &str,
+    key: &str,
+) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
+    let url = format!(
+        "{}/rustfs/admin/v3/replication/diff?bucket={}&prefix={}",
+        env.url,
+        urlencoding::encode(bucket),
+        urlencoding::encode(key)
+    );
+
+    loop {
+        let response = signed_request(http::Method::POST, &url, &env.access_key, &env.secret_key, None, None).await?;
+        if response.status() != StatusCode::OK {
+            return Err(format!("replication diff failed with status {}", response.status()).into());
+        }
+        let diff: serde_json::Value = response.json().await?;
+        let failed = diff["Entries"].as_array().is_some_and(|entries| {
+            entries.iter().any(|entry| {
+                entry["Object"].as_str() == Some(key)
+                    && entry["IsDeleteMarker"].as_bool() == Some(true)
+                    && entry["ReplicationStatus"].as_str() == Some("FAILED")
+            })
+        });
+        if failed {
+            return Ok(());
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!("source delete marker {key} never reported FAILED; last diff={diff}").into());
+        }
+        sleep(Duration::from_millis(200)).await;
+    }
+}
+
 /// Return the `LastModified` of the (single) delete marker for `key`, if present.
 async fn delete_marker_last_modified(
     client: &Client,
@@ -1935,7 +1971,10 @@ async fn wait_for_site_replication_enabled(
 ) -> Result<SiteReplicationInfo, Box<dyn Error + Send + Sync>> {
     for _ in 0..40 {
         let info = site_replication_info(env).await?;
-        if info.enabled && info.sites.len() == expected_sites {
+        if info.enabled
+            && info.sites.len() == expected_sites
+            && info.sites.iter().all(|peer| peer.sync_state == SyncStatus::Enable)
+        {
             return Ok(info);
         }
         sleep(Duration::from_millis(250)).await;
@@ -3691,23 +3730,9 @@ async fn test_bucket_replication_replays_failed_entries_after_source_restart() -
     Ok(())
 }
 
-/// backlog#1147 repl-5, scenario (c) — replayed delete marker keeps the source
-/// mtime (mirrors backlog#867).
-///
-/// A delete marker is created while the target is down; the source is then
-/// restarted (data preserved) and the target brought back, so the marker
-/// replicates through the failure-replay path. The replayed marker must carry
-/// the SOURCE's `LastModified`, not the replay time. A deliberate gap before
-/// recovery makes any regression (replay-time stamping) obvious.
-///
-/// The source restart is load-bearing, not just paranoia: on a live
-/// (never-restarted) source, the failed delete-marker replication wedges the
-/// per-object `/[replicate]/<key>` namespace lock and the marker never
-/// replicates even after the target recovers — tracked as backlog#1278. Once
-/// that is fixed, a restart-free variant of this scenario should be added.
 #[tokio::test]
 #[serial]
-async fn test_bucket_replication_replayed_delete_marker_preserves_source_mtime() -> TestResult {
+async fn test_bucket_replication_replayed_delete_marker_preserves_source_mtime_without_source_restart() -> TestResult {
     init_logging();
 
     let mut source_env = RustFSTestEnvironment::new().await?;
@@ -3759,13 +3784,11 @@ async fn test_bucket_replication_replayed_delete_marker_preserves_source_mtime()
         .await?
         .ok_or("source has no delete marker after DELETE")?;
 
+    wait_for_source_delete_marker_replication_failed(&source_env, source_bucket, object_key).await?;
+
     // Widen the gap so a replay-time-stamping regression is unmistakable.
     sleep(Duration::from_secs(3)).await;
 
-    // Restart the source (see the doc comment: live-source replay is wedged by
-    // backlog#1278), then bring the target back; the restarted source's scanner
-    // heal pass replays the failed delete marker.
-    source_env.restart_server_preserving_data(vec![], &source_env_vars).await?;
     target_env.restart_server_preserving_data(vec![], &[]).await?;
 
     let target_mtime = wait_for_target_delete_marker(&target_client, target_bucket, object_key).await?;
@@ -4025,17 +4048,23 @@ async fn test_site_replication_allows_private_ca_https_with_ca_cert_pem_real_dua
 
 #[tokio::test]
 #[serial]
-async fn test_site_replication_resync_start_cancel_restart_real_dual_node() -> Result<(), Box<dyn Error + Send + Sync>> {
+async fn test_site_replication_resync_lifecycle_survives_real_server_restart() -> Result<(), Box<dyn Error + Send + Sync>> {
     init_logging();
+    let resync_process_env = [
+        ("RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET", "true"),
+        // Verbose server logging can block startup when this focused test is run
+        // through a captured test process rather than nextest.
+        ("RUST_LOG", "error"),
+    ];
 
     let mut source_env = RustFSTestEnvironment::new().await?;
-    source_env
-        .start_rustfs_server_with_env(vec![], LOOPBACK_REPLICATION_TARGET_ENV)
-        .await?;
+    source_env.capture_log_path = Some(format!("{}/server.log", source_env.temp_dir));
+    source_env.start_rustfs_server_with_env(vec![], &resync_process_env).await?;
 
     let mut target_env = RustFSTestEnvironment::new().await?;
+    target_env.capture_log_path = Some(format!("{}/server.log", target_env.temp_dir));
     target_env
-        .start_rustfs_server_without_cleanup_with_env(LOOPBACK_REPLICATION_TARGET_ENV)
+        .start_rustfs_server_without_cleanup_with_env(&resync_process_env)
         .await?;
 
     let source_bucket = "site-repl-resync-src";
@@ -4083,12 +4112,12 @@ async fn test_site_replication_resync_start_cancel_restart_real_dual_node() -> R
     wait_for_bucket_on_target(&source_client, source_bucket).await?;
     let target_arn = wait_for_remote_target_arn(&source_env, source_bucket).await?;
 
-    for idx in 0..32 {
+    for idx in 0..96 {
         source_client
             .put_object()
             .bucket(source_bucket)
             .key(format!("resync-object-{idx:02}"))
-            .body(ByteStream::from(vec![b'x'; 256 * 1024]))
+            .body(ByteStream::from(vec![b'x'; 512 * 1024]))
             .send()
             .await?;
     }
@@ -4096,18 +4125,31 @@ async fn test_site_replication_resync_start_cancel_restart_real_dual_node() -> R
     let started = site_replication_resync_op(&source_env, "start", &remote_peer).await?;
     assert_eq!(started.status, "success", "unexpected start result: {:?}", started);
     assert!(
-        started
-            .buckets
-            .iter()
-            .any(|bucket| bucket.bucket == source_bucket && matches!(bucket.status.as_str(), "started" | "success")),
+        started.buckets.iter().any(|bucket| {
+            bucket.bucket == source_bucket && matches!(bucket.status.as_str(), "started" | "running" | "completed" | "success")
+        }),
         "source bucket start status missing: {:?}",
         started
     );
     assert!(!started.resync_id.is_empty(), "start response omitted the resync id: {:?}", started);
     let started_reset_id = started.resync_id.clone();
 
+    assert!(
+        matches!(started.state.as_str(), "pending" | "running"),
+        "the fixture must keep the first generation active long enough to test duplicate start: {:?}",
+        started
+    );
+    let duplicate_err = site_replication_resync_op(&source_env, "start", &remote_peer)
+        .await
+        .expect_err("duplicate start must be rejected while a generation is active");
+    assert!(
+        duplicate_err.to_string().contains("already active"),
+        "unexpected duplicate start error: {duplicate_err}"
+    );
+
     let canceled = site_replication_resync_op(&source_env, "cancel", &remote_peer).await?;
     assert_eq!(canceled.status, "success", "unexpected cancel result: {:?}", canceled);
+    assert_eq!(canceled.state, "canceled");
     assert!(
         canceled
             .buckets
@@ -4116,34 +4158,56 @@ async fn test_site_replication_resync_start_cancel_restart_real_dual_node() -> R
         "source bucket cancel status missing: {:?}",
         canceled
     );
+    let canceled_again = site_replication_resync_op(&source_env, "cancel", &remote_peer).await?;
+    assert_eq!(canceled_again.resync_id, canceled.resync_id, "repeated cancel must be idempotent");
+    assert_eq!(canceled_again.state, "canceled");
 
     let canceled_target =
         wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |target| target.status == "Canceled").await?;
-    assert_eq!(canceled_target.status, "Canceled");
     assert_eq!(canceled_target.reset_id, started_reset_id);
 
     let restarted = site_replication_resync_op(&source_env, "start", &remote_peer).await?;
     assert_eq!(restarted.status, "success", "unexpected restart result: {:?}", restarted);
+    assert_ne!(restarted.resync_id, started_reset_id);
     assert!(
+        matches!(restarted.state.as_str(), "pending" | "running"),
+        "the second generation must be active before the process restart: {:?}",
         restarted
-            .buckets
-            .iter()
-            .any(|bucket| bucket.bucket == source_bucket && matches!(bucket.status.as_str(), "started" | "success")),
-        "source bucket restart status missing: {:?}",
-        restarted
+    );
+    let restarted_reset_id = restarted.resync_id.clone();
+
+    source_env.restart_server_preserving_data(vec![], &resync_process_env).await?;
+    wait_for_site_replication_enabled(&source_env, 2).await?;
+
+    let after_restart = site_replication_resync_op(&source_env, "status", &remote_peer).await?;
+    assert_eq!(
+        after_restart.resync_id, restarted_reset_id,
+        "server restart changed the durable resync id"
+    );
+    assert_eq!(after_restart.generation, restarted.generation);
+    assert_eq!(after_restart.created_at, restarted.created_at);
+    assert!(
+        matches!(after_restart.state.as_str(), "pending" | "running" | "completed" | "failed"),
+        "unexpected recovered lifecycle state: {:?}",
+        after_restart
+    );
+    assert!(
+        after_restart.buckets.iter().any(|bucket| bucket.bucket == source_bucket),
+        "durable status lost the source bucket after restart: {:?}",
+        after_restart
     );
     let restart_snapshot = get_replication_reset_status(&source_env, source_bucket, &target_arn).await?;
     let restarted_target = wait_for_replication_reset_target(&source_env, source_bucket, &target_arn, |target| {
-        !target.reset_id.is_empty() && target.reset_id != started_reset_id
+        target.reset_id == restarted_reset_id
     })
     .await
     .map_err(|err| {
         format!(
             "restart ids: start={} restart={} snapshot={:?}; {err}",
-            started_reset_id, restarted.resync_id, restart_snapshot.targets
+            started_reset_id, restarted_reset_id, restart_snapshot.targets
         )
     })?;
-    assert_ne!(restarted_target.reset_id, started_reset_id);
+    assert_eq!(restarted_target.reset_id, restarted_reset_id);
 
     Ok(())
 }

crates/ecstore/Cargo.toml

@@ -61,6 +61,7 @@ rustfs-kms.workspace = true
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
 rustfs-object-capacity.workspace = true
+arc-swap.workspace = true
 async-trait.workspace = true
 bytes = { workspace = true, features = ["serde"] }
 byteorder = { workspace = true }
@@ -122,7 +123,7 @@ libc.workspace = true
 # (backlog#1178); "fs" comes from the workspace default.
 rustix = { workspace = true, features = ["process", "fs"] }
 rustfs-madmin.workspace = true
-reqwest = { workspace = true, default-features = false, features = ["rustls", "http2", "system-proxy"] }
+reqwest = { workspace = true }
 aes-gcm = { workspace = true, features = ["rand_core"] }
 chacha20poly1305.workspace = true
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }

crates/ecstore/src/api/mod.rs

@@ -67,7 +67,11 @@ pub mod bucket {
         }
 
         pub mod tier_delete_journal {
-            pub use crate::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry;
+            #[cfg(feature = "test-util")]
+            pub use crate::bucket::lifecycle::tier_delete_journal::recover_tier_delete_journal_entries;
+            pub use crate::bucket::lifecycle::tier_delete_journal::{
+                persist_tier_delete_journal_entry, record_tier_delete_journal_backend_identity,
+            };
         }
 
         pub mod tier_last_day_stats {
@@ -134,7 +138,7 @@ pub mod bucket {
     }
 
     pub mod quota {
-        pub use crate::bucket::quota::{BucketQuota, QuotaError, QuotaOperation};
+        pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
 
         pub mod checker {
             pub use crate::bucket::quota::checker::QuotaChecker;
@@ -237,8 +241,10 @@ pub mod config {
         pub use crate::config::com::{
             COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
             ServerConfigCorruptError, delete_config, is_server_config_corrupt_error, lookup_configs, read_config,
-            read_config_no_lock, read_config_with_metadata, read_config_without_migrate, save_config, save_config_with_opts,
-            save_server_config, try_migrate_server_config,
+            read_config_no_lock, read_config_with_metadata, read_config_without_migrate, read_config_without_migrate_no_lock,
+            read_existing_server_config_no_lock, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
+            save_server_config_no_lock, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
+            with_server_config_read_lock, with_server_config_write_lock,
         };
     }
 
@@ -247,7 +253,8 @@ pub mod config {
             CLASS_RRS, CLASS_STANDARD, Config, DEEP_ARCHIVE, DEFAULT_INLINE_BLOCK, DEFAULT_KVS, DEFAULT_RRS_PARITY,
             EXPRESS_ONEZONE, GLACIER, GLACIER_IR, INLINE_BLOCK, INLINE_BLOCK_ENV, INTELLIGENT_TIERING, MIN_PARITY_DRIVES,
             ONEZONE_IA, OPTIMIZE, OPTIMIZE_ENV, OUTPOSTS, RRS, RRS_ENV, SCHEME_PREFIX, SNOW, STANDARD, STANDARD_ENV, STANDARD_IA,
-            StorageClass, default_parity_count, lookup_config, parse_storage_class, validate_parity, validate_parity_inner,
+            StorageClass, default_parity_count, lookup_config, lookup_config_for_pools, parse_storage_class, validate_parity,
+            validate_parity_inner,
         };
     }
 
@@ -258,12 +265,14 @@ pub mod config {
 
 pub mod data_usage {
     pub use crate::data_usage::{
-        DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, init_compression_total_memory_from_backend,
-        load_compression_total_from_memory, load_data_usage_from_backend, record_bucket_delete_marker_memory,
+        DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
+        init_compression_total_memory_from_backend, live_bucket_usage_computations, load_compression_total_from_memory,
+        load_data_usage_from_backend, load_data_usage_from_backend_cached, record_bucket_delete_marker_memory,
         record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
         record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
         refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
         remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
+        store_data_usage_in_backend,
     };
 }
 
@@ -308,8 +317,8 @@ pub mod error {
 
 pub mod erasure {
     pub use crate::erasure::coding::{
-        BitrotReader, BitrotWriter, BitrotWriterWrapper, CustomWriter, Erasure, ReedSolomonEncoder, calc_shard_size,
-        calc_shard_size_legacy,
+        BitrotReader, BitrotWriter, BitrotWriterWrapper, CustomWriter, Erasure, ErasureConstructionError, ReedSolomonEncoder,
+        calc_shard_size, calc_shard_size_legacy,
     };
 }
 
@@ -351,9 +360,12 @@ pub mod notification {
 
 pub mod object {
     pub use crate::object_api::{
-        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions,
-        PutObjReader, RangedDecompressReader, StreamConsumer, register_get_object_body_cache_hook, register_object_mutation_hook,
+        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, GetObjectBodySource,
+        GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, StreamConsumer,
+        get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
+        register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
     };
+    pub use crate::store::PreparedGetObjectReader;
 }
 
 pub mod rebalance {
@@ -373,9 +385,12 @@ pub mod rio {
 
 pub mod rpc {
     pub use crate::cluster::rpc::{
-        LocalPeerS3Client, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, SERVICE_SIGNAL_REFRESH_CONFIG,
-        SERVICE_SIGNAL_RELOAD_DYNAMIC, TONIC_RPC_PREFIX, TonicInterceptor, gen_tonic_signature_interceptor,
-        node_service_time_out_client, node_service_time_out_client_no_auth, verify_rpc_signature,
+        LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client,
+        SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor,
+        gen_signature_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
+        node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
+        sign_tonic_rpc_response_proof, verify_rpc_signature, verify_tonic_canonical_body_digest, verify_tonic_rpc_response_proof,
+        verify_tonic_rpc_signature,
     };
 }
 
@@ -402,7 +417,7 @@ pub mod tier {
         pub use crate::services::tier::tier::{
             ERR_TIER_BACKEND_IN_USE, ERR_TIER_BACKEND_NOT_EMPTY, ERR_TIER_INVALID_CONFIG, ERR_TIER_MISSING_CREDENTIALS,
             ERR_TIER_TYPE_UNSUPPORTED, TIER_CONFIG_FILE, TIER_CONFIG_FORMAT, TIER_CONFIG_V1, TIER_CONFIG_VERSION, TierConfigMgr,
-            is_err_config_not_found, try_migrate_tiering_config,
+            TierConfigUpdateError, is_err_config_not_found, try_migrate_tiering_config,
         };
     }
 
@@ -413,7 +428,7 @@ pub mod tier {
     pub mod tier_config {
         pub use crate::services::tier::tier_config::{
             ServicePrincipalAuth, TierAliyun, TierAzure, TierConfig, TierGCS, TierHuaweicloud, TierMinIO, TierR2, TierRustFS,
-            TierS3, TierTencent, TierType,
+            TierS3, TierTencent, TierType, TierWasabi,
         };
     }
 
@@ -424,6 +439,13 @@ pub mod tier {
         };
     }
 
+    pub mod tier_mutation_peer {
+        pub use crate::services::tier::tier_mutation_peer::{
+            MAX_TIER_MUTATION_PEER_COMMIT_ETAG_SIZE, TierMutationPeerError, TierMutationPeerOutcome, TierMutationPeerResult,
+            TierMutationPeerState, handle_tier_mutation_peer_request,
+        };
+    }
+
     pub mod warm_backend {
         pub use crate::services::tier::warm_backend::{
             WarmBackend, WarmBackendGetOpts, WarmBackendImpl, build_transition_put_options, check_warm_backend, new_warm_backend,
@@ -433,9 +455,9 @@ pub mod tier {
     #[cfg(feature = "test-util")]
     pub mod test_util {
         pub use crate::services::tier::test_util::{
-            FaultConfig, MockStoredObject, MockWarmBackend, MockWarmOp, TransitionMeta, assert_transition_meta_consistent,
-            free_version_count, read_transition_meta, register_mock_tier, register_mock_tier_backend,
-            wait_for_free_version_absence,
+            FaultConfig, MockStoredObject, MockWarmBackend, MockWarmOp, TransitionCleanupStoreBarrier, TransitionMeta,
+            assert_transition_meta_consistent, free_version_count, read_transition_meta, register_mock_tier,
+            register_mock_tier_backend, wait_for_free_version_absence,
         };
     }
 }

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -280,6 +280,7 @@ pub struct BucketTargetSys {
     pub hc_client: Arc<HttpClient>,
     pub a_mutex: Arc<Mutex<HashMap<String, ArnErrs>>>,
     pub arn_errs_map: Arc<RwLock<HashMap<String, ArnErrs>>>,
+    heartbeat_started: OnceLock<()>,
 }
 
 impl BucketTargetSys {
@@ -295,9 +296,20 @@ impl BucketTargetSys {
             hc_client: Arc::new(HttpClient::new()),
             a_mutex: Arc::new(Mutex::new(HashMap::new())),
             arn_errs_map: Arc::new(RwLock::new(HashMap::new())),
+            heartbeat_started: OnceLock::new(),
         }
     }
 
+    pub(crate) fn start_heartbeat(&'static self) {
+        if self.heartbeat_started.set(()).is_err() {
+            return;
+        }
+
+        tokio::spawn(async move {
+            self.heartbeat().await;
+        });
+    }
+
     pub async fn is_offline(&self, url: &Url) -> bool {
         let key = endpoint_health_key(url);
         {
@@ -367,7 +379,7 @@ impl BucketTargetSys {
     async fn check_endpoint_health(&self, endpoint: &str, scheme: &str) -> bool {
         let scheme = if scheme.is_empty() { "https" } else { scheme };
         let url = format!("{scheme}://{endpoint}/");
-        match self.hc_client.head(url).timeout(Duration::from_secs(3)).send().await {
+        match self.hc_client.get(url).timeout(Duration::from_secs(3)).send().await {
             Ok(response) => response.status().as_u16() < 500,
             Err(_) => false,
         }

crates/ecstore/src/bucket/lifecycle/mod.rs

@@ -28,3 +28,4 @@ pub mod tier_delete_journal;
 pub mod tier_free_version_recovery;
 pub mod tier_last_day_stats;
 pub mod tier_sweeper;
+pub mod transition_transaction;

crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use std::sync::Arc;
+use std::{future::Future, sync::Arc, time::Duration};
 
 use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
@@ -20,10 +20,11 @@ use tokio_util::sync::CancellationToken;
 use tracing::{debug, warn};
 
 use crate::bucket::lifecycle::config_boundary;
-use crate::bucket::lifecycle::tier_sweeper::{Jentry, delete_object_from_remote_tier_idempotent};
+use crate::bucket::lifecycle::tier_sweeper::{Jentry, delete_object_from_remote_tier_idempotent_with_manager_and_identity};
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::{Error, Result};
 use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
+use crate::services::tier::tier::tier_destination_id_from_metadata;
 use crate::storage_api_contracts::{
     list::ListOperations as _,
     object::{DeletedObject, ObjectIO, ObjectOperations, ObjectToDelete},
@@ -37,8 +38,11 @@ const LOG_SUBSYSTEM_LIFECYCLE: &str = "lifecycle";
 const EVENT_LIFECYCLE_TIER_DELETE_JOURNAL: &str = "lifecycle_tier_delete_journal";
 
 pub const DEFAULT_TIER_DELETE_JOURNAL_RECOVERY_LIMIT: usize = 1_000;
-const TIER_DELETE_JOURNAL_VERSION: u8 = 1;
-const TIER_DELETE_JOURNAL_PREFIX: &str = "ilm/tier-delete-journal/";
+const TIER_DELETE_JOURNAL_RECOVERY_INTERVAL: Duration = Duration::from_secs(60);
+const TIER_DELETE_JOURNAL_RECOVERY_TIMEOUT: Duration = Duration::from_secs(300);
+const TIER_DELETE_JOURNAL_VERSION: u8 = 2;
+const TIER_DELETE_JOURNAL_EXACT_VERSION: u8 = 3;
+pub(crate) const TIER_DELETE_JOURNAL_PREFIX: &str = "ilm/tier-delete-journal/";
 
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
 #[serde(deny_unknown_fields)]
@@ -47,22 +51,31 @@ struct PersistedTierDeleteJournalEntry {
     obj_name: String,
     version_id: String,
     tier_name: String,
+    #[serde(default)]
+    backend_identity: Option<[u8; 32]>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    version_id_exact: Option<bool>,
 }
 
 impl PersistedTierDeleteJournalEntry {
     fn from_jentry(je: &Jentry) -> Self {
         Self {
-            version: TIER_DELETE_JOURNAL_VERSION,
+            version: if je.version_id_exact {
+                TIER_DELETE_JOURNAL_EXACT_VERSION
+            } else if je.backend_identity.is_some() {
+                TIER_DELETE_JOURNAL_VERSION
+            } else {
+                1
+            },
             obj_name: je.obj_name.clone(),
             version_id: je.version_id.clone(),
             tier_name: je.tier_name.clone(),
+            backend_identity: je.backend_identity,
+            version_id_exact: je.version_id_exact.then_some(true),
         }
     }
 
     fn into_jentry(self) -> Result<Jentry> {
-        if self.version != TIER_DELETE_JOURNAL_VERSION {
-            return Err(Error::other(format!("unsupported tier delete journal version {}", self.version)));
-        }
         // Empty `version_id` is a legal sentinel for objects transitioned to an
         // unversioned remote tier (see CLAUDE.md: a tier version of `None`/`""`
         // means the tier bucket is unversioned, so the remote delete is issued
@@ -71,10 +84,40 @@ impl PersistedTierDeleteJournalEntry {
         if self.obj_name.is_empty() || self.tier_name.is_empty() {
             return Err(Error::other("tier delete journal entry is incomplete"));
         }
+        if self.version != TIER_DELETE_JOURNAL_EXACT_VERSION && self.version_id_exact.unwrap_or(false) {
+            return Err(Error::other(
+                "legacy tier delete journal entry has an unsupported exact version constraint",
+            ));
+        }
+        let (backend_identity, version_id_exact) = match self.version {
+            1 => (None, false),
+            TIER_DELETE_JOURNAL_VERSION => (
+                Some(
+                    self.backend_identity
+                        .ok_or_else(|| Error::other("tier delete journal v2 entry is missing its backend identity"))?,
+                ),
+                false,
+            ),
+            TIER_DELETE_JOURNAL_EXACT_VERSION => {
+                if self.version_id.is_empty() || self.version_id_exact != Some(true) {
+                    return Err(Error::other("tier delete journal v3 entry is missing its exact version constraint"));
+                }
+                (
+                    Some(
+                        self.backend_identity
+                            .ok_or_else(|| Error::other("tier delete journal v3 entry is missing its backend identity"))?,
+                    ),
+                    true,
+                )
+            }
+            version => return Err(Error::other(format!("unsupported tier delete journal version {version}"))),
+        };
         Ok(Jentry {
             obj_name: self.obj_name,
             version_id: self.version_id,
             tier_name: self.tier_name,
+            backend_identity,
+            version_id_exact,
         })
     }
 }
@@ -95,23 +138,41 @@ pub(crate) fn tier_delete_journal_object_name(je: &Jentry) -> String {
     hasher.update(je.obj_name.as_bytes());
     hasher.update([0]);
     hasher.update(je.version_id.as_bytes());
+    if let Some(backend_identity) = je.backend_identity {
+        hasher.update([0]);
+        hasher.update(backend_identity);
+    }
+    if je.version_id_exact {
+        hasher.update([0]);
+        hasher.update(b"exact-version-id");
+    }
     format!(
         "{TIER_DELETE_JOURNAL_PREFIX}{}.json",
         rustfs_utils::crypto::hex(hasher.finalize().as_slice())
     )
 }
 
-fn decode_tier_delete_journal_entry(data: &[u8]) -> Result<Jentry> {
+pub(crate) fn decode_tier_delete_journal_entry(data: &[u8]) -> Result<Jentry> {
     let persisted: PersistedTierDeleteJournalEntry =
         serde_json::from_slice(data).map_err(|err| Error::other(format!("decode tier delete journal failed: {err}")))?;
     persisted.into_jentry()
 }
 
-fn encode_tier_delete_journal_entry(je: &Jentry) -> Result<Vec<u8>> {
+pub(crate) fn encode_tier_delete_journal_entry(je: &Jentry) -> Result<Vec<u8>> {
     serde_json::to_vec(&PersistedTierDeleteJournalEntry::from_jentry(je))
         .map_err(|err| Error::other(format!("encode tier delete journal failed: {err}")))
 }
 
+pub fn record_tier_delete_journal_backend_identity(
+    je: &mut Jentry,
+    metadata: &std::collections::HashMap<String, String>,
+) -> std::io::Result<()> {
+    if let Some(identity) = tier_destination_id_from_metadata(metadata)? {
+        je.backend_identity = Some(identity);
+    }
+    Ok(())
+}
+
 pub async fn persist_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
 where
     S: ObjectIO<
@@ -148,7 +209,18 @@ where
 }
 
 pub async fn process_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
-    delete_object_from_remote_tier_idempotent(&je.obj_name, &je.version_id, &je.tier_name).await?;
+    let backend_identity = je
+        .backend_identity
+        .ok_or_else(|| std::io::Error::other("legacy tier delete journal has no durable backend identity"))?;
+    delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+        &je.obj_name,
+        &je.version_id,
+        &je.tier_name,
+        backend_identity,
+        &api.tier_config_mgr(),
+        je.version_id_exact,
+    )
+    .await?;
     remove_tier_delete_journal_entry(api, je).await
 }
 
@@ -218,6 +290,21 @@ pub async fn recover_tier_delete_journal_entries(
             }
         };
 
+        if je.backend_identity.is_none() {
+            stats.failed += 1;
+            warn!(
+                event = EVENT_LIFECYCLE_TIER_DELETE_JOURNAL,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                journal_object = %object.name,
+                remote_object = %je.obj_name,
+                remote_version_id = %je.version_id,
+                tier = %je.tier_name,
+                "Legacy tier delete journal entry has no durable backend identity and will be retained"
+            );
+            continue;
+        }
+
         match process_tier_delete_journal_entry(api.clone(), &je).await {
             Ok(()) => stats.deleted += 1,
             Err(err) => {
@@ -241,16 +328,33 @@ pub async fn recover_tier_delete_journal_entries(
 }
 
 pub async fn run_tier_delete_journal_recovery_loop(api: Arc<ECStore>, cancel_token: CancellationToken) {
-    let mut interval = tokio::time::interval(std::time::Duration::from_secs(60));
+    let mut interval = tokio::time::interval(TIER_DELETE_JOURNAL_RECOVERY_INTERVAL);
+    interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
     let mut marker: Option<String> = None;
 
     loop {
+        #[cfg(test)]
         tokio::select! {
+            biased;
             _ = cancel_token.cancelled() => return,
-            _ = interval.tick() => {}
+            _ = interval.tick() => {},
+            _ = api.ctx.wait_for_tier_delete_journal_recovery() => {},
+        }
+        #[cfg(not(test))]
+        tokio::select! {
+            biased;
+            _ = cancel_token.cancelled() => return,
+            _ = interval.tick() => {},
         }
 
-        match recover_tier_delete_journal_entries(api.clone(), DEFAULT_TIER_DELETE_JOURNAL_RECOVERY_LIMIT, marker.clone()).await {
+        let recovery =
+            recover_tier_delete_journal_entries(api.clone(), DEFAULT_TIER_DELETE_JOURNAL_RECOVERY_LIMIT, marker.clone());
+        let Some(result) =
+            await_tier_delete_journal_recovery(&cancel_token, TIER_DELETE_JOURNAL_RECOVERY_TIMEOUT, recovery).await
+        else {
+            return;
+        };
+        match result {
             Ok(stats) => {
                 marker = stats.next_marker;
                 debug!(
@@ -279,16 +383,44 @@ pub async fn run_tier_delete_journal_recovery_loop(api: Arc<ECStore>, cancel_tok
     }
 }
 
+async fn await_tier_delete_journal_recovery<T, F>(
+    cancel_token: &CancellationToken,
+    timeout: Duration,
+    recovery: F,
+) -> Option<Result<T>>
+where
+    F: Future<Output = Result<T>>,
+{
+    tokio::select! {
+        _ = cancel_token.cancelled() => None,
+        result = tokio::time::timeout(timeout, recovery) => Some(match result {
+            Ok(result) => result,
+            Err(_) => Err(Error::other(format!(
+                "tier delete journal recovery timed out after {} seconds",
+                timeout.as_secs()
+            ))),
+        }),
+    }
+}
+
 #[cfg(test)]
 mod tests {
-    use super::{decode_tier_delete_journal_entry, encode_tier_delete_journal_entry, tier_delete_journal_object_name};
+    use super::{
+        TIER_DELETE_JOURNAL_EXACT_VERSION, await_tier_delete_journal_recovery, decode_tier_delete_journal_entry,
+        encode_tier_delete_journal_entry, record_tier_delete_journal_backend_identity, tier_delete_journal_object_name,
+    };
     use crate::bucket::lifecycle::tier_sweeper::Jentry;
+    use crate::error::Result;
+    use std::time::Duration;
+    use tokio_util::sync::CancellationToken;
 
     fn journal_entry() -> Jentry {
         Jentry {
             obj_name: "remote/object".to_string(),
             version_id: "remote-version".to_string(),
             tier_name: "WARM".to_string(),
+            backend_identity: Some([7; 32]),
+            version_id_exact: false,
         }
     }
 
@@ -302,6 +434,83 @@ mod tests {
         assert_eq!(decoded.obj_name, je.obj_name);
         assert_eq!(decoded.version_id, je.version_id);
         assert_eq!(decoded.tier_name, je.tier_name);
+        assert_eq!(decoded.backend_identity, je.backend_identity);
+        assert_eq!(decoded.version_id_exact, je.version_id_exact);
+    }
+
+    #[test]
+    fn tier_delete_journal_roundtrips_exact_put_response_constraint() {
+        let mut exact = journal_entry();
+        exact.version_id = uuid::Uuid::nil().to_string();
+        exact.version_id_exact = true;
+        let mut normalized = exact.clone();
+        normalized.version_id_exact = false;
+
+        let encoded = encode_tier_delete_journal_entry(&exact).expect("exact journal entry should encode");
+        let persisted: serde_json::Value = serde_json::from_slice(&encoded).expect("exact journal JSON should decode");
+        let decoded = decode_tier_delete_journal_entry(&encoded).expect("exact journal entry should decode");
+
+        assert_eq!(persisted["version"], TIER_DELETE_JOURNAL_EXACT_VERSION);
+        assert_eq!(persisted["version_id_exact"], true);
+        assert!(decoded.version_id_exact);
+        assert_ne!(tier_delete_journal_object_name(&exact), tier_delete_journal_object_name(&normalized));
+    }
+
+    #[test]
+    fn tier_delete_journal_rejects_invalid_exact_version_constraints() {
+        let identity = vec![7_u8; 32];
+        let invalid = [
+            serde_json::json!({
+                "version": 1,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "version_id_exact": true,
+            }),
+            serde_json::json!({
+                "version": 2,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+                "version_id_exact": true,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+                "version_id_exact": true,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+                "version_id_exact": false,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "version_id_exact": true,
+            }),
+        ];
+
+        for persisted in invalid {
+            let encoded = serde_json::to_vec(&persisted).expect("invalid journal fixture should encode");
+            decode_tier_delete_journal_entry(&encoded).expect_err("invalid exact journal constraint must fail closed");
+        }
     }
 
     #[test]
@@ -317,6 +526,63 @@ mod tests {
         assert!(!first.contains("remote/object"));
     }
 
+    #[test]
+    fn tier_delete_journal_paths_separate_legacy_and_backend_identities() {
+        let mut legacy = journal_entry();
+        legacy.backend_identity = None;
+        let mut backend_a = journal_entry();
+        backend_a.backend_identity = Some([1; 32]);
+        let mut backend_b = journal_entry();
+        backend_b.backend_identity = Some([2; 32]);
+
+        assert_eq!(
+            tier_delete_journal_object_name(&legacy),
+            "ilm/tier-delete-journal/5ba6a7eb6338412b771613a6845a42ae5b8e26b5d201323eb01b38c5b42ff300.json"
+        );
+        assert_ne!(tier_delete_journal_object_name(&legacy), tier_delete_journal_object_name(&backend_a));
+        assert_ne!(tier_delete_journal_object_name(&backend_a), tier_delete_journal_object_name(&backend_b));
+    }
+
+    #[test]
+    fn tier_delete_journal_v2_requires_backend_identity() {
+        let payload = br#"{"version":2,"obj_name":"remote/object","version_id":"v1","tier_name":"WARM"}"#;
+
+        let err = decode_tier_delete_journal_entry(payload).expect_err("v2 entry without identity must fail closed");
+
+        assert!(err.to_string().contains("backend identity"));
+    }
+
+    #[test]
+    fn tier_delete_journal_uses_persisted_transition_destination_identity() {
+        let mut je = journal_entry();
+        je.backend_identity = None;
+        let identity = [9_u8; 32];
+        let mut metadata = std::collections::HashMap::new();
+        rustfs_utils::http::metadata_compat::insert_str(
+            &mut metadata,
+            rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
+            rustfs_utils::crypto::hex(identity),
+        );
+
+        record_tier_delete_journal_backend_identity(&mut je, &metadata).expect("persisted transition identity should decode");
+        let encoded = encode_tier_delete_journal_entry(&je).expect("identity-bound journal should encode");
+        let decoded = decode_tier_delete_journal_entry(&encoded).expect("identity-bound journal should decode");
+
+        assert_eq!(decoded.backend_identity, Some(identity));
+    }
+
+    #[test]
+    fn tier_delete_journal_without_transition_identity_stays_legacy() {
+        let mut je = journal_entry();
+        je.backend_identity = None;
+
+        let encoded = encode_tier_delete_journal_entry(&je).expect("legacy journal should remain encodable");
+        let persisted: serde_json::Value = serde_json::from_slice(&encoded).expect("journal JSON should decode");
+
+        assert_eq!(persisted["version"], 1);
+        assert!(persisted["backend_identity"].is_null());
+    }
+
     #[test]
     fn tier_delete_journal_rejects_incomplete_entry() {
         let payload = br#"{"version":1,"obj_name":"","version_id":"v1","tier_name":"WARM"}"#;
@@ -339,6 +605,7 @@ mod tests {
         assert_eq!(decoded.obj_name, "remote/object");
         assert!(decoded.version_id.is_empty());
         assert_eq!(decoded.tier_name, "WARM");
+        assert_eq!(decoded.backend_identity, None);
     }
 
     #[test]
@@ -360,4 +627,29 @@ mod tests {
 
         assert!(err.to_string().contains("decode tier delete journal failed"));
     }
+
+    #[tokio::test]
+    async fn tier_delete_journal_recovery_has_a_hard_outer_timeout() {
+        let result = await_tier_delete_journal_recovery(
+            &CancellationToken::new(),
+            Duration::from_millis(10),
+            std::future::pending::<Result<()>>(),
+        )
+        .await
+        .expect("an elapsed timeout should return a recovery error")
+        .expect_err("a permanently pending recovery must time out");
+
+        assert!(result.to_string().contains("recovery timed out"), "{result}");
+    }
+
+    #[tokio::test]
+    async fn tier_delete_journal_recovery_drops_in_flight_work_on_shutdown() {
+        let cancel = CancellationToken::new();
+        cancel.cancel();
+
+        let result =
+            await_tier_delete_journal_recovery(&cancel, Duration::from_secs(30), std::future::pending::<Result<()>>()).await;
+
+        assert!(result.is_none(), "shutdown must cancel the in-flight recovery future");
+    }
 }

crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs

@@ -13,11 +13,14 @@
 // limitations under the License.
 
 use std::sync::Arc;
+#[cfg(test)]
+use std::sync::Mutex;
 use std::time::Duration;
 
 use tokio::sync::mpsc;
 use tokio_util::sync::CancellationToken;
 
+use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version;
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::Result;
 use crate::object_api::ObjectInfo;
@@ -31,10 +34,78 @@ use rustfs_filemeta::FileInfo;
 pub const DEFAULT_FREE_VERSION_RECOVERY_LIMIT: usize = 1_000;
 const DEFAULT_FREE_VERSION_RECOVERY_SCAN_LIMIT: usize = 10_000;
 const BACKGROUND_WALKDIR_TIMEOUT: Duration = Duration::from_secs(60);
+#[cfg(not(test))]
+const BACKGROUND_WALK_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(5);
+#[cfg(test)]
+const BACKGROUND_WALK_SHUTDOWN_TIMEOUT: Duration = Duration::from_millis(100);
 
 type ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, crate::error::Error>;
 type WalkOptions = StorageWalkOptions<fn(&FileInfo) -> bool>;
 
+#[cfg(test)]
+pub(super) enum RecoveryWalkTestAction {
+    SendItemsThenError(Vec<ObjectInfo>, crate::error::Error),
+    SendItemsThenHang(Vec<ObjectInfo>, Arc<tokio::sync::Notify>),
+    SendItemsUntilReceiverCloses(Arc<tokio::sync::Notify>),
+    ReturnError(crate::error::Error),
+    WaitForCancellation(Arc<tokio::sync::Notify>),
+}
+
+#[cfg(test)]
+type RecoveryWalkTestHook = Box<dyn Fn(&str) -> Option<RecoveryWalkTestAction> + Send + Sync>;
+
+#[cfg(test)]
+static RECOVERY_WALK_TEST_HOOK: Mutex<Option<RecoveryWalkTestHook>> = Mutex::new(None);
+
+#[cfg(test)]
+static RECOVERY_BUCKET_LIST_WAIT_HOOK: Mutex<Option<Arc<tokio::sync::Notify>>> = Mutex::new(None);
+
+#[cfg(test)]
+pub(super) struct RecoveryWalkHookGuard;
+
+#[cfg(test)]
+impl Drop for RecoveryWalkHookGuard {
+    fn drop(&mut self) {
+        let mut hook = RECOVERY_WALK_TEST_HOOK
+            .lock()
+            .expect("recovery walk test hook lock should not poison");
+        *hook = None;
+    }
+}
+
+#[cfg(test)]
+pub(super) fn set_recovery_walk_test_hook(
+    hook_fn: impl Fn(&str) -> Option<RecoveryWalkTestAction> + Send + Sync + 'static,
+) -> RecoveryWalkHookGuard {
+    let mut hook = RECOVERY_WALK_TEST_HOOK
+        .lock()
+        .expect("recovery walk test hook lock should not poison");
+    *hook = Some(Box::new(hook_fn));
+    RecoveryWalkHookGuard
+}
+
+#[cfg(test)]
+pub(super) struct RecoveryBucketListWaitHookGuard;
+
+#[cfg(test)]
+impl Drop for RecoveryBucketListWaitHookGuard {
+    fn drop(&mut self) {
+        let mut hook = RECOVERY_BUCKET_LIST_WAIT_HOOK
+            .lock()
+            .expect("recovery bucket-list test hook lock should not poison");
+        *hook = None;
+    }
+}
+
+#[cfg(test)]
+pub(super) fn set_recovery_bucket_list_wait_hook(started: Arc<tokio::sync::Notify>) -> RecoveryBucketListWaitHookGuard {
+    let mut hook = RECOVERY_BUCKET_LIST_WAIT_HOOK
+        .lock()
+        .expect("recovery bucket-list test hook lock should not poison");
+    *hook = Some(started);
+    RecoveryBucketListWaitHookGuard
+}
+
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct FreeVersionRecoveryStats {
     pub scanned: usize,
@@ -68,12 +139,22 @@ pub async fn recover_tier_free_versions(
     limit: usize,
     bucket_marker: Option<String>,
     object_marker: Option<String>,
+) -> Result<FreeVersionRecoveryStats> {
+    recover_tier_free_versions_with_cancel(api, limit, bucket_marker, object_marker, CancellationToken::new()).await
+}
+
+pub(super) async fn recover_tier_free_versions_with_cancel(
+    api: Arc<ECStore>,
+    limit: usize,
+    bucket_marker: Option<String>,
+    object_marker: Option<String>,
+    cancel_token: CancellationToken,
 ) -> Result<FreeVersionRecoveryStats> {
     if limit == 0 {
         return Err(std::io::Error::other("free-version recovery limit must be greater than zero").into());
     }
 
-    let page = list_tier_free_versions(api, limit, bucket_marker.clone(), object_marker.clone()).await?;
+    let page = list_tier_free_versions(api, limit, bucket_marker.clone(), object_marker.clone(), cancel_token.clone()).await?;
     let mut stats = FreeVersionRecoveryStats {
         scanned: 0,
         enqueued: 0,
@@ -87,8 +168,11 @@ pub async fn recover_tier_free_versions(
 
     let mut retry_cursor = RetryCursor::new(bucket_marker, object_marker);
     for oi in page.items {
+        if cancel_token.is_cancelled() {
+            return Err(tier_free_version_recovery_cancelled());
+        }
         retry_cursor.visit(&oi);
-        if !record_recovered_free_version_enqueue(&mut stats, queue_recovered_free_version(oi).await) {
+        if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(oi).await) {
             let (bucket_marker, object_marker) = retry_cursor.retry_markers();
             stats.truncated = true;
             stats.next_bucket_marker = bucket_marker;
@@ -100,6 +184,18 @@ pub async fn recover_tier_free_versions(
     Ok(stats)
 }
 
+fn tier_free_version_recovery_cancelled() -> crate::error::Error {
+    std::io::Error::new(std::io::ErrorKind::Interrupted, "tier free-version recovery cancelled").into()
+}
+
+fn tier_free_version_recovery_walk_shutdown_timed_out() -> crate::error::Error {
+    std::io::Error::new(
+        std::io::ErrorKind::TimedOut,
+        "tier free-version recovery walk did not stop after cancellation",
+    )
+    .into()
+}
+
 fn record_recovered_free_version_enqueue(stats: &mut FreeVersionRecoveryStats, queued: bool) -> bool {
     stats.scanned += 1;
     if queued {
@@ -111,10 +207,6 @@ fn record_recovered_free_version_enqueue(stats: &mut FreeVersionRecoveryStats, q
     }
 }
 
-async fn queue_recovered_free_version(oi: ObjectInfo) -> bool {
-    crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version(oi).await
-}
-
 #[derive(Debug, Clone, PartialEq, Eq)]
 struct RetryCursor {
     input_bucket_marker: Option<String>,
@@ -160,11 +252,12 @@ impl RetryCursor {
     }
 }
 
-async fn list_tier_free_versions(
+pub(super) async fn list_tier_free_versions(
     api: Arc<ECStore>,
     limit: usize,
     bucket_marker: Option<String>,
     object_marker: Option<String>,
+    cancel_token: CancellationToken,
 ) -> Result<FreeVersionRecoveryPage> {
     let mut page = FreeVersionRecoveryPage {
         items: Vec::new(),
@@ -179,21 +272,42 @@ async fn list_tier_free_versions(
         return Ok(page);
     }
 
-    let buckets = api.list_bucket(&BucketOptions::default()).await?;
+    let bucket_options = BucketOptions::default();
+    let list_buckets = async {
+        #[cfg(test)]
+        let wait_hook = RECOVERY_BUCKET_LIST_WAIT_HOOK
+            .lock()
+            .expect("recovery bucket-list test hook lock should not poison")
+            .clone();
+        #[cfg(test)]
+        if let Some(started) = wait_hook {
+            started.notify_one();
+            std::future::pending::<()>().await;
+        }
+        api.list_bucket(&bucket_options).await
+    };
+    tokio::pin!(list_buckets);
+    let buckets = tokio::select! {
+        biased;
+        _ = cancel_token.cancelled() => return Err(tier_free_version_recovery_cancelled()),
+        result = &mut list_buckets => result?,
+    };
     let mut bucket_seen = bucket_marker.is_none();
     let mut truncated_after: Option<RecoveryCursor> = None;
     let walk_scan_limit = recovery_walk_scan_limit(limit);
 
     for bucket in buckets {
+        if cancel_token.is_cancelled() {
+            return Err(tier_free_version_recovery_cancelled());
+        }
         if bucket.name == RUSTFS_META_BUCKET {
             continue;
         }
         if !bucket_seen {
-            if bucket_marker.as_deref() == Some(bucket.name.as_str()) {
-                bucket_seen = true;
-            } else {
+            if bucket_marker.as_deref().is_some_and(|marker| bucket.name.as_str() < marker) {
                 continue;
             }
+            bucket_seen = true;
         }
 
         page.buckets_scanned += 1;
@@ -204,16 +318,94 @@ async fn list_tier_free_versions(
         };
 
         let (tx, mut rx) = mpsc::channel::<ObjectInfoOrErr>(100);
-        let cancel = CancellationToken::new();
+        let cancel = cancel_token.child_token();
         let mut draining_after_truncation = false;
+        let mut drain_deadline = None;
         let mut last_seen_object: Option<String> = None;
         let mut scanned_objects = 0usize;
-        let walk = tokio::spawn({
+        let mut walk = tokio::spawn({
             let api = api.clone();
             let bucket_name = bucket.name.clone();
             let object_marker = bucket_object_marker.clone();
             let cancel = cancel.clone();
             async move {
+                #[cfg(test)]
+                let test_action = {
+                    let hook = RECOVERY_WALK_TEST_HOOK
+                        .lock()
+                        .expect("recovery walk test hook lock should not poison");
+                    hook.as_ref().and_then(|hook| hook(&bucket_name))
+                };
+                #[cfg(test)]
+                if let Some(action) = test_action {
+                    match action {
+                        RecoveryWalkTestAction::SendItemsThenError(items, err) => {
+                            for item in items {
+                                if tx
+                                    .send(ObjectInfoOrErr {
+                                        item: Some(item),
+                                        err: None,
+                                    })
+                                    .await
+                                    .is_err()
+                                {
+                                    return Ok(());
+                                }
+                            }
+                            let _ = tx
+                                .send(ObjectInfoOrErr {
+                                    item: None,
+                                    err: Some(err),
+                                })
+                                .await;
+                            return Ok(());
+                        }
+                        RecoveryWalkTestAction::SendItemsThenHang(items, started) => {
+                            for item in items {
+                                if tx
+                                    .send(ObjectInfoOrErr {
+                                        item: Some(item),
+                                        err: None,
+                                    })
+                                    .await
+                                    .is_err()
+                                {
+                                    return Ok(());
+                                }
+                            }
+                            started.notify_one();
+                            return std::future::pending().await;
+                        }
+                        RecoveryWalkTestAction::SendItemsUntilReceiverCloses(started) => {
+                            started.notify_one();
+                            let mut index = 0usize;
+                            loop {
+                                if tx
+                                    .send(ObjectInfoOrErr {
+                                        item: Some(ObjectInfo {
+                                            bucket: bucket_name.clone(),
+                                            name: format!("nonrecoverable-{index:08}"),
+                                            ..Default::default()
+                                        }),
+                                        err: None,
+                                    })
+                                    .await
+                                    .is_err()
+                                {
+                                    return Ok(());
+                                }
+                                index = index.saturating_add(1);
+                            }
+                        }
+                        RecoveryWalkTestAction::ReturnError(err) => return Err(err),
+                        RecoveryWalkTestAction::WaitForCancellation(started) => {
+                            started.notify_one();
+                            cancel.cancelled().await;
+                            return Err(tier_free_version_recovery_cancelled());
+                        }
+                    }
+                }
+
                 api.walk(
                     cancel,
                     &bucket_name,
@@ -231,47 +423,88 @@ async fn list_tier_free_versions(
             }
         });
 
-        while let Some(item) = rx.recv().await {
+        let mut receive_error = None;
+        loop {
+            let item = tokio::select! {
+                biased;
+                _ = cancel_token.cancelled() => {
+                    cancel.cancel();
+                    receive_error = Some(tier_free_version_recovery_cancelled());
+                    break;
+                }
+                _ = async {
+                    if let Some(deadline) = drain_deadline {
+                        tokio::time::sleep_until(deadline).await;
+                    } else {
+                        std::future::pending::<()>().await;
+                    }
+                }, if drain_deadline.is_some() => {
+                    receive_error = Some(tier_free_version_recovery_walk_shutdown_timed_out());
+                    break;
+                }
+                item = rx.recv() => match item {
+                    Some(item) => item,
+                    None => break,
+                },
+            };
             page.scanned_entries += 1;
-            if draining_after_truncation {
-                continue;
-            }
             if let Some(err) = item.err {
                 cancel.cancel();
-                walk.await.map_err(|err| std::io::Error::other(err.to_string()))??;
-                return Err(err);
+                receive_error = Some(err);
+                break;
+            }
+            if draining_after_truncation {
+                continue;
             }
             let Some(oi) = item.item else {
                 continue;
             };
             record_scanned_object(&mut last_seen_object, &mut scanned_objects, &oi.name);
             if let Some(cursor) = &truncated_after
-                && cursor.object != oi.name
+                && (cursor.bucket.as_str() != bucket.name.as_str() || cursor.object.as_str() != oi.name.as_str())
             {
                 page.truncated = true;
                 cancel.cancel();
                 draining_after_truncation = true;
+                drain_deadline = Some(tokio::time::Instant::now() + BACKGROUND_WALK_SHUTDOWN_TIMEOUT);
                 continue;
             }
             if is_recoverable_tier_free_version(&oi) {
-                let cursor = RecoveryCursor {
+                let current_cursor = RecoveryCursor {
                     bucket: bucket.name.clone(),
                     object: oi.name.clone(),
                 };
                 page.items.push(oi);
-                page.next_bucket_marker = Some(cursor.bucket.clone());
-                page.next_object_marker = Some(cursor.object.clone());
+                page.next_bucket_marker = Some(current_cursor.bucket.clone());
+                page.next_object_marker = Some(current_cursor.object.clone());
                 if page.items.len() >= limit && truncated_after.is_none() {
-                    truncated_after = Some(cursor);
+                    truncated_after = Some(current_cursor);
                 }
             }
         }
 
-        walk.await.map_err(|err| std::io::Error::other(err.to_string()))??;
+        drop(rx);
+        let walk_shutdown_timeout = drain_deadline
+            .map(|deadline| deadline.saturating_duration_since(tokio::time::Instant::now()))
+            .unwrap_or(BACKGROUND_WALK_SHUTDOWN_TIMEOUT);
+        let walk_result = match tokio::time::timeout(walk_shutdown_timeout, &mut walk).await {
+            Ok(result) => result.map_err(|err| std::io::Error::other(err.to_string()))?,
+            Err(_) => {
+                walk.abort();
+                let _ = walk.await;
+                if let Some(err) = receive_error {
+                    return Err(err);
+                }
+                return Err(tier_free_version_recovery_walk_shutdown_timed_out());
+            }
+        };
+        if let Some(err) = receive_error {
+            return Err(err);
+        }
+        walk_result?;
         mark_scan_truncated_if_needed(&mut page, scanned_objects, walk_scan_limit, &bucket.name, last_seen_object.as_deref());
 
         if page.truncated {
-            page.next_bucket_marker = Some(bucket.name.clone());
             break;
         }
     }

crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs

@@ -23,6 +23,7 @@ use crate::bucket::lifecycle::bucket_lifecycle_ops::ExpiryOp;
 use crate::bucket::lifecycle::lifecycle::{self, ObjectOpts};
 use crate::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry;
 use crate::client::signer_error::error_chain_contains_signer_header_marker;
+use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease};
 use crate::storage_api_contracts::lifecycle::TransitionedObject;
 use crate::store::ECStore;
 use rustfs_utils::get_env_usize;
@@ -247,6 +248,8 @@ impl ObjSweeper {
                 obj_name: self.remote_object.clone(),
                 version_id: self.transition_version_id.clone(),
                 tier_name: self.transition_tier.clone(),
+                backend_identity: None,
+                version_id_exact: false,
             });
         }
         None
@@ -281,6 +284,8 @@ pub struct Jentry {
     pub(crate) obj_name: String,
     pub(crate) version_id: String,
     pub(crate) tier_name: String,
+    pub(crate) backend_identity: Option<TierDestinationId>,
+    pub(crate) version_id_exact: bool,
 }
 
 impl ExpiryOp for Jentry {
@@ -312,6 +317,30 @@ async fn delete_object_from_remote_tier_raw(obj_name: &str, rv_id: &str, tier_na
         return result;
     }
 
+    let tier_config_mgr = runtime_sources::tier_config_mgr_handle();
+    delete_object_from_remote_tier_raw_with_manager(obj_name, rv_id, tier_name, &tier_config_mgr).await
+}
+
+async fn delete_object_from_remote_tier_raw_with_manager(
+    obj_name: &str,
+    rv_id: &str,
+    tier_name: &str,
+    tier_config_mgr: &Arc<tokio::sync::RwLock<TierConfigMgr>>,
+) -> Result<(), std::io::Error> {
+    let lease = TierConfigMgr::acquire_operation_lease(&tier_config_mgr, tier_name)
+        .await
+        .map_err(std::io::Error::other)?;
+    delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, &lease, false).await
+}
+
+async fn delete_object_from_remote_tier_raw_with_lease(
+    obj_name: &str,
+    rv_id: &str,
+    lease: &TierOperationLease,
+    version_id_exact: bool,
+) -> Result<(), std::io::Error> {
+    lease.validate_remote_version_id(rv_id)?;
+
     if remote_delete_breaker_is_open(Instant::now()).await {
         metrics::counter!(METRIC_DELETE_REMOTE_BREAKER_TOTAL).increment(1);
         return Err(std::io::Error::other(ERR_REMOTE_DELETE_BREAKER_OPEN));
@@ -323,13 +352,11 @@ async fn delete_object_from_remote_tier_raw(obj_name: &str, rv_id: &str, tier_na
         .map_err(|_| std::io::Error::other(ERR_REMOTE_DELETE_LIMITER_CLOSED))?;
     let _inflight = RemoteDeleteInflightGuard::new();
 
-    let tier_config_mgr = runtime_sources::tier_config_mgr_handle();
-    let mut config_mgr = tier_config_mgr.write().await;
-    let w = match config_mgr.get_driver(tier_name).await {
-        Ok(w) => w,
-        Err(e) => return Err(std::io::Error::other(e)),
-    };
-    w.remove(obj_name, rv_id).await
+    if version_id_exact {
+        lease.remove_exact(obj_name, rv_id).await
+    } else {
+        lease.remove(obj_name, rv_id).await
+    }
 }
 
 #[cfg(test)]
@@ -341,6 +368,30 @@ fn run_remote_tier_delete_test_hook(obj_name: &str, rv_id: &str, tier_name: &str
         .map(|hook| hook(obj_name, rv_id, tier_name))
 }
 
+#[cfg(test)]
+pub(super) struct RemoteTierDeleteHookGuard;
+
+#[cfg(test)]
+impl Drop for RemoteTierDeleteHookGuard {
+    fn drop(&mut self) {
+        let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
+            .lock()
+            .expect("remote tier delete test hook lock should not poison");
+        *hook = None;
+    }
+}
+
+#[cfg(test)]
+pub(super) fn set_remote_tier_delete_test_hook(
+    hook_fn: impl Fn(&str, &str, &str) -> std::io::Result<()> + Send + Sync + 'static,
+) -> RemoteTierDeleteHookGuard {
+    let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
+        .lock()
+        .expect("remote tier delete test hook lock should not poison");
+    *hook = Some(Box::new(hook_fn));
+    RemoteTierDeleteHookGuard
+}
+
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum RemoteTierDeleteOutcome {
     Deleted,
@@ -364,6 +415,38 @@ pub async fn delete_object_from_remote_tier_idempotent(
     }
 }
 
+pub(crate) async fn delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+    obj_name: &str,
+    rv_id: &str,
+    tier_name: &str,
+    backend_identity: TierDestinationId,
+    tier_config_mgr: &Arc<tokio::sync::RwLock<TierConfigMgr>>,
+    version_id_exact: bool,
+) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
+    let lease = TierConfigMgr::acquire_operation_lease_for_backend_identity(tier_config_mgr, tier_name, backend_identity)
+        .await
+        .map_err(std::io::Error::other)?;
+    delete_object_from_remote_tier_with_lease_idempotent(obj_name, rv_id, &lease, version_id_exact).await
+}
+
+pub(crate) async fn delete_object_from_remote_tier_with_lease_idempotent(
+    obj_name: &str,
+    rv_id: &str,
+    lease: &TierOperationLease,
+    version_id_exact: bool,
+) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
+    match delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, lease, version_id_exact).await {
+        Ok(()) => Ok(RemoteTierDeleteOutcome::Deleted),
+        Err(err) if is_remote_tier_not_found_error(&err) => Ok(RemoteTierDeleteOutcome::AlreadyRemoved),
+        Err(err) => {
+            if should_record_remote_delete_failure(&err) {
+                record_remote_delete_failure(&err, Instant::now()).await;
+            }
+            Err(err)
+        }
+    }
+}
+
 pub(crate) fn is_remote_tier_not_found_error(err: &std::io::Error) -> bool {
     let message = err.to_string();
     message.contains("NoSuchKey")
@@ -401,6 +484,8 @@ pub fn transitioned_force_delete_journal_entry(transitioned: &TransitionedObject
         obj_name: transitioned.name.clone(),
         version_id: transitioned.version_id.clone(),
         tier_name: transitioned.tier.clone(),
+        backend_identity: None,
+        version_id_exact: false,
     })
 }
 
@@ -409,34 +494,14 @@ mod test {
     use crate::client::signer_error::invalid_utf8_header_error;
 
     use super::{
-        ERR_REMOTE_DELETE_BREAKER_OPEN, ERR_REMOTE_DELETE_LIMITER_CLOSED, REMOTE_TIER_DELETE_TEST_HOOK, RemoteDeleteBreaker,
-        RemoteTierDeleteOutcome, delete_object_from_remote_tier_idempotent, is_remote_tier_not_found_error,
-        is_signer_header_error, should_record_remote_delete_failure,
+        ERR_REMOTE_DELETE_BREAKER_OPEN, ERR_REMOTE_DELETE_LIMITER_CLOSED, RemoteDeleteBreaker, RemoteTierDeleteOutcome,
+        delete_object_from_remote_tier_idempotent, delete_object_from_remote_tier_idempotent_with_manager_and_identity,
+        is_remote_tier_not_found_error, is_signer_header_error, set_remote_tier_delete_test_hook,
+        should_record_remote_delete_failure,
     };
     use std::io::{Error, ErrorKind};
     use std::time::{Duration, Instant};
 
-    struct RemoteTierDeleteHookGuard;
-
-    impl Drop for RemoteTierDeleteHookGuard {
-        fn drop(&mut self) {
-            let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
-                .lock()
-                .expect("remote tier delete test hook lock should not poison");
-            *hook = None;
-        }
-    }
-
-    fn set_remote_tier_delete_test_hook(
-        hook_fn: impl Fn(&str, &str, &str) -> std::io::Result<()> + Send + Sync + 'static,
-    ) -> RemoteTierDeleteHookGuard {
-        let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
-            .lock()
-            .expect("remote tier delete test hook lock should not poison");
-        *hook = Some(Box::new(hook_fn));
-        RemoteTierDeleteHookGuard
-    }
-
     #[test]
     fn signer_header_error_detection_matches_utf8_failures() {
         let err = Error::new(
@@ -506,6 +571,99 @@ mod test {
         assert!(err.to_string().contains("driver not found"));
     }
 
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    async fn journal_delete_rejects_backend_identity_mismatch() {
+        let manager = crate::services::tier::tier::TierConfigMgr::new();
+        crate::services::tier::test_util::register_mock_tier(&manager, "WARM").await;
+        let lease = crate::services::tier::tier::TierConfigMgr::acquire_operation_lease(&manager, "WARM")
+            .await
+            .expect("test tier lease should be available");
+        let mut mismatched = lease.backend_identity();
+        mismatched[0] ^= 1;
+        drop(lease);
+
+        let err = delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+            "remote/object",
+            "remote-version",
+            "WARM",
+            mismatched,
+            &manager,
+            false,
+        )
+        .await
+        .expect_err("journal recovery must fail closed when the tier name was rebound");
+
+        assert!(err.to_string().contains("identity no longer matches"));
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    async fn journal_delete_dispatches_an_exact_version_constraint() {
+        let manager = crate::services::tier::tier::TierConfigMgr::new();
+        let backend = crate::services::tier::test_util::register_mock_tier(&manager, "WARM").await;
+        let lease = crate::services::tier::tier::TierConfigMgr::acquire_operation_lease(&manager, "WARM")
+            .await
+            .expect("test tier lease should be available");
+        let identity = lease.backend_identity();
+        drop(lease);
+
+        let outcome = delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+            "remote/object",
+            "exact-version",
+            "WARM",
+            identity,
+            &manager,
+            true,
+        )
+        .await
+        .expect("an exact journal delete should reach the backend");
+
+        assert_eq!(outcome, RemoteTierDeleteOutcome::Deleted);
+        assert_eq!(backend.exact_remove_count(), 1);
+        assert_eq!(backend.remove_count().await, 1);
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    async fn journal_delete_rejects_nonempty_remote_version_before_backend_io() {
+        let manager = crate::services::tier::tier::TierConfigMgr::new();
+        let backend = crate::services::tier::test_util::register_mock_tier(&manager, "WARM").await;
+        let lease = crate::services::tier::tier::TierConfigMgr::acquire_operation_lease(&manager, "WARM")
+            .await
+            .expect("test tier lease should be available");
+        let identity = lease.backend_identity();
+        drop(lease);
+        backend.set_reject_non_empty_remote_versions(true);
+
+        let err = delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+            "remote/object",
+            "remote-version",
+            "WARM",
+            identity,
+            &manager,
+            true,
+        )
+        .await
+        .expect_err("a provider that rejects a versioned delete must fail before remote IO");
+
+        assert!(err.to_string().contains("requires an unversioned remote object"));
+        assert_eq!(backend.remove_count().await, 0);
+
+        delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+            "remote/object",
+            "",
+            "WARM",
+            identity,
+            &manager,
+            false,
+        )
+        .await
+        .expect("unversioned remote delete should continue without a version ID");
+
+        assert_eq!(backend.remove_versions().await, vec![("remote/object".to_string(), String::new())]);
+    }
+
     #[test]
     fn breaker_opens_at_threshold_and_recovers_after_window() {
         let mut breaker = RemoteDeleteBreaker::new(3, Duration::from_secs(30));

crates/ecstore/src/bucket/metadata_sys.rs

@@ -412,6 +412,10 @@ impl BucketMetadataSys {
         }
     }
 
+    pub(crate) fn object_store(&self) -> Arc<ECStore> {
+        self.api.clone()
+    }
+
     pub async fn init(&mut self, buckets: Vec<String>) {
         let _ = self.init_internal(buckets).await;
     }

crates/ecstore/src/bucket/utils.rs

@@ -183,6 +183,16 @@ pub fn is_valid_object_name(object: &str) -> bool {
     is_valid_object_prefix(object)
 }
 
+/// Client-facing reason attached to rejections of object keys that Win32/NTFS
+/// cannot represent as file paths (issue #3299). Deployments on Linux/macOS
+/// accept the full S3 key character set.
+pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str =
+    "object key contains characters unsupported on Windows hosts (one of ':', '*', '?', '\"', '|', '<', '>')";
+
+/// Client-facing reason for path segments Windows can store but not address
+/// afterwards (issue #3449): trailing dot/space or reserved DOS device names.
+pub const WINDOWS_RESERVED_SEGMENT_REASON: &str = "object key contains a path segment unsupported on Windows hosts (trailing dot or space, or a reserved device name such as NUL/CON/COM1)";
+
 /// Reserved DOS device names that shadow regular files on Windows, even when
 /// an extension is appended (e.g. `NUL.txt` resolves to the `NUL` device).
 const WINDOWS_RESERVED_NAMES: &[&str] = &[
@@ -226,13 +236,21 @@ pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Res
             || object.contains('>')
         // || object.contains('\\')
         {
-            return Err(StorageError::ObjectNameInvalid(bucket.to_owned(), object.to_owned()));
+            return Err(StorageError::InvalidArgument(
+                bucket.to_owned(),
+                object.to_owned(),
+                WINDOWS_RESERVED_CHARACTERS_REASON.to_owned(),
+            ));
         }
 
         // Reject names that NTFS would happily create but the Win32 path
         // layer cannot read back (os error 3), e.g. `baddir.` or `NUL.txt`.
         if object_name_has_windows_incompatible_segment(object) {
-            return Err(StorageError::ObjectNameInvalid(bucket.to_owned(), object.to_owned()));
+            return Err(StorageError::InvalidArgument(
+                bucket.to_owned(),
+                object.to_owned(),
+                WINDOWS_RESERVED_SEGMENT_REASON.to_owned(),
+            ));
         }
     }
 
@@ -537,8 +555,8 @@ mod tests {
             let result = check_object_name_for_length_and_slash("test-bucket", object);
             if cfg!(target_os = "windows") {
                 assert!(
-                    matches!(result, Err(StorageError::ObjectNameInvalid(..))),
-                    "object name must be rejected on Windows: {object:?}"
+                    matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_SEGMENT_REASON),
+                    "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}"
                 );
             } else {
                 assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}");
@@ -554,6 +572,23 @@ mod tests {
         }
     }
 
+    #[test]
+    fn test_check_object_name_windows_reserved_characters() {
+        // Keys containing Win32-reserved characters are rejected on Windows
+        // with a descriptive reason (issue #3299); valid elsewhere.
+        for object in ["path/*sUt*mykey", "a:b", "what?", "pipe|name", "quote\"d", "lt<gt>"] {
+            let result = check_object_name_for_length_and_slash("test-bucket", object);
+            if cfg!(target_os = "windows") {
+                assert!(
+                    matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_CHARACTERS_REASON),
+                    "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}"
+                );
+            } else {
+                assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}");
+            }
+        }
+    }
+
     #[test]
     fn test_check_bucket_and_object_names() {
         // Valid names

crates/ecstore/src/cache_value/metacache_set.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::disk::disk_store::get_drive_walkdir_stall_timeout;
+use crate::disk::disk_store::{get_drive_walkdir_peek_timeout, get_drive_walkdir_stall_timeout};
 use crate::disk::error::DiskError;
 use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
 use metrics::counter;
@@ -124,6 +124,24 @@ fn classify_listing_quorum_failure(errors: &[DiskError]) -> DiskError {
     DiskError::ErasureReadQuorum
 }
 
+/// Returns true when a metacache listing missed quorum purely because the
+/// volume or path is absent on a quorum of drives, i.e. every recorded failure
+/// is [`DiskError::VolumeNotFound`] or [`DiskError::FileNotFound`].
+///
+/// This is a benign, expected outcome: the caller is handed
+/// `VolumeNotFound`/`FileNotFound` and decides how to react. The most common
+/// trigger is a startup race where the system bucket has not yet been created
+/// on every drive when an early reader (e.g. the IAM config loader) lists it.
+/// It must be distinguished from a listing that failed for a real reason (I/O,
+/// timeout, corruption) so the former is not surfaced at `error`. See
+/// rustfs/rustfs#5076.
+fn is_benign_not_found_listing_failure(errors: &[DiskError]) -> bool {
+    !errors.is_empty()
+        && errors
+            .iter()
+            .all(|err| matches!(err, DiskError::VolumeNotFound | DiskError::FileNotFound))
+}
+
 struct PublishedBytesWriter<W> {
     inner: W,
     published: bool,
@@ -175,6 +193,10 @@ pub(crate) enum TestReaderBehavior {
     ProducerError(DiskError),
     PrimaryErrorThenFallback(DiskError),
     PartialThenTimeout(Vec<MetaCacheEntry>),
+    DelayedEntries {
+        delay: Duration,
+        entries: Vec<MetaCacheEntry>,
+    },
 }
 
 #[cfg(test)]
@@ -339,6 +361,13 @@ async fn list_path_raw_inner(
                         drop(out);
                         Some(err)
                     }
+                    TestReaderBehavior::DelayedEntries { delay, entries } => {
+                        tokio::time::sleep(delay).await;
+                        let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
+                        out.write(&entries).await.expect("delayed test entries should be written");
+                        out.close().await.expect("delayed test entries should close");
+                        return Ok(());
+                    }
                 }
             } else {
                 None
@@ -473,6 +502,17 @@ async fn list_path_raw_inner(
                             record_producer_error(&producer_errs_clone, disk_idx, &err);
                             return Err(err);
                         }
+                        TestReaderBehavior::DelayedEntries { delay, entries } => {
+                            tokio::time::sleep(delay).await;
+                            let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
+                            out.write(&entries)
+                                .await
+                                .expect("delayed test fallback entries should be written");
+                            out.close().await.expect("delayed test fallback entries should close");
+                            need_fallback = false;
+                            last_err = None;
+                            continue;
+                        }
                     }
                 }
 
@@ -609,47 +649,23 @@ async fn list_path_raw_inner(
         // Consumer-side peek timeout, and a caveat worth understanding
         // (rustfs/backlog#1217).
         //
-        // This budget is the SAME SOURCE and SAME VALUE as the producer-side
-        // walk stall budget: both default to `walkdir_stall_timeout` and fall
-        // back to `get_drive_walkdir_stall_timeout()` (5s). But the two measure
-        // different things:
+        // This budget must not be stricter than the producer-side walk stall
+        // budget. The two measure different things:
         //   * producer stall: bounds a single drive READ inside the walk (see
         //     `with_walk_stall_deadline` in `disk/local.rs`);
         //   * this consumer peek: bounds the interval between two ADJACENT
         //     entries arriving from a drive's reader (`peek_with_timeout`).
         //
-        // Because they are coupled to the same value, the consumer cannot wait
-        // meaningfully longer for the next entry than the producer is allowed to
-        // spend producing one. When a drive walks a region dense with
-        // non-listable internal items (many entries the producer filters out
-        // before emitting the next visible one), a HEALTHY drive can take longer
-        // than one budget to hand the consumer its next entry. The consumer then
-        // classifies it as `PeekOutcome::TimedOut` and DETACHES that reader (it
-        // is replaced by a drained duplex below), dropping a good drive from the
-        // merge. That caps the "large prefix always succeeds" guarantee: a wide
-        // enough non-listable stretch can knock healthy drives out of quorum.
-        //
-        // This is left documented, not decoupled. Giving the consumer peek an
-        // independent, strictly-larger budget would reduce these false detaches,
-        // but it also delays detaching a genuinely dead drive by the same amount
-        // and shifts listing tail-latency semantics; that trade-off wants soak
-        // data before it changes the default, so it is deferred to a follow-up.
-        // The constraint for any such change: the consumer peek must be >= the
-        // producer stall (never stricter), so it can never declare a drive
-        // stalled before the producer itself would have failed.
-        let peek_timeout = opts
-            .walkdir_stall_timeout
-            .or({
-                #[cfg(test)]
-                {
-                    opts.peek_timeout
-                }
-                #[cfg(not(test))]
-                {
-                    None
-                }
-            })
-            .unwrap_or_else(get_drive_walkdir_stall_timeout);
+        // A drive can spend one stall budget walking a dense non-listable region
+        // before it can publish the next visible entry. Use an independent,
+        // profile-aware consumer budget so the merge does not detach that reader
+        // before the producer itself would have failed.
+        let producer_stall_timeout = opts.walkdir_stall_timeout.unwrap_or_else(get_drive_walkdir_stall_timeout);
+        let configured_peek_timeout = get_drive_walkdir_peek_timeout().max(producer_stall_timeout);
+        #[cfg(not(test))]
+        let peek_timeout = configured_peek_timeout;
+        #[cfg(test)]
+        let peek_timeout = opts.peek_timeout.unwrap_or(configured_peek_timeout);
         let mut errs: Vec<Option<DiskError>> = Vec::with_capacity(readers.len());
         for _ in 0..readers.len() {
             errs.push(None);
@@ -842,17 +858,38 @@ async fn list_path_raw_inner(
                     _ => {}
                 });
 
-                error!(
-                    event = EVENT_METACACHE_LISTING,
-                    component = LOG_COMPONENT_ECSTORE,
-                    subsystem = LOG_SUBSYSTEM_METACACHE,
-                    bucket = %opts.bucket,
-                    path = %opts.path,
-                    state = "quorum_failed",
-                    error = %combined_err.join(", "),
-                    "Metacache listing quorum failed"
-                );
                 let failures = errs.iter().flatten().cloned().collect::<Vec<_>>();
+                // A listing that misses quorum purely because the volume/path is
+                // absent on a quorum of drives is benign and expected — the caller
+                // receives VolumeNotFound/FileNotFound and decides how to react.
+                // The common trigger is a startup race where the system bucket is
+                // not yet created on every drive when an early reader (e.g. the IAM
+                // config loader) lists it, so surfacing it at `error` is misleading
+                // noise (rustfs/rustfs#5076). Keep `error` for listings that failed
+                // for a real reason (I/O, timeout, corruption).
+                if is_benign_not_found_listing_failure(&failures) {
+                    debug!(
+                        event = EVENT_METACACHE_LISTING,
+                        component = LOG_COMPONENT_ECSTORE,
+                        subsystem = LOG_SUBSYSTEM_METACACHE,
+                        bucket = %opts.bucket,
+                        path = %opts.path,
+                        state = "quorum_not_found",
+                        error = %combined_err.join(", "),
+                        "Metacache listing quorum not reached (volume/path absent)"
+                    );
+                } else {
+                    error!(
+                        event = EVENT_METACACHE_LISTING,
+                        component = LOG_COMPONENT_ECSTORE,
+                        subsystem = LOG_SUBSYSTEM_METACACHE,
+                        bucket = %opts.bucket,
+                        path = %opts.path,
+                        state = "quorum_failed",
+                        error = %combined_err.join(", "),
+                        "Metacache listing quorum failed"
+                    );
+                }
                 return Err(classify_listing_quorum_failure(&failures));
             }
 
@@ -1039,6 +1076,21 @@ mod tests {
     use time::OffsetDateTime;
     use uuid::Uuid;
 
+    #[test]
+    fn benign_not_found_listing_failure_detection() {
+        // Pure not-found quorum misses are benign (the volume/path simply does
+        // not exist on a quorum of drives) and must not be logged at ERROR.
+        assert!(is_benign_not_found_listing_failure(&[DiskError::VolumeNotFound]));
+        assert!(is_benign_not_found_listing_failure(
+            &[DiskError::VolumeNotFound, DiskError::FileNotFound,]
+        ));
+        // No recorded failure is not a not-found case.
+        assert!(!is_benign_not_found_listing_failure(&[]));
+        // Any real error must keep the failure at ERROR severity.
+        assert!(!is_benign_not_found_listing_failure(&[DiskError::VolumeNotFound, DiskError::Timeout,]));
+        assert!(!is_benign_not_found_listing_failure(&[DiskError::DiskNotFound]));
+    }
+
     #[tokio::test]
     async fn list_path_raw_empty_disks_returns_read_quorum() {
         let err = list_path_raw(CancellationToken::new(), ListPathRawOptions::default())
@@ -1246,6 +1298,57 @@ mod tests {
         assert_eq!(err, DiskError::Timeout);
     }
 
+    #[tokio::test]
+    async fn list_path_raw_waits_past_producer_stall_for_slow_progressing_reader() {
+        let entry = MetaCacheEntry {
+            name: "bucket/visible-object".to_string(),
+            metadata: vec![1, 2, 3],
+            cached: None,
+            reusable: false,
+        };
+        let seen = Arc::new(Mutex::new(Vec::new()));
+        let seen_clone = seen.clone();
+
+        timeout(
+            Duration::from_secs(1),
+            list_path_raw(
+                CancellationToken::new(),
+                ListPathRawOptions {
+                    disks: vec![None, None],
+                    min_disks: 2,
+                    walkdir_stall_timeout: Some(Duration::from_millis(20)),
+                    test_reader_behaviors: vec![
+                        TestReaderBehavior::DelayedEntries {
+                            delay: Duration::from_millis(60),
+                            entries: vec![entry.clone()],
+                        },
+                        TestReaderBehavior::Entries(vec![entry]),
+                    ],
+                    partial: Some(Box::new(move |entries: MetaCacheEntries, _: &[Option<DiskError>]| {
+                        let seen = seen_clone.clone();
+                        Box::pin(async move {
+                            let mut names = entries.0.iter().flatten().map(|entry| entry.name.as_str());
+                            if let (Some(first), Some(second), None) = (names.next(), names.next(), names.next())
+                                && first == second
+                            {
+                                seen.lock().expect("seen mutex poisoned").push(first.to_owned());
+                            }
+                        })
+                    })),
+                    ..Default::default()
+                },
+            ),
+        )
+        .await
+        .expect("slow-progressing reader should complete inside the consumer peek budget")
+        .expect("slow-progressing reader should not be detached by the producer stall budget");
+
+        assert_eq!(
+            seen.lock().expect("seen mutex poisoned").as_slice(),
+            &["bucket/visible-object".to_string()]
+        );
+    }
+
     #[tokio::test]
     async fn list_path_raw_prefers_timeout_for_mixed_errors_beyond_quorum_budget() {
         let err = list_path_raw(

crates/ecstore/src/client/api_error_response.rs

@@ -103,7 +103,7 @@ pub fn http_resp_to_error_response(
     object_name: &str,
 ) -> ErrorResponse {
     let err_body = String::from_utf8_lossy(&b).to_string();
-    if h.is_empty() || resp_status.is_client_error() || resp_status.is_server_error() {
+    if h.is_empty() || !(resp_status.is_client_error() || resp_status.is_server_error()) {
         return ErrorResponse {
             status_code: resp_status,
             code: S3ErrorCode::ResponseInterrupted,
@@ -329,4 +329,21 @@ mod tests {
         assert_eq!(value["RequestId"], Value::String("req-xml-123".to_string()));
         assert!(value.get("request_id").is_none(), "external error contract must not expose request_id");
     }
+
+    #[test]
+    fn parses_s3_error_code_from_client_error_response() {
+        let mut headers = HeaderMap::new();
+        headers.insert("x-amz-request-id", "request-id".parse().expect("request ID header should parse"));
+
+        let response = http_resp_to_error_response(
+            StatusCode::NOT_FOUND,
+            &headers,
+            b"<Error><Code>NoSuchVersion</Code><Message>remote detail</Message></Error>".to_vec(),
+            "bucket",
+            "object",
+        );
+
+        assert_eq!(response.code, S3ErrorCode::NoSuchVersion);
+        assert_eq!(response.status_code, StatusCode::NOT_FOUND);
+    }
 }

crates/ecstore/src/client/api_get_object.rs

@@ -30,7 +30,7 @@ use tokio_util::io::StreamReader;
 use crate::client::{
     api_error_response::err_invalid_argument,
     api_get_options::GetObjectOptions,
-    transition_api::{ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, to_object_info},
+    transition_api::{ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, to_object_info_for_provider},
 };
 use futures_util::StreamExt;
 use http_body_util::BodyExt;
@@ -77,7 +77,8 @@ impl TransitionClient {
             )
             .await?;
 
-        let object_stat = to_object_info(bucket_name, object_name, resp.headers())?;
+        let object_stat =
+            to_object_info_for_provider(bucket_name, object_name, resp.headers(), self.provider_version_capabilities())?;
 
         let h = resp.headers().clone();
 

crates/ecstore/src/client/api_list.rs

@@ -24,7 +24,7 @@ use crate::client::{
         ListBucketResult, ListBucketV2Result, ListMultipartUploadsResult, ListObjectPartsResult, ListVersionsResult, ObjectPart,
     },
     credentials,
-    transition_api::{ReaderImpl, RequestMetadata, TransitionClient},
+    transition_api::{ReaderImpl, RequestMetadata, TransitionClient, collect_response_body},
 };
 use crate::storage_api_contracts::bucket::BucketInfo;
 use http::{HeaderMap, StatusCode};
@@ -88,7 +88,7 @@ impl TransitionClient {
             url_values.insert("max-keys".to_string(), max_keys.to_string());
         }
 
-        let mut resp = self
+        let resp = self
             .execute_method(
                 http::Method::GET,
                 &mut RequestMetadata {
@@ -164,7 +164,7 @@ impl TransitionClient {
         Ok(list_bucket_result)
     }
 
-    pub fn list_object_versions_query(
+    pub async fn list_object_versions_query(
         &self,
         bucket_name: &str,
         opts: &ListObjectsOptions,
@@ -172,93 +172,88 @@ impl TransitionClient {
         version_id_marker: &str,
         delimiter: &str,
     ) -> Result<ListVersionsResult, std::io::Error> {
-        /*if err := s3utils.CheckValidBucketName(bucketName); err != nil {
-          return ListVersionsResult{}, err
+        let mut url_values = HashMap::new();
+        url_values.insert("versions".to_string(), "".to_string());
+        url_values.insert("prefix".to_string(), opts.prefix.clone());
+        url_values.insert("delimiter".to_string(), delimiter.to_string());
+        url_values.insert("encoding-type".to_string(), "url".to_string());
+
+        if !key_marker.is_empty() {
+            url_values.insert("key-marker".to_string(), key_marker.to_string());
         }
-        if err := s3utils.CheckValidObjectNamePrefix(opts.Prefix); err != nil {
-          return ListVersionsResult{}, err
-        }
-        urlValues := make(url.Values)
-
-        urlValues.Set("versions", "")
-
-        urlValues.Set("prefix", opts.Prefix)
-
-        urlValues.Set("delimiter", delimiter)
-
-        if keyMarker != "" {
-          urlValues.Set("key-marker", keyMarker)
-        }
-
         if opts.max_keys > 0 {
-          urlValues.Set("max-keys", fmt.Sprintf("%d", opts.max_keys))
+            url_values.insert("max-keys".to_string(), opts.max_keys.to_string());
+        }
+        if !version_id_marker.is_empty() {
+            url_values.insert("version-id-marker".to_string(), version_id_marker.to_string());
+        }
+        if opts.with_metadata {
+            url_values.insert("metadata".to_string(), "true".to_string());
         }
 
-        if versionIDMarker != "" {
-          urlValues.Set("version-id-marker", versionIDMarker)
+        let mut resp = self
+            .execute_method(
+                http::Method::GET,
+                &mut RequestMetadata {
+                    bucket_name: bucket_name.to_string(),
+                    object_name: "".to_string(),
+                    query_values: url_values,
+                    content_sha256_hex: EMPTY_STRING_SHA256_HASH.to_string(),
+                    custom_header: opts.headers.clone(),
+                    content_body: ReaderImpl::Body(Bytes::new()),
+                    content_length: 0,
+                    content_md5_base64: "".to_string(),
+                    stream_sha256: false,
+                    trailer: HeaderMap::new(),
+                    pre_sign_url: Default::default(),
+                    add_crc: Default::default(),
+                    extra_pre_sign_header: Default::default(),
+                    bucket_location: Default::default(),
+                    expires: Default::default(),
+                },
+            )
+            .await?;
+
+        let resp_status = resp.status();
+        let headers = resp.headers().clone();
+        let body = collect_response_body(resp.into_body(), MAX_S3_CLIENT_RESPONSE_SIZE).await?;
+        if resp_status != StatusCode::OK {
+            return Err(std::io::Error::other(http_resp_to_error_response(
+                resp_status,
+                &headers,
+                body,
+                bucket_name,
+                "",
+            )));
         }
 
-        if opts.WithMetadata {
-          urlValues.Set("metadata", "true")
+        let mut versions = quick_xml::de::from_reader::<_, ListVersionsResult>(body.as_slice())
+            .map_err(|err| std::io::Error::new(std::io::ErrorKind::InvalidData, err))?;
+        for version in &mut versions.versions {
+            version.key = decode_s3_name(&version.key, &versions.encoding_type)?;
+        }
+        for marker in &mut versions.delete_markers {
+            marker.key = decode_s3_name(&marker.key, &versions.encoding_type)?;
+        }
+        for prefix in &mut versions.common_prefixes {
+            prefix.prefix = decode_s3_name(&prefix.prefix, &versions.encoding_type)?;
+        }
+        if !versions.next_key_marker.is_empty() {
+            versions.next_key_marker = decode_s3_name(&versions.next_key_marker, &versions.encoding_type)?;
         }
 
-        urlValues.Set("encoding-type", "url")
-
-        let resp = self.executeMethod(http::Method::GET, &mut RequestMetadata{
-            bucketName:       bucketName,
-            queryValues:      urlValues,
-            contentSHA256Hex: emptySHA256Hex,
-            customHeader:     opts.headers,
-        }).await?;
-        defer closeResponse(resp)
-        if err != nil {
-          return ListVersionsResult{}, err
-        }
-        if resp != nil {
-          if resp.StatusCode != http.StatusOK {
-            return ListVersionsResult{}, httpRespToErrorResponse(resp, bucketName, "")
-          }
-        }
-
-        listObjectVersionsOutput := ListVersionsResult{}
-        err = xml_decoder(resp.Body, &listObjectVersionsOutput)
-        if err != nil {
-          return ListVersionsResult{}, err
-        }
-
-        for i, obj := range listObjectVersionsOutput.Versions {
-          listObjectVersionsOutput.Versions[i].Key, err = decode_s3_name(obj.Key, listObjectVersionsOutput.EncodingType)
-          if err != nil {
-            return listObjectVersionsOutput, err
-          }
-        }
-
-        for i, obj := range listObjectVersionsOutput.CommonPrefixes {
-          listObjectVersionsOutput.CommonPrefixes[i].Prefix, err = decode_s3_name(obj.Prefix, listObjectVersionsOutput.EncodingType)
-          if err != nil {
-            return listObjectVersionsOutput, err
-          }
-        }
-
-        if listObjectVersionsOutput.NextKeyMarker != "" {
-          listObjectVersionsOutput.NextKeyMarker, err = decode_s3_name(listObjectVersionsOutput.NextKeyMarker, listObjectVersionsOutput.EncodingType)
-          if err != nil {
-            return listObjectVersionsOutput, err
-          }
-        }
-
-        Ok(listObjectVersionsOutput)*/
-        Err(std::io::Error::new(
-            ErrorKind::Unsupported,
-            credentials::ErrorResponse {
+        if versions.is_truncated && versions.next_key_marker.is_empty() {
+            return Err(std::io::Error::other(credentials::ErrorResponse {
                 sts_error: credentials::STSError {
                     r#type: "".to_string(),
                     code: "NotImplemented".to_string(),
-                    message: format!("list_object_versions_query is not implemented for bucket {bucket_name}"),
+                    message: "Truncated ListObjectVersions response should have next key marker set".to_string(),
                 },
                 request_id: "".to_string(),
-            },
-        ))
+            }));
+        }
+
+        Ok(versions)
     }
 
     pub fn list_objects_query(
@@ -364,6 +359,7 @@ impl TransitionClient {
     }
 }
 
+#[derive(Default)]
 #[allow(dead_code)]
 pub struct ListObjectsOptions {
     reverse_versions: bool,
@@ -431,3 +427,51 @@ fn decode_s3_name(name: &str, encoding_type: &str) -> Result<String, std::io::Er
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn list_versions_xml_preserves_versions_and_delete_markers() {
+        let xml = br#"
+            <ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+                <Name>tier-bucket</Name>
+                <Prefix>archive/object</Prefix>
+                <KeyMarker></KeyMarker>
+                <VersionIdMarker></VersionIdMarker>
+                <MaxKeys>2</MaxKeys>
+                <IsTruncated>true</IsTruncated>
+                <NextKeyMarker>archive/object</NextKeyMarker>
+                <NextVersionIdMarker>version-a</NextVersionIdMarker>
+                <Version>
+                    <Key>archive/object</Key>
+                    <VersionId>version-a</VersionId>
+                    <IsLatest>true</IsLatest>
+                    <LastModified>2026-07-22T00:00:00Z</LastModified>
+                    <ETag>&quot;etag-a&quot;</ETag>
+                    <Size>5</Size>
+                    <StorageClass>STANDARD</StorageClass>
+                </Version>
+                <DeleteMarker>
+                    <Key>archive/object</Key>
+                    <VersionId>marker-a</VersionId>
+                    <IsLatest>false</IsLatest>
+                    <LastModified>2026-07-22T00:00:01Z</LastModified>
+                </DeleteMarker>
+            </ListVersionsResult>
+        "#;
+
+        let parsed =
+            quick_xml::de::from_reader::<_, ListVersionsResult>(xml.as_slice()).expect("ListObjectVersions XML should parse");
+
+        assert!(parsed.is_truncated);
+        assert_eq!(parsed.next_key_marker, "archive/object");
+        assert_eq!(parsed.next_version_id_marker, "version-a");
+        assert_eq!(parsed.versions.len(), 1);
+        assert_eq!(parsed.versions[0].key, "archive/object");
+        assert_eq!(parsed.versions[0].version_id, "version-a");
+        assert_eq!(parsed.delete_markers.len(), 1);
+        assert_eq!(parsed.delete_markers[0].version_id, "marker-a");
+    }
+}

crates/ecstore/src/client/api_put_object_multipart.rs

@@ -41,7 +41,7 @@ use crate::client::{
     transition_api::{ReaderImpl, RequestMetadata, TransitionClient, UploadInfo},
 };
 use rustfs_utils::path::trim_etag;
-use s3s::header::{X_AMZ_EXPIRATION, X_AMZ_VERSION_ID};
+use s3s::header::X_AMZ_EXPIRATION;
 
 impl TransitionClient {
     pub async fn put_object_multipart(
@@ -417,11 +417,7 @@ impl TransitionClient {
             bucket: complete_multipart_upload_result.bucket,
             key: complete_multipart_upload_result.key,
             etag: trim_etag(&complete_multipart_upload_result.etag),
-            version_id: if let Some(h_x_amz_version_id) = h.get(X_AMZ_VERSION_ID) {
-                h_x_amz_version_id.to_str().unwrap_or("").to_string()
-            } else {
-                "".to_string()
-            },
+            version_id: self.raw_version_id(&h)?.unwrap_or_default().to_string(),
             location: complete_multipart_upload_result.location,
             expiration: exp_time,
             expiration_rule_id: rule_id,

crates/ecstore/src/client/api_put_object_streaming.rs

@@ -44,7 +44,7 @@ use crate::client::{
 
 use crate::client::utils::base64_encode;
 use rustfs_utils::path::trim_etag;
-use s3s::header::{X_AMZ_EXPIRATION, X_AMZ_VERSION_ID};
+use s3s::header::X_AMZ_EXPIRATION;
 
 /// Read exactly `want` bytes for a single multipart part, or fewer if the reader
 /// reaches EOF first. Advances the reader so the next call returns the following
@@ -567,11 +567,7 @@ impl TransitionClient {
             key: object_name.to_string(),
             etag: trim_etag(h.get("ETag").and_then(|v| v.to_str().ok()).unwrap_or("")),
 
-            version_id: if let Some(h_x_amz_version_id) = h.get(X_AMZ_VERSION_ID) {
-                h_x_amz_version_id.to_str().unwrap_or("").to_string()
-            } else {
-                "".to_string()
-            },
+            version_id: self.raw_version_id(h)?.unwrap_or_default().to_string(),
             size,
             expiration: exp_time,
             expiration_rule_id: rule_id,

crates/ecstore/src/client/api_remove.rs

@@ -201,12 +201,7 @@ impl TransitionClient {
             object_name: object_name.to_string(),
             object_version_id: opts.version_id,
             delete_marker: resp.headers().get(X_AMZ_DELETE_MARKER).map_or(false, |v| v == "true"),
-            delete_marker_version_id: resp
-                .headers()
-                .get(X_AMZ_VERSION_ID)
-                .and_then(|v| v.to_str().ok())
-                .unwrap_or_default()
-                .to_string(),
+            delete_marker_version_id: self.raw_version_id(resp.headers())?.unwrap_or_default().to_string(),
             ..Default::default()
         })
     }

crates/ecstore/src/client/api_s3_datatypes.rs

@@ -55,34 +55,38 @@ pub struct ListBucketV2Result {
     pub start_after: String,
 }
 
-#[allow(dead_code)]
+#[derive(Debug, Default, Serialize, Deserialize)]
+#[serde(default, rename_all = "PascalCase")]
 pub struct Version {
-    etag: String,
-    is_latest: bool,
-    key: String,
-    last_modified: OffsetDateTime,
-    owner: Owner,
-    size: i64,
-    storage_class: String,
-    version_id: String,
-    user_metadata: HashMap<String, String>,
-    user_tags: HashMap<String, String>,
-    is_delete_marker: bool,
+    #[serde(rename = "ETag")]
+    pub etag: String,
+    pub is_latest: bool,
+    pub key: String,
+    pub size: i64,
+    pub storage_class: String,
+    pub version_id: String,
+    pub user_metadata: HashMap<String, String>,
+    pub user_tags: HashMap<String, String>,
 }
 
+#[derive(Debug, Default, Serialize, Deserialize)]
+#[serde(default, rename_all = "PascalCase")]
 pub struct ListVersionsResult {
-    versions: Vec<Version>,
-    common_prefixes: Vec<CommonPrefix>,
-    name: String,
-    prefix: String,
-    delimiter: String,
-    max_keys: i64,
-    encoding_type: String,
-    is_truncated: bool,
-    key_marker: String,
-    version_id_marker: String,
-    next_key_marker: String,
-    next_version_id_marker: String,
+    #[serde(rename = "Version")]
+    pub versions: Vec<Version>,
+    #[serde(rename = "DeleteMarker")]
+    pub delete_markers: Vec<Version>,
+    pub common_prefixes: Vec<CommonPrefix>,
+    pub name: String,
+    pub prefix: String,
+    pub delimiter: String,
+    pub max_keys: i64,
+    pub encoding_type: String,
+    pub is_truncated: bool,
+    pub key_marker: String,
+    pub version_id_marker: String,
+    pub next_key_marker: String,
+    pub next_version_id_marker: String,
 }
 
 pub struct ListBucketResult {

crates/ecstore/src/client/api_stat.rs

@@ -18,7 +18,7 @@
 #![allow(unused_must_use)]
 #![allow(clippy::all)]
 
-use http::{HeaderMap, HeaderValue};
+use http::{HeaderMap, HeaderValue, StatusCode};
 use http_body_util::BodyExt;
 use hyper::body::Body;
 use hyper::body::Bytes;
@@ -31,13 +31,62 @@ use uuid::Uuid;
 use crate::client::{
     api_error_response::{ErrorResponse, err_invalid_argument, http_resp_to_error_response},
     api_get_options::GetObjectOptions,
-    transition_api::{ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, to_object_info},
+    transition_api::{
+        ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, collect_response_body, to_object_info_for_provider,
+    },
 };
 use s3s::{
-    dto::VersioningConfiguration,
+    dto::{BucketVersioningStatus, MFADelete, VersioningConfiguration},
     header::{X_AMZ_DELETE_MARKER, X_AMZ_VERSION_ID},
 };
 
+const S3_XML_NAMESPACE: &str = "http://s3.amazonaws.com/doc/2006-03-01/";
+
+#[derive(serde::Deserialize)]
+#[serde(deny_unknown_fields)]
+struct StrictVersioningConfiguration {
+    #[serde(rename = "@xmlns")]
+    namespace: Option<String>,
+    #[serde(rename = "MfaDelete")]
+    mfa_delete: Option<String>,
+    #[serde(rename = "Status")]
+    status: Option<String>,
+}
+
+#[derive(serde::Deserialize)]
+enum StrictVersioningResponse {
+    VersioningConfiguration(StrictVersioningConfiguration),
+}
+
+fn parse_bucket_versioning_response(
+    status: StatusCode,
+    headers: &HeaderMap,
+    body: Vec<u8>,
+    bucket_name: &str,
+) -> Result<VersioningConfiguration, std::io::Error> {
+    if status != StatusCode::OK {
+        return Err(std::io::Error::other(http_resp_to_error_response(status, headers, body, bucket_name, "")));
+    }
+
+    let StrictVersioningResponse::VersioningConfiguration(parsed) =
+        quick_xml::de::from_reader(body.as_slice()).map_err(|err| std::io::Error::new(std::io::ErrorKind::InvalidData, err))?;
+    if parsed
+        .namespace
+        .as_deref()
+        .is_some_and(|namespace| namespace != S3_XML_NAMESPACE)
+    {
+        return Err(std::io::Error::new(
+            std::io::ErrorKind::InvalidData,
+            "unexpected GetBucketVersioning XML namespace",
+        ));
+    }
+    Ok(VersioningConfiguration {
+        mfa_delete: parsed.mfa_delete.map(MFADelete::from),
+        status: parsed.status.map(BucketVersioningStatus::from),
+        ..Default::default()
+    })
+}
+
 impl TransitionClient {
     pub async fn bucket_exists(&self, bucket_name: &str) -> Result<bool, std::io::Error> {
         let resp = self
@@ -123,19 +172,8 @@ impl TransitionClient {
                 let resp_status = resp.status();
                 let h = resp.headers().clone();
 
-                let mut body_vec = Vec::new();
-                let mut body = resp.into_body();
-                while let Some(frame) = body.frame().await {
-                    let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-                    if let Some(data) = frame.data_ref() {
-                        body_vec.extend_from_slice(data);
-                    }
-                }
-                let resperr = http_resp_to_error_response(resp_status, &h, body_vec, bucket_name, "");
-
-                warn!("get bucket versioning, resperr: {:?}", resperr);
-
-                Ok(VersioningConfiguration::default())
+                let body_vec = collect_response_body(resp.into_body(), rustfs_config::MAX_S3_CLIENT_RESPONSE_SIZE).await?;
+                parse_bucket_versioning_response(resp_status, &h, body_vec, bucket_name)
             }
 
             Err(err) => Err(std::io::Error::other(err)),
@@ -206,20 +244,20 @@ impl TransitionClient {
                             ..Default::default()
                         };
                         return Ok(ObjectInfo {
-                            version_id: h
-                                .get(X_AMZ_VERSION_ID)
-                                .and_then(|v| v.to_str().ok())
-                                .and_then(|s| Uuid::from_str(s).ok()),
+                            version_id: self
+                                .raw_version_id(h)?
+                                .and_then(|s| Uuid::from_str(s).ok())
+                                .filter(|v| !v.is_nil()),
                             is_delete_marker: delete_marker,
                             ..Default::default()
                         });
                         //err_resp
                     }
                     return Ok(ObjectInfo {
-                        version_id: h
-                            .get(X_AMZ_VERSION_ID)
-                            .and_then(|v| v.to_str().ok())
-                            .and_then(|s| Uuid::from_str(s).ok()),
+                        version_id: self
+                            .raw_version_id(h)?
+                            .and_then(|s| Uuid::from_str(s).ok())
+                            .filter(|v| !v.is_nil()),
                         is_delete_marker: delete_marker,
                         replication_ready: replication_ready,
                         ..Default::default()
@@ -227,7 +265,7 @@ impl TransitionClient {
                     //http_resp_to_error_response(resp, bucket_name, object_name)
                 }
 
-                Ok(to_object_info(bucket_name, object_name, h).expect("operation should succeed"))
+                to_object_info_for_provider(bucket_name, object_name, h, self.provider_version_capabilities())
             }
             Err(err) => {
                 return Err(std::io::Error::other(err));
@@ -235,3 +273,72 @@ impl TransitionClient {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::parse_bucket_versioning_response;
+    use http::{HeaderMap, StatusCode};
+    use s3s::dto::BucketVersioningStatus;
+
+    #[test]
+    fn parses_bucket_versioning_statuses_mfa_delete_and_unversioned_state() {
+        for (xml, expected) in [
+            (
+                br#"<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Enabled</Status></VersioningConfiguration>"#
+                    .as_slice(),
+                Some(BucketVersioningStatus::ENABLED),
+            ),
+            (
+                br#"<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Suspended</Status></VersioningConfiguration>"#
+                    .as_slice(),
+                Some(BucketVersioningStatus::SUSPENDED),
+            ),
+            (
+                br#"<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>"#.as_slice(),
+                None,
+            ),
+        ] {
+            let config = parse_bucket_versioning_response(StatusCode::OK, &HeaderMap::new(), xml.to_vec(), "tier-bucket")
+                .expect("valid GetBucketVersioning response should parse");
+            assert_eq!(config.status.as_ref().map(|status| status.as_str()), expected);
+        }
+
+        let config = parse_bucket_versioning_response(
+            StatusCode::OK,
+            &HeaderMap::new(),
+            br#"<VersioningConfiguration><MfaDelete>Enabled</MfaDelete></VersioningConfiguration>"#.to_vec(),
+            "tier-bucket",
+        )
+        .expect("valid MfaDelete should parse");
+        assert_eq!(config.mfa_delete.as_ref().map(|status| status.as_str()), Some("Enabled"));
+    }
+
+    #[test]
+    fn rejects_non_ok_and_malformed_bucket_versioning_responses() {
+        let valid = br#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#.to_vec();
+        for status in [StatusCode::NO_CONTENT, StatusCode::PARTIAL_CONTENT] {
+            let status_err = parse_bucket_versioning_response(status, &HeaderMap::new(), valid.clone(), "tier-bucket")
+                .expect_err("GetBucketVersioning must return exactly HTTP 200");
+            assert_eq!(status_err.kind(), std::io::ErrorKind::Other);
+        }
+
+        let parse_err = parse_bucket_versioning_response(
+            StatusCode::OK,
+            &HeaderMap::new(),
+            b"<VersioningConfiguration>".to_vec(),
+            "tier-bucket",
+        )
+        .expect_err("malformed GetBucketVersioning XML must fail closed");
+        assert_eq!(parse_err.kind(), std::io::ErrorKind::InvalidData);
+
+        for xml in [
+            b"<VersioningConfiguration><Statuz>Enabled</Statuz></VersioningConfiguration>".as_slice(),
+            b"<WrongRoot><Status>Enabled</Status></WrongRoot>".as_slice(),
+            b"<VersioningConfiguration xmlns=\"https://example.invalid\"/>".as_slice(),
+        ] {
+            let strict_err = parse_bucket_versioning_response(StatusCode::OK, &HeaderMap::new(), xml.to_vec(), "tier-bucket")
+                .expect_err("unknown GetBucketVersioning XML must fail closed");
+            assert_eq!(strict_err.kind(), std::io::ErrorKind::InvalidData);
+        }
+    }
+}

crates/ecstore/src/client/mod.rs

@@ -38,6 +38,7 @@ pub mod constants;
 pub mod credentials;
 pub mod object_api_utils;
 pub mod object_handlers_common;
+pub(crate) mod provider_versions;
 pub(crate) mod runtime_sources;
 pub mod signer_error;
 pub mod transition_api;

crates/ecstore/src/client/provider_versions.rs

@@ -0,0 +1,236 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use std::io::{Error, ErrorKind};
+
+use http::HeaderMap;
+
+const X_AMZ_VERSION_ID: &str = "x-amz-version-id";
+const X_OSS_VERSION_ID: &str = "x-oss-version-id";
+const X_COS_VERSION_ID: &str = "x-cos-version-id";
+const X_OBS_VERSION_ID: &str = "x-obs-version-id";
+const MAX_REMOTE_VERSION_ID_LEN: usize = 1024;
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(crate) enum BucketVersioningState {
+    Unknown,
+    Disabled,
+    Suspended,
+    Enabled,
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub(crate) enum RemoteVersion {
+    Unknown,
+    Disabled,
+    SuspendedNull,
+    Exact(String),
+}
+
+impl RemoteVersion {
+    pub(crate) fn exact_id(&self) -> Option<&str> {
+        match self {
+            Self::SuspendedNull => Some("null"),
+            Self::Exact(version_id) => Some(version_id),
+            Self::Unknown | Self::Disabled => None,
+        }
+    }
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(crate) struct ProviderVersionCapabilities {
+    raw_version_header: Option<&'static str>,
+    pub(crate) exact_get_delete: bool,
+}
+
+impl ProviderVersionCapabilities {
+    pub(crate) fn for_tier_type(tier_type: &str) -> Self {
+        if tier_type.eq_ignore_ascii_case("s3")
+            || tier_type.eq_ignore_ascii_case("rustfs")
+            || tier_type.eq_ignore_ascii_case("minio")
+            || tier_type.eq_ignore_ascii_case("r2")
+            || tier_type.eq_ignore_ascii_case("wasabi")
+        {
+            Self {
+                raw_version_header: Some(X_AMZ_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else if tier_type.eq_ignore_ascii_case("aliyun") {
+            Self {
+                raw_version_header: Some(X_OSS_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else if tier_type.eq_ignore_ascii_case("tencent") {
+            Self {
+                raw_version_header: Some(X_COS_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else if tier_type.eq_ignore_ascii_case("huaweicloud") {
+            Self {
+                raw_version_header: Some(X_OBS_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else {
+            Self {
+                raw_version_header: None,
+                exact_get_delete: false,
+            }
+        }
+    }
+
+    pub(crate) fn raw_version_id(self, headers: &HeaderMap) -> Result<Option<&str>, Error> {
+        let Some(header_name) = self.raw_version_header else {
+            return Ok(None);
+        };
+        let Some(value) = headers.get(header_name) else {
+            return Ok(None);
+        };
+        let value = value
+            .to_str()
+            .map_err(|_| Error::new(ErrorKind::InvalidData, "remote object version id is not valid ASCII"))?;
+        validate_remote_version_id(value)?;
+        Ok(Some(value))
+    }
+
+    pub(crate) fn remote_version(self, headers: &HeaderMap, versioning: BucketVersioningState) -> Result<RemoteVersion, Error> {
+        let Some(value) = self.raw_version_id(headers)? else {
+            return Ok(match versioning {
+                BucketVersioningState::Disabled => RemoteVersion::Disabled,
+                BucketVersioningState::Unknown | BucketVersioningState::Suspended | BucketVersioningState::Enabled => {
+                    RemoteVersion::Unknown
+                }
+            });
+        };
+        if value == "null" {
+            return Ok(RemoteVersion::SuspendedNull);
+        }
+        Ok(RemoteVersion::Exact(value.to_string()))
+    }
+}
+
+fn validate_remote_version_id(version_id: &str) -> Result<(), Error> {
+    if version_id.is_empty() {
+        return Err(Error::new(
+            ErrorKind::InvalidData,
+            "remote tier returned an empty object version id header",
+        ));
+    }
+    if version_id.len() > MAX_REMOTE_VERSION_ID_LEN {
+        return Err(Error::new(
+            ErrorKind::InvalidData,
+            "remote tier returned an oversized object version id header",
+        ));
+    }
+    if version_id.chars().any(char::is_control) {
+        return Err(Error::new(
+            ErrorKind::InvalidData,
+            "remote tier returned an object version id containing control characters",
+        ));
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion};
+    use http::{HeaderMap, HeaderValue};
+
+    #[test]
+    fn provider_version_header_matrix_preserves_opaque_versions() {
+        for (tier_type, header_name) in [
+            ("s3", "x-amz-version-id"),
+            ("S3", "x-amz-version-id"),
+            ("rustfs", "x-amz-version-id"),
+            ("RustFS", "x-amz-version-id"),
+            ("minio", "x-amz-version-id"),
+            ("MinIO", "x-amz-version-id"),
+            ("r2", "x-amz-version-id"),
+            ("R2", "x-amz-version-id"),
+            ("wasabi", "x-amz-version-id"),
+            ("Wasabi", "x-amz-version-id"),
+            ("aliyun", "x-oss-version-id"),
+            ("Aliyun", "x-oss-version-id"),
+            ("tencent", "x-cos-version-id"),
+            ("Tencent", "x-cos-version-id"),
+            ("huaweicloud", "x-obs-version-id"),
+            ("Huaweicloud", "x-obs-version-id"),
+        ] {
+            let mut headers = HeaderMap::new();
+            headers.insert(header_name, HeaderValue::from_static("opaque.version_01"));
+            let capabilities = ProviderVersionCapabilities::for_tier_type(tier_type);
+
+            assert_eq!(capabilities.raw_version_id(&headers).expect("raw version"), Some("opaque.version_01"));
+            assert_eq!(
+                capabilities
+                    .remote_version(&headers, BucketVersioningState::Enabled)
+                    .expect("remote version"),
+                RemoteVersion::Exact("opaque.version_01".to_string())
+            );
+        }
+    }
+
+    #[test]
+    fn provider_version_header_matrix_does_not_cross_read_sibling_headers() {
+        let mut headers = HeaderMap::new();
+        headers.insert("x-amz-version-id", HeaderValue::from_static("aws-version"));
+        headers.insert("x-cos-version-id", HeaderValue::from_static("cos-version"));
+
+        assert_eq!(
+            ProviderVersionCapabilities::for_tier_type("s3")
+                .raw_version_id(&headers)
+                .expect("aws raw version"),
+            Some("aws-version")
+        );
+        assert_eq!(
+            ProviderVersionCapabilities::for_tier_type("tencent")
+                .raw_version_id(&headers)
+                .expect("cos raw version"),
+            Some("cos-version")
+        );
+    }
+
+    #[test]
+    fn provider_version_missing_header_is_unknown_until_bucket_state_is_known() {
+        let headers = HeaderMap::new();
+        let capabilities = ProviderVersionCapabilities::for_tier_type("aliyun");
+
+        assert_eq!(
+            capabilities
+                .remote_version(&headers, BucketVersioningState::Unknown)
+                .expect("unknown versioning"),
+            RemoteVersion::Unknown
+        );
+        assert_eq!(
+            capabilities
+                .remote_version(&headers, BucketVersioningState::Disabled)
+                .expect("disabled versioning"),
+            RemoteVersion::Disabled
+        );
+    }
+
+    #[test]
+    fn provider_version_rejects_empty_or_oversized_headers() {
+        let oversized = "v".repeat(1025);
+        for bad in ["", oversized.as_str()] {
+            let mut headers = HeaderMap::new();
+            headers.insert("x-oss-version-id", HeaderValue::from_str(bad).expect("test header value"));
+
+            assert!(
+                ProviderVersionCapabilities::for_tier_type("aliyun")
+                    .raw_version_id(&headers)
+                    .is_err()
+            );
+        }
+    }
+}

crates/ecstore/src/client/transition_api.rs

@@ -31,6 +31,7 @@ use crate::client::{
     },
     constants::{UNSIGNED_PAYLOAD, UNSIGNED_PAYLOAD_TRAILER},
     credentials::{CredContext, Credentials, SignatureType, Static},
+    provider_versions::{BucketVersioningState, ProviderVersionCapabilities},
     signer_error,
 };
 use crate::{client::checksum::ChecksumMode, object_api::GetObjectReader};
@@ -41,7 +42,7 @@ use http::{
     request::{Builder, Request},
 };
 use http_body::Body;
-use http_body_util::BodyExt;
+use http_body_util::{BodyExt, LengthLimitError, Limited};
 use hyper::body::Bytes;
 use hyper::body::Incoming;
 use hyper_rustls::{ConfigBuilderExt, HttpsConnector};
@@ -54,9 +55,7 @@ use rustfs_rio::HashReader;
 use rustfs_utils::HashAlgorithm;
 use rustfs_utils::{
     net::get_endpoint_url,
-    retry::{
-        DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, MAX_RETRY, RetryTimer, is_http_status_retryable, is_s3code_retryable,
-    },
+    retry::{DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, MAX_RETRY, RetryTimer},
 };
 use rustls_pki_types::PrivateKeyDer;
 use rustls_pki_types::pem::PemObject;
@@ -81,9 +80,25 @@ use url::{Url, form_urlencoded};
 use uuid::Uuid;
 
 const C_USER_AGENT: &str = "RustFS (linux; x86)";
+pub(crate) const MAX_S3_ERROR_RESPONSE_SIZE: usize = 64 * 1024;
 
 const SUCCESS_STATUS: [StatusCode; 3] = [StatusCode::OK, StatusCode::NO_CONTENT, StatusCode::PARTIAL_CONTENT];
 
+pub(crate) async fn collect_response_body<B>(body: B, limit: usize) -> Result<Vec<u8>, std::io::Error>
+where
+    B: Body<Data = Bytes>,
+    B::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
+{
+    let body = Limited::new(body, limit).collect().await.map_err(|err| {
+        if err.is::<LengthLimitError>() {
+            std::io::Error::new(std::io::ErrorKind::InvalidData, "remote tier response body exceeds limit")
+        } else {
+            std::io::Error::other(err)
+        }
+    })?;
+    Ok(body.to_bytes().to_vec())
+}
+
 const C_UNKNOWN: i32 = -1;
 const C_OFFLINE: i32 = 0;
 const C_ONLINE: i32 = 1;
@@ -92,9 +107,18 @@ fn invalid_utf8_header_error(scope: &str, header_name: &str) -> std::io::Error {
     signer_error::invalid_utf8_header_error(scope, header_name)
 }
 
-fn validate_header_values(headers: &HeaderMap, scope: &str) -> Result<(), std::io::Error> {
+fn validate_header_values(headers: &HeaderMap, scope: &str, signer_type: &SignatureType) -> Result<(), std::io::Error> {
     for (name, value) in headers {
-        value.to_str().map_err(|_| invalid_utf8_header_error(scope, name.as_str()))?;
+        if signer_type == &SignatureType::SignatureV2 {
+            // The SigV2 canonicalizer only supports visible ASCII values. Keep rejecting
+            // non-ASCII here so it cannot silently omit a value that is sent on the wire.
+            value.to_str().map_err(|_| invalid_utf8_header_error(scope, name.as_str()))?;
+        } else {
+            let value = std::str::from_utf8(value.as_bytes()).map_err(|_| invalid_utf8_header_error(scope, name.as_str()))?;
+            if value.chars().any(|ch| !ch.is_ascii() && ch.is_whitespace()) {
+                return Err(invalid_utf8_header_error(scope, name.as_str()));
+            }
+        }
     }
     Ok(())
 }
@@ -300,6 +324,14 @@ impl TransitionClient {
         self.endpoint_url.clone()
     }
 
+    pub(crate) fn provider_version_capabilities(&self) -> ProviderVersionCapabilities {
+        ProviderVersionCapabilities::for_tier_type(&self.tier_type)
+    }
+
+    pub(crate) fn raw_version_id<'a>(&self, headers: &'a HeaderMap) -> Result<Option<&'a str>, std::io::Error> {
+        self.provider_version_capabilities().raw_version_id(headers)
+    }
+
     fn trace_errors_only_off(&self) {
         if let Ok(mut trace_errors_only) = self.trace_errors_only.lock() {
             *trace_errors_only = false;
@@ -379,20 +411,17 @@ impl TransitionClient {
     pub async fn doit(&self, req: Request<s3s::Body>) -> Result<Response<Incoming>, std::io::Error> {
         let req_method;
         let req_uri;
-        let req_headers;
         let resp;
         let http_client = self.http_client.clone();
         {
             req_method = req.method().clone();
             req_uri = req.uri().clone();
-            req_headers = req.headers().clone();
 
             debug!("endpoint_url: {}", self.endpoint_url.as_str().to_string());
             resp = http_client.request(req);
         }
         let resp = resp.await;
         debug!("http_client url: {} {}", req_method, req_uri);
-        debug!("http_client headers: {:?}", req_headers);
         if let Err(err) = resp {
             error!("http_client call error: {:?}", err);
             return Err(std::io::Error::other(err));
@@ -402,28 +431,23 @@ impl TransitionClient {
             Ok(r) => r,
             Err(_) => return Err(std::io::Error::other("Unexpected error in response")),
         };
-        debug!("http_resp: {:?}", resp);
+        debug!(status = %resp.status(), "remote tier response received");
 
         //let b = resp.body_mut().store_all_unlimited().await.unwrap().to_vec();
         //debug!("http_resp_body: {}", String::from_utf8(b).unwrap());
 
         //if self.is_trace_enabled && !(self.trace_errors_only && resp.status() == StatusCode::OK) {
         if !resp.status().is_success() {
-            //self.dump_http(&cloned_req, &resp)?;
-            let mut body_vec = Vec::new();
-            let mut body = resp.into_body();
-            while let Some(frame) = body.frame().await {
-                let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-                if let Some(data) = frame.data_ref() {
-                    body_vec.extend_from_slice(data);
-                }
-            }
-            let body_str = String::from_utf8_lossy(&body_vec);
-            warn!("err_body: {}", body_str);
-            Err(std::io::Error::other(format!("http_client call error: {}", body_str)))
-        } else {
-            Ok(resp)
+            let status = resp.status();
+            let request_id = resp
+                .headers()
+                .get("x-amz-request-id")
+                .and_then(|value| value.to_str().ok())
+                .unwrap_or_default()
+                .to_string();
+            warn!(status = %status, request_id, "remote tier request rejected");
         }
+        Ok(resp)
     }
 
     pub async fn execute_method(
@@ -465,17 +489,22 @@ impl TransitionClient {
             let resp_status = resp.status();
             let h = resp.headers().clone();
 
-            let mut body_vec = Vec::new();
-            let mut body = resp.into_body();
-            while let Some(frame) = body.frame().await {
-                let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-                if let Some(data) = frame.data_ref() {
-                    body_vec.extend_from_slice(data);
-                }
-            }
-            let mut err_response =
-                http_resp_to_error_response(resp_status, &h, body_vec.clone(), &metadata.bucket_name, &metadata.object_name);
-            err_response.message = format!("remote tier error: {}", err_response.message);
+            let body_vec = collect_response_body(resp.into_body(), MAX_S3_ERROR_RESPONSE_SIZE).await?;
+            let parsed_error =
+                http_resp_to_error_response(resp_status, &h, body_vec, &metadata.bucket_name, &metadata.object_name);
+            let routing_region = parsed_error.region;
+            let code = match parsed_error.code {
+                S3ErrorCode::Custom(_) => S3ErrorCode::ResponseInterrupted,
+                code => code,
+            };
+            let err_response = ErrorResponse {
+                status_code: resp_status,
+                message: format!("remote tier request failed with status {resp_status}: {}", code.as_str()),
+                code,
+                bucket_name: metadata.bucket_name.clone(),
+                key: metadata.object_name.clone(),
+                ..Default::default()
+            };
 
             if self.region == "" {
                 return match err_response.code {
@@ -484,20 +513,20 @@ impl TransitionClient {
                         Err(std::io::Error::other(err_response))
                     }
                     S3ErrorCode::AccessDenied => {
-                        if err_response.region == "" {
+                        if routing_region.is_empty() {
                             return Err(std::io::Error::other(err_response));
                         }
                         if metadata.bucket_name != "" {
                             if let Ok(mut bucket_loc_cache) = self.bucket_loc_cache.lock() {
                                 if let Some(location) = bucket_loc_cache.get(&metadata.bucket_name) {
-                                    if location != err_response.region {
-                                        bucket_loc_cache.set(&metadata.bucket_name, &err_response.region);
+                                    if location != routing_region {
+                                        bucket_loc_cache.set(&metadata.bucket_name, &routing_region);
                                         //continue;
                                     }
                                 }
                             }
-                        } else if err_response.region != metadata.bucket_location {
-                            metadata.bucket_location = err_response.region.clone();
+                        } else if routing_region != metadata.bucket_location {
+                            metadata.bucket_location = routing_region;
                             //continue;
                         }
                         Err(std::io::Error::other(err_response))
@@ -508,18 +537,10 @@ impl TransitionClient {
                 };
             }
 
-            if is_s3code_retryable(err_response.code.as_str()) {
-                continue;
-            }
-
-            if is_http_status_retryable(&resp_status) {
-                continue;
-            }
-
-            break;
+            return Err(std::io::Error::other(err_response));
         }
 
-        Err(std::io::Error::other("resp err"))
+        Err(std::io::Error::other("remote tier request did not produce a response"))
     }
 
     async fn new_request(
@@ -586,7 +607,7 @@ impl TransitionClient {
                     )));
                 }
                 if let Some(extra_headers) = metadata.extra_pre_sign_header.as_ref() {
-                    validate_header_values(extra_headers, "presign extra header")?;
+                    validate_header_values(extra_headers, "presign extra header", &signer_type)?;
                     let headers = req.headers_mut();
                     for (k, v) in extra_headers {
                         headers.insert(k, v.clone());
@@ -611,7 +632,7 @@ impl TransitionClient {
         }
 
         self.set_user_agent(&mut req);
-        validate_header_values(&metadata.custom_header, "request custom header")?;
+        validate_header_values(&metadata.custom_header, "request custom header", &signer_type)?;
 
         for (k, v) in metadata.custom_header.clone() {
             if let Some(key) = k {
@@ -1144,6 +1165,15 @@ impl Default for UploadInfo {
 /// This function parses various S3 response headers to construct an ObjectInfo struct
 /// containing metadata about an S3 object.
 pub fn to_object_info(bucket_name: &str, object_name: &str, h: &HeaderMap) -> Result<ObjectInfo, std::io::Error> {
+    to_object_info_for_provider(bucket_name, object_name, h, ProviderVersionCapabilities::for_tier_type("s3"))
+}
+
+pub(crate) fn to_object_info_for_provider(
+    bucket_name: &str,
+    object_name: &str,
+    h: &HeaderMap,
+    version_capabilities: ProviderVersionCapabilities,
+) -> Result<ObjectInfo, std::io::Error> {
     // Helper function to get header value as string
     let get_header = |name: &str| -> String { h.get(name).and_then(|val| val.to_str().ok()).unwrap_or("").to_string() };
 
@@ -1261,14 +1291,11 @@ pub fn to_object_info(bucket_name: &str, object_name: &str, h: &HeaderMap) -> Re
     };
 
     // Extract version ID
-    let version_id = {
-        let version_id_str = get_header("x-amz-version-id");
-        if !version_id_str.is_empty() {
-            Some(Uuid::parse_str(&version_id_str).unwrap_or_else(|_| Uuid::nil()))
-        } else {
-            None
-        }
-    };
+    let version_id = version_capabilities
+        .remote_version(h, BucketVersioningState::Unknown)?
+        .exact_id()
+        .and_then(|version_id| Uuid::parse_str(version_id).ok())
+        .filter(|version_id| !version_id.is_nil());
 
     // Check if it's a delete marker
     let is_delete_marker = get_header("x-amz-delete-marker") == "true";
@@ -1383,8 +1410,43 @@ pub struct CreateBucketConfiguration {
 
 #[cfg(test)]
 mod tests {
-    use super::{build_tls_config, signer_error_to_io_error, validate_header_values, with_rustls_init_guard};
+    use super::{
+        MAX_S3_CLIENT_RESPONSE_SIZE, MAX_S3_ERROR_RESPONSE_SIZE, SignatureType, build_tls_config, collect_response_body,
+        signer_error_to_io_error, to_object_info_for_provider, validate_header_values, with_rustls_init_guard,
+    };
+    use crate::client::provider_versions::ProviderVersionCapabilities;
     use http::{HeaderMap, HeaderValue};
+    use http_body_util::Full;
+    use hyper::body::Bytes;
+    use uuid::Uuid;
+
+    #[tokio::test]
+    async fn response_body_limit_rejects_the_first_byte_past_the_boundary() {
+        let exact = collect_response_body(
+            Full::new(Bytes::from(vec![0_u8; MAX_S3_CLIENT_RESPONSE_SIZE])),
+            MAX_S3_CLIENT_RESPONSE_SIZE,
+        )
+        .await
+        .expect("a response exactly at the limit should be accepted");
+        assert_eq!(exact.len(), MAX_S3_CLIENT_RESPONSE_SIZE);
+
+        let err = collect_response_body(
+            Full::new(Bytes::from(vec![0_u8; MAX_S3_CLIENT_RESPONSE_SIZE + 1])),
+            MAX_S3_CLIENT_RESPONSE_SIZE,
+        )
+        .await
+        .expect_err("oversized remote response must fail closed");
+
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
+
+        let err = collect_response_body(
+            Full::new(Bytes::from(vec![0_u8; MAX_S3_ERROR_RESPONSE_SIZE + 1])),
+            MAX_S3_ERROR_RESPONSE_SIZE,
+        )
+        .await
+        .expect_err("oversized S3 error response must use the smaller error limit");
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
+    }
 
     #[test]
     fn rustls_guard_converts_panics_to_io_errors() {
@@ -1427,11 +1489,34 @@ mod tests {
             HeaderValue::from_bytes(&[0xFF]).expect("invalid utf8 bytes should be accepted by HeaderValue"),
         );
 
-        let err =
-            validate_header_values(&headers, "request custom header").expect_err("invalid header value should fail validation");
+        let err = validate_header_values(&headers, "request custom header", &SignatureType::SignatureV4)
+            .expect_err("invalid header value should fail validation");
         assert!(err.to_string().contains("x-amz-meta-invalid"));
     }
 
+    #[test]
+    fn validate_header_values_accepts_utf8_for_v4_but_not_v2() {
+        let mut headers = HeaderMap::new();
+        headers.insert(
+            "x-amz-meta-name",
+            HeaderValue::from_bytes("20260715/鲁A12345/object".as_bytes()).expect("valid utf8 metadata header"),
+        );
+
+        assert!(validate_header_values(&headers, "request custom header", &SignatureType::SignatureV4).is_ok());
+        assert!(validate_header_values(&headers, "request custom header", &SignatureType::SignatureV2).is_err());
+    }
+
+    #[test]
+    fn validate_header_values_rejects_non_ascii_whitespace_for_v4() {
+        let mut headers = HeaderMap::new();
+        headers.insert(
+            "x-amz-meta-name",
+            HeaderValue::from_bytes("tier/a\u{00a0}b/object".as_bytes()).expect("valid utf8 metadata header"),
+        );
+
+        assert!(validate_header_values(&headers, "request custom header", &SignatureType::SignatureV4).is_err());
+    }
+
     #[test]
     fn signer_error_mapping_preserves_header_name() {
         let err = signer_error_to_io_error(
@@ -1443,4 +1528,39 @@ mod tests {
         assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
         assert!(err.to_string().contains("x-amz-meta-invalid"));
     }
+
+    #[test]
+    fn object_info_uses_provider_version_header_without_uuid_coercion() {
+        let mut headers = HeaderMap::new();
+        headers.insert("x-cos-version-id", HeaderValue::from_static("opaque.version_01"));
+
+        let info =
+            to_object_info_for_provider("bucket", "object", &headers, ProviderVersionCapabilities::for_tier_type("tencent"))
+                .expect("opaque provider version should parse");
+
+        assert_eq!(info.version_id, None);
+        assert_eq!(
+            info.metadata.get("x-cos-version-id").and_then(|value| value.to_str().ok()),
+            Some("opaque.version_01")
+        );
+    }
+
+    #[test]
+    fn object_info_keeps_s3_uuid_version_id_and_filters_nil() {
+        let version_id = Uuid::new_v4();
+        let mut headers = HeaderMap::new();
+        headers.insert(
+            "x-amz-version-id",
+            HeaderValue::from_str(&version_id.to_string()).expect("uuid header should be valid"),
+        );
+
+        let info = to_object_info_for_provider("bucket", "object", &headers, ProviderVersionCapabilities::for_tier_type("s3"))
+            .expect("s3 uuid version should parse");
+        assert_eq!(info.version_id, Some(version_id));
+
+        headers.insert("x-amz-version-id", HeaderValue::from_static("00000000-0000-0000-0000-000000000000"));
+        let info = to_object_info_for_provider("bucket", "object", &headers, ProviderVersionCapabilities::for_tier_type("s3"))
+            .expect("nil s3 uuid version should parse");
+        assert_eq!(info.version_id, None);
+    }
 }

crates/ecstore/src/cluster/control_plane.rs

@@ -32,6 +32,9 @@ const FAILURE_DOMAIN_NOT_REPORTED: &str = "failure domain labels not reported by
 const NUMA_NOT_WIRED: &str = "NUMA topology not wired into runtime";
 const PROFILING_NOT_WIRED: &str = "profiling capability not wired into ECStore";
 const PEER_HEALTH_NOT_REPORTED: &str = "peer health not reported by endpoints";
+const PEER_HEALTH_LOCAL_NODE: &str = "local node does not require peer health probing";
+const PEER_HEALTH_REACHABLE: &str = "peer marked reachable by internode health tracker";
+const PEER_HEALTH_UNREACHABLE: &str = "peer marked unreachable by internode health tracker";
 const CONTROL_RPC_SEPARATED: &str = "control RPC remains on the gRPC control plane";
 const DATA_STREAM_RPC_SEPARATED: &str = "remote disk data streams remain on the internode data transport";
 
@@ -341,12 +344,24 @@ pub fn peer_health_snapshot_from_membership(membership: &ClusterMembershipSnapsh
             .map(|node| ClusterPeerHealth {
                 node_id: node.node_id.clone(),
                 is_local: node.is_local,
-                status: CapabilityStatus::unknown().with_reason(PEER_HEALTH_NOT_REPORTED),
+                status: peer_health_status_for_node(node),
             })
             .collect(),
     }
 }
 
+fn peer_health_status_for_node(node: &ClusterNodeMembership) -> CapabilityStatus {
+    if node.is_local {
+        return CapabilityStatus::supported().with_reason(PEER_HEALTH_LOCAL_NODE);
+    }
+
+    match rustfs_io_metrics::internode_metrics::cluster_peer_observed_online_status(&node.grid_host) {
+        Some(true) => CapabilityStatus::supported().with_reason(PEER_HEALTH_REACHABLE),
+        Some(false) => CapabilityStatus::unknown().with_reason(PEER_HEALTH_UNREACHABLE),
+        None => CapabilityStatus::disabled().with_reason(PEER_HEALTH_NOT_REPORTED),
+    }
+}
+
 pub fn rpc_boundary_snapshot() -> ClusterRpcBoundarySnapshot {
     ClusterRpcBoundarySnapshot {
         control_channels: ["metadata", "lock", "health", "administrative"]
@@ -498,6 +513,7 @@ mod tests {
     use std::collections::BTreeSet;
 
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
+    use crate::storage_api_contracts::topology::CapabilityState;
 
     #[test]
     fn topology_snapshot_maps_endpoint_sets_without_local_paths() {
@@ -588,16 +604,55 @@ mod tests {
     }
 
     #[test]
-    fn peer_health_snapshot_reports_static_unknown_status() {
-        let membership = membership_snapshot_from_endpoint_pools(&sample_url_endpoint_pools());
+    fn peer_health_snapshot_reports_observed_peer_status() {
+        let online_host = "http://cluster-control-plane-peer-online-test:9000";
+        let offline_host = "http://cluster-control-plane-peer-offline-test:9000";
+        rustfs_io_metrics::internode_metrics::record_peer_reachable(online_host);
+        rustfs_io_metrics::internode_metrics::record_peer_unreachable(offline_host, 1);
+        let membership = ClusterMembershipSnapshot {
+            nodes: vec![
+                ClusterNodeMembership {
+                    node_id: LOCAL_NODE_ID.to_string(),
+                    grid_host: String::new(),
+                    is_local: true,
+                    pools: vec![0],
+                },
+                ClusterNodeMembership {
+                    node_id: "cluster-control-plane-peer-online-test:9000".to_string(),
+                    grid_host: online_host.to_string(),
+                    is_local: false,
+                    pools: vec![0],
+                },
+                ClusterNodeMembership {
+                    node_id: "cluster-control-plane-peer-offline-test:9000".to_string(),
+                    grid_host: offline_host.to_string(),
+                    is_local: false,
+                    pools: vec![0],
+                },
+                ClusterNodeMembership {
+                    node_id: "cluster-control-plane-peer-unknown-test:9000".to_string(),
+                    grid_host: "http://cluster-control-plane-peer-unknown-test:9000".to_string(),
+                    is_local: false,
+                    pools: vec![0],
+                },
+            ],
+            drives: Vec::new(),
+        };
         let snapshot = peer_health_snapshot_from_membership(&membership);
 
-        assert_eq!(snapshot.peers.len(), 2);
-        assert_eq!(snapshot.peers[0].node_id, "node1.example:9000");
+        assert_eq!(snapshot.peers.len(), 4);
+        assert_eq!(snapshot.peers[0].node_id, LOCAL_NODE_ID);
         assert!(snapshot.peers[0].is_local);
-        assert_eq!(snapshot.peers[0].status.reason.as_deref(), Some(PEER_HEALTH_NOT_REPORTED));
-        assert_eq!(snapshot.peers[1].node_id, "node2.example:9000");
+        assert!(snapshot.peers[0].status.state.is_supported());
+        assert_eq!(snapshot.peers[0].status.reason.as_deref(), Some(PEER_HEALTH_LOCAL_NODE));
+        assert_eq!(snapshot.peers[1].node_id, "cluster-control-plane-peer-online-test:9000");
         assert!(!snapshot.peers[1].is_local);
+        assert!(snapshot.peers[1].status.state.is_supported());
+        assert_eq!(snapshot.peers[1].status.reason.as_deref(), Some(PEER_HEALTH_REACHABLE));
+        assert_eq!(snapshot.peers[2].status.reason.as_deref(), Some(PEER_HEALTH_UNREACHABLE));
+        assert!(!snapshot.peers[2].status.state.is_supported());
+        assert_eq!(snapshot.peers[3].status.reason.as_deref(), Some(PEER_HEALTH_NOT_REPORTED));
+        assert_eq!(snapshot.peers[3].status.state, CapabilityState::Disabled);
     }
 
     #[test]

crates/ecstore/src/cluster/rpc/client.rs

@@ -12,12 +12,17 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::cluster::rpc::{TONIC_RPC_PREFIX, gen_signature_headers};
+use crate::cluster::rpc::http_auth::RPC_CONTENT_SHA256_HEADER;
+use crate::cluster::rpc::{gen_tonic_signature_headers, normalize_tonic_rpc_audience};
 use crate::disk::error::{DiskError, Error as DiskErrorType};
 use crate::runtime::sources as runtime_sources;
-use http::Method;
+use http::Uri;
 use rustfs_protos::{
-    ChannelClass, create_new_channel, get_channel_for_class, proto_gen::node_service::node_service_client::NodeServiceClient,
+    ChannelClass, create_new_channel, get_channel_for_class,
+    proto_gen::node_service::{
+        heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
+        tier_mutation_control_service_client::TierMutationControlServiceClient,
+    },
 };
 use std::{error::Error, io::ErrorKind};
 use tonic::{service::interceptor::InterceptedService, transport::Channel};
@@ -36,6 +41,36 @@ pub async fn node_service_time_out_client(
     node_service_time_out_client_for_class(addr, interceptor, ChannelClass::Control).await
 }
 
+pub async fn heal_control_time_out_client(
+    addr: &str,
+    interceptor: TonicInterceptor,
+) -> Result<HealControlServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
+    let interceptor = interceptor.with_rpc_audience(addr)?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
+    let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
+    Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
+        .max_decoding_message_size(max_message_size)
+        .max_encoding_message_size(max_message_size))
+}
+
+pub async fn tier_mutation_control_time_out_client(
+    addr: &str,
+    interceptor: TonicInterceptor,
+) -> Result<TierMutationControlServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
+    let interceptor = interceptor.with_rpc_audience(addr)?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
+    let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
+    Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
+        .max_decoding_message_size(max_message_size)
+        .max_encoding_message_size(max_message_size))
+}
+
 /// Build a `NodeServiceClient` bound to the [`ChannelClass`]-appropriate channel for `addr`.
 ///
 /// Bulk `bytes`-carrying RPCs (ReadAll/WriteAll/ReadMultiple/BatchReadVersion) pass
@@ -47,6 +82,7 @@ pub async fn node_service_time_out_client_for_class(
     interceptor: TonicInterceptor,
     class: ChannelClass,
 ) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
+    let interceptor = interceptor.with_rpc_audience(addr)?;
     let channel = match class {
         ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
             Some(channel) => {
@@ -111,11 +147,25 @@ pub(crate) fn is_network_like_disk_error(err: &DiskErrorType) -> bool {
     }
 }
 
-pub struct TonicSignatureInterceptor;
+pub struct TonicSignatureInterceptor {
+    audience: Option<String>,
+}
 
 impl tonic::service::Interceptor for TonicSignatureInterceptor {
     fn call(&mut self, mut req: tonic::Request<()>) -> Result<tonic::Request<()>, tonic::Status> {
-        let headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET)
+        let method = req
+            .extensions()
+            .get::<tonic::GrpcMethod<'_>>()
+            .ok_or_else(|| tonic::Status::unauthenticated("Missing gRPC method metadata"))?;
+        let audience = self
+            .audience
+            .as_deref()
+            .ok_or_else(|| tonic::Status::unauthenticated("Missing gRPC audience"))?;
+        let content_sha256 = req
+            .metadata()
+            .get(RPC_CONTENT_SHA256_HEADER)
+            .and_then(|value| value.to_str().ok());
+        let headers = gen_tonic_signature_headers(audience, method.service(), method.method(), content_sha256)
             .map_err(|_| tonic::Status::unauthenticated("No valid auth token"))?;
         req.metadata_mut().as_mut().extend(headers);
         inject_trace_context_into_metadata(req.metadata_mut());
@@ -125,7 +175,7 @@ impl tonic::service::Interceptor for TonicSignatureInterceptor {
 }
 
 pub fn gen_tonic_signature_interceptor() -> TonicSignatureInterceptor {
-    TonicSignatureInterceptor
+    TonicSignatureInterceptor { audience: None }
 }
 
 pub struct NoOpInterceptor;
@@ -141,6 +191,22 @@ pub enum TonicInterceptor {
     NoOp(NoOpInterceptor),
 }
 
+impl TonicInterceptor {
+    fn with_rpc_audience(mut self, addr: &str) -> std::io::Result<Self> {
+        if let Self::Signature(interceptor) = &mut self {
+            let uri = addr
+                .parse::<Uri>()
+                .map_err(|_| std::io::Error::other("Invalid gRPC peer URI"))?;
+            let audience = uri
+                .authority()
+                .map(|authority| normalize_tonic_rpc_audience(authority.as_str()))
+                .ok_or_else(|| std::io::Error::other("Missing gRPC peer authority"))?;
+            interceptor.audience = Some(audience?);
+        }
+        Ok(self)
+    }
+}
+
 impl tonic::service::Interceptor for TonicInterceptor {
     fn call(&mut self, req: tonic::Request<()>) -> Result<tonic::Request<()>, tonic::Status> {
         match self {
@@ -165,6 +231,20 @@ mod tests {
         runtime_sources::ensure_test_rpc_secret();
     }
 
+    fn test_request() -> tonic::Request<()> {
+        let mut request = tonic::Request::new(());
+        request
+            .extensions_mut()
+            .insert(tonic::GrpcMethod::new("node_service.NodeService", "Ping"));
+        request
+    }
+
+    fn test_interceptor() -> TonicSignatureInterceptor {
+        TonicSignatureInterceptor {
+            audience: Some("node-a:9000".to_string()),
+        }
+    }
+
     fn with_trace_parent<F>(trace_id_hex: &str, f: F)
     where
         F: FnOnce(),
@@ -192,20 +272,55 @@ mod tests {
     #[test]
     fn test_signature_interceptor_keeps_auth_headers() {
         ensure_test_rpc_secret();
-        let mut interceptor = TonicSignatureInterceptor;
-        let req = tonic::Request::new(());
+        let mut interceptor = test_interceptor();
+        let req = test_request();
 
         let req = interceptor.call(req).expect("interceptor call should succeed");
 
         assert!(req.metadata().contains_key("x-rustfs-signature"));
         assert!(req.metadata().contains_key("x-rustfs-timestamp"));
+        assert!(req.metadata().contains_key("x-rustfs-rpc-signature-v2"));
+        assert!(req.metadata().contains_key("x-rustfs-rpc-nonce"));
+        assert!(
+            crate::cluster::rpc::verify_tonic_rpc_signature(
+                "node-a:9000",
+                "/node_service.NodeService/Ping",
+                req.metadata().as_ref(),
+            )
+            .is_ok(),
+            "interceptor signature should bind the configured peer audience and generated method"
+        );
+    }
+
+    #[test]
+    fn test_signature_interceptor_binds_audience_from_peer_uri() {
+        let interceptor = TonicInterceptor::Signature(gen_tonic_signature_interceptor())
+            .with_rpc_audience("http://node-a:9000")
+            .expect("peer URI should provide an audience");
+        let TonicInterceptor::Signature(interceptor) = interceptor else {
+            panic!("signature interceptor variant should be preserved");
+        };
+
+        assert_eq!(interceptor.audience.as_deref(), Some("node-a:9000"));
+    }
+
+    #[test]
+    fn test_signature_interceptor_requires_generated_method_metadata() {
+        ensure_test_rpc_secret();
+        let mut interceptor = test_interceptor();
+        let error = interceptor
+            .call(tonic::Request::new(()))
+            .expect_err("requests without an exact generated method must fail closed");
+
+        assert_eq!(error.code(), tonic::Code::Unauthenticated);
+        assert_eq!(error.message(), "Missing gRPC method metadata");
     }
 
     #[test]
     fn test_signature_interceptor_may_inject_request_id() {
         ensure_test_rpc_secret();
-        let mut interceptor = TonicSignatureInterceptor;
-        let req = tonic::Request::new(());
+        let mut interceptor = test_interceptor();
+        let req = test_request();
 
         let span = tracing::info_span!("grpc-rpc-test-span");
         let _guard = span.enter();
@@ -219,8 +334,8 @@ mod tests {
     #[test]
     fn test_signature_interceptor_injects_traceparent_metadata() {
         ensure_test_rpc_secret();
-        let mut interceptor = TonicSignatureInterceptor;
-        let req = tonic::Request::new(());
+        let mut interceptor = test_interceptor();
+        let req = test_request();
 
         with_trace_parent("4bf92f3577b34da6a3ce929d0e0e4736", || {
             let req = interceptor.call(req).expect("interceptor call should succeed");

crates/ecstore/src/cluster/rpc/http_auth.rs

@@ -19,8 +19,9 @@
 //! GHSA-r5qv-rc46-hv8q (internode RPC authentication must fail closed, fixed in
 //! rustfs/rustfs#4402) is anchored by the `ghsa_r5qv_*` tests in the module
 //! below, plus the broader negative-signature suite. The advisory class is: a
-//! node must never accept an RPC whose auth is missing, malformed, replayed, or
-//! signed with the default/empty shared secret. See
+//! node must never accept an RPC whose auth is missing, malformed, or signed
+//! with the default/empty shared secret. Body-bound v2 requests additionally
+//! receive process-local replay protection. See
 //! `docs/testing/security-regressions.md` for the full advisory -> test map.
 //!
 //! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-r5qv-rc46-hv8q>
@@ -29,23 +30,90 @@ use crate::cluster::rpc::context_propagation::{inject_request_id_into_http_heade
 use base64::Engine as _;
 use base64::engine::general_purpose;
 use hmac::{Hmac, KeyInit, Mac};
+use http::uri::Authority;
 use http::{HeaderMap, HeaderValue, Method, Uri};
 #[cfg(test)]
 use rustfs_credentials::{DEFAULT_SECRET_KEY, RPC_SECRET_REQUIRED_MESSAGE};
 use rustfs_credentials::{RPC_SECRET_REQUIRED_OPERATOR_MESSAGE, try_get_rpc_token};
+use sha2::Digest as _;
 use sha2::Sha256;
-use std::sync::Once;
+use std::collections::{HashSet, VecDeque};
+use std::sync::{LazyLock, Mutex, Once};
+use std::time::{Duration, Instant};
 use time::OffsetDateTime;
 use tracing::error;
+use uuid::Uuid;
 
 type HmacSha256 = Hmac<Sha256>;
 
 const SIGNATURE_HEADER: &str = "x-rustfs-signature";
 const TIMESTAMP_HEADER: &str = "x-rustfs-timestamp";
+const RPC_AUTH_VERSION_HEADER: &str = "x-rustfs-rpc-auth-version";
+const RPC_SIGNATURE_V2_HEADER: &str = "x-rustfs-rpc-signature-v2";
+const RPC_NONCE_HEADER: &str = "x-rustfs-rpc-nonce";
+pub(crate) const RPC_CONTENT_SHA256_HEADER: &str = "x-rustfs-content-sha256";
+const RPC_AUTH_VERSION_V2: &str = "2";
+const RPC_RESPONSE_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-response-proof-v1\0";
+const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
+const UNSIGNED_PAYLOAD_NONCE: &str = "unsigned";
 const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
+const REPLAY_CACHE_RETENTION: Duration = Duration::from_secs(601);
+const MAX_REPLAY_PROTECTED_NONCES: usize = 65_536;
 pub const TONIC_RPC_PREFIX: &str = "/node_service.NodeService";
 static RPC_SECRET_RESOLUTION_LOG_ONCE: Once = Once::new();
 
+#[derive(Default)]
+struct RpcNonceCache {
+    nonces: HashSet<Uuid>,
+    expirations: VecDeque<(Instant, i64, Uuid)>,
+    max_wall_time: i64,
+}
+
+impl RpcNonceCache {
+    fn remove_expired(&mut self, now: Instant, wall_time: i64) {
+        while matches!(
+            self.expirations.front(),
+            Some((expires_at, valid_until, _)) if *expires_at < now && *valid_until < wall_time
+        ) {
+            let Some((_, _, nonce)) = self.expirations.pop_front() else {
+                break;
+            };
+            self.nonces.remove(&nonce);
+        }
+    }
+
+    fn check_and_record(
+        &mut self,
+        nonce: Uuid,
+        signed_at: i64,
+        now: Instant,
+        wall_time: i64,
+        expires_at: Instant,
+        capacity: usize,
+    ) -> std::io::Result<()> {
+        self.max_wall_time = self.max_wall_time.max(wall_time);
+        if self.max_wall_time.saturating_sub(signed_at) > SIGNATURE_VALID_DURATION {
+            return Err(std::io::Error::other("RPC request timestamp expired after clock regression"));
+        }
+        self.remove_expired(now, self.max_wall_time);
+        if self.nonces.contains(&nonce) {
+            return Err(std::io::Error::other("RPC request replay detected"));
+        }
+        if self.nonces.len() >= capacity {
+            return Err(std::io::Error::other("RPC replay cache capacity exceeded"));
+        }
+        self.nonces.insert(nonce);
+        self.expirations
+            .push_back((expires_at, signed_at.saturating_add(SIGNATURE_VALID_DURATION), nonce));
+        Ok(())
+    }
+}
+
+// This cache is a process-local wire-replay defense only. Mutation handlers
+// still need a stable operation ID and coordinator-owned idempotency across
+// retries, node failover, and restart.
+static LOCAL_RPC_NONCE_CACHE: LazyLock<Mutex<RpcNonceCache>> = LazyLock::new(|| Mutex::new(RpcNonceCache::default()));
+
 /// Get the shared secret for HMAC signing
 #[cfg(test)]
 fn resolve_shared_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> std::io::Result<String> {
@@ -71,6 +139,30 @@ fn get_shared_secret() -> std::io::Result<String> {
     })
 }
 
+fn rpc_response_proof_mac(canonical_body: &[u8]) -> std::io::Result<HmacSha256> {
+    let secret = get_shared_secret()?;
+    let mut mac =
+        <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
+    mac.update(RPC_RESPONSE_PROOF_DOMAIN);
+    mac.update(
+        &u64::try_from(canonical_body.len())
+            .map_err(|_| std::io::Error::other("RPC response proof length cannot be represented"))?
+            .to_be_bytes(),
+    );
+    mac.update(canonical_body);
+    Ok(mac)
+}
+
+pub fn sign_tonic_rpc_response_proof(canonical_body: &[u8]) -> std::io::Result<Vec<u8>> {
+    Ok(rpc_response_proof_mac(canonical_body)?.finalize().into_bytes().to_vec())
+}
+
+pub fn verify_tonic_rpc_response_proof(canonical_body: &[u8], proof: &[u8]) -> std::io::Result<()> {
+    rpc_response_proof_mac(canonical_body)?
+        .verify_slice(proof)
+        .map_err(|_| std::io::Error::other("Invalid RPC response proof"))
+}
+
 /// Build the canonical payload covered by the RPC HMAC.
 fn signature_payload(url: &str, method: &Method, timestamp: i64) -> String {
     let uri: Uri = url.parse().expect("Invalid URL");
@@ -109,6 +201,96 @@ fn verify_signature(secret: &str, url: &str, method: &Method, timestamp: i64, si
     mac.verify_slice(&signature).is_ok()
 }
 
+#[derive(Clone, Copy)]
+struct SignatureV2Scope<'a> {
+    audience: &'a str,
+    service: &'a str,
+    rpc_method: &'a str,
+    timestamp: &'a str,
+    nonce: &'a str,
+    content_sha256: &'a str,
+}
+
+fn update_signature_v2(mac: &mut HmacSha256, scope: SignatureV2Scope<'_>) {
+    for part in [
+        b"rustfs-rpc-auth-v2|".as_slice(),
+        scope.audience.as_bytes(),
+        b"|/",
+        scope.service.as_bytes(),
+        b"/",
+        scope.rpc_method.as_bytes(),
+        b"|POST|",
+        scope.timestamp.as_bytes(),
+        b"|",
+        scope.nonce.as_bytes(),
+        b"|",
+        scope.content_sha256.as_bytes(),
+    ] {
+        mac.update(part);
+    }
+}
+
+fn generate_signature_v2(secret: &str, scope: SignatureV2Scope<'_>) -> std::io::Result<String> {
+    let mut mac =
+        <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
+    update_signature_v2(&mut mac, scope);
+    Ok(general_purpose::STANDARD.encode(mac.finalize().into_bytes()))
+}
+
+fn verify_signature_v2(secret: &str, scope: SignatureV2Scope<'_>, signature: &str) -> bool {
+    let Ok(signature) = general_purpose::STANDARD.decode(signature) else {
+        return false;
+    };
+    let Ok(mut mac) = <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()) else {
+        return false;
+    };
+    update_signature_v2(&mut mac, scope);
+    mac.verify_slice(&signature).is_ok()
+}
+
+fn valid_content_sha256(value: &str) -> bool {
+    value == UNSIGNED_PAYLOAD
+        || (value.len() == 64
+            && value
+                .bytes()
+                .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)))
+}
+
+fn header_value(value: &str, name: &str) -> std::io::Result<HeaderValue> {
+    HeaderValue::from_str(value).map_err(|_| std::io::Error::other(format!("Invalid {name} header value")))
+}
+
+pub fn normalize_tonic_rpc_audience(value: &str) -> std::io::Result<String> {
+    let authority = value
+        .parse::<Authority>()
+        .map_err(|_| std::io::Error::other("Invalid gRPC peer authority"))?;
+    Ok(authority.as_str().to_ascii_lowercase())
+}
+
+fn check_timestamp(timestamp: i64) -> std::io::Result<()> {
+    let current_time = OffsetDateTime::now_utc().unix_timestamp();
+    if current_time.saturating_sub(timestamp) > SIGNATURE_VALID_DURATION
+        || timestamp.saturating_sub(current_time) > SIGNATURE_VALID_DURATION
+    {
+        return Err(std::io::Error::other("Request timestamp expired"));
+    }
+    Ok(())
+}
+
+fn check_and_record_nonce(nonce: Uuid, signed_at: i64) -> std::io::Result<()> {
+    let wall_time = OffsetDateTime::now_utc().unix_timestamp();
+    let mut cache = LOCAL_RPC_NONCE_CACHE
+        .lock()
+        .map_err(|_| std::io::Error::other("RPC replay cache unavailable"))?;
+    // Take the monotonic timestamp after acquiring the lock so expiration
+    // entries remain ordered by the same serialization point as insertion.
+    let now = Instant::now();
+    let expires_at = now
+        .checked_add(REPLAY_CACHE_RETENTION)
+        .ok_or_else(|| std::io::Error::other("RPC replay expiry overflow"))?;
+    cache.check_and_record(nonce, signed_at, now, wall_time, expires_at, MAX_REPLAY_PROTECTED_NONCES)
+}
+
 /// Build headers with authentication signature
 pub fn build_auth_headers(url: &str, method: &Method, headers: &mut HeaderMap) -> std::io::Result<()> {
     let auth_headers = gen_signature_headers(url, method)?;
@@ -135,6 +317,184 @@ pub fn gen_signature_headers(url: &str, method: &Method) -> std::io::Result<Head
     Ok(headers)
 }
 
+/// Generate rolling-upgrade-safe gRPC auth metadata.
+///
+/// The legacy signature remains present for old servers. New servers prefer the
+/// v2 signature and bind it to the destination authority and exact generated
+/// gRPC method. A versioned canonical mutation payload can opt into the
+/// additional body-digest capability.
+pub fn gen_tonic_signature_headers(
+    audience: &str,
+    service: &str,
+    rpc_method: &str,
+    content_sha256: Option<&str>,
+) -> std::io::Result<HeaderMap> {
+    if audience.is_empty() || service.is_empty() || rpc_method.is_empty() || service.contains('/') || rpc_method.contains('/') {
+        return Err(std::io::Error::other("Invalid RPC v2 signing scope"));
+    }
+    let content_sha256 = content_sha256.unwrap_or(UNSIGNED_PAYLOAD);
+    if !valid_content_sha256(content_sha256) {
+        return Err(std::io::Error::other("Invalid RPC content SHA-256"));
+    }
+
+    let secret = get_shared_secret()?;
+    let timestamp = OffsetDateTime::now_utc().unix_timestamp();
+    let timestamp_header = timestamp.to_string();
+    let body_nonce = (content_sha256 != UNSIGNED_PAYLOAD).then(|| Uuid::new_v4().to_string());
+    let nonce = body_nonce.as_deref().unwrap_or(UNSIGNED_PAYLOAD_NONCE);
+    let legacy_signature = generate_signature(&secret, TONIC_RPC_PREFIX, &Method::GET, timestamp);
+    let signature_v2 = generate_signature_v2(
+        &secret,
+        SignatureV2Scope {
+            audience,
+            service,
+            rpc_method,
+            timestamp: &timestamp_header,
+            nonce,
+            content_sha256,
+        },
+    )?;
+
+    let mut headers = HeaderMap::new();
+    headers.insert(SIGNATURE_HEADER, header_value(&legacy_signature, SIGNATURE_HEADER)?);
+    headers.insert(TIMESTAMP_HEADER, header_value(&timestamp_header, TIMESTAMP_HEADER)?);
+    headers.insert(RPC_AUTH_VERSION_HEADER, HeaderValue::from_static(RPC_AUTH_VERSION_V2));
+    headers.insert(RPC_SIGNATURE_V2_HEADER, header_value(&signature_v2, RPC_SIGNATURE_V2_HEADER)?);
+    headers.insert(RPC_NONCE_HEADER, header_value(nonce, RPC_NONCE_HEADER)?);
+    headers.insert(RPC_CONTENT_SHA256_HEADER, header_value(content_sha256, RPC_CONTENT_SHA256_HEADER)?);
+    Ok(headers)
+}
+
+/// Bind a mutation to a versioned, deterministic canonical payload.
+///
+/// Do not pass a protobuf re-encoding here: unknown fields and map ordering are
+/// not a stable mixed-version contract.
+pub fn set_tonic_canonical_body_digest<T>(request: &mut tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
+    let digest = hex_simd::encode_to_string(Sha256::digest(canonical_body), hex_simd::AsciiCase::Lower);
+    request
+        .metadata_mut()
+        .as_mut()
+        .insert(RPC_CONTENT_SHA256_HEADER, header_value(&digest, RPC_CONTENT_SHA256_HEADER)?);
+    Ok(())
+}
+
+pub fn verify_tonic_canonical_body_digest<T>(request: &tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
+    let version = request
+        .metadata()
+        .get(RPC_AUTH_VERSION_HEADER)
+        .and_then(|value| value.to_str().ok());
+    if version != Some(RPC_AUTH_VERSION_V2) {
+        return Err(std::io::Error::other("RPC mutation requires v2 authentication"));
+    }
+    let expected = request
+        .metadata()
+        .get(RPC_CONTENT_SHA256_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC content SHA-256"))?;
+    if expected == UNSIGNED_PAYLOAD || !valid_content_sha256(expected) {
+        return Err(std::io::Error::other("RPC body is not bound to the signature"));
+    }
+    let actual = hex_simd::encode_to_string(Sha256::digest(canonical_body), hex_simd::AsciiCase::Lower);
+    if actual != expected {
+        return Err(std::io::Error::other("RPC content SHA-256 mismatch"));
+    }
+    Ok(())
+}
+
+fn has_v2_auth_headers(headers: &HeaderMap) -> bool {
+    [
+        RPC_AUTH_VERSION_HEADER,
+        RPC_SIGNATURE_V2_HEADER,
+        RPC_NONCE_HEADER,
+        RPC_CONTENT_SHA256_HEADER,
+    ]
+    .iter()
+    .any(|name| headers.contains_key(*name))
+}
+
+/// Verify gRPC authentication, preferring v2 without downgrade on malformed v2 metadata.
+pub fn verify_tonic_rpc_signature(audience: &str, path: &str, headers: &HeaderMap) -> std::io::Result<()> {
+    if !has_v2_auth_headers(headers) {
+        // RUSTFS_COMPAT_TODO(heal-rpc-auth-v2): accept old peers during rolling upgrades. Remove after the minimum
+        // supported RustFS peer version sends v2 authentication on every internode gRPC request.
+        return verify_rpc_signature(TONIC_RPC_PREFIX, &Method::GET, headers);
+    }
+
+    let path = path
+        .strip_prefix('/')
+        .ok_or_else(|| std::io::Error::other("Invalid RPC request path"))?;
+    let (service, rpc_method) = path
+        .split_once('/')
+        .filter(|(service, rpc_method)| !service.is_empty() && !rpc_method.is_empty() && !rpc_method.contains('/'))
+        .ok_or_else(|| std::io::Error::other("Invalid RPC request path"))?;
+    if audience.is_empty() {
+        return Err(std::io::Error::other("Missing RPC audience"));
+    }
+
+    let version = headers
+        .get(RPC_AUTH_VERSION_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC auth version"))?;
+    if version != RPC_AUTH_VERSION_V2 {
+        return Err(std::io::Error::other("Unsupported RPC auth version"));
+    }
+    let signature = headers
+        .get(RPC_SIGNATURE_V2_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC v2 signature"))?;
+    let timestamp_header = headers
+        .get(TIMESTAMP_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing timestamp header"))?;
+    let timestamp = timestamp_header
+        .parse::<i64>()
+        .map_err(|_| std::io::Error::other("Invalid timestamp format"))?;
+    check_timestamp(timestamp)?;
+    let nonce = headers
+        .get(RPC_NONCE_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC nonce"))?;
+    let content_sha256 = headers
+        .get(RPC_CONTENT_SHA256_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC content SHA-256"))?;
+    if !valid_content_sha256(content_sha256) {
+        return Err(std::io::Error::other("Invalid RPC content SHA-256"));
+    }
+    let parsed_nonce = if content_sha256 == UNSIGNED_PAYLOAD {
+        if nonce != UNSIGNED_PAYLOAD_NONCE {
+            return Err(std::io::Error::other("Invalid unsigned RPC nonce"));
+        }
+        None
+    } else {
+        let parsed_nonce = Uuid::parse_str(nonce).map_err(|_| std::io::Error::other("Invalid RPC nonce"))?;
+        if parsed_nonce.is_nil() {
+            return Err(std::io::Error::other("Invalid RPC nonce"));
+        }
+        Some(parsed_nonce)
+    };
+
+    let secret = get_shared_secret()?;
+    if !verify_signature_v2(
+        &secret,
+        SignatureV2Scope {
+            audience,
+            service,
+            rpc_method,
+            timestamp: timestamp_header,
+            nonce,
+            content_sha256,
+        },
+        signature,
+    ) {
+        return Err(std::io::Error::other("Invalid RPC v2 signature"));
+    }
+    if let Some(nonce) = parsed_nonce {
+        check_and_record_nonce(nonce, timestamp)?;
+    }
+    Ok(())
+}
+
 /// Verify the request signature for RPC requests
 pub fn verify_rpc_signature(url: &str, method: &Method, headers: &HeaderMap) -> std::io::Result<()> {
     // Get signature from header
@@ -153,14 +513,7 @@ pub fn verify_rpc_signature(url: &str, method: &Method, headers: &HeaderMap) ->
         .parse()
         .map_err(|_| std::io::Error::other("Invalid timestamp format"))?;
 
-    // Check timestamp validity (prevent replay attacks)
-    let current_time = OffsetDateTime::now_utc().unix_timestamp();
-
-    if current_time.saturating_sub(timestamp) > SIGNATURE_VALID_DURATION
-        || timestamp.saturating_sub(current_time) > SIGNATURE_VALID_DURATION
-    {
-        return Err(std::io::Error::other("Request timestamp expired"));
-    }
+    check_timestamp(timestamp)?;
 
     // Verify signature with constant-time HMAC comparison.
     let secret = get_shared_secret()?;
@@ -707,4 +1060,203 @@ mod tests {
             assert!(result.is_ok(), "Round-trip test failed for {method} {url}");
         }
     }
+
+    #[test]
+    fn tonic_v2_signature_is_bound_to_exact_method() {
+        ensure_test_rpc_secret();
+        let headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+
+        assert!(verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers).is_ok());
+        let error = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/SignalService", &headers)
+            .expect_err("signature replayed to a different method must fail");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn tonic_v2_signature_is_bound_to_exact_service() {
+        ensure_test_rpc_secret();
+        let headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+
+        let error = verify_tonic_rpc_signature("node-a:9000", "/other.NodeService/Ping", &headers)
+            .expect_err("signature replayed to a different service must fail");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn tonic_v2_signature_is_bound_to_destination_audience() {
+        ensure_test_rpc_secret();
+        let headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+
+        let error = verify_tonic_rpc_signature("node-b:9000", "/node_service.NodeService/Ping", &headers)
+            .expect_err("signature replayed to a different node must fail");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn malformed_v2_auth_does_not_downgrade_to_valid_legacy_signature() {
+        ensure_test_rpc_secret();
+        let mut headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+        headers.insert(RPC_SIGNATURE_V2_HEADER, HeaderValue::from_static("invalid"));
+
+        assert!(
+            verify_rpc_signature(TONIC_RPC_PREFIX, &Method::GET, &headers).is_ok(),
+            "the compatibility signature should remain valid for old servers"
+        );
+        let error = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers)
+            .expect_err("new servers must not downgrade malformed v2 auth");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn legacy_tonic_signature_remains_accepted_during_rolling_upgrade() {
+        ensure_test_rpc_secret();
+        let headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
+
+        assert!(verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers).is_ok());
+    }
+
+    #[test]
+    fn body_bound_tonic_request_rejects_replay_and_body_tampering() {
+        ensure_test_rpc_secret();
+        let body = b"heal-control-request";
+        let mut request = tonic::Request::new(());
+        set_tonic_canonical_body_digest(&mut request, body).expect("canonical body digest should be attached");
+        let content_sha256 = request
+            .metadata()
+            .get(RPC_CONTENT_SHA256_HEADER)
+            .and_then(|value| value.to_str().ok());
+        let headers =
+            gen_tonic_signature_headers("node-a:9000", "node_service.HealControlService", "HealControl", content_sha256)
+                .expect("body-bound auth headers should build");
+        request.metadata_mut().as_mut().extend(headers.clone());
+
+        assert!(verify_tonic_rpc_signature("node-a:9000", "/node_service.HealControlService/HealControl", &headers).is_ok());
+        let replay = verify_tonic_rpc_signature("node-a:9000", "/node_service.HealControlService/HealControl", &headers)
+            .expect_err("reusing a body-bound nonce must fail");
+        assert_eq!(replay.to_string(), "RPC request replay detected");
+        assert!(verify_tonic_canonical_body_digest(&request, body).is_ok());
+        let tampered = verify_tonic_canonical_body_digest(&request, b"different-body")
+            .expect_err("a different canonical request body must fail");
+        assert_eq!(tampered.to_string(), "RPC content SHA-256 mismatch");
+    }
+
+    #[test]
+    fn tier_mutation_rpc_contract_requires_method_bound_v2_body_digest() {
+        ensure_test_rpc_secret();
+        let mutation_id = uuid::uuid!("12345678-1234-5678-9abc-def012345678");
+        let body = rustfs_protos::canonical_tier_mutation_rpc_body(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            rustfs_protos::TierMutationRpcPhase::Prepare,
+            mutation_id,
+            b"canonical-tier-mutation-prepare",
+        )
+        .expect("small tier mutation body should encode");
+        let mut request = tonic::Request::new(());
+        set_tonic_canonical_body_digest(&mut request, &body).expect("canonical body digest should be attached");
+        let content_sha256 = request
+            .metadata()
+            .get(RPC_CONTENT_SHA256_HEADER)
+            .and_then(|value| value.to_str().ok());
+        let headers = gen_tonic_signature_headers(
+            "node-a:9000",
+            "node_service.TierMutationControlService",
+            "PrepareTierMutation",
+            content_sha256,
+        )
+        .expect("body-bound tier mutation auth headers should build");
+        request.metadata_mut().as_mut().extend(headers.clone());
+
+        assert!(
+            verify_tonic_rpc_signature("node-a:9000", "/node_service.TierMutationControlService/PrepareTierMutation", &headers)
+                .is_ok(),
+            "tier mutation RPC signature must bind destination, service, method, nonce, and body digest"
+        );
+        let method_replay =
+            verify_tonic_rpc_signature("node-a:9000", "/node_service.TierMutationControlService/CommitTierMutation", &headers)
+                .expect_err("prepare auth must not replay to commit");
+        assert_eq!(method_replay.to_string(), "Invalid RPC v2 signature");
+        let service_replay = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/PrepareTierMutation", &headers)
+            .expect_err("tier mutation auth must not replay to the legacy node service path");
+        assert_eq!(service_replay.to_string(), "Invalid RPC v2 signature");
+        let tampered_body = rustfs_protos::canonical_tier_mutation_rpc_body(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            rustfs_protos::TierMutationRpcPhase::Commit,
+            mutation_id,
+            b"canonical-tier-mutation-prepare",
+        )
+        .expect("small tier mutation body should encode");
+        let tampered =
+            verify_tonic_canonical_body_digest(&request, &tampered_body).expect_err("commit body must not match prepare digest");
+        assert_eq!(tampered.to_string(), "RPC content SHA-256 mismatch");
+    }
+
+    #[test]
+    fn partial_v2_metadata_fails_closed() {
+        ensure_test_rpc_secret();
+        let mut headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
+        headers.insert(RPC_AUTH_VERSION_HEADER, HeaderValue::from_static(RPC_AUTH_VERSION_V2));
+
+        let error = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers)
+            .expect_err("partial v2 metadata must not fall back to legacy auth");
+        assert_eq!(error.to_string(), "Missing RPC v2 signature");
+    }
+
+    #[test]
+    fn canonical_mutation_digest_rejects_legacy_only_auth() {
+        let mut request = tonic::Request::new(());
+        set_tonic_canonical_body_digest(&mut request, b"heal-control-v1\0start").expect("canonical digest should be attached");
+
+        let error = verify_tonic_canonical_body_digest(&request, b"heal-control-v1\0start")
+            .expect_err("mutation body verification must also require v2 auth");
+        assert_eq!(error.to_string(), "RPC mutation requires v2 authentication");
+    }
+
+    #[test]
+    fn nonce_cache_expires_by_monotonic_deadline_and_fails_closed_at_capacity() {
+        let now = Instant::now();
+        let expiry = now.checked_add(REPLAY_CACHE_RETENTION).expect("test expiry should fit");
+        let after_expiry = expiry.checked_add(Duration::from_secs(1)).expect("test expiry should fit");
+        let nonce_a = Uuid::new_v4();
+        let nonce_b = Uuid::new_v4();
+        let mut cache = RpcNonceCache::default();
+
+        cache
+            .check_and_record(nonce_a, 100, now, 100, expiry, 1)
+            .expect("first nonce should be recorded");
+        let capacity = cache
+            .check_and_record(nonce_b, 100, now, 100, expiry, 1)
+            .expect_err("a full replay cache must fail closed");
+        assert_eq!(capacity.to_string(), "RPC replay cache capacity exceeded");
+        cache
+            .check_and_record(nonce_b, 702, after_expiry, 702, after_expiry, 1)
+            .expect("expired nonce should release capacity");
+        assert!(!cache.nonces.contains(&nonce_a));
+        assert!(cache.nonces.contains(&nonce_b));
+    }
+
+    #[test]
+    fn nonce_cache_rejects_replay_after_wall_clock_regression() {
+        let now = Instant::now();
+        let expiry = now.checked_add(REPLAY_CACHE_RETENTION).expect("test expiry should fit");
+        let after_expiry = expiry.checked_add(Duration::from_secs(1)).expect("test expiry should fit");
+        let nonce = Uuid::new_v4();
+        let mut cache = RpcNonceCache::default();
+
+        cache
+            .check_and_record(nonce, 1_000, now, 1_000, expiry, 2)
+            .expect("first nonce should be recorded");
+        let replay = cache
+            .check_and_record(nonce, 1_000, after_expiry, 900, after_expiry, 2)
+            .expect_err("wall clock regression must not make an old signature reusable");
+        assert_eq!(replay.to_string(), "RPC request replay detected");
+
+        let stale = cache
+            .check_and_record(Uuid::new_v4(), 600, after_expiry, 900, after_expiry, 2)
+            .expect_err("the monotonic wall-clock high-water mark must fail closed");
+        assert_eq!(stale.to_string(), "RPC request timestamp expired after clock regression");
+    }
 }

crates/ecstore/src/cluster/rpc/mod.rs

@@ -28,12 +28,17 @@ pub(crate) use background_monitor::spawn_background_monitor;
 pub use client::{
     TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
 };
-pub use http_auth::{TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, verify_rpc_signature};
+pub use http_auth::{
+    TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience,
+    set_tonic_canonical_body_digest, sign_tonic_rpc_response_proof, verify_rpc_signature, verify_tonic_canonical_body_digest,
+    verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
+};
 #[cfg(test)]
 pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
 pub use internode_data_transport::build_internode_data_transport_from_env;
 pub use peer_rest_client::{
-    PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
+    PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG,
+    SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerPeerActivity,
 };
 pub(crate) use peer_s3_client::heal_bucket_local_on_disks;
 pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, S3PeerSys};

crates/ecstore/src/cluster/rpc/peer_rest_client.rs

@@ -12,7 +12,11 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::cluster::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
+use crate::cluster::rpc::client::{
+    TonicInterceptor, gen_tonic_signature_interceptor, heal_control_time_out_client, node_service_time_out_client,
+    tier_mutation_control_time_out_client,
+};
+use crate::cluster::rpc::{set_tonic_canonical_body_digest, verify_tonic_rpc_response_proof};
 use crate::error::{Error, Result};
 use crate::{
     disk::disk_store::{get_drive_active_check_interval, get_drive_active_check_timeout},
@@ -20,6 +24,7 @@ use crate::{
     runtime::sources as runtime_sources,
     services::metrics_realtime::{CollectMetricsOpts, MetricType},
 };
+use bytes::Bytes;
 use rmp_serde::{Deserializer, Serializer};
 use rustfs_madmin::{
     ServerProperties,
@@ -27,17 +32,19 @@ use rustfs_madmin::{
     metrics::RealtimeMetrics,
     net::NetInfo,
 };
-use rustfs_protos::evict_failed_connection;
 use rustfs_protos::proto_gen::node_service::{
-    CancelDecommissionRequest, ClearDecommissionRequest, DeleteBucketMetadataRequest, DeletePolicyRequest,
-    DeleteServiceAccountRequest, DeleteUserRequest, GetCpusRequest, GetLiveEventsRequest, GetMemInfoRequest, GetMetricsRequest,
-    GetNetInfoRequest, GetOsInfoRequest, GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest, GetSysConfigRequest,
-    GetSysErrorsRequest, LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, LoadPolicyRequest,
-    LoadRebalanceMetaRequest, LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest,
-    LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, ServerInfoRequest,
-    SignalServiceRequest, StartDecommissionRequest, StartProfilingRequest, StopRebalanceRequest,
-    node_service_client::NodeServiceClient,
+    BackgroundHealStatusRequest, CancelDecommissionRequest, ClearDecommissionRequest, DeleteBucketMetadataRequest,
+    DeletePolicyRequest, DeleteServiceAccountRequest, DeleteUserRequest, GetCpusRequest, GetLiveEventsRequest, GetMemInfoRequest,
+    GetMetricsRequest, GetNetInfoRequest, GetOsInfoRequest, GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest,
+    GetSysConfigRequest, GetSysErrorsRequest, HealControlRequest, LoadBucketMetadataRequest, LoadGroupRequest,
+    LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, LoadServiceAccountRequest,
+    LoadTransitionTierConfigRequest, LoadUserRequest, LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest,
+    ReloadSiteReplicationConfigRequest, ScannerActivityRequest, ScannerActivityResponse, ServerInfoRequest, SignalServiceRequest,
+    StartDecommissionRequest, StartProfilingRequest, StopRebalanceRequest, TierMutationAbortRequest, TierMutationCommitRequest,
+    TierMutationControlResponse, TierMutationPeerState, TierMutationPrepareRequest, node_service_client::NodeServiceClient,
+    tier_mutation_control_service_client::TierMutationControlServiceClient,
 };
+use rustfs_protos::{TierMutationRpcPhase, evict_failed_connection};
 use rustfs_utils::XHost;
 use serde::{Deserialize, Serialize as _};
 use std::{
@@ -54,14 +61,53 @@ use tonic::Request;
 use tonic::service::interceptor::InterceptedService;
 use tonic::transport::Channel;
 use tracing::{debug, info, warn};
+use uuid::Uuid;
 
 pub const PEER_RESTSIGNAL: &str = "signal";
 pub const PEER_RESTSUB_SYS: &str = "sub-sys";
 pub const PEER_RESTDRY_RUN: &str = "dry-run";
 pub const SERVICE_SIGNAL_REFRESH_CONFIG: u64 = 1;
 pub const SERVICE_SIGNAL_RELOAD_DYNAMIC: u64 = 2;
+const BACKGROUND_HEAL_STATUS_MAX_MESSAGE_SIZE: usize = 64 * 1024;
+const HEAL_CONTROL_FINGERPRINT_MAX_SIZE: usize = 256;
+const HEAL_CONTROL_PAYLOAD_MAX_SIZE: usize = 64 * 1024;
 const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
 const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
+const SCANNER_ACTIVITY_MAX_MESSAGE_SIZE: usize = 1024;
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ScannerPeerActivity {
+    pub instance_id: String,
+    pub namespace_generation: u64,
+    pub maintenance_generation: u64,
+}
+
+fn decode_scanner_activity(response: ScannerActivityResponse) -> Result<ScannerPeerActivity> {
+    let instance_id = response.instance_id;
+    if instance_id.len() != 32
+        || !instance_id
+            .as_bytes()
+            .iter()
+            .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(byte))
+    {
+        return Err(Error::other("peer returned an invalid scanner activity instance ID"));
+    }
+    Ok(ScannerPeerActivity {
+        instance_id,
+        namespace_generation: response.namespace_generation,
+        maintenance_generation: response.maintenance_generation,
+    })
+}
+
+fn validate_heal_control_capability_proof(canonical_ack: &[u8], proof: &[u8]) -> Result<()> {
+    verify_tonic_rpc_response_proof(canonical_ack, proof)
+        .map_err(|_| Error::other("peer returned an invalid heal control capability proof"))
+}
+
+fn validate_heal_control_response_proof(canonical_response: &[u8], proof: &[u8]) -> Result<()> {
+    verify_tonic_rpc_response_proof(canonical_response, proof)
+        .map_err(|_| Error::other("peer returned an invalid heal control response proof"))
+}
 
 #[derive(Clone, Debug)]
 pub struct PeerLiveEventsBatch {
@@ -78,6 +124,82 @@ pub struct PeerRestClient {
     recovery_running: Arc<AtomicBool>,
 }
 
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum PeerTierMutationState {
+    Prepared,
+    Committed,
+    Aborted,
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct PeerTierMutationOutcome {
+    pub state: PeerTierMutationState,
+    pub applied: bool,
+}
+
+fn validate_tier_mutation_response_proof(
+    version: u32,
+    phase: TierMutationRpcPhase,
+    mutation_id: Uuid,
+    canonical_payload: &[u8],
+    response: &TierMutationControlResponse,
+) -> Result<()> {
+    let canonical_response =
+        rustfs_protos::canonical_tier_mutation_rpc_response_body(rustfs_protos::TierMutationRpcResponseProofInput {
+            version,
+            phase,
+            mutation_id,
+            canonical_payload,
+            success: response.success,
+            state: response.state,
+            applied: response.applied,
+            error_info: response.error_info.as_deref(),
+        })
+        .map_err(|_| Error::other("tier mutation response length cannot be represented"))?;
+    verify_tonic_rpc_response_proof(&canonical_response, &response.response_proof)
+        .map_err(|_| Error::other("peer returned an invalid tier mutation response proof"))
+}
+
+fn decode_tier_mutation_peer_state(state: i32) -> Result<PeerTierMutationState> {
+    match TierMutationPeerState::try_from(state).map_err(|_| Error::other("peer returned an invalid tier mutation state"))? {
+        TierMutationPeerState::Prepared => Ok(PeerTierMutationState::Prepared),
+        TierMutationPeerState::Committed => Ok(PeerTierMutationState::Committed),
+        TierMutationPeerState::Aborted => Ok(PeerTierMutationState::Aborted),
+        TierMutationPeerState::Unspecified => Err(Error::other("peer returned an unspecified tier mutation state")),
+    }
+}
+
+fn validate_tier_mutation_payload_len(phase: TierMutationRpcPhase, payload_len: usize) -> Result<()> {
+    let limit = match phase {
+        TierMutationRpcPhase::Prepare => rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE,
+        TierMutationRpcPhase::Commit => rustfs_protos::TIER_MUTATION_RPC_MAX_COMMIT_PAYLOAD_SIZE,
+        TierMutationRpcPhase::Abort => {
+            if payload_len == 0 {
+                return Ok(());
+            }
+            return Err(Error::other("tier mutation abort payload must be empty"));
+        }
+        _ => return Err(Error::other("tier mutation rpc phase is unsupported")),
+    };
+    if payload_len > limit {
+        return Err(Error::other("tier mutation payload exceeds size limit"));
+    }
+    Ok(())
+}
+
+fn tier_mutation_phase_label(phase: TierMutationRpcPhase) -> &'static str {
+    match phase {
+        TierMutationRpcPhase::Prepare => "prepare",
+        TierMutationRpcPhase::Commit => "commit",
+        TierMutationRpcPhase::Abort => "abort",
+        _ => "unknown",
+    }
+}
+
+fn tier_mutation_control_status_error(phase: TierMutationRpcPhase, status: tonic::Status) -> Error {
+    Error::other(format!("peer tier mutation {} RPC failed: {status}", tier_mutation_phase_label(phase)))
+}
+
 impl PeerRestClient {
     fn recovery_monitor_span(grid_host: &str) -> tracing::Span {
         tracing::info_span!(
@@ -97,31 +219,62 @@ impl PeerRestClient {
             recovery_running: Arc::new(AtomicBool::new(false)),
         }
     }
-    pub async fn new_clients(eps: EndpointServerPools) -> (Vec<Option<Self>>, Vec<Option<Self>>) {
-        if !runtime_sources::setup_is_dist_erasure().await {
-            return (Vec::new(), Vec::new());
-        }
 
-        let eps = eps.clone();
-        let hosts = eps.hosts_sorted();
-        let mut remote = Vec::with_capacity(hosts.len());
-        let mut all = vec![None; hosts.len()];
-        for (i, hs_host) in hosts.iter().enumerate() {
-            if let Some(host) = hs_host
-                && let Some(grid_host) = eps.find_grid_hosts_from_peer(host)
-            {
-                let client = PeerRestClient::new(host.clone(), grid_host);
+    fn build_clients_from_slots(
+        slots: Vec<(String, Option<String>, bool)>,
+    ) -> (Vec<Option<Self>>, Vec<Option<Self>>, Vec<String>) {
+        let mut remote = Vec::with_capacity(slots.len().saturating_sub(1));
+        let mut all = vec![None; slots.len()];
+        let mut remote_topology_hosts = Vec::with_capacity(slots.len().saturating_sub(1));
 
-                all[i] = Some(client.clone());
-                remote.push(Some(client));
+        for (idx, (peer_host_port, grid_host, is_local)) in slots.into_iter().enumerate() {
+            if is_local {
+                continue;
             }
+
+            let client = match grid_host {
+                Some(grid_host) => match XHost::try_from(peer_host_port.clone()) {
+                    Ok(host) => Some(PeerRestClient::new(host, grid_host)),
+                    Err(err) => {
+                        warn!(peer = %peer_host_port, "Xhost parse failed while constructing peer client: {err:?}");
+                        None
+                    }
+                },
+                None => {
+                    warn!(peer = %peer_host_port, "grid host is missing while constructing peer client");
+                    None
+                }
+            };
+
+            all[idx] = client.clone();
+            remote.push(client);
+            remote_topology_hosts.push(peer_host_port);
         }
 
+        (remote, all, remote_topology_hosts)
+    }
+
+    pub async fn new_clients(eps: EndpointServerPools) -> (Vec<Option<Self>>, Vec<Option<Self>>) {
+        let (remote, all, _) = Self::new_clients_with_topology(eps).await;
+        (remote, all)
+    }
+
+    pub async fn new_clients_with_topology(eps: EndpointServerPools) -> (Vec<Option<Self>>, Vec<Option<Self>>, Vec<String>) {
+        if !runtime_sources::setup_is_dist_erasure().await {
+            return (Vec::new(), Vec::new(), Vec::new());
+        }
+
+        let (remote, all, remote_topology_hosts) = Self::build_clients_from_slots(eps.peer_grid_host_slots_sorted());
+
         if all.len() != remote.len() + 1 {
-            warn!("Expected number of all hosts ({}) to be remote +1 ({})", all.len(), remote.len());
+            warn!(
+                all_hosts = all.len(),
+                remote_slots = remote.len(),
+                "Expected number of all hosts to be remote slots + local node"
+            );
         }
 
-        (remote, all)
+        (remote, all, remote_topology_hosts)
     }
 
     pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
@@ -141,6 +294,48 @@ impl PeerRestClient {
             })
     }
 
+    async fn get_heal_control_client(
+        &self,
+    ) -> Result<
+        rustfs_protos::proto_gen::node_service::heal_control_service_client::HealControlServiceClient<
+            InterceptedService<Channel, TonicInterceptor>,
+        >,
+    > {
+        if self.offline.load(Ordering::Acquire) {
+            self.mark_offline_and_spawn_recovery();
+            return Err(Error::other(format!("peer {} is temporarily offline", self.grid_host)));
+        }
+
+        heal_control_time_out_client(&self.grid_host, TonicInterceptor::Signature(gen_tonic_signature_interceptor()))
+            .await
+            .map_err(|err| {
+                let storage_err = Error::other(format!("can not get heal control client, err: {err}"));
+                if Self::is_network_like_error(&storage_err) {
+                    self.mark_offline_and_spawn_recovery();
+                }
+                storage_err
+            })
+    }
+
+    async fn get_tier_mutation_control_client(
+        &self,
+    ) -> Result<TierMutationControlServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
+        if self.offline.load(Ordering::Acquire) {
+            self.mark_offline_and_spawn_recovery();
+            return Err(Error::other(format!("peer {} is temporarily offline", self.grid_host)));
+        }
+
+        tier_mutation_control_time_out_client(&self.grid_host, TonicInterceptor::Signature(gen_tonic_signature_interceptor()))
+            .await
+            .map_err(|err| {
+                let storage_err = Error::other(format!("can not get tier mutation control client, err: {err}"));
+                if Self::is_network_like_error(&storage_err) {
+                    self.mark_offline_and_spawn_recovery();
+                }
+                storage_err
+            })
+    }
+
     /// Evict the connection to this peer from the global cache.
     /// This should be called when communication with this peer fails.
     pub async fn evict_connection(&self) {
@@ -639,12 +834,208 @@ impl PeerRestClient {
         Err(Error::NotImplemented)
     }
 
-    pub async fn load_bucket_metadata(&self, bucket: &str) -> Result<()> {
+    pub async fn background_heal_status(&self) -> Result<Option<Vec<u8>>> {
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_client()
+                    .await?
+                    .max_decoding_message_size(BACKGROUND_HEAL_STATUS_MAX_MESSAGE_SIZE);
+                let response = match client
+                    .background_heal_status(Request::new(BackgroundHealStatusRequest::default()))
+                    .await
+                {
+                    Ok(response) => response.into_inner(),
+                    Err(status) if status.code() == tonic::Code::Unimplemented => {
+                        // RUSTFS_COMPAT_TODO(heal-status-rpc-v1): accept old peers without node heal snapshots. Remove after the minimum supported RustFS peer version implements BackgroundHealStatus.
+                        return Ok(None);
+                    }
+                    Err(status) => return Err(status.into()),
+                };
+                if !response.success {
+                    return Err(Error::other(
+                        response
+                            .error_info
+                            .unwrap_or_else(|| "peer background heal status failed without an error".to_string()),
+                    ));
+                }
+                Ok(Some(response.bg_heal_state.to_vec()))
+            }
+            .await,
+        )
+        .await
+    }
+
+    pub async fn prepare_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Result<PeerTierMutationOutcome> {
+        self.tier_mutation_control(TierMutationRpcPhase::Prepare, mutation_id, canonical_payload)
+            .await
+    }
+
+    pub async fn commit_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Result<PeerTierMutationOutcome> {
+        self.tier_mutation_control(TierMutationRpcPhase::Commit, mutation_id, canonical_payload)
+            .await
+    }
+
+    pub async fn abort_tier_mutation(&self, mutation_id: Uuid) -> Result<PeerTierMutationOutcome> {
+        self.tier_mutation_control(TierMutationRpcPhase::Abort, mutation_id, Bytes::new())
+            .await
+    }
+
+    async fn tier_mutation_control(
+        &self,
+        phase: TierMutationRpcPhase,
+        mutation_id: Uuid,
+        canonical_payload: Bytes,
+    ) -> Result<PeerTierMutationOutcome> {
+        validate_tier_mutation_payload_len(phase, canonical_payload.len())?;
+        let version = rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION;
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_tier_mutation_control_client()
+                    .await?
+                    .max_encoding_message_size(rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE)
+                    .max_decoding_message_size(rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE);
+                let canonical_body =
+                    rustfs_protos::canonical_tier_mutation_rpc_body(version, phase, mutation_id, canonical_payload.as_ref())
+                        .map_err(|_| Error::other("tier mutation request length cannot be represented"))?;
+                let mutation_id_text = mutation_id.to_string();
+                let response = match phase {
+                    TierMutationRpcPhase::Prepare => {
+                        let mut request = Request::new(TierMutationPrepareRequest {
+                            version,
+                            mutation_id: mutation_id_text.clone(),
+                            canonical_payload: canonical_payload.clone(),
+                        });
+                        set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                        client
+                            .prepare_tier_mutation(request)
+                            .await
+                            .map_err(|status| tier_mutation_control_status_error(phase, status))?
+                            .into_inner()
+                    }
+                    TierMutationRpcPhase::Commit => {
+                        let mut request = Request::new(TierMutationCommitRequest {
+                            version,
+                            mutation_id: mutation_id_text.clone(),
+                            canonical_payload: canonical_payload.clone(),
+                        });
+                        set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                        client
+                            .commit_tier_mutation(request)
+                            .await
+                            .map_err(|status| tier_mutation_control_status_error(phase, status))?
+                            .into_inner()
+                    }
+                    TierMutationRpcPhase::Abort => {
+                        let mut request = Request::new(TierMutationAbortRequest {
+                            version,
+                            mutation_id: mutation_id_text,
+                            canonical_payload: canonical_payload.clone(),
+                        });
+                        set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                        client
+                            .abort_tier_mutation(request)
+                            .await
+                            .map_err(|status| tier_mutation_control_status_error(phase, status))?
+                            .into_inner()
+                    }
+                    _ => return Err(Error::other("tier mutation rpc phase is unsupported")),
+                };
+                validate_tier_mutation_response_proof(version, phase, mutation_id, &canonical_payload, &response)?;
+                if !response.success {
+                    return Err(Error::other(
+                        response
+                            .error_info
+                            .unwrap_or_else(|| "peer tier mutation failed without an error".to_string()),
+                    ));
+                }
+                let state = decode_tier_mutation_peer_state(response.state)?;
+                Ok(PeerTierMutationOutcome {
+                    state,
+                    applied: response.applied,
+                })
+            }
+            .await,
+        )
+        .await
+    }
+
+    pub async fn heal_control(&self, version: u32, topology_fingerprint: String, command: Vec<u8>) -> Result<Vec<u8>> {
+        if topology_fingerprint.len() > HEAL_CONTROL_FINGERPRINT_MAX_SIZE {
+            return Err(Error::other("heal control topology fingerprint exceeds size limit"));
+        }
+        if command.len() > HEAL_CONTROL_PAYLOAD_MAX_SIZE {
+            return Err(Error::other("heal control command exceeds size limit"));
+        }
+        let capability_probe = rustfs_protos::is_heal_control_capability_probe(&command);
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_heal_control_client()
+                    .await?
+                    .max_encoding_message_size(rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE)
+                    .max_decoding_message_size(rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE);
+                let canonical_body = rustfs_protos::canonical_heal_control_request_body(version, &topology_fingerprint, &command)
+                    .map_err(|_| Error::other("heal control request length cannot be represented"))?;
+                let mut request = Request::new(HealControlRequest {
+                    version,
+                    topology_fingerprint: topology_fingerprint.clone(),
+                    command: command.clone().into(),
+                });
+                request.set_timeout(rustfs_protos::heal_control_execution_timeout());
+                set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                let response = client.heal_control(request).await?.into_inner();
+                if !response.success {
+                    return Err(Error::other(
+                        response
+                            .error_info
+                            .unwrap_or_else(|| "peer heal control failed without an error".to_string()),
+                    ));
+                }
+                if !capability_probe {
+                    let canonical_response = rustfs_protos::canonical_heal_control_response_body(
+                        version,
+                        &topology_fingerprint,
+                        &command,
+                        &response.result,
+                    )
+                    .map_err(|_| Error::other("heal control response length cannot be represented"))?;
+                    validate_heal_control_response_proof(&canonical_response, &response.response_proof)?;
+                }
+                Ok(response.result.to_vec())
+            }
+            .await,
+        )
+        .await
+    }
+
+    /// Confirms that a peer supports the current heal-control coordination
+    /// contract and has the same storage
+    /// topology. Every non-success response is an error so old or divergent
+    /// peers cannot be mistaken for compatible ones.
+    pub async fn probe_heal_control(&self, topology_fingerprint: String) -> Result<()> {
+        let nonce = uuid::Uuid::new_v4();
+        let probe = rustfs_protos::heal_control_capability_probe(nonce.as_bytes());
+        let canonical_ack = rustfs_protos::canonical_heal_control_capability_ack(
+            rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION,
+            &topology_fingerprint,
+            &probe,
+        )
+        .map_err(|_| Error::other("heal control capability acknowledgement length cannot be represented"))?;
+        let proof = self
+            .heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
+            .await?;
+        validate_heal_control_capability_proof(&canonical_ack, &proof)
+    }
+
+    pub async fn load_bucket_metadata(&self, bucket: &str, scanner_maintenance_change: bool) -> Result<()> {
         self.finalize_result(
             async {
                 let mut client = self.get_client().await?;
                 let request = Request::new(LoadBucketMetadataRequest {
                     bucket: bucket.to_string(),
+                    scanner_maintenance_change,
                 });
 
                 let response = client.load_bucket_metadata(request).await?.into_inner();
@@ -908,6 +1299,25 @@ impl PeerRestClient {
         .await
     }
 
+    pub async fn scanner_activity(&self) -> Result<ScannerPeerActivity> {
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_client()
+                    .await?
+                    .max_decoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE)
+                    .max_encoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE);
+                let response = client
+                    .scanner_activity(Request::new(ScannerActivityRequest {}))
+                    .await?
+                    .into_inner();
+                decode_scanner_activity(response)
+            }
+            .await,
+        )
+        .await
+    }
+
     pub async fn get_metacache_listing(&self) -> Result<()> {
         warn!("get_metacache_listing is not implemented in PeerRestClient");
         Err(Error::NotImplemented)
@@ -1157,6 +1567,78 @@ mod tests {
         )
     }
 
+    #[test]
+    fn build_clients_from_slots_preserves_missing_remote_topology_slots() {
+        let slots = vec![
+            ("127.0.0.1:9000".to_string(), None, true),
+            ("127.0.0.1:9001".to_string(), Some("http://127.0.0.1:9001".to_string()), false),
+            ("127.0.0.1:notaport".to_string(), Some("http://127.0.0.1:notaport".to_string()), false),
+            ("127.0.0.1:9003".to_string(), None, false),
+        ];
+
+        let (remote, all, remote_topology_hosts) = PeerRestClient::build_clients_from_slots(slots);
+
+        assert_eq!(remote.len(), 3, "local node is excluded but remote slots are not compacted away");
+        assert_eq!(all.len(), 4, "all slots preserve the sorted cluster topology shape");
+        assert_eq!(
+            remote_topology_hosts,
+            vec![
+                "127.0.0.1:9001".to_string(),
+                "127.0.0.1:notaport".to_string(),
+                "127.0.0.1:9003".to_string()
+            ]
+        );
+        assert!(remote[0].is_some(), "valid remote peer should get a client");
+        assert!(remote[1].is_none(), "unparseable remote peer should remain observable as a missing slot");
+        assert!(remote[2].is_none(), "missing grid host should remain observable as a missing slot");
+        assert!(all[0].is_none(), "local node is represented by the local server_info row");
+        assert!(all[1].is_some());
+        assert!(all[2].is_none());
+        assert!(all[3].is_none());
+    }
+
+    #[test]
+    fn scanner_activity_requires_restart_safe_peer_identity() {
+        let missing_instance = ScannerActivityResponse {
+            instance_id: String::new(),
+            namespace_generation: 7,
+            maintenance_generation: 3,
+        };
+        assert!(
+            decode_scanner_activity(missing_instance)
+                .expect_err("an empty instance ID is not restart safe")
+                .to_string()
+                .contains("instance ID")
+        );
+
+        let malformed_instance = ScannerActivityResponse {
+            instance_id: "ABCDEF0123456789ABCDEF0123456789".to_string(),
+            namespace_generation: 7,
+            maintenance_generation: 3,
+        };
+        assert!(
+            decode_scanner_activity(malformed_instance)
+                .expect_err("activity instance IDs must use the canonical lowercase hex form")
+                .to_string()
+                .contains("instance ID")
+        );
+
+        let activity = decode_scanner_activity(ScannerActivityResponse {
+            instance_id: "0123456789abcdef0123456789abcdef".to_string(),
+            namespace_generation: 7,
+            maintenance_generation: 3,
+        })
+        .expect("complete activity responses should be accepted");
+        assert_eq!(
+            activity,
+            ScannerPeerActivity {
+                instance_id: "0123456789abcdef0123456789abcdef".to_string(),
+                namespace_generation: 7,
+                maintenance_generation: 3,
+            }
+        );
+    }
+
     #[test]
     fn peer_rest_client_marks_network_like_errors() {
         assert!(PeerRestClient::is_network_like_error(&Error::other("transport error")));
@@ -1177,6 +1659,230 @@ mod tests {
         assert!(err.to_string().contains("temporarily offline"));
     }
 
+    #[tokio::test]
+    async fn peer_rest_client_rejects_oversized_heal_control_before_dialing() {
+        let client = test_peer_client();
+        let err = client
+            .heal_control(1, "fingerprint".to_string(), vec![0; HEAL_CONTROL_PAYLOAD_MAX_SIZE + 1])
+            .await
+            .expect_err("oversized heal control payload must fail locally");
+
+        assert!(err.to_string().contains("exceeds size limit"));
+        assert!(!client.offline.load(Ordering::Acquire));
+    }
+
+    #[test]
+    fn heal_control_capability_proof_must_authenticate_exact_ack() {
+        runtime_sources::ensure_test_rpc_secret();
+        let proof = crate::cluster::rpc::sign_tonic_rpc_response_proof(b"expected").expect("test proof should sign");
+        assert!(validate_heal_control_capability_proof(b"expected", &proof).is_ok());
+        let err = validate_heal_control_capability_proof(b"different", &proof)
+            .expect_err("a proof for a different acknowledgement must fail closed");
+        assert!(err.to_string().contains("invalid heal control capability proof"));
+    }
+
+    #[test]
+    fn heal_control_response_proof_binds_command_and_result() {
+        runtime_sources::ensure_test_rpc_secret();
+        let canonical = rustfs_protos::canonical_heal_control_response_body(2, "fingerprint", b"query", b"result")
+            .expect("small response should encode");
+        let proof = crate::cluster::rpc::sign_tonic_rpc_response_proof(&canonical).expect("test proof should sign");
+        assert!(validate_heal_control_response_proof(&canonical, &proof).is_ok());
+
+        for tampered in [
+            rustfs_protos::canonical_heal_control_response_body(2, "fingerprint", b"cancel", b"result").unwrap(),
+            rustfs_protos::canonical_heal_control_response_body(2, "fingerprint", b"query", b"tampered").unwrap(),
+        ] {
+            let err = validate_heal_control_response_proof(&tampered, &proof)
+                .expect_err("proof must not authenticate a different command or result");
+            assert!(err.to_string().contains("invalid heal control response proof"));
+        }
+    }
+
+    struct TierMutationResponseFixture<'a> {
+        version: u32,
+        phase: TierMutationRpcPhase,
+        mutation_id: Uuid,
+        canonical_payload: &'a [u8],
+        success: bool,
+        state: i32,
+        applied: bool,
+        error_info: Option<&'a str>,
+    }
+
+    fn signed_tier_mutation_response(input: TierMutationResponseFixture<'_>) -> TierMutationControlResponse {
+        let canonical =
+            rustfs_protos::canonical_tier_mutation_rpc_response_body(rustfs_protos::TierMutationRpcResponseProofInput {
+                version: input.version,
+                phase: input.phase,
+                mutation_id: input.mutation_id,
+                canonical_payload: input.canonical_payload,
+                success: input.success,
+                state: input.state,
+                applied: input.applied,
+                error_info: input.error_info,
+            })
+            .expect("small tier mutation response should encode");
+        let response_proof =
+            crate::cluster::rpc::sign_tonic_rpc_response_proof(&canonical).expect("tier mutation response should sign");
+        TierMutationControlResponse {
+            success: input.success,
+            state: input.state,
+            applied: input.applied,
+            error_info: input.error_info.map(str::to_string),
+            response_proof: response_proof.into(),
+        }
+    }
+
+    #[test]
+    fn tier_mutation_response_proof_binds_phase_payload_state_and_error() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mutation_id = Uuid::new_v4();
+        let payload = b"tier-mutation-prepare";
+        let response = signed_tier_mutation_response(TierMutationResponseFixture {
+            version: rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            phase: TierMutationRpcPhase::Prepare,
+            mutation_id,
+            canonical_payload: payload,
+            success: true,
+            state: TierMutationPeerState::Prepared as i32,
+            applied: true,
+            error_info: None,
+        });
+        validate_tier_mutation_response_proof(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            TierMutationRpcPhase::Prepare,
+            mutation_id,
+            payload,
+            &response,
+        )
+        .expect("matching tier mutation response proof should verify");
+
+        for tampered in [
+            TierMutationControlResponse {
+                success: false,
+                error_info: Some("peer failed".to_string()),
+                ..response.clone()
+            },
+            TierMutationControlResponse {
+                state: TierMutationPeerState::Committed as i32,
+                ..response.clone()
+            },
+            TierMutationControlResponse {
+                applied: false,
+                ..response.clone()
+            },
+        ] {
+            let err = validate_tier_mutation_response_proof(
+                rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+                TierMutationRpcPhase::Prepare,
+                mutation_id,
+                payload,
+                &tampered,
+            )
+            .expect_err("tampered tier mutation response proof must fail");
+            assert!(err.to_string().contains("invalid tier mutation response proof"));
+        }
+
+        let err = validate_tier_mutation_response_proof(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            TierMutationRpcPhase::Commit,
+            mutation_id,
+            payload,
+            &response,
+        )
+        .expect_err("response proof must bind request phase");
+        assert!(err.to_string().contains("invalid tier mutation response proof"));
+    }
+
+    #[test]
+    fn tier_mutation_peer_state_decode_fails_closed() {
+        assert_eq!(
+            decode_tier_mutation_peer_state(TierMutationPeerState::Prepared as i32).expect("prepared state should decode"),
+            PeerTierMutationState::Prepared
+        );
+        assert_eq!(
+            decode_tier_mutation_peer_state(TierMutationPeerState::Committed as i32).expect("committed state should decode"),
+            PeerTierMutationState::Committed
+        );
+        assert_eq!(
+            decode_tier_mutation_peer_state(TierMutationPeerState::Aborted as i32).expect("aborted state should decode"),
+            PeerTierMutationState::Aborted
+        );
+        assert!(decode_tier_mutation_peer_state(TierMutationPeerState::Unspecified as i32).is_err());
+        assert!(decode_tier_mutation_peer_state(99).is_err());
+    }
+
+    #[test]
+    fn tier_mutation_payload_guard_rejects_invalid_lengths() {
+        validate_tier_mutation_payload_len(
+            TierMutationRpcPhase::Prepare,
+            rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE,
+        )
+        .expect("max prepare payload should fit");
+        assert!(
+            validate_tier_mutation_payload_len(
+                TierMutationRpcPhase::Prepare,
+                rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE + 1,
+            )
+            .is_err()
+        );
+        validate_tier_mutation_payload_len(
+            TierMutationRpcPhase::Commit,
+            rustfs_protos::TIER_MUTATION_RPC_MAX_COMMIT_PAYLOAD_SIZE,
+        )
+        .expect("max commit payload should fit");
+        assert!(
+            validate_tier_mutation_payload_len(
+                TierMutationRpcPhase::Commit,
+                rustfs_protos::TIER_MUTATION_RPC_MAX_COMMIT_PAYLOAD_SIZE + 1,
+            )
+            .is_err()
+        );
+        validate_tier_mutation_payload_len(TierMutationRpcPhase::Abort, 0).expect("empty abort payload should fit");
+        assert!(validate_tier_mutation_payload_len(TierMutationRpcPhase::Abort, 1).is_err());
+    }
+
+    #[test]
+    fn tier_mutation_rpc_status_matrix_fails_closed_for_old_or_unresponsive_peers() {
+        for (phase, label) in [
+            (TierMutationRpcPhase::Prepare, "prepare"),
+            (TierMutationRpcPhase::Commit, "commit"),
+            (TierMutationRpcPhase::Abort, "abort"),
+        ] {
+            for status in [
+                tonic::Status::unimplemented("old peer has no tier mutation control service"),
+                tonic::Status::deadline_exceeded("peer tier mutation control timed out"),
+                tonic::Status::unavailable("peer tier mutation control unavailable"),
+            ] {
+                let err = tier_mutation_control_status_error(phase, status);
+                let rendered = err.to_string();
+                assert!(rendered.contains(&format!("peer tier mutation {label} RPC failed")), "{rendered}");
+                assert!(
+                    rendered.contains("old peer")
+                        || rendered.contains("timed out")
+                        || rendered.contains("unavailable")
+                        || rendered.contains("Unavailable"),
+                    "{rendered}"
+                );
+            }
+        }
+    }
+
+    #[tokio::test]
+    async fn peer_rest_client_rejects_oversized_tier_prepare_before_dialing() {
+        let client = test_peer_client();
+        let err = client
+            .prepare_tier_mutation(
+                Uuid::new_v4(),
+                Bytes::from(vec![0; rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE + 1]),
+            )
+            .await
+            .expect_err("oversized tier prepare should fail before dialing");
+        assert!(err.to_string().contains("tier mutation payload exceeds size limit"));
+        assert!(!client.offline.load(Ordering::Acquire));
+    }
+
     #[tokio::test]
     async fn peer_rest_client_prepare_retry_clears_offline_gate() {
         // finalize_result sets the offline gate on a network error; without

crates/ecstore/src/cluster/rpc/peer_s3_client.rs

@@ -23,7 +23,7 @@ use crate::disk::{DiskAPI, DiskStore, disk_store::get_max_timeout_duration};
 use crate::runtime::instance::{InstanceContext, bootstrap_ctx};
 use crate::runtime::sources as runtime_sources;
 use crate::storage_api_contracts::bucket::{BucketInfo, BucketOptions, DeleteBucketOptions, MakeBucketOptions};
-use crate::store::utils::is_reserved_or_invalid_bucket;
+use crate::store::{has_xlmeta_files, utils::is_reserved_or_invalid_bucket};
 use crate::{
     disk::{
         self, VolumeInfo,
@@ -614,13 +614,24 @@ impl PeerS3Client for LocalPeerS3Client {
             return Err(Error::ErasureWriteQuorum);
         }
 
+        let force = if opts.force_if_empty && !opts.force {
+            for disk in local_disks.iter() {
+                if has_xlmeta_files(&disk.path().join(bucket)).await.map_err(Error::Io)? {
+                    return Err(Error::VolumeNotEmpty);
+                }
+            }
+            true
+        } else {
+            opts.force
+        };
+
         let mut futures = Vec::with_capacity(local_disks.len());
 
         for disk in local_disks.iter() {
             // Non-force delete refuses a non-empty bucket (VolumeNotEmpty), which
             // the recreate loop below turns into BucketNotEmpty; only an explicit
             // force delete removes recursively (backlog#799 B1).
-            futures.push(disk.delete_volume(bucket, opts.force));
+            futures.push(disk.delete_volume(bucket, force));
         }
 
         let results = join_all(futures).await;
@@ -988,13 +999,15 @@ impl PeerS3Client for RemotePeerS3Client {
         .await
     }
 
-    async fn delete_bucket(&self, bucket: &str, _opts: &DeleteBucketOptions) -> Result<()> {
+    async fn delete_bucket(&self, bucket: &str, opts: &DeleteBucketOptions) -> Result<()> {
         self.execute_with_timeout(
             || async {
+                let options = serde_json::to_string(opts)?;
                 let mut client = self.get_client().await?;
 
                 let request = Request::new(DeleteBucketRequest {
                     bucket: bucket.to_string(),
+                    options,
                 });
                 let response = client.delete_bucket(request).await?.into_inner();
                 if !response.success {

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -1129,6 +1129,9 @@ fn decode_batch_read_version_response_items(
             let resp = decode_msgpack_or_json::<BatchReadVersionResp>(buf, "", "BatchReadVersionResp").map_err(|err| {
                 Error::other(format!("decode BatchReadVersionResp msgpack item {index} from {endpoint} failed: {err}"))
             })?;
+            if resp.success {
+                validate_decoded_file_info(&resp.file_info)?;
+            }
             batch_read_version_resps.push(resp);
         }
         return Ok(batch_read_version_resps);
@@ -1143,12 +1146,19 @@ fn decode_batch_read_version_response_items(
         let resp = serde_json::from_str::<BatchReadVersionResp>(json_str).map_err(|err| {
             Error::other(format!("decode BatchReadVersionResp json item {index} from {endpoint} failed: {err}"))
         })?;
+        if resp.success {
+            validate_decoded_file_info(&resp.file_info)?;
+        }
         batch_read_version_resps.push(resp);
     }
 
     Ok(batch_read_version_resps)
 }
 
+fn validate_decoded_file_info(file_info: &FileInfo) -> Result<()> {
+    file_info.validate_for_metadata_read().map_err(Into::into)
+}
+
 #[async_trait::async_trait]
 impl DiskAPI for RemoteDisk {
     #[tracing::instrument(level = "trace", skip_all)]
@@ -1825,6 +1835,7 @@ impl DiskAPI for RemoteDisk {
                 }
 
                 let file_info = decode_msgpack_or_json::<FileInfo>(&response.file_info_bin, &response.file_info, "FileInfo")?;
+                validate_decoded_file_info(&file_info)?;
 
                 Ok(file_info)
             },
@@ -2712,6 +2723,25 @@ mod tests {
 
     static INIT: Once = Once::new();
 
+    #[test]
+    fn decoded_remote_metadata_rejects_default_like_delete_marker() {
+        let forged = FileInfo {
+            deleted: true,
+            ..Default::default()
+        };
+        assert!(matches!(validate_decoded_file_info(&forged), Err(DiskError::FileCorrupt)));
+
+        let marker = FileInfo {
+            volume: "bucket".to_string(),
+            name: "object".to_string(),
+            version_id: Some(Uuid::new_v4()),
+            deleted: true,
+            mod_time: Some(::time::OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+        validate_decoded_file_info(&marker).expect("canonical remote delete marker should validate");
+    }
+
     #[derive(Clone, Default)]
     struct CapturedLogs {
         buffer: Arc<Mutex<Vec<u8>>>,
@@ -3021,17 +3051,19 @@ mod tests {
     }
 
     fn sample_batch_read_version_resp(index: usize, path: &str, success: bool) -> BatchReadVersionResp {
+        let mut file_info = FileInfo::new(path, 1, 0);
+        file_info.erasure.index = 1;
         BatchReadVersionResp {
             index,
             path: path.to_string(),
             version_id: "version-a".to_string(),
             success,
+            file_info,
             error: if success {
                 String::new()
             } else {
                 "file version not found".to_string()
             },
-            ..Default::default()
         }
     }
 
@@ -3055,6 +3087,25 @@ mod tests {
         assert!(decoded[0].success);
     }
 
+    #[test]
+    fn batch_read_version_response_rejects_invalid_success_metadata() {
+        let endpoint = sample_remote_endpoint();
+        let mut response_item = sample_batch_read_version_resp(0, "invalid-object", true);
+        response_item.file_info.erasure.data_blocks = 0;
+        response_item.file_info.erasure.parity_blocks = 2;
+        let response = BatchReadVersionResponse {
+            success: true,
+            batch_read_version_resps: Vec::new(),
+            batch_read_version_resps_bin: vec![encode_msgpack(&response_item).expect("msgpack response should encode").into()],
+            error: None,
+        };
+
+        let err = decode_batch_read_version_response_items(response, &endpoint)
+            .expect_err("successful remote response with invalid metadata must fail closed");
+
+        assert_eq!(err, DiskError::FileCorrupt);
+    }
+
     #[test]
     fn batch_read_version_response_decode_reports_corrupt_msgpack_item() {
         let endpoint = sample_remote_endpoint();

crates/ecstore/src/config/com.rs

@@ -17,12 +17,15 @@ use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result};
 use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
 use crate::runtime::sources as runtime_sources;
+use crate::set_disk::get_lock_acquire_timeout;
 use crate::storage_api_contracts::{
     admin::StorageAdminApi,
     heal::HealOperations,
+    namespace::NamespaceLocking,
     object::{DeletedObject, EcstoreObjectIO, ObjectIO, ObjectOperations, ObjectToDelete},
     range::HTTPRangeSpec,
 };
+use crate::store::ECStore;
 use http::HeaderMap;
 use rustfs_common::heal_channel::{HealOpts, HealScanMode};
 use rustfs_config::audit::{
@@ -45,10 +48,94 @@ use serde_json::{Map, Value};
 use std::collections::{HashMap, HashSet};
 use std::sync::LazyLock;
 use std::sync::{Arc, RwLock};
+use tokio::sync::RwLock as AsyncRwLock;
 use tracing::{debug, error, info, instrument, warn};
 
 pub const CONFIG_PREFIX: &str = "config";
-const CONFIG_FILE: &str = "config.json";
+const SERVER_CONFIG_OBJECT: &str = "config/config.json";
+
+// Server-config lock order: SERVER_CONFIG_LOCK -> distributed namespace lock
+// for SERVER_CONFIG_OBJECT. Readers and writers must never reverse this order.
+static SERVER_CONFIG_LOCK: LazyLock<AsyncRwLock<()>> = LazyLock::new(|| AsyncRwLock::new(()));
+
+fn config_task_join_error(operation: &'static str, error: tokio::task::JoinError) -> Error {
+    let outcome = if error.is_cancelled() { "cancelled" } else { "panicked" };
+    Error::other(format!("{operation} task {outcome}"))
+}
+
+/// Runs one complete server-config transaction while holding both the local
+/// process guard and the distributed namespace write lock for `config.json`.
+/// The operation must use the corresponding no-lock read/save functions.
+pub async fn with_server_config_write_lock<F, Fut, T>(store: Arc<ECStore>, operation: F) -> Result<T>
+where
+    F: FnOnce() -> Fut + Send + 'static,
+    Fut: std::future::Future<Output = T> + Send + 'static,
+    T: Send + 'static,
+{
+    tokio::spawn(async move {
+        // Lock order: SERVER_CONFIG_LOCK -> namespace write lock.
+        let _local_guard = SERVER_CONFIG_LOCK.write().await;
+        let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, SERVER_CONFIG_OBJECT).await?;
+        let _write_guard = namespace_lock.get_write_lock(get_lock_acquire_timeout()).await?;
+        Ok(operation().await)
+    })
+    .await
+    .map_err(|error| config_task_join_error("server config write", error))?
+}
+
+/// Reads and synchronously publishes one server-config snapshot while holding
+/// a shared namespace lock. Concurrent peer reloads may proceed together, but
+/// no writer can interleave between the snapshot read and generation accept.
+pub async fn with_server_config_read_lock<F, Fut, T>(store: Arc<ECStore>, operation: F) -> Result<T>
+where
+    F: FnOnce() -> Fut + Send + 'static,
+    Fut: std::future::Future<Output = T> + Send + 'static,
+    T: Send + 'static,
+{
+    tokio::spawn(async move {
+        // Lock order: SERVER_CONFIG_LOCK -> namespace read lock.
+        let _local_guard = SERVER_CONFIG_LOCK.read().await;
+        let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, SERVER_CONFIG_OBJECT).await?;
+        let _read_guard = namespace_lock.get_read_lock(get_lock_acquire_timeout()).await?;
+        Ok(operation().await)
+    })
+    .await
+    .map_err(|error| config_task_join_error("server config read", error))?
+}
+
+/// Runs a cancellation-safe transaction under the distributed write lock for
+/// one metadata config object. The operation must use no-lock object I/O.
+pub async fn with_config_object_write_lock<F, Fut, T>(store: Arc<ECStore>, object: String, operation: F) -> Result<T>
+where
+    F: FnOnce() -> Fut + Send + 'static,
+    Fut: std::future::Future<Output = T> + Send + 'static,
+    T: Send + 'static,
+{
+    tokio::spawn(async move {
+        let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &object).await?;
+        let _write_guard = namespace_lock.get_write_lock(get_lock_acquire_timeout()).await?;
+        Ok(operation().await)
+    })
+    .await
+    .map_err(|error| config_task_join_error("config object write", error))?
+}
+
+/// Runs one read-and-publish transaction under the distributed read lock for
+/// a metadata config object. The operation must use no-lock object I/O.
+pub async fn with_config_object_read_lock<F, Fut, T>(store: Arc<ECStore>, object: String, operation: F) -> Result<T>
+where
+    F: FnOnce() -> Fut + Send + 'static,
+    Fut: std::future::Future<Output = T> + Send + 'static,
+    T: Send + 'static,
+{
+    tokio::spawn(async move {
+        let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &object).await?;
+        let _read_guard = namespace_lock.get_read_lock(get_lock_acquire_timeout()).await?;
+        Ok(operation().await)
+    })
+    .await
+    .map_err(|error| config_task_join_error("config object read", error))?
+}
 
 /// Environment variable gating the startup fallback to the default server
 /// config when the persisted `config.json` object is corrupt beyond repair
@@ -279,6 +366,17 @@ where
     Ok(data)
 }
 
+/// Read an existing config object without treating an empty payload as absent.
+/// Callers that validate their own payload format need to distinguish corruption
+/// from `ConfigNotFound`.
+pub(crate) async fn read_config_preserve_empty<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
+where
+    S: EcstoreObjectIO,
+{
+    let (data, _obj) = read_config_with_metadata_inner(api, file, &ObjectOptions::default(), true).await?;
+    Ok(data)
+}
+
 pub async fn read_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
 where
     S: ObjectIO<
@@ -304,6 +402,26 @@ where
 }
 
 pub async fn read_config_with_metadata<S>(api: Arc<S>, file: &str, opts: &ObjectOptions) -> Result<(Vec<u8>, ObjectInfo)>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        >,
+{
+    read_config_with_metadata_inner(api, file, opts, false).await
+}
+
+async fn read_config_with_metadata_inner<S>(
+    api: Arc<S>,
+    file: &str,
+    opts: &ObjectOptions,
+    preserve_empty: bool,
+) -> Result<(Vec<u8>, ObjectInfo)>
 where
     S: ObjectIO<
             Error = Error,
@@ -330,7 +448,7 @@ where
 
     let data = rd.read_all().await?;
 
-    if data.is_empty() {
+    if data.is_empty() && !preserve_empty {
         return Err(Error::ConfigNotFound);
     }
 
@@ -362,6 +480,31 @@ where
     .await
 }
 
+pub async fn save_config_no_lock<S>(api: Arc<S>, file: &str, data: Vec<u8>) -> Result<()>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        >,
+{
+    save_config_with_opts(
+        api,
+        file,
+        data,
+        &ObjectOptions {
+            max_parity: true,
+            no_lock: true,
+            ..Default::default()
+        },
+    )
+    .await
+}
+
 #[instrument(skip(api))]
 pub async fn delete_config<S>(api: Arc<S>, file: &str) -> Result<()>
 where
@@ -425,15 +568,23 @@ async fn new_and_save_server_config<S>(api: Arc<S>) -> Result<Config>
 where
     S: EcstoreObjectIO + StorageAdminApi,
 {
-    let mut cfg = new_server_config();
-    lookup_configs(&mut cfg, api.clone()).await;
+    let cfg = new_server_config();
     save_server_config(api, &cfg).await?;
 
     Ok(cfg)
 }
 
+async fn new_and_save_server_config_no_lock<S>(api: Arc<S>) -> Result<Config>
+where
+    S: EcstoreObjectIO + StorageAdminApi,
+{
+    let cfg = new_server_config();
+    save_server_config_no_lock(api, &cfg).await?;
+    Ok(cfg)
+}
+
 fn get_config_file() -> String {
-    format!("{CONFIG_PREFIX}{SLASH_SEPARATOR}{CONFIG_FILE}")
+    SERVER_CONFIG_OBJECT.to_string()
 }
 
 fn storage_class_kvs_mut(cfg: &mut Config) -> &mut KVS {
@@ -1237,17 +1388,19 @@ where
 }
 
 /// Handle the situation where the configuration file does not exist, create and save a new configuration
-async fn handle_missing_config<S>(api: Arc<S>, context: &str) -> Result<Config>
+async fn handle_missing_config<S>(api: Arc<S>, context: &str, namespace_lock_held: bool) -> Result<Config>
 where
     S: EcstoreObjectIO + StorageAdminApi,
 {
     warn!("Configuration not found ({}): Start initializing new configuration", context);
     let cfg = if runtime_sources::first_cluster_node_is_local().await {
-        new_and_save_server_config(api.clone()).await?
+        if namespace_lock_held {
+            new_and_save_server_config_no_lock(api.clone()).await?
+        } else {
+            new_and_save_server_config(api.clone()).await?
+        }
     } else {
-        let mut cfg = new_server_config();
-        lookup_configs(&mut cfg, api).await;
-        cfg
+        new_server_config()
     };
     warn!("Configuration initialization complete ({})", context);
     Ok(cfg)
@@ -1260,6 +1413,46 @@ fn handle_config_read_error(err: Error, file_path: &str) -> Result<Config> {
 }
 
 pub async fn read_config_without_migrate<S>(api: Arc<S>) -> Result<Config>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        > + StorageAdminApi,
+{
+    read_config_without_migrate_inner(api, false).await
+}
+
+/// Reads the server config while an upper layer holds the namespace write lock
+/// for [`SERVER_CONFIG_OBJECT`]. Missing-config initialization uses the matching
+/// no-lock save path and therefore cannot recursively acquire the same lock.
+pub async fn read_config_without_migrate_no_lock<S>(api: Arc<S>) -> Result<Config>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        > + StorageAdminApi,
+{
+    read_config_without_migrate_inner(api, true).await
+}
+
+/// Reads an already-initialized server config while a caller owns a namespace
+/// read lock. This never initializes or migrates a missing object.
+pub async fn read_existing_server_config_no_lock(api: Arc<ECStore>) -> Result<Config> {
+    let data = read_config_no_lock(api, SERVER_CONFIG_OBJECT).await?;
+    Ok(decode_persisted_server_config(&data)?.merge())
+}
+
+async fn read_config_without_migrate_inner<S>(api: Arc<S>, namespace_lock_held: bool) -> Result<Config>
 where
     S: ObjectIO<
             Error = Error,
@@ -1275,13 +1468,13 @@ where
 
     // Try to read the configuration file
     match read_config_no_lock(api.clone(), &config_file).await {
-        Ok(data) => read_server_config(api, &data).await,
-        Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration").await,
+        Ok(data) => read_server_config(api, &data, namespace_lock_held).await,
+        Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration", namespace_lock_held).await,
         Err(err) => handle_config_read_error(err, &config_file),
     }
 }
 
-async fn read_server_config<S>(api: Arc<S>, data: &[u8]) -> Result<Config>
+async fn read_server_config<S>(api: Arc<S>, data: &[u8], namespace_lock_held: bool) -> Result<Config>
 where
     S: EcstoreObjectIO + StorageAdminApi,
 {
@@ -1296,7 +1489,9 @@ where
                 let cfg = decode_persisted_server_config(&cfg_data)?;
                 return Ok(cfg.merge());
             }
-            Err(Error::ConfigNotFound) => return handle_missing_config(api, "Read alternate configuration").await,
+            Err(Error::ConfigNotFound) => {
+                return handle_missing_config(api, "Read alternate configuration", namespace_lock_held).await;
+            }
             Err(err) => return handle_config_read_error(err, &config_file),
         }
     }
@@ -1506,6 +1701,38 @@ fn fallback_server_config_after_corruption(err: Error, config_file: &str, recove
 }
 
 pub async fn save_server_config<S>(api: Arc<S>, cfg: &Config) -> Result<()>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        >,
+{
+    save_server_config_inner(api, cfg, false).await
+}
+
+/// Saves the server config while an upper layer holds the namespace write
+/// lock for [`SERVER_CONFIG_OBJECT`].
+pub async fn save_server_config_no_lock<S>(api: Arc<S>, cfg: &Config) -> Result<()>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        >,
+{
+    save_server_config_inner(api, cfg, true).await
+}
+
+async fn save_server_config_inner<S>(api: Arc<S>, cfg: &Config, no_lock: bool) -> Result<()>
 where
     S: ObjectIO<
             Error = Error,
@@ -1518,7 +1745,11 @@ where
         >,
 {
     let config_file = get_config_file();
-    let existing = match read_config(api.clone(), &config_file).await {
+    let existing = match if no_lock {
+        read_config_no_lock(api.clone(), &config_file).await
+    } else {
+        read_config(api.clone(), &config_file).await
+    } {
         Ok(v) => Some(v),
         Err(Error::ConfigNotFound) => None,
         Err(err) => {
@@ -1542,17 +1773,28 @@ where
         return Ok(());
     }
 
-    save_config(api, &config_file, data).await
+    if no_lock {
+        save_config_with_opts(
+            api,
+            &config_file,
+            data,
+            &ObjectOptions {
+                max_parity: true,
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+    } else {
+        save_config(api, &config_file, data).await
+    }
 }
 
-pub async fn lookup_configs<S>(cfg: &mut Config, api: Arc<S>)
+pub async fn lookup_configs<S>(cfg: &mut Config, api: Arc<S>) -> Result<()>
 where
     S: StorageAdminApi,
 {
-    // TODO: from etcd
-    if let Err(err) = apply_dynamic_config(cfg, api).await {
-        error!("apply_dynamic_config err {:?}", &err);
-    }
+    apply_dynamic_config(cfg, api).await
 }
 
 async fn apply_dynamic_config<S>(cfg: &mut Config, api: Arc<S>) -> Result<()>
@@ -1569,24 +1811,34 @@ where
 async fn apply_dynamic_config_for_sub_sys<S>(cfg: &mut Config, api: Arc<S>, subsys: &str) -> Result<()>
 where
     S: StorageAdminApi,
+{
+    apply_dynamic_config_for_sub_sys_with(
+        cfg,
+        api,
+        subsys,
+        storageclass::lookup_config_for_pools,
+        runtime_sources::set_storage_class_config,
+    )
+    .await
+}
+
+async fn apply_dynamic_config_for_sub_sys_with<S, R, F>(
+    cfg: &mut Config,
+    api: Arc<S>,
+    subsys: &str,
+    resolve_storage_class: R,
+    publish_storage_class: F,
+) -> Result<()>
+where
+    S: StorageAdminApi,
+    R: FnOnce(&KVS, &[usize]) -> Result<storageclass::Config>,
+    F: FnOnce(storageclass::Config),
 {
     let set_drive_counts = StorageAdminApi::set_drive_counts(api.as_ref());
     if subsys == STORAGE_CLASS_SUB_SYS {
         let kvs = cfg.get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER).unwrap_or_default();
-
-        for (i, count) in set_drive_counts.iter().enumerate() {
-            match storageclass::lookup_config(&kvs, *count) {
-                Ok(res) => {
-                    if i == 0 {
-                        runtime_sources::set_storage_class_config(res);
-                    }
-                }
-                Err(err) => {
-                    error!("init storage class err:{:?}", &err);
-                    break;
-                }
-            }
-        }
+        let candidate = resolve_storage_class(&kvs, &set_drive_counts)?;
+        publish_storage_class(candidate);
     }
 
     Ok(())
@@ -1595,7 +1847,8 @@ where
 #[cfg(test)]
 mod tests {
     use super::{
-        configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
+        apply_dynamic_config_for_sub_sys_with, config_task_join_error, configs_semantically_equal, decode_server_config_blob,
+        encode_server_config_blob, is_standard_object_server_config, lookup_configs, read_config, read_config_preserve_empty,
         read_config_with_metadata, storage_class_kvs_mut,
     };
     use crate::config::{audit, notify, oidc};
@@ -1616,6 +1869,17 @@ mod tests {
     use rustfs_config::{
         DEFAULT_DELIMITER, ENABLE_KEY, EnableState, MYSQL_DSN_STRING, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_TABLE,
     };
+
+    #[tokio::test]
+    async fn config_task_join_error_does_not_expose_panic_payload() {
+        let join_error = tokio::spawn(async { panic!("do-not-expose-payload") })
+            .await
+            .expect_err("test task should panic");
+
+        let rendered = config_task_join_error("server config write", join_error).to_string();
+        assert!(rendered.contains("panicked"));
+        assert!(!rendered.contains("do-not-expose-payload"));
+    }
     use rustfs_lock::client::LockClient;
     use rustfs_lock::client::local::LocalClient;
     use rustfs_lock::{LockError, LockInfo, LockResponse, LockStats};
@@ -2977,6 +3241,7 @@ mod tests {
         state: Mutex<RecoveryReadState>,
         heal_replacement: Option<Vec<u8>>,
         heal_calls: AtomicUsize,
+        drive_counts: Vec<usize>,
     }
 
     impl RecoveryMockStore {
@@ -2985,8 +3250,14 @@ mod tests {
                 state: Mutex::new(state),
                 heal_replacement,
                 heal_calls: AtomicUsize::new(0),
+                drive_counts: vec![2],
             }
         }
+
+        fn with_drive_counts(mut self, drive_counts: Vec<usize>) -> Self {
+            self.drive_counts = drive_counts;
+            self
+        }
     }
 
     struct ServerConfigDecryptHookGuard {
@@ -3059,6 +3330,21 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn read_config_preserve_empty_distinguishes_empty_object() {
+        let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None));
+
+        let err = read_config(store.clone(), "config/empty.json")
+            .await
+            .expect_err("the existing config contract treats empty objects as missing");
+        assert!(matches!(err, Error::ConfigNotFound));
+
+        let data = read_config_preserve_empty(store, "config/empty.json")
+            .await
+            .expect("payload-validating callers must observe the empty object");
+        assert!(data.is_empty());
+    }
+
     #[async_trait::async_trait]
     impl StorageAdminApi for RecoveryMockStore {
         type BackendInfo = rustfs_madmin::BackendInfo;
@@ -3086,7 +3372,7 @@ mod tests {
         }
 
         fn set_drive_counts(&self) -> Vec<usize> {
-            vec![2]
+            self.drive_counts.clone()
         }
     }
 
@@ -3127,6 +3413,68 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn invalid_later_pool_does_not_partially_publish_storage_class() {
+        temp_env::async_with_vars(
+            [
+                (crate::config::storageclass::STANDARD_ENV, None::<&str>),
+                (crate::config::storageclass::RRS_ENV, None::<&str>),
+                (crate::config::storageclass::OPTIMIZE_ENV, None::<&str>),
+                (crate::config::storageclass::INLINE_BLOCK_ENV, None::<&str>),
+            ],
+            async {
+                let mut cfg = Config::new();
+                storage_class_kvs_mut(&mut cfg)
+                    .insert(crate::config::storageclass::CLASS_STANDARD.to_string(), "EC:2".to_string());
+                let store =
+                    Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None).with_drive_counts(vec![4, 2]));
+
+                let result = lookup_configs(&mut cfg, store.clone()).await;
+                assert!(result.is_err(), "public lookup entry must propagate an invalid later pool");
+
+                let mut publish_count = 0;
+                let result = apply_dynamic_config_for_sub_sys_with(
+                    &mut cfg,
+                    store,
+                    STORAGE_CLASS_SUB_SYS,
+                    crate::config::storageclass::lookup_config_for_pools_without_env,
+                    |_| {
+                        publish_count += 1;
+                    },
+                )
+                .await;
+                assert!(result.is_err(), "invalid later pool must fail the complete candidate");
+                assert_eq!(publish_count, 0, "failed reload must not publish any candidate");
+            },
+        )
+        .await;
+    }
+
+    #[tokio::test]
+    async fn dynamic_storage_class_publishes_one_multi_pool_snapshot() {
+        let mut cfg = Config::new();
+        let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None).with_drive_counts(vec![4, 2]));
+
+        let mut published = None;
+        let result = apply_dynamic_config_for_sub_sys_with(
+            &mut cfg,
+            store,
+            STORAGE_CLASS_SUB_SYS,
+            crate::config::storageclass::lookup_config_for_pools_without_env,
+            |candidate| {
+                published = Some(candidate);
+            },
+        )
+        .await;
+
+        result.expect("valid multi-pool candidate should publish");
+        assert_eq!(
+            published.and_then(|cfg| cfg.parities_for_sc(crate::config::storageclass::STANDARD)),
+            Some(vec![2, 1])
+        );
+    }
+
     fn encrypted_current_server_config_blob() -> Vec<u8> {
         let mut cfg = Config::new();
         let kvs = storage_class_kvs_mut(&mut cfg);

crates/ecstore/src/config/mod.rs

@@ -26,6 +26,7 @@ pub mod storageclass;
 
 use crate::error::Result;
 use crate::store::ECStore;
+use arc_swap::ArcSwap;
 use com::{STORAGE_CLASS_SUB_SYS, lookup_configs, read_config_without_migrate_with_recovery};
 use rustfs_config::HEAL_SUB_SYS;
 use rustfs_config::audit::{
@@ -39,11 +40,12 @@ use rustfs_config::notify::{
 use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
 use rustfs_config::server_config::{register_default_kvs, set_global_server_config};
 use std::collections::HashMap;
+use std::sync::Arc;
 use std::sync::LazyLock;
-use std::sync::{Arc, RwLock};
+use tracing::warn;
 
-pub static GLOBAL_STORAGE_CLASS: LazyLock<RwLock<storageclass::Config>> =
-    LazyLock::new(|| RwLock::new(storageclass::Config::default()));
+pub static GLOBAL_STORAGE_CLASS: LazyLock<ArcSwap<storageclass::Config>> =
+    LazyLock::new(|| ArcSwap::from_pointee(storageclass::Config::default()));
 pub static GLOBAL_CONFIG_SYS: LazyLock<ConfigSys> = LazyLock::new(ConfigSys::new);
 
 pub static RUSTFS_CONFIG_PREFIX: &str = "config";
@@ -63,7 +65,7 @@ impl ConfigSys {
     pub async fn init(&self, api: Arc<ECStore>) -> Result<()> {
         let mut cfg = read_config_without_migrate_with_recovery(api.clone()).await?;
 
-        lookup_configs(&mut cfg, api).await;
+        lookup_configs(&mut cfg, api).await?;
 
         set_global_server_config(cfg);
 
@@ -72,12 +74,166 @@ impl ConfigSys {
 }
 
 pub fn get_global_storage_class() -> Option<storageclass::Config> {
-    GLOBAL_STORAGE_CLASS.read().ok().map(|guard| (*guard).clone())
+    Some(GLOBAL_STORAGE_CLASS.load().as_ref().clone())
+}
+
+pub(crate) fn get_global_storage_class_snapshot() -> Arc<storageclass::Config> {
+    GLOBAL_STORAGE_CLASS.load_full()
+}
+
+fn publish_storage_class_config(target: &ArcSwap<storageclass::Config>, cfg: storageclass::Config) {
+    let cfg = Arc::new(cfg);
+    target.store(cfg.clone());
+
+    for (pool_index, drives_per_set) in cfg.automatic_zero_parity_pools() {
+        warn!(
+            event = "storage_class_zero_redundancy",
+            component = "ecstore",
+            subsystem = "storage_class",
+            state = "degraded",
+            pool_index,
+            drives_per_set,
+            parity = 0,
+            "automatic storage class has no parity"
+        );
+    }
 }
 
 pub fn set_global_storage_class(cfg: storageclass::Config) {
-    if let Ok(mut guard) = GLOBAL_STORAGE_CLASS.write() {
-        *guard = cfg;
+    publish_storage_class_config(&GLOBAL_STORAGE_CLASS, cfg);
+}
+
+#[cfg(test)]
+mod storage_class_publish_tests {
+    use super::publish_storage_class_config;
+    use crate::config::storageclass::{self, CLASS_STANDARD, STANDARD, lookup_config_for_pools_without_env};
+    use arc_swap::ArcSwap;
+    use rustfs_config::server_config::KVS;
+    use std::io::{self, Write};
+    use std::sync::{Arc, Mutex};
+    use tracing_subscriber::fmt::MakeWriter;
+
+    #[derive(Clone, Default)]
+    struct CapturedLogs {
+        output: Arc<Mutex<Vec<u8>>>,
+        published: Option<Arc<ArcSwap<storageclass::Config>>>,
+    }
+
+    struct CapturedWriter(CapturedLogs);
+
+    impl Write for CapturedWriter {
+        fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+            if let Some(published) = &self.0.published {
+                assert_eq!(
+                    published.load_full().parities_for_sc(STANDARD),
+                    Some(vec![2, 0]),
+                    "warning must be emitted after the new snapshot is visible"
+                );
+            }
+            self.0.output.lock().expect("log buffer lock poisoned").extend_from_slice(buf);
+            Ok(buf.len())
+        }
+
+        fn flush(&mut self) -> io::Result<()> {
+            Ok(())
+        }
+    }
+
+    impl<'a> MakeWriter<'a> for CapturedLogs {
+        type Writer = CapturedWriter;
+
+        fn make_writer(&'a self) -> Self::Writer {
+            CapturedWriter(self.clone())
+        }
+    }
+
+    impl CapturedLogs {
+        fn observing(published: Arc<ArcSwap<storageclass::Config>>) -> Self {
+            Self {
+                published: Some(published),
+                ..Default::default()
+            }
+        }
+
+        fn output(&self) -> String {
+            String::from_utf8(self.output.lock().expect("log buffer lock poisoned").clone()).expect("logs should be UTF-8")
+        }
+    }
+
+    #[test]
+    fn automatic_single_disk_publish_emits_structured_warning() {
+        let cfg = lookup_config_for_pools_without_env(&KVS::new(), &[4, 1]).expect("automatic config should resolve");
+        let target = Arc::new(ArcSwap::from_pointee(Default::default()));
+        let logs = CapturedLogs::observing(target.clone());
+        let subscriber = tracing_subscriber::fmt().json().with_writer(logs.clone()).finish();
+
+        tracing::subscriber::with_default(subscriber, || publish_storage_class_config(&target, cfg));
+
+        assert_eq!(target.load_full().parities_for_sc(STANDARD), Some(vec![2, 0]));
+        let output = logs.output();
+        assert_eq!(
+            output.matches("storage_class_zero_redundancy").count(),
+            1,
+            "unexpected warning count: {output}"
+        );
+        assert!(output.contains("\"level\":\"WARN\""), "unexpected warning level: {output}");
+        assert!(output.contains("\"pool_index\":1"), "missing pool index: {output}");
+        assert!(output.contains("\"drives_per_set\":1"), "missing drive count: {output}");
+        assert!(output.contains("\"parity\":0"), "missing parity: {output}");
+    }
+
+    #[test]
+    fn explicit_zero_parity_publish_does_not_emit_automatic_warning() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:0".to_string());
+        let cfg = lookup_config_for_pools_without_env(&kvs, &[1]).expect("explicit EC:0 should resolve");
+        let target = ArcSwap::from_pointee(Default::default());
+        let logs = CapturedLogs::default();
+        let subscriber = tracing_subscriber::fmt().json().with_writer(logs.clone()).finish();
+
+        tracing::subscriber::with_default(subscriber, || publish_storage_class_config(&target, cfg));
+
+        assert_eq!(target.load_full().parities_for_sc(STANDARD), Some(vec![0]));
+        assert!(!logs.output().contains("storage_class_zero_redundancy"));
+    }
+
+    #[test]
+    fn storage_class_arc_swap_never_exposes_mixed_pool_state() {
+        let automatic = lookup_config_for_pools_without_env(&KVS::new(), &[4, 6]).expect("automatic config should resolve");
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:1".to_string());
+        let explicit = lookup_config_for_pools_without_env(&kvs, &[4, 6]).expect("explicit config should resolve");
+        let target = Arc::new(ArcSwap::from_pointee(automatic.clone()));
+        let old_snapshot = target.load_full();
+
+        let writer_target = target.clone();
+        let writer = std::thread::spawn(move || {
+            for index in 0..2_000 {
+                let next = if index % 2 == 0 { automatic.clone() } else { explicit.clone() };
+                writer_target.store(Arc::new(next));
+            }
+        });
+
+        let readers: Vec<_> = (0..4)
+            .map(|_| {
+                let reader_target = target.clone();
+                std::thread::spawn(move || {
+                    for _ in 0..2_000 {
+                        let pair = reader_target
+                            .load_full()
+                            .parities_for_sc(STANDARD)
+                            .expect("published config must have pool topology");
+                        assert!(pair == [2, 3] || pair == [1, 1], "mixed pool snapshot: {pair:?}");
+                    }
+                })
+            })
+            .collect();
+
+        writer.join().expect("writer thread should complete");
+        for reader in readers {
+            reader.join().expect("reader thread should complete");
+        }
+        assert_eq!(old_snapshot.parities_for_sc(STANDARD), Some(vec![2, 3]));
     }
 }
 

crates/ecstore/src/config/storageclass.rs

@@ -15,7 +15,7 @@
 use crate::error::{Error, Result};
 use rustfs_config::server_config::{KV, KVS};
 use serde::{Deserialize, Serialize};
-use std::env;
+use std::env::{self, VarError};
 use std::sync::LazyLock;
 use tracing::warn;
 
@@ -69,6 +69,8 @@ pub const MIN_PARITY_DRIVES: usize = 0;
 
 // Default RRS parity is always minimum parity.
 pub const DEFAULT_RRS_PARITY: usize = 1;
+const DEFAULT_RRS_STORAGE_CLASS: &str = "EC:1";
+const ZERO_SET_DRIVE_COUNT_ERROR: &str = "set drive count must be greater than zero";
 
 pub static DEFAULT_INLINE_BLOCK: usize = 128 * 1024;
 
@@ -81,7 +83,7 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
         },
         KV {
             key: CLASS_RRS.to_owned(),
-            value: "EC:1".to_owned(),
+            value: DEFAULT_RRS_STORAGE_CLASS.to_owned(),
             hidden_if_empty: false,
         },
         KV {
@@ -105,6 +107,13 @@ pub struct StorageClass {
     parity: usize,
 }
 
+#[derive(Debug, Clone)]
+struct PoolParity {
+    drives_per_set: usize,
+    parity: usize,
+    automatic: bool,
+}
+
 // Config storage class configuration
 #[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Config {
@@ -113,37 +122,89 @@ pub struct Config {
     optimize: Option<String>,
     inline_block: usize,
     initialized: bool,
+    #[serde(skip)]
+    standard_parities: Vec<PoolParity>,
+    #[serde(skip)]
+    rrs_parities: Vec<PoolParity>,
 }
 
 impl Config {
-    pub fn get_parity_for_sc(&self, sc: &str) -> Option<usize> {
+    fn storage_class(&self, sc: &str) -> (&StorageClass, &[PoolParity]) {
         match sc.trim() {
-            RRS => {
-                if self.initialized {
-                    Some(self.rrs.parity)
-                } else {
-                    None
-                }
-            }
-            // All these storage classes use standard parity configuration
-            STANDARD | DEEP_ARCHIVE | EXPRESS_ONEZONE | GLACIER | GLACIER_IR | INTELLIGENT_TIERING | ONEZONE_IA | OUTPOSTS
-            | SNOW | STANDARD_IA => {
-                if self.initialized {
-                    Some(self.standard.parity)
-                } else {
-                    None
-                }
-            }
-            _ => {
-                if self.initialized {
-                    Some(self.standard.parity)
-                } else {
-                    None
-                }
-            }
+            RRS => (&self.rrs, &self.rrs_parities),
+            _ => (&self.standard, &self.standard_parities),
         }
     }
 
+    pub(crate) fn is_initialized(&self) -> bool {
+        self.initialized
+    }
+
+    pub fn get_parity_for_sc(&self, sc: &str) -> Option<usize> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (storage_class, pools) = self.storage_class(sc);
+        let Some(first) = pools.first() else {
+            return Some(storage_class.parity);
+        };
+
+        pools.iter().all(|pool| pool.parity == first.parity).then_some(first.parity)
+    }
+
+    /// Returns the resolved parity for a pool topology.
+    ///
+    /// A topology-bound lookup fails closed for unknown drive counts and for
+    /// deserialized legacy configurations that have no pool topology. Legacy
+    /// callers retain scalar compatibility through [`Self::get_parity_for_sc`].
+    pub(crate) fn parity_for_sc(&self, sc: &str, drives_per_set: usize) -> Option<usize> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (_, pools) = self.storage_class(sc);
+
+        pools
+            .iter()
+            .find(|pool| pool.drives_per_set == drives_per_set)
+            .map(|pool| pool.parity)
+    }
+
+    pub(crate) fn parity_for_pool(&self, sc: &str, pool_index: usize, drives_per_set: usize) -> Option<usize> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (_, pools) = self.storage_class(sc);
+
+        pools
+            .get(pool_index)
+            .filter(|pool| pool.drives_per_set == drives_per_set)
+            .map(|pool| pool.parity)
+    }
+
+    pub fn parities_for_sc(&self, sc: &str) -> Option<Vec<usize>> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (_, pools) = self.storage_class(sc);
+        if pools.is_empty() {
+            return None;
+        }
+
+        Some(pools.iter().map(|pool| pool.parity).collect())
+    }
+
+    pub(crate) fn automatic_zero_parity_pools(&self) -> impl Iterator<Item = (usize, usize)> + '_ {
+        self.standard_parities
+            .iter()
+            .enumerate()
+            .filter(|(_, pool)| pool.automatic && pool.parity == 0)
+            .map(|(pool_index, pool)| (pool_index, pool.drives_per_set))
+    }
+
     pub fn should_inline(&self, shard_size: i64, versioned: bool) -> bool {
         if shard_size < 0 {
             return false;
@@ -180,74 +241,175 @@ impl Config {
     }
 }
 
-pub fn lookup_config(kvs: &KVS, set_drive_count: usize) -> Result<Config> {
-    let standard = {
-        let ssc_str = {
-            if let Ok(ssc_str) = env::var(STANDARD_ENV) {
-                ssc_str
-            } else {
-                kvs.get(CLASS_STANDARD)
-            }
-        };
+#[derive(Debug, Default)]
+struct StorageClassEnvOverrides {
+    standard: Option<String>,
+    rrs: Option<String>,
+    optimize: Option<String>,
+    inline_block: Option<String>,
+}
 
-        if !ssc_str.is_empty() {
-            parse_storage_class(&ssc_str)?
-        } else {
-            StorageClass {
-                parity: default_parity_count(set_drive_count),
-            }
-        }
-    };
+impl StorageClassEnvOverrides {
+    fn from_process() -> Result<Self> {
+        Ok(Self {
+            standard: read_optional_env(STANDARD_ENV)?,
+            rrs: read_optional_env(RRS_ENV)?,
+            optimize: read_optional_env(OPTIMIZE_ENV)?,
+            inline_block: read_optional_env(INLINE_BLOCK_ENV)?,
+        })
+    }
+}
 
-    let rrs = {
-        let ssc_str = {
-            if let Ok(ssc_str) = env::var(RRS_ENV) {
-                ssc_str
-            } else {
-                kvs.get(CLASS_RRS)
-            }
-        };
+fn read_optional_env(name: &str) -> Result<Option<String>> {
+    parse_optional_env(name, env::var(name))
+}
 
-        if !ssc_str.is_empty() {
-            parse_storage_class(&ssc_str)?
-        } else {
-            StorageClass {
-                parity: { if set_drive_count == 1 { 0 } else { DEFAULT_RRS_PARITY } },
-            }
-        }
-    };
+fn parse_optional_env(name: &str, value: std::result::Result<String, VarError>) -> Result<Option<String>> {
+    match value {
+        Ok(value) => Ok(Some(value)),
+        Err(VarError::NotPresent) => Ok(None),
+        Err(VarError::NotUnicode(_)) => Err(Error::other(format!("{name} contains non-Unicode data"))),
+    }
+}
 
-    validate_parity_inner(standard.parity, rrs.parity, set_drive_count)?;
+#[derive(Debug, Clone, Copy)]
+enum ParityPolicy {
+    AutomaticStandard,
+    LegacyRrs,
+    Explicit(usize),
+}
 
-    let optimize = { env::var(OPTIMIZE_ENV).ok() };
-
-    let inline_block = {
-        if let Ok(ev) = env::var(INLINE_BLOCK_ENV) {
-            if let Ok(block) = ev.parse::<bytesize::ByteSize>() {
-                if block.as_u64() as usize > DEFAULT_INLINE_BLOCK {
-                    warn!(
-                        "inline block value bigger than recommended max of 128KiB -> {}, performance may degrade for PUT please benchmark the changes",
-                        block
-                    );
+impl ParityPolicy {
+    fn resolve(self, drives_per_set: usize) -> usize {
+        match self {
+            Self::AutomaticStandard => default_parity_count(drives_per_set),
+            Self::LegacyRrs => {
+                if drives_per_set == 1 {
+                    0
+                } else {
+                    DEFAULT_RRS_PARITY
                 }
-                block.as_u64() as usize
-            } else {
-                return Err(Error::other(format!("parse {INLINE_BLOCK_ENV} format failed")));
             }
-        } else {
-            DEFAULT_INLINE_BLOCK
+            Self::Explicit(parity) => parity,
         }
+    }
+}
+
+fn standard_policy(kvs: &KVS, value: Option<String>) -> Result<ParityPolicy> {
+    match value {
+        Some(value) if value.is_empty() => Ok(ParityPolicy::AutomaticStandard),
+        Some(value) => Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity)),
+        None => {
+            let value = kvs.get(CLASS_STANDARD);
+            if value.is_empty() {
+                Ok(ParityPolicy::AutomaticStandard)
+            } else {
+                Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity))
+            }
+        }
+    }
+}
+
+fn rrs_policy(kvs: &KVS, value: Option<String>) -> Result<ParityPolicy> {
+    match value {
+        Some(value) if value.is_empty() => Ok(ParityPolicy::LegacyRrs),
+        Some(value) => Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity)),
+        None => {
+            let value = kvs.get(CLASS_RRS);
+            if value.is_empty() || value == DEFAULT_RRS_STORAGE_CLASS {
+                Ok(ParityPolicy::LegacyRrs)
+            } else {
+                Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity))
+            }
+        }
+    }
+}
+
+pub fn lookup_config_for_pools(kvs: &KVS, set_drive_counts: &[usize]) -> Result<Config> {
+    lookup_config_for_pools_with_env(kvs, set_drive_counts, StorageClassEnvOverrides::from_process()?)
+}
+
+fn lookup_config_for_pools_with_env(
+    kvs: &KVS,
+    set_drive_counts: &[usize],
+    overrides: StorageClassEnvOverrides,
+) -> Result<Config> {
+    if set_drive_counts.is_empty() {
+        return Err(Error::other("storage class requires at least one pool"));
+    }
+
+    let standard_policy = standard_policy(kvs, overrides.standard)?;
+    let rrs_policy = rrs_policy(kvs, overrides.rrs)?;
+    let mut standard_parities = Vec::with_capacity(set_drive_counts.len());
+    let mut rrs_parities = Vec::with_capacity(set_drive_counts.len());
+
+    for (pool_index, &drives_per_set) in set_drive_counts.iter().enumerate() {
+        let standard_parity = standard_policy.resolve(drives_per_set);
+        let rrs_parity = rrs_policy.resolve(drives_per_set);
+        validate_parity_inner(standard_parity, rrs_parity, drives_per_set).map_err(|err| {
+            Error::other(format!(
+                "storage class validation failed for pool {pool_index} ({drives_per_set} drives): {err}"
+            ))
+        })?;
+        standard_parities.push(PoolParity {
+            drives_per_set,
+            parity: standard_parity,
+            automatic: matches!(standard_policy, ParityPolicy::AutomaticStandard),
+        });
+        rrs_parities.push(PoolParity {
+            drives_per_set,
+            parity: rrs_parity,
+            automatic: matches!(rrs_policy, ParityPolicy::LegacyRrs),
+        });
+    }
+
+    let optimize = overrides.optimize;
+    let inline_block = if let Some(value) = overrides.inline_block {
+        let block = value
+            .parse::<bytesize::ByteSize>()
+            .map_err(|_| Error::other(format!("parse {INLINE_BLOCK_ENV} format failed")))?;
+        let inline_block = usize::try_from(block.as_u64())
+            .map_err(|_| Error::other(format!("{INLINE_BLOCK_ENV} exceeds the platform size limit")))?;
+        if inline_block > DEFAULT_INLINE_BLOCK {
+            warn!(
+                event = "storage_class_inline_block_large",
+                component = "ecstore",
+                subsystem = "storage_class",
+                state = "configured",
+                inline_block,
+                recommended_max = DEFAULT_INLINE_BLOCK,
+                "storage class inline block exceeds recommendation"
+            );
+        }
+        inline_block
+    } else {
+        DEFAULT_INLINE_BLOCK
     };
 
     Ok(Config {
-        standard,
-        rrs,
+        standard: StorageClass {
+            parity: standard_parities[0].parity,
+        },
+        rrs: StorageClass {
+            parity: rrs_parities[0].parity,
+        },
         optimize,
         inline_block,
         initialized: true,
+        standard_parities,
+        rrs_parities,
     })
 }
 
+pub fn lookup_config(kvs: &KVS, set_drive_count: usize) -> Result<Config> {
+    lookup_config_for_pools(kvs, &[set_drive_count])
+}
+
+#[cfg(test)]
+pub(crate) fn lookup_config_for_pools_without_env(kvs: &KVS, set_drive_counts: &[usize]) -> Result<Config> {
+    lookup_config_for_pools_with_env(kvs, set_drive_counts, StorageClassEnvOverrides::default())
+}
+
 pub fn parse_storage_class(env: &str) -> Result<StorageClass> {
     let s: Vec<&str> = env.split(':').collect();
 
@@ -281,6 +443,10 @@ pub fn validate_parity(ss_parity: usize, set_drive_count: usize) -> Result<()> {
     //     )));
     // }
 
+    if set_drive_count == 0 {
+        return Err(Error::other(ZERO_SET_DRIVE_COUNT_ERROR));
+    }
+
     if ss_parity > set_drive_count / 2 {
         return Err(Error::other(format!(
             "parity {} should be less than or equal to {}",
@@ -310,22 +476,24 @@ pub fn validate_parity_inner(ss_parity: usize, rrs_parity: usize, set_drive_coun
     //     )));
     // }
 
-    if set_drive_count > 2 {
-        if ss_parity > set_drive_count / 2 {
-            return Err(Error::other(format!(
-                "Standard storage class parity {} should be less than or equal to {}",
-                ss_parity,
-                set_drive_count / 2
-            )));
-        }
+    if set_drive_count == 0 {
+        return Err(Error::other(ZERO_SET_DRIVE_COUNT_ERROR));
+    }
 
-        if rrs_parity > set_drive_count / 2 {
-            return Err(Error::other(format!(
-                "Reduced redundancy storage class parity {} should be less than or equal to {}",
-                rrs_parity,
-                set_drive_count / 2
-            )));
-        }
+    if ss_parity > set_drive_count / 2 {
+        return Err(Error::other(format!(
+            "Standard storage class parity {} should be less than or equal to {}",
+            ss_parity,
+            set_drive_count / 2
+        )));
+    }
+
+    if rrs_parity > set_drive_count / 2 {
+        return Err(Error::other(format!(
+            "Reduced redundancy storage class parity {} should be less than or equal to {}",
+            rrs_parity,
+            set_drive_count / 2
+        )));
     }
 
     if ss_parity > 0 && rrs_parity > 0 && ss_parity < rrs_parity {
@@ -340,6 +508,171 @@ pub fn validate_parity_inner(ss_parity: usize, rrs_parity: usize, set_drive_coun
 mod tests {
     use super::*;
 
+    fn no_env_overrides() -> StorageClassEnvOverrides {
+        StorageClassEnvOverrides::default()
+    }
+
+    #[test]
+    fn automatic_parity_is_resolved_per_pool() {
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], no_env_overrides())
+            .expect("automatic storage class should support heterogeneous pools");
+
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(2));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 2), Some(1));
+        assert_eq!(cfg.parity_for_sc(RRS, 4), Some(1));
+        assert_eq!(cfg.parity_for_sc(RRS, 2), Some(1));
+        assert_eq!(cfg.get_parity_for_sc(STANDARD), None, "heterogeneous parity has no truthful scalar");
+
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[4, 6], no_env_overrides())
+            .expect("automatic parity should be valid for both pools");
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(2));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 6), Some(3));
+    }
+
+    #[test]
+    fn automatic_single_disk_pool_uses_zero_parity() {
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[4, 1], no_env_overrides())
+            .expect("automatic single-disk storage should remain supported");
+
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(2));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 1), Some(0));
+        assert_eq!(cfg.parity_for_sc(RRS, 1), Some(0));
+    }
+
+    #[test]
+    fn explicit_standard_parity_is_validated_against_every_pool() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:2".to_string());
+
+        let err = lookup_config_for_pools_with_env(&kvs, &[4, 2], no_env_overrides())
+            .expect_err("EC:2 must be rejected by the two-drive pool");
+        assert!(
+            err.to_string().contains("pool 1") && err.to_string().contains("2 drives"),
+            "error must identify the rejecting pool: {err}"
+        );
+
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:1".to_string());
+        let cfg = lookup_config_for_pools_with_env(&kvs, &[4, 2], no_env_overrides()).expect("EC:1 is valid for both pools");
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(1));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 2), Some(1));
+        assert_eq!(cfg.get_parity_for_sc(STANDARD), Some(1));
+    }
+
+    #[test]
+    fn environment_rrs_value_is_explicit_but_persisted_default_is_legacy() {
+        let persisted_default = lookup_config_for_pools_with_env(&DEFAULT_KVS, &[1], no_env_overrides())
+            .expect("persisted default RRS EC:1 should retain single-disk compatibility");
+        assert_eq!(persisted_default.parity_for_sc(RRS, 1), Some(0));
+
+        let env = StorageClassEnvOverrides {
+            rrs: Some("EC:1".to_string()),
+            ..Default::default()
+        };
+        let err = lookup_config_for_pools_with_env(&DEFAULT_KVS, &[1], env)
+            .expect_err("environment RRS EC:1 is explicit and must fail on a single disk");
+        assert!(err.to_string().contains("pool 0") && err.to_string().contains("1 drives"));
+    }
+
+    #[test]
+    fn explicit_environment_standard_parity_is_not_clamped() {
+        let env = StorageClassEnvOverrides {
+            standard: Some("EC:2".to_string()),
+            ..Default::default()
+        };
+        let err = lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], env)
+            .expect_err("explicit environment parity must not be clamped for a smaller pool");
+        assert!(err.to_string().contains("pool 1") && err.to_string().contains("2 drives"));
+    }
+
+    #[test]
+    fn empty_environment_values_retain_automatic_semantics() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:2".to_string());
+        kvs.insert(CLASS_RRS.to_string(), "EC:2".to_string());
+        let env = StorageClassEnvOverrides {
+            standard: Some(String::new()),
+            rrs: Some(String::new()),
+            ..Default::default()
+        };
+
+        let cfg = lookup_config_for_pools_with_env(&kvs, &[4, 2], env)
+            .expect("empty environment values should override persisted parity with automatic defaults");
+        assert_eq!(cfg.parities_for_sc(STANDARD), Some(vec![2, 1]));
+        assert_eq!(cfg.parities_for_sc(RRS), Some(vec![1, 1]));
+    }
+
+    #[test]
+    fn explicit_zero_parity_is_not_reported_as_automatic() {
+        let env = StorageClassEnvOverrides {
+            standard: Some("EC:0".to_string()),
+            ..Default::default()
+        };
+
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[1], env)
+            .expect("explicit zero parity should remain valid for a single-drive pool");
+        assert_eq!(cfg.parities_for_sc(STANDARD), Some(vec![0]));
+        assert_eq!(cfg.automatic_zero_parity_pools().count(), 0);
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn non_unicode_environment_value_fails_closed() {
+        use std::ffi::OsString;
+        use std::os::unix::ffi::OsStringExt;
+
+        let err = parse_optional_env(STANDARD_ENV, Err(VarError::NotUnicode(OsString::from_vec(vec![0xff]))))
+            .expect_err("non-Unicode parity must fail closed");
+        assert!(err.to_string().contains(STANDARD_ENV));
+    }
+
+    #[test]
+    fn environment_standard_override_can_rescue_persisted_config() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:2".to_string());
+        let env = StorageClassEnvOverrides {
+            standard: Some("EC:1".to_string()),
+            ..Default::default()
+        };
+
+        let cfg = lookup_config_for_pools_with_env(&kvs, &[4, 2], env)
+            .expect("environment override should replace the persisted parity before validation");
+        assert_eq!(cfg.parity_for_pool(STANDARD, 0, 4), Some(1));
+        assert_eq!(cfg.parity_for_pool(STANDARD, 1, 2), Some(1));
+    }
+
+    #[test]
+    fn invalid_or_unknown_pool_topology_fails_closed() {
+        assert!(lookup_config_for_pools_with_env(&KVS::new(), &[], no_env_overrides()).is_err());
+
+        let err = lookup_config_for_pools_with_env(&KVS::new(), &[4, 0], no_env_overrides())
+            .expect_err("zero-drive pools must be rejected");
+        assert!(err.to_string().contains("pool 1") && err.to_string().contains("0 drives"));
+
+        let cfg =
+            lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], no_env_overrides()).expect("valid topology should resolve");
+        assert_eq!(cfg.parity_for_pool(STANDARD, 2, 6), None);
+        assert_eq!(cfg.parity_for_pool(STANDARD, 1, 6), None);
+        assert_eq!(cfg.parity_for_pool(STANDARD, 0, 2), None);
+        assert_eq!(cfg.parity_for_pool(STANDARD, 1, 4), None);
+        assert_eq!(cfg.parity_for_sc(STANDARD, 6), None);
+    }
+
+    #[test]
+    fn resolved_pool_state_is_not_serialized() {
+        let cfg =
+            lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], no_env_overrides()).expect("valid topology should resolve");
+        let encoded = serde_json::to_string(&cfg).expect("config should serialize");
+        assert!(!encoded.contains("standard_parities"));
+        assert!(!encoded.contains("rrs_parities"));
+
+        let decoded: Config = serde_json::from_str(&encoded).expect("legacy scalar config should deserialize");
+        assert_eq!(decoded.get_parity_for_sc(STANDARD), Some(2));
+        assert_eq!(decoded.parity_for_pool(STANDARD, 0, 4), None);
+        assert_eq!(decoded.parity_for_sc(STANDARD, 4), None);
+        assert_eq!(decoded.parities_for_sc(STANDARD), None);
+        assert!(validate_parity(0, 0).is_err());
+    }
+
     #[test]
     fn lookup_config_reads_rrs_from_class_rrs_key() {
         // Regression: kvs.get(RRS) used RRS="REDUCED_REDUNDANCY" instead of
@@ -348,7 +681,7 @@ mod tests {
         kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
         kvs.insert(CLASS_RRS.to_string(), "EC:2".to_string());
 
-        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        let cfg = lookup_config_for_pools_without_env(&kvs, &[8]).expect("lookup should succeed");
         assert_eq!(cfg.standard.parity, 4, "standard parity should be 4");
         assert_eq!(cfg.rrs.parity, 2, "rrs parity should be 2");
     }
@@ -360,7 +693,7 @@ mod tests {
         kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
         kvs.insert(RRS.to_string(), "EC:2".to_string());
 
-        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        let cfg = lookup_config_for_pools_without_env(&kvs, &[8]).expect("lookup should succeed");
         assert_eq!(cfg.standard.parity, 4);
         assert_eq!(
             cfg.rrs.parity, DEFAULT_RRS_PARITY,

crates/ecstore/src/core/pools_test.rs

@@ -87,50 +87,57 @@ mod capacity_dedup_tests {
     }
 
     #[test]
-    fn test_four_disk_erasure_coding() {
-        // 4-disk erasure coding: 2 data disks + 2 parity disks
+    fn test_four_node_ec_2_2_reports_stable_usable_capacity() {
+        const TIB: u64 = 1 << 40;
+        const DISK_TOTAL: u64 = 2 * TIB;
+        const DISK_FREE: u64 = TIB / 5;
+
         let disks = vec![
             rustfs_madmin::Disk {
                 endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk1".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 0,
-                total_space: 1_000_000_000_000, // 1TB
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
             rustfs_madmin::Disk {
-                endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk2".to_string(),
+                endpoint: "node2".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 1,
-                total_space: 1_000_000_000_000,
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
             rustfs_madmin::Disk {
-                endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk3".to_string(),
+                endpoint: "node3".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 2,
-                total_space: 1_000_000_000_000,
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
             rustfs_madmin::Disk {
-                endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk4".to_string(),
+                endpoint: "node4".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 3,
-                total_space: 1_000_000_000_000,
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
@@ -146,9 +153,17 @@ mod capacity_dedup_tests {
         };
 
         let total = get_total_usable_capacity(&disks, &info);
+        let free = get_total_usable_capacity_free(&disks, &info);
+        let used = total.saturating_sub(free);
+        let expected_total = usize::try_from(4 * TIB).expect("4 TiB must fit the supported platform's usize");
+        let expected_free = usize::try_from(2 * DISK_FREE).expect("usable free capacity must fit usize");
+        let single_node_used = usize::try_from(DISK_TOTAL - DISK_FREE).expect("single-node used capacity must fit usize");
 
-        // Only count data disks (disk_index < 2)
-        assert_eq!(total, 2_000_000_000_000, "Should count only data disks (2 × 1TB)");
+        assert_eq!(total, expected_total, "usable total must count the two data disks");
+        assert_eq!(free, expected_free, "usable free must count the two data disks");
+        assert_eq!(used, 2 * single_node_used, "usable used must be approximately 3.6 TiB");
+        assert_ne!(used, single_node_used, "used must not collapse to one node's approximately 1.8 TiB");
+        assert_ne!(used, 4 * single_node_used, "used must not report the approximately 7.2 TiB raw aggregate");
     }
 
     #[test]

crates/ecstore/src/core/sets.rs

@@ -35,7 +35,7 @@ use crate::{
     object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader},
     runtime::instance::{InstanceContext, bootstrap_ctx},
     runtime::sources as runtime_sources,
-    set_disk::SetDisks,
+    set_disk::{PreparedGetObjectMetadata, SetDisks},
     store::init_format::{check_format_erasure_values, get_format_erasure_in_quorum, load_format_erasure_all, save_format_file},
 };
 use futures::{
@@ -403,6 +403,31 @@ impl crate::storage_api_contracts::object::ObjectIO for Sets {
 }
 
 impl Sets {
+    pub(crate) async fn prepare_get_object_reader_metadata(
+        &self,
+        bucket: &str,
+        object: &str,
+        opts: &ObjectOptions,
+    ) -> Result<PreparedGetObjectMetadata> {
+        self.get_disks_by_key(object)
+            .prepare_get_object_metadata(bucket, object, opts)
+            .await
+    }
+
+    pub(crate) async fn get_object_reader_with_prepared_metadata(
+        &self,
+        bucket: &str,
+        object: &str,
+        range: Option<HTTPRangeSpec>,
+        headers: HeaderMap,
+        opts: &ObjectOptions,
+        metadata: PreparedGetObjectMetadata,
+    ) -> Result<GetObjectReader> {
+        self.get_disks_by_key(object)
+            .get_object_reader_with_prepared_metadata(bucket, object, range, headers, opts, metadata)
+            .await
+    }
+
     /// `put_object` plus the rename_data old-size backfill
     /// (rustfs/backlog#1009); see `SetDisks::put_object_with_old_current_size`.
     pub async fn put_object_with_old_current_size(

crates/ecstore/src/data_usage/mod.rs

@@ -24,7 +24,7 @@ use crate::storage_api_contracts::{
 };
 use crate::{
     bucket::{metadata_sys::get_replication_config, versioning::VersioningApi as _, versioning_sys::BucketVersioningSys},
-    config::com::read_config,
+    config::com::{read_config, read_config_preserve_empty},
     disk::DiskAPI,
     error::{Error, classify_system_path_failure_reason},
     object_api::ObjectInfo,
@@ -41,7 +41,10 @@ use rustfs_utils::path::SLASH_SEPARATOR;
 use std::{
     collections::{HashMap, HashSet, hash_map::Entry},
     future::Future,
-    sync::{Arc, LazyLock, OnceLock},
+    sync::{
+        Arc, LazyLock, OnceLock,
+        atomic::{AtomicU64, Ordering},
+    },
     time::{Duration, SystemTime},
 };
 use tokio::fs;
@@ -76,6 +79,67 @@ static USAGE_MEMORY_CACHE: OnceLock<UsageMemoryCache> = OnceLock::new();
 static USAGE_CACHE_UPDATING: OnceLock<CacheUpdating> = OnceLock::new();
 static LIVE_BUCKET_USAGE_CACHE: OnceLock<LiveBucketUsageCache> = OnceLock::new();
 
+/// Cached copy of the last persisted data usage snapshot, served to admin
+/// endpoints for up to `DATA_USAGE_CACHE_TTL_SECS` between backend reads.
+#[derive(Debug, Clone)]
+struct CachedDataUsageSnapshot {
+    info: Option<DataUsageInfo>,
+    loaded_at: tokio::time::Instant,
+}
+
+impl CachedDataUsageSnapshot {
+    fn result(&self) -> Result<DataUsageInfo, Error> {
+        self.info
+            .clone()
+            .ok_or_else(|| Error::other("data usage snapshot load recently failed"))
+    }
+}
+
+fn fresh_cached_data_usage_snapshot(
+    cache: &Option<CachedDataUsageSnapshot>,
+    now: tokio::time::Instant,
+    ttl: Duration,
+) -> Option<Result<DataUsageInfo, Error>> {
+    cache
+        .as_ref()
+        .filter(|cached| now.duration_since(cached.loaded_at) < ttl)
+        .map(CachedDataUsageSnapshot::result)
+}
+
+fn cache_data_usage_snapshot_result(
+    cache: &mut Option<CachedDataUsageSnapshot>,
+    result: Result<DataUsageInfo, Error>,
+    loaded_at: tokio::time::Instant,
+) -> Result<DataUsageInfo, Error> {
+    match result {
+        Ok(info) => {
+            *cache = Some(CachedDataUsageSnapshot {
+                info: Some(info.clone()),
+                loaded_at,
+            });
+            Ok(info)
+        }
+        Err(e) => {
+            *cache = Some(CachedDataUsageSnapshot { info: None, loaded_at });
+            Err(e)
+        }
+    }
+}
+
+type DataUsageSnapshotCache = Arc<RwLock<Option<CachedDataUsageSnapshot>>>;
+
+static DATA_USAGE_SNAPSHOT_CACHE: OnceLock<DataUsageSnapshotCache> = OnceLock::new();
+
+// Always-on revert detector for rustfs/backlog#1306: one relaxed increment per
+// full-bucket version listing is negligible and lets tests prove that admin
+// request paths never trigger live listings.
+static LIVE_BUCKET_USAGE_COMPUTATIONS: AtomicU64 = AtomicU64::new(0);
+
+/// Number of live full-bucket usage computations performed by this process.
+pub fn live_bucket_usage_computations() -> u64 {
+    LIVE_BUCKET_USAGE_COMPUTATIONS.load(Ordering::Relaxed)
+}
+
 /// Deferred persist thresholds for compression totals: persist after this many
 /// operations recorded, but no more often than the min interval.
 const COMPRESSION_PERSIST_BATCH_SIZE: u64 = 100;
@@ -115,6 +179,10 @@ fn cache_updating() -> &'static CacheUpdating {
     USAGE_CACHE_UPDATING.get_or_init(|| Arc::new(RwLock::new(false)))
 }
 
+fn data_usage_snapshot_cache() -> &'static DataUsageSnapshotCache {
+    DATA_USAGE_SNAPSHOT_CACHE.get_or_init(|| Arc::new(RwLock::new(None)))
+}
+
 fn live_bucket_usage_cache() -> &'static LiveBucketUsageCache {
     LIVE_BUCKET_USAGE_CACHE.get_or_init(|| {
         moka::future::Cache::builder()
@@ -135,6 +203,7 @@ lazy_static::lazy_static! {
         SLASH_SEPARATOR,
         DATA_USAGE_OBJ_NAME
     );
+    static ref DATA_USAGE_OBJ_BACKUP_PATH: String = format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str());
     pub static ref DATA_USAGE_BLOOM_NAME_PATH: String = format!("{}{}{}",
         crate::disk::BUCKET_META_PREFIX,
         SLASH_SEPARATOR,
@@ -147,18 +216,56 @@ lazy_static::lazy_static! {
     );
 }
 
+/// Decide whether an incoming usage snapshot must be skipped as stale, given the local
+/// wall clock `now`. Mirrors `stale_data_usage_update_reason` in
+/// `crates/scanner/src/scanner.rs` — keep the two consistent.
+///
+/// If the persisted `existing.last_update` is future-dated beyond
+/// [`rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE`] (clock step-back or a
+/// slower-clock scanner leader), it is untrustworthy: the save is allowed so usage
+/// stats cannot freeze forever.
+fn stale_data_usage_persist_reason(incoming: &DataUsageInfo, existing: &DataUsageInfo, now: SystemTime) -> Option<&'static str> {
+    match (incoming.last_update, existing.last_update) {
+        (Some(new_ts), Some(existing_ts))
+            if new_ts <= existing_ts && !rustfs_data_usage::usage_last_update_is_untrusted_future(existing_ts, now) =>
+        {
+            Some("older_or_equal_last_update")
+        }
+        _ => None,
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum UsageSnapshotSource {
+    Primary,
+    Backup,
+    Missing,
+}
+
+fn stale_data_usage_persist_reason_for_source(
+    incoming: &DataUsageInfo,
+    existing: &DataUsageInfo,
+    source: UsageSnapshotSource,
+    now: SystemTime,
+) -> Option<&'static str> {
+    let reason = stale_data_usage_persist_reason(incoming, existing, now);
+    if source == UsageSnapshotSource::Backup && incoming.last_update == existing.last_update {
+        None
+    } else {
+        reason
+    }
+}
+
 /// Store data usage info to backend storage
 #[instrument(skip(store))]
 pub async fn store_data_usage_in_backend(data_usage_info: DataUsageInfo, store: Arc<ECStore>) -> Result<(), Error> {
     // Prevent older data from overwriting newer persisted stats
-    if let Ok(buf) = read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await
-        && let Ok(existing) = serde_json::from_slice::<DataUsageInfo>(&buf)
-        && let (Some(new_ts), Some(existing_ts)) = (data_usage_info.last_update, existing.last_update)
-        && new_ts <= existing_ts
+    if let Ok((existing, source)) = load_data_usage_snapshot(store.clone()).await
+        && let Some(reason) = stale_data_usage_persist_reason_for_source(&data_usage_info, &existing, source, SystemTime::now())
     {
         info!(
-            "Skip persisting data usage: incoming last_update {:?} <= existing {:?}",
-            new_ts, existing_ts
+            "Skip persisting data usage ({reason}): incoming last_update {:?} <= existing {:?}",
+            data_usage_info.last_update, existing.last_update
         );
         return Ok(());
     }
@@ -175,6 +282,12 @@ async fn save_data_usage_in_backend(data_usage_info: DataUsageInfo, store: Arc<E
         .await
         .map_err(Error::other)?;
 
+    // Invalidate the cached snapshot so readers observe the new save on their
+    // next request instead of waiting out the remaining TTL. The next cached
+    // read reloads through `load_data_usage_from_backend`, keeping its
+    // backward-compatibility post-processing.
+    *data_usage_snapshot_cache().write().await = None;
+
     Ok(())
 }
 
@@ -224,8 +337,8 @@ pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str)
     live_bucket_usage_cache().invalidate(bucket).await;
     clear_bucket_usage_memory(bucket).await;
 
-    let data_usage_info = load_data_usage_from_backend(store.clone()).await?;
-    let existing = load_data_usage_from_backend(store.clone()).await.ok();
+    let data_usage_info = load_primary_data_usage_from_backend(store.clone()).await?;
+    let existing = load_primary_data_usage_from_backend(store.clone()).await.ok();
 
     if let Some(data_usage_info) = merge_bucket_usage_removal(data_usage_info, existing, bucket) {
         save_data_usage_in_backend(data_usage_info, store).await?;
@@ -234,47 +347,123 @@ pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str)
     Ok(())
 }
 
+fn record_usage_snapshot_failure(operation: &'static str, object: &str, e: &Error) {
+    let reason = classify_system_path_failure_reason(e);
+    record_system_path_failure("data_usage", operation, reason);
+    error!(
+        event = "data_usage_snapshot_load_failed",
+        component = "ecstore",
+        subsystem = "data_usage",
+        state = "read_failed",
+        operation,
+        reason,
+        object = %object,
+        error = %e,
+        "data usage snapshot load failed"
+    );
+}
+
+fn record_usage_snapshot_decode_failure(operation: &'static str, object: &str, e: &Error) {
+    const REASON: &str = "decode_error";
+    record_system_path_failure("data_usage", operation, REASON);
+    error!(
+        event = "data_usage_snapshot_load_failed",
+        component = "ecstore",
+        subsystem = "data_usage",
+        state = "decode_failed",
+        operation,
+        reason = REASON,
+        object = %object,
+        error = %e,
+        "data usage snapshot load failed"
+    );
+}
+
+fn parse_usage_snapshot(buf: &[u8]) -> Result<DataUsageInfo, Error> {
+    serde_json::from_slice(buf).map_err(|e| {
+        Error::other(format!(
+            "Failed to deserialize data usage info: {:?} at line {} column {}",
+            e.classify(),
+            e.line(),
+            e.column()
+        ))
+    })
+}
+
+/// Decide which usage snapshot to serve: the primary `.usage.json` or its
+/// `.bkp` backup. The backup future is polled only when the primary is unusable,
+/// so the healthy path performs a single read.
+///
+/// `Ok(DataUsageInfo::default())` is returned only when BOTH the primary and
+/// the backup are `ConfigNotFound` — a genuine "no snapshot yet". A real
+/// primary failure with a missing backup propagates as an error instead of
+/// being rendered as confirmed all-zero stats.
+async fn resolve_loaded_snapshot_with_source(
+    primary: Result<Vec<u8>, Error>,
+    backup: impl Future<Output = Result<Vec<u8>, Error>>,
+) -> Result<(DataUsageInfo, UsageSnapshotSource), Error> {
+    let primary_failure = match primary {
+        Ok(buf) => match parse_usage_snapshot(&buf) {
+            Ok(info) => return Ok((info, UsageSnapshotSource::Primary)),
+            Err(e) => {
+                record_usage_snapshot_decode_failure("parse_primary", DATA_USAGE_OBJ_NAME_PATH.as_str(), &e);
+                e
+            }
+        },
+        Err(e) => {
+            if e != Error::ConfigNotFound {
+                record_usage_snapshot_failure("read_primary", DATA_USAGE_OBJ_NAME_PATH.as_str(), &e);
+            }
+            e
+        }
+    };
+    match backup.await {
+        Ok(buf) => match parse_usage_snapshot(&buf) {
+            Ok(info) => Ok((info, UsageSnapshotSource::Backup)),
+            Err(e) => {
+                record_usage_snapshot_decode_failure("parse_backup", &DATA_USAGE_OBJ_BACKUP_PATH, &e);
+                Err(e)
+            }
+        },
+        Err(Error::ConfigNotFound) if primary_failure == Error::ConfigNotFound => {
+            Ok((DataUsageInfo::default(), UsageSnapshotSource::Missing))
+        }
+        Err(Error::ConfigNotFound) => Err(primary_failure),
+        Err(e) => {
+            record_usage_snapshot_failure("read_backup", &DATA_USAGE_OBJ_BACKUP_PATH, &e);
+            Err(e)
+        }
+    }
+}
+
+async fn resolve_loaded_snapshot(
+    primary: Result<Vec<u8>, Error>,
+    backup: impl Future<Output = Result<Vec<u8>, Error>>,
+) -> Result<DataUsageInfo, Error> {
+    Ok(resolve_loaded_snapshot_with_source(primary, backup).await?.0)
+}
+
+async fn load_data_usage_snapshot(store: Arc<ECStore>) -> Result<(DataUsageInfo, UsageSnapshotSource), Error> {
+    let primary = read_config_preserve_empty(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await;
+    resolve_loaded_snapshot_with_source(
+        primary,
+        async move { read_config_preserve_empty(store, &DATA_USAGE_OBJ_BACKUP_PATH).await },
+    )
+    .await
+}
+
+async fn load_primary_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
+    let buf = read_config_preserve_empty(store, &DATA_USAGE_OBJ_NAME_PATH).await?;
+    Ok(normalize_loaded_data_usage(parse_usage_snapshot(&buf)?).await)
+}
+
 /// Load data usage info from backend storage
 #[instrument(skip(store))]
 pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
-    let buf: Vec<u8> = match read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await {
-        Ok(data) => data,
-        Err(e) => {
-            let reason = classify_system_path_failure_reason(&e);
-            record_system_path_failure("data_usage", "read_primary", reason);
-            error!(
-                path_kind = "data_usage",
-                operation = "read_primary",
-                reason,
-                object = %DATA_USAGE_OBJ_NAME_PATH.as_str(),
-                error = %e,
-                "system path read failed"
-            );
-
-            match read_config(store.clone(), format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()).as_str()).await {
-                Ok(data) => data,
-                Err(e) => {
-                    if e == Error::ConfigNotFound {
-                        return Ok(DataUsageInfo::default());
-                    }
-                    let reason = classify_system_path_failure_reason(&e);
-                    record_system_path_failure("data_usage", "read_backup", reason);
-                    error!(
-                        path_kind = "data_usage",
-                        operation = "read_backup",
-                        reason,
-                        object = %format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()),
-                        error = %e,
-                        "system path read failed"
-                    );
-                    return Err(Error::other(e));
-                }
-            }
-        }
-    };
-    let mut data_usage_info: DataUsageInfo =
-        serde_json::from_slice(&buf).map_err(|e| Error::other(format!("Failed to deserialize data usage info: {e}")))?;
+    Ok(normalize_loaded_data_usage(load_data_usage_snapshot(store).await?.0).await)
+}
 
+async fn normalize_loaded_data_usage(mut data_usage_info: DataUsageInfo) -> DataUsageInfo {
     info!("Loaded data usage info from backend with {} buckets", data_usage_info.buckets_count);
 
     // Handle backward compatibility
@@ -325,7 +514,34 @@ pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsa
         }
     }
 
-    Ok(data_usage_info)
+    data_usage_info
+}
+
+/// Load the persisted data usage snapshot through a small in-process cache.
+///
+/// Admin read endpoints call this on every request; the cache bounds backend
+/// reads (and the associated JSON parse and INFO log) to once per
+/// `DATA_USAGE_CACHE_TTL_SECS` per process. `save_data_usage_in_backend`
+/// invalidates the cache so a fresh scanner save is visible immediately.
+pub async fn load_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
+    let ttl = Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
+
+    {
+        let cache = data_usage_snapshot_cache().read().await;
+        if let Some(result) = fresh_cached_data_usage_snapshot(&cache, tokio::time::Instant::now(), ttl) {
+            return result;
+        }
+    }
+
+    // Re-check under the write lock so concurrent expirations trigger a single
+    // backend read instead of a stampede.
+    let mut cache = data_usage_snapshot_cache().write().await;
+    if let Some(result) = fresh_cached_data_usage_snapshot(&cache, tokio::time::Instant::now(), ttl) {
+        return result;
+    }
+
+    let result = load_data_usage_from_backend(store).await;
+    cache_data_usage_snapshot_result(&mut cache, result, tokio::time::Instant::now())
 }
 
 /// Aggregate usage information from local disk snapshots.
@@ -530,6 +746,7 @@ impl BucketUsageAccumulator {
 type UsageVersionPage = StorageListObjectVersionsInfo<ObjectInfo>;
 
 pub async fn compute_bucket_usage(store: Arc<ECStore>, bucket_name: &str) -> Result<BucketUsageInfo, Error> {
+    LIVE_BUCKET_USAGE_COMPUTATIONS.fetch_add(1, Ordering::Relaxed);
     let bucket = bucket_name.to_string();
     compute_bucket_usage_with_pages(bucket_name, move |marker, version_marker| {
         let store = Arc::clone(&store);
@@ -923,7 +1140,7 @@ async fn update_usage_cache_if_needed() {
         let updating_clone = (*cache_updating()).clone();
         tokio::spawn(async move {
             if let Some(store) = runtime_sources::object_store_handle()
-                && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
+                && let Ok(data_usage_info) = load_data_usage_from_backend(store).await
             {
                 replace_bucket_usage_memory_from_info(&data_usage_info).await;
             }
@@ -947,7 +1164,7 @@ async fn update_usage_cache_if_needed() {
     drop(updating);
 
     if let Some(store) = runtime_sources::object_store_handle()
-        && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
+        && let Ok(data_usage_info) = load_data_usage_from_backend(store).await
     {
         replace_bucket_usage_memory_from_info(&data_usage_info).await;
     }
@@ -1372,6 +1589,250 @@ mod tests {
         info
     }
 
+    fn usage_with_last_update(last_update: Option<SystemTime>) -> DataUsageInfo {
+        DataUsageInfo {
+            last_update,
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_allows_newer_incoming() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let incoming = usage_with_last_update(Some(now));
+        let existing = usage_with_last_update(Some(now - Duration::from_secs(60)));
+        assert_eq!(stale_data_usage_persist_reason(&incoming, &existing, now), None);
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_skips_older_or_equal_incoming() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing = usage_with_last_update(Some(now - Duration::from_secs(60)));
+
+        let older = usage_with_last_update(Some(now - Duration::from_secs(120)));
+        assert_eq!(
+            stale_data_usage_persist_reason(&older, &existing, now),
+            Some("older_or_equal_last_update")
+        );
+
+        let equal = usage_with_last_update(existing.last_update);
+        assert_eq!(
+            stale_data_usage_persist_reason(&equal, &existing, now),
+            Some("older_or_equal_last_update")
+        );
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_allows_save_when_existing_is_future_dated() {
+        // Existing snapshot timestamp beyond the clock tolerance is untrustworthy
+        // (clock step-back / slower-clock leader): the save must be allowed even
+        // though incoming <= existing, otherwise usage stats freeze forever.
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing =
+            usage_with_last_update(Some(now + rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE + Duration::from_secs(1)));
+        let incoming = usage_with_last_update(Some(now));
+        assert_eq!(stale_data_usage_persist_reason(&incoming, &existing, now), None);
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_skips_at_exact_tolerance_boundary() {
+        // Exactly at now + tolerance is still within the trusted window.
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing = usage_with_last_update(Some(now + rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE));
+        let incoming = usage_with_last_update(Some(now));
+        assert_eq!(
+            stale_data_usage_persist_reason(&incoming, &existing, now),
+            Some("older_or_equal_last_update")
+        );
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_preserves_none_handling() {
+        // This call site (unlike the scanner sibling) allows saves when either
+        // timestamp is missing — pin that behavior.
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+
+        let incoming_none = usage_with_last_update(None);
+        let existing_some = usage_with_last_update(Some(now - Duration::from_secs(60)));
+        assert_eq!(stale_data_usage_persist_reason(&incoming_none, &existing_some, now), None);
+
+        let incoming_some = usage_with_last_update(Some(now));
+        let existing_none = usage_with_last_update(None);
+        assert_eq!(stale_data_usage_persist_reason(&incoming_some, &existing_none, now), None);
+
+        let both_none = usage_with_last_update(None);
+        assert_eq!(stale_data_usage_persist_reason(&both_none, &usage_with_last_update(None), now), None);
+    }
+
+    #[test]
+    fn backup_stale_guard_allows_equal_snapshot_to_repair_primary() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing = usage_with_last_update(Some(now - Duration::from_secs(60)));
+        let equal = usage_with_last_update(existing.last_update);
+        let older = usage_with_last_update(Some(now - Duration::from_secs(120)));
+
+        assert_eq!(
+            stale_data_usage_persist_reason_for_source(&equal, &existing, UsageSnapshotSource::Backup, now),
+            None
+        );
+        assert_eq!(
+            stale_data_usage_persist_reason_for_source(&older, &existing, UsageSnapshotSource::Backup, now),
+            Some("older_or_equal_last_update")
+        );
+    }
+
+    fn snapshot_bytes(bucket: &str) -> Vec<u8> {
+        let info = data_usage_info_for_test(bucket, 3, 42, SystemTime::UNIX_EPOCH);
+        serde_json::to_vec(&info).expect("test snapshot must serialize")
+    }
+
+    #[test]
+    fn data_usage_backup_path_appends_suffix_to_primary() {
+        assert_eq!(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()));
+    }
+
+    fn assert_snapshot_bucket(info: &DataUsageInfo, bucket: &str) {
+        assert!(
+            info.buckets_usage.contains_key(bucket),
+            "expected snapshot for bucket {bucket}, got buckets {:?}",
+            info.buckets_usage.keys().collect::<Vec<_>>()
+        );
+    }
+
+    #[test]
+    fn parse_snapshot_error_does_not_include_payload_value() {
+        let err =
+            parse_usage_snapshot(br#"{"buckets_count":"secret-marker"}"#).expect_err("an invalid field type must fail decoding");
+        assert!(!err.to_string().contains("secret-marker"));
+    }
+
+    #[test]
+    fn cached_snapshot_failure_is_reused_until_ttl_expires() {
+        let loaded_at = tokio::time::Instant::now();
+        let mut cache = None;
+
+        let first = cache_data_usage_snapshot_result(&mut cache, Err(Error::ErasureReadQuorum), loaded_at);
+        assert!(matches!(first, Err(Error::ErasureReadQuorum)));
+
+        let cached = fresh_cached_data_usage_snapshot(&cache, loaded_at + Duration::from_secs(1), Duration::from_secs(30))
+            .expect("the failed load must remain cached within the TTL");
+        assert!(cached.is_err());
+
+        assert!(fresh_cached_data_usage_snapshot(&cache, loaded_at + Duration::from_secs(30), Duration::from_secs(30)).is_none());
+    }
+
+    #[test]
+    fn cached_snapshot_success_is_reused_until_ttl_expires() {
+        let loaded_at = tokio::time::Instant::now();
+        let expected = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH);
+        let mut cache = None;
+
+        let first =
+            cache_data_usage_snapshot_result(&mut cache, Ok(expected), loaded_at).expect("successful load must be returned");
+        assert_snapshot_bucket(&first, "bucket");
+
+        let cached = fresh_cached_data_usage_snapshot(&cache, loaded_at + Duration::from_secs(1), Duration::from_secs(30))
+            .expect("successful load must remain cached within the TTL")
+            .expect("cached success must remain successful");
+        assert_snapshot_bucket(&cached, "bucket");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_ok_is_used_without_backup_read() {
+        let backup_read = Arc::new(std::sync::atomic::AtomicBool::new(false));
+        let backup_read_probe = Arc::clone(&backup_read);
+
+        let info = resolve_loaded_snapshot(Ok(snapshot_bytes("primary-bucket")), async move {
+            backup_read_probe.store(true, std::sync::atomic::Ordering::SeqCst);
+            Ok(snapshot_bytes("backup-bucket"))
+        })
+        .await
+        .expect("healthy primary snapshot must load");
+
+        assert_snapshot_bucket(&info, "primary-bucket");
+        assert!(
+            !backup_read.load(std::sync::atomic::Ordering::SeqCst),
+            "backup must not be read when the primary parses"
+        );
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_corrupt_falls_back_to_backup() {
+        let (info, source) =
+            resolve_loaded_snapshot_with_source(Ok(b"{not json".to_vec()), async { Ok(snapshot_bytes("backup-bucket")) })
+                .await
+                .expect("backup snapshot must be served when the primary is corrupt");
+
+        assert_snapshot_bucket(&info, "backup-bucket");
+        assert_eq!(source, UsageSnapshotSource::Backup);
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_corrupt_backup_corrupt_is_error() {
+        let err = resolve_loaded_snapshot(Ok(b"{not json".to_vec()), async { Ok(b"also not json".to_vec()) })
+            .await
+            .expect_err("two corrupt snapshots must not produce stats");
+        assert!(err.to_string().contains("deserialize"), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_corrupt_backup_missing_is_error() {
+        let err = resolve_loaded_snapshot(Ok(b"{not json".to_vec()), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect_err("a corrupt primary without a backup must not produce stats");
+        assert!(err.to_string().contains("deserialize"), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_empty_backup_missing_is_error() {
+        let err = resolve_loaded_snapshot(Ok(Vec::new()), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect_err("an empty primary object is corrupt, not an absent snapshot");
+        assert!(err.to_string().contains("deserialize"), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_read_error_falls_back_to_backup() {
+        let info = resolve_loaded_snapshot(Err(Error::ErasureReadQuorum), async { Ok(snapshot_bytes("backup-bucket")) })
+            .await
+            .expect("backup snapshot must be served when the primary read fails");
+
+        assert_snapshot_bucket(&info, "backup-bucket");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_real_error_backup_missing_is_error_not_zeros() {
+        let err = resolve_loaded_snapshot(Err(Error::ErasureReadQuorum), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect_err("primary corruption must not be rendered as all-zero stats");
+        assert!(matches!(err, Error::ErasureReadQuorum), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_missing_uses_backup() {
+        let info = resolve_loaded_snapshot(Err(Error::ConfigNotFound), async { Ok(snapshot_bytes("backup-bucket")) })
+            .await
+            .expect("an existing backup must be used when the primary is missing");
+        assert_snapshot_bucket(&info, "backup-bucket");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_backup_read_error_is_preserved() {
+        let err = resolve_loaded_snapshot(Err(Error::ConfigNotFound), async { Err(Error::ErasureReadQuorum) })
+            .await
+            .expect_err("a backup read failure must be returned");
+        assert!(matches!(err, Error::ErasureReadQuorum), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_both_missing_is_default() {
+        let info = resolve_loaded_snapshot(Err(Error::ConfigNotFound), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect("no snapshot yet must resolve to empty stats");
+        assert_eq!(info.buckets_count, 0);
+        assert!(info.buckets_usage.is_empty());
+    }
+
     #[tokio::test]
     async fn compute_usage_preserves_same_object_across_1000_entry_page_boundary() {
         let first_page = UsageVersionPage {

crates/ecstore/src/diagnostics/admin_server_info.rs

@@ -13,9 +13,13 @@
 // limitations under the License.
 
 use crate::cluster::rpc::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
-use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend};
+use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
 use crate::error::{Error, Result};
-use crate::{disk::endpoint::Endpoint, runtime::sources as runtime_sources};
+use crate::{
+    disk::endpoint::{Endpoint, EndpointType},
+    layout::endpoints::EndpointServerPools,
+    runtime::sources as runtime_sources,
+};
 
 use crate::data_usage::load_data_usage_cache;
 use crate::storage_api_contracts::admin::StorageAdminApi;
@@ -42,6 +46,33 @@ use shadow_rs::shadow;
 shadow!(build);
 
 const SERVER_PING_TIMEOUT: Duration = Duration::from_secs(1);
+const DATA_USAGE_UNAVAILABLE_ERROR: &str = "data usage snapshot unavailable";
+
+fn apply_data_usage_result(
+    result: Result<rustfs_data_usage::DataUsageInfo>,
+    buckets: &mut rustfs_madmin::Buckets,
+    objects: &mut rustfs_madmin::Objects,
+    versions: &mut rustfs_madmin::Versions,
+    delete_markers: &mut rustfs_madmin::DeleteMarkers,
+    usage: &mut rustfs_madmin::Usage,
+) {
+    match result {
+        Ok(info) => {
+            buckets.count = info.buckets_count;
+            objects.count = info.objects_total_count;
+            versions.count = info.versions_total_count;
+            delete_markers.count = info.delete_markers_total_count;
+            usage.size = info.objects_total_size;
+        }
+        Err(_) => {
+            buckets.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            objects.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            versions.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            delete_markers.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            usage.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+        }
+    }
+}
 
 // pub const ITEM_OFFLINE: &str = "offline";
 // pub const ITEM_INITIALIZING: &str = "initializing";
@@ -246,22 +277,14 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
 
     if let Some(store) = runtime_sources::object_store_handle() {
         mode = ITEM_ONLINE;
-        match load_data_usage_from_backend(store.clone()).await {
-            Ok(res) => {
-                buckets.count = res.buckets_count;
-                objects.count = res.objects_total_count;
-                versions.count = res.versions_total_count;
-                delete_markers.count = res.delete_markers_total_count;
-                usage.size = res.objects_total_size;
-            }
-            Err(err) => {
-                buckets.error = Some(err.to_string());
-                objects.error = Some(err.to_string());
-                versions.error = Some(err.to_string());
-                delete_markers.error = Some(err.to_string());
-                usage.error = Some(err.to_string());
-            }
-        }
+        apply_data_usage_result(
+            load_data_usage_from_backend_cached(store.clone()).await,
+            &mut buckets,
+            &mut objects,
+            &mut versions,
+            &mut delete_markers,
+            &mut usage,
+        );
 
         let after3 = OffsetDateTime::now_utc();
 
@@ -272,6 +295,20 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
         let after4 = OffsetDateTime::now_utc();
 
         warn!("backend_info end {:?}", after4 - after3);
+        if let Some(endpoints) = runtime_sources::endpoint_pools() {
+            let (added, report) = reconcile_servers_with_endpoint_topology(&mut servers, &endpoints);
+            if added > 0 || !report.is_complete() {
+                warn!(
+                    event = "admin_v3_info_topology_incomplete",
+                    synthesized_servers = added,
+                    expected_drives = report.expected_drives,
+                    observed_drives = report.observed_drives,
+                    missing_drives = report.missing_drive_ids.len(),
+                    duplicate_drives = report.duplicate_drive_ids.len(),
+                    "admin v3 server_info reconciled endpoint topology before computing backend counters"
+                );
+            }
+        }
 
         let mut all_disks: Vec<Disk> = Vec::new();
         for server in servers.iter() {
@@ -378,6 +415,228 @@ fn get_online_offline_disks_stats(disks_info: &[Disk]) -> (BackendDisks, Backend
     (BackendDisks(online_disks), BackendDisks(offline_disks), BackendDisks(unknown_disks))
 }
 
+#[derive(Debug, Default, PartialEq, Eq)]
+struct TopologyCompletenessReport {
+    expected_drives: usize,
+    observed_drives: usize,
+    missing_drive_ids: Vec<TopologyDriveKey>,
+    duplicate_drive_ids: Vec<TopologyDriveKey>,
+}
+
+impl TopologyCompletenessReport {
+    fn is_complete(&self) -> bool {
+        self.expected_drives == self.observed_drives && self.missing_drive_ids.is_empty() && self.duplicate_drive_ids.is_empty()
+    }
+}
+
+#[derive(Debug)]
+struct TopologyMember {
+    display_endpoint: String,
+    disks: Vec<Disk>,
+    drive_keys: Vec<TopologyDriveKey>,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
+struct TopologyDriveKey {
+    pool_index: i32,
+    set_index: i32,
+    disk_index: i32,
+    member_index: usize,
+}
+
+#[derive(Debug, Default)]
+struct EndpointTopology {
+    members: Vec<TopologyMember>,
+    aliases: HashMap<String, usize>,
+    expected_drive_ids: HashSet<TopologyDriveKey>,
+}
+
+impl EndpointTopology {
+    fn from_endpoint_pools(endpoints: &EndpointServerPools) -> Self {
+        let mut topology = Self::default();
+        let mut by_host_port = HashMap::new();
+        let mut display_aliases: HashMap<String, Option<usize>> = HashMap::new();
+
+        for pool in endpoints.as_ref() {
+            for ep in pool.endpoints.as_ref() {
+                if ep.get_type() != EndpointType::Url {
+                    continue;
+                }
+                let host_port = ep.host_port();
+                if host_port.is_empty() {
+                    continue;
+                }
+
+                let member_index = match by_host_port.get(&host_port).copied() {
+                    Some(index) => index,
+                    None => {
+                        let display_endpoint = ep.url.host_str().map(str::to_owned).unwrap_or_else(|| host_port.clone());
+                        let index = topology.members.len();
+                        topology.members.push(TopologyMember {
+                            display_endpoint,
+                            disks: Vec::new(),
+                            drive_keys: Vec::new(),
+                        });
+                        by_host_port.insert(host_port.clone(), index);
+                        index
+                    }
+                };
+
+                topology.aliases.entry(host_port.clone()).or_insert(member_index);
+                topology.aliases.entry(ep.to_string()).or_insert(member_index);
+                if let Some(display_endpoint) = ep.url.host_str() {
+                    match display_aliases.entry(display_endpoint.to_owned()) {
+                        std::collections::hash_map::Entry::Vacant(entry) => {
+                            entry.insert(Some(member_index));
+                        }
+                        std::collections::hash_map::Entry::Occupied(mut entry) => {
+                            if entry.get().is_some_and(|index| index != member_index) {
+                                entry.insert(None);
+                            }
+                        }
+                    }
+                }
+
+                let drive_key = TopologyDriveKey {
+                    pool_index: ep.pool_idx,
+                    set_index: ep.set_idx,
+                    disk_index: ep.disk_idx,
+                    member_index,
+                };
+                topology.expected_drive_ids.insert(drive_key.clone());
+                topology.members[member_index].drive_keys.push(drive_key);
+                topology.members[member_index].disks.push(Disk {
+                    endpoint: ep.to_string(),
+                    state: ITEM_UNKNOWN.to_string(),
+                    pool_index: ep.pool_idx,
+                    set_index: ep.set_idx,
+                    disk_index: ep.disk_idx,
+                    ..Default::default()
+                });
+            }
+        }
+
+        for (display_endpoint, member_index) in display_aliases {
+            if let Some(member_index) = member_index {
+                topology.aliases.entry(display_endpoint).or_insert(member_index);
+            }
+        }
+
+        topology
+    }
+
+    fn is_empty(&self) -> bool {
+        self.members.is_empty()
+    }
+
+    fn observe_servers(&self, servers: &[ServerProperties]) -> (Vec<bool>, HashMap<TopologyDriveKey, usize>) {
+        let mut observed_members = vec![false; self.members.len()];
+        let mut observed_drive_counts = HashMap::with_capacity(self.expected_drive_ids.len());
+
+        for server in servers {
+            if let Some(member_index) = self.member_index_from_endpoint(&server.endpoint) {
+                observed_members[member_index] = true;
+            }
+
+            for disk in &server.disks {
+                if let Some(member_index) = self.member_index_from_endpoint(&disk.endpoint) {
+                    observed_members[member_index] = true;
+                    let drive_key = TopologyDriveKey {
+                        pool_index: disk.pool_index,
+                        set_index: disk.set_index,
+                        disk_index: disk.disk_index,
+                        member_index,
+                    };
+                    if self.expected_drive_ids.contains(&drive_key) {
+                        *observed_drive_counts.entry(drive_key).or_insert(0) += 1;
+                    }
+                }
+            }
+        }
+
+        (observed_members, observed_drive_counts)
+    }
+
+    fn member_index_from_endpoint(&self, endpoint: &str) -> Option<usize> {
+        if let Some(index) = self.aliases.get(endpoint) {
+            return Some(*index);
+        }
+
+        Endpoint::try_from(endpoint)
+            .ok()
+            .and_then(|ep| self.aliases.get(&ep.host_port()).copied())
+    }
+
+    fn report_from_counts(&self, observed_drive_counts: HashMap<TopologyDriveKey, usize>) -> TopologyCompletenessReport {
+        let mut missing_drive_ids = Vec::new();
+        let mut duplicate_drive_ids = Vec::new();
+        for id in &self.expected_drive_ids {
+            match observed_drive_counts.get(id).copied().unwrap_or(0) {
+                0 => missing_drive_ids.push(id.clone()),
+                1 => {}
+                _ => duplicate_drive_ids.push(id.clone()),
+            }
+        }
+        missing_drive_ids.sort();
+        duplicate_drive_ids.sort();
+
+        TopologyCompletenessReport {
+            expected_drives: self.expected_drive_ids.len(),
+            observed_drives: observed_drive_counts.values().sum(),
+            missing_drive_ids,
+            duplicate_drive_ids,
+        }
+    }
+}
+
+fn reconcile_servers_with_endpoint_topology(
+    servers: &mut Vec<ServerProperties>,
+    endpoints: &EndpointServerPools,
+) -> (usize, TopologyCompletenessReport) {
+    let topology = EndpointTopology::from_endpoint_pools(endpoints);
+    if topology.is_empty() {
+        return (0, TopologyCompletenessReport::default());
+    }
+
+    let (observed_members, mut observed_drive_counts) = topology.observe_servers(servers);
+    let mut missing: Vec<_> = topology
+        .members
+        .iter()
+        .enumerate()
+        .filter(|(index, _)| !observed_members[*index])
+        .map(|(_, member)| {
+            for drive_key in &member.drive_keys {
+                *observed_drive_counts.entry(drive_key.clone()).or_insert(0) += 1;
+            }
+            ServerProperties {
+                endpoint: member.display_endpoint.clone(),
+                state: ITEM_UNKNOWN.to_string(),
+                disks: member.disks.clone(),
+                ..Default::default()
+            }
+        })
+        .collect();
+    missing.sort_by(|a, b| a.endpoint.cmp(&b.endpoint));
+
+    let added = missing.len();
+    servers.extend(missing);
+    let report = topology.report_from_counts(observed_drive_counts);
+
+    (added, report)
+}
+
+fn server_topology_completeness_report(
+    servers: &[ServerProperties],
+    endpoints: &EndpointServerPools,
+) -> TopologyCompletenessReport {
+    let topology = EndpointTopology::from_endpoint_pools(endpoints);
+    if topology.is_empty() {
+        return TopologyCompletenessReport::default();
+    }
+    let (_, observed_drive_counts) = topology.observe_servers(servers);
+    topology.report_from_counts(observed_drive_counts)
+}
+
 async fn get_pools_info(all_disks: &[Disk]) -> Result<HashMap<i32, HashMap<i32, ErasureSetInfo>>> {
     let Some(store) = runtime_sources::object_store_handle() else {
         return Err(Error::other("ServerNotInitialized"));
@@ -430,10 +689,17 @@ pub fn get_commit_id() -> String {
 mod tests {
     use serial_test::serial;
 
+    use crate::layout::{
+        endpoint::Endpoint,
+        endpoints::{EndpointServerPools, Endpoints, PoolEndpoints},
+    };
     use crate::runtime::sources as runtime_sources;
-    use rustfs_madmin::{Disk, ITEM_OFFLINE, ITEM_UNKNOWN};
+    use rustfs_madmin::{Disk, ITEM_OFFLINE, ITEM_ONLINE, ITEM_UNKNOWN, ServerProperties};
 
-    use super::{get_local_server_property, get_online_offline_disks_stats, get_server_info};
+    use super::{
+        DATA_USAGE_UNAVAILABLE_ERROR, apply_data_usage_result, get_local_server_property, get_online_offline_disks_stats,
+        get_server_info, reconcile_servers_with_endpoint_topology, server_topology_completeness_report,
+    };
 
     fn disk_with_state(endpoint: &str, state: &str) -> Disk {
         Disk {
@@ -443,6 +709,50 @@ mod tests {
         }
     }
 
+    fn topology_endpoint(host: &str, pool_index: usize, set_index: usize, disk_index: usize) -> Endpoint {
+        topology_endpoint_url(format!("http://{host}:9000/data{disk_index}").as_str(), pool_index, set_index, disk_index)
+    }
+
+    fn topology_endpoint_url(url: &str, pool_index: usize, set_index: usize, disk_index: usize) -> Endpoint {
+        let mut endpoint = Endpoint::try_from(url).expect("URL endpoint should parse");
+        endpoint.set_pool_index(pool_index);
+        endpoint.set_set_index(set_index);
+        endpoint.set_disk_index(disk_index);
+        endpoint
+    }
+
+    fn topology_with_hosts(hosts: &[&str]) -> EndpointServerPools {
+        let endpoints: Vec<Endpoint> = hosts
+            .iter()
+            .enumerate()
+            .map(|(disk_index, host)| topology_endpoint(host, 0, 0, disk_index))
+            .collect();
+        EndpointServerPools::from(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: hosts.len(),
+            endpoints: Endpoints::from(endpoints),
+            cmd_line: String::new(),
+            platform: String::new(),
+        }])
+    }
+
+    fn server_with_disk(host: &str, disk_index: i32, state: &str) -> ServerProperties {
+        ServerProperties {
+            endpoint: host.to_string(),
+            state: ITEM_ONLINE.to_string(),
+            disks: vec![Disk {
+                endpoint: format!("http://{host}:9000/data{disk_index}"),
+                state: state.to_string(),
+                pool_index: 0,
+                set_index: 0,
+                disk_index,
+                ..Default::default()
+            }],
+            ..Default::default()
+        }
+    }
+
     #[test]
     fn disk_stats_split_unknown_into_its_own_bucket() {
         // A member whose properties RPC could not be answered contributes
@@ -471,6 +781,142 @@ mod tests {
         );
     }
 
+    #[test]
+    fn topology_reconcile_synthesizes_missing_member_as_unknown() {
+        let endpoints = topology_with_hosts(&["rustfs-1", "rustfs-2", "rustfs-3", "rustfs-4"]);
+        let mut servers = vec![
+            server_with_disk("rustfs-1", 0, "ok"),
+            server_with_disk("rustfs-2", 1, "ok"),
+            server_with_disk("rustfs-3", 2, "ok"),
+        ];
+
+        let (added, report) = reconcile_servers_with_endpoint_topology(&mut servers, &endpoints);
+        let (_, _, unknown) =
+            get_online_offline_disks_stats(&servers.iter().flat_map(|server| server.disks.clone()).collect::<Vec<_>>());
+
+        assert_eq!(added, 1, "the missing fourth topology member must be synthesized");
+        assert!(report.is_complete(), "synthesized drives should complete the topology report");
+        assert_eq!(servers.len(), 4, "v3 server list must preserve topology membership length");
+        let synthesized = servers
+            .iter()
+            .find(|server| server.endpoint == "rustfs-4")
+            .expect("missing member should be present");
+        assert_eq!(synthesized.state, ITEM_UNKNOWN);
+        assert_eq!(synthesized.disks.len(), 1);
+        assert_eq!(synthesized.disks[0].endpoint, "http://rustfs-4:9000/data3");
+        assert_eq!(synthesized.disks[0].disk_index, 3);
+        assert_eq!(unknown.sum(), 1, "the synthesized drive must land in unknownDisks");
+    }
+
+    #[test]
+    fn topology_reconcile_does_not_duplicate_existing_synthesized_rows() {
+        let endpoints = topology_with_hosts(&["rustfs-1", "rustfs-2"]);
+        let mut servers = vec![
+            server_with_disk("rustfs-1", 0, "ok"),
+            server_with_disk("rustfs-2", 1, ITEM_UNKNOWN),
+        ];
+        servers[1].state = ITEM_UNKNOWN.to_string();
+
+        let (added, report) = reconcile_servers_with_endpoint_topology(&mut servers, &endpoints);
+
+        assert_eq!(
+            added, 0,
+            "an existing unknown/degraded/offline row with topology drives already represents the member"
+        );
+        assert!(report.is_complete(), "existing synthesized rows should already complete the topology");
+        assert_eq!(servers.len(), 2);
+    }
+
+    #[test]
+    fn topology_reconcile_does_not_over_match_ambiguous_host_only_aliases() {
+        let endpoints = EndpointServerPools::from(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 2,
+            endpoints: Endpoints::from(vec![
+                topology_endpoint_url("http://rustfs-1:9000/data0", 0, 0, 0),
+                topology_endpoint_url("http://rustfs-1:9001/data1", 0, 0, 1),
+            ]),
+            cmd_line: String::new(),
+            platform: String::new(),
+        }]);
+        let mut servers = vec![server_with_disk("rustfs-1", 0, "ok")];
+
+        let (added, report) = reconcile_servers_with_endpoint_topology(&mut servers, &endpoints);
+
+        assert_eq!(added, 1, "host-only aliases are ambiguous when one host has multiple topology ports");
+        assert!(report.is_complete());
+        assert!(
+            servers
+                .iter()
+                .any(|server| server.disks.iter().any(|disk| disk.endpoint == "http://rustfs-1:9001/data1")),
+            "the second host:port member should be synthesized from exact topology"
+        );
+    }
+
+    #[test]
+    fn topology_report_detects_duplicate_drive_identity_with_balanced_total() {
+        let endpoints = topology_with_hosts(&["rustfs-1", "rustfs-2"]);
+        let servers = vec![server_with_disk("rustfs-1", 0, "ok"), server_with_disk("rustfs-1", 0, "ok")];
+
+        let report = server_topology_completeness_report(&servers, &endpoints);
+
+        assert_eq!(report.expected_drives, 2);
+        assert_eq!(report.observed_drives, 2, "a plain total-count check would look balanced");
+        assert_eq!(report.missing_drive_ids.len(), 1, "rustfs-2's drive identity is absent");
+        assert_eq!(report.duplicate_drive_ids.len(), 1, "rustfs-1's drive identity is duplicated");
+        assert!(!report.is_complete());
+    }
+
+    #[test]
+    fn data_usage_errors_are_sanitized_in_server_info() {
+        let mut buckets = rustfs_madmin::Buckets::default();
+        let mut objects = rustfs_madmin::Objects::default();
+        let mut versions = rustfs_madmin::Versions::default();
+        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
+        let mut usage = rustfs_madmin::Usage::default();
+
+        apply_data_usage_result(
+            Err(crate::error::Error::other("sensitive disk path")),
+            &mut buckets,
+            &mut objects,
+            &mut versions,
+            &mut delete_markers,
+            &mut usage,
+        );
+
+        assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+    }
+
+    #[test]
+    fn data_usage_counts_are_mapped_into_server_info() {
+        let mut buckets = rustfs_madmin::Buckets::default();
+        let mut objects = rustfs_madmin::Objects::default();
+        let mut versions = rustfs_madmin::Versions::default();
+        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
+        let mut usage = rustfs_madmin::Usage::default();
+        let info = rustfs_data_usage::DataUsageInfo {
+            buckets_count: 2,
+            objects_total_count: 3,
+            versions_total_count: 4,
+            delete_markers_total_count: 5,
+            objects_total_size: 6,
+            ..Default::default()
+        };
+
+        apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
+
+        assert_eq!(buckets.count, 2);
+        assert_eq!(objects.count, 3);
+        assert_eq!(versions.count, 4);
+        assert_eq!(delete_markers.count, 5);
+        assert_eq!(usage.size, 6);
+    }
+
     #[serial]
     #[tokio::test]
     async fn server_info_includes_global_deployment_id() {

crates/ecstore/src/diagnostics/get.rs

@@ -137,6 +137,7 @@ pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_fou
 pub(crate) const GET_METADATA_CACHE_REASON_NOT_READ_DATA: &str = "not_read_data";
 pub(crate) const GET_METADATA_CACHE_REASON_PART_NUMBER: &str = "part_number";
 pub(crate) const GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ: &str = "raw_data_movement_read";
+pub(crate) const GET_METADATA_CACHE_REASON_STALE_PUBLICATION: &str = "stale_publication";
 pub(crate) const GET_METADATA_CACHE_REASON_USABLE: &str = "usable";
 pub(crate) const GET_METADATA_CACHE_REASON_VERSION_ID: &str = "version_id";
 pub(crate) const GET_METADATA_CACHE_REASON_VERSION_SUSPENDED: &str = "version_suspended";
@@ -386,6 +387,7 @@ mod tests {
         assert_eq!(GET_METADATA_CACHE_REASON_NOT_READ_DATA, "not_read_data");
         assert_eq!(GET_METADATA_CACHE_REASON_PART_NUMBER, "part_number");
         assert_eq!(GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ, "raw_data_movement_read");
+        assert_eq!(GET_METADATA_CACHE_REASON_STALE_PUBLICATION, "stale_publication");
         assert_eq!(GET_METADATA_CACHE_REASON_USABLE, "usable");
         assert_eq!(GET_METADATA_CACHE_REASON_VERSION_ID, "version_id");
         assert_eq!(GET_METADATA_CACHE_REASON_VERSION_SUSPENDED, "version_suspended");

crates/ecstore/src/disk/disk_store.rs

@@ -209,6 +209,16 @@ pub fn get_drive_walkdir_stall_timeout() -> Duration {
     )
 }
 
+pub fn get_drive_walkdir_peek_timeout() -> Duration {
+    let stall_timeout = get_drive_walkdir_stall_timeout();
+    let configured = get_drive_timeout_duration(
+        rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS,
+        rustfs_config::DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS.saturating_mul(2)),
+    );
+    configured.max(stall_timeout)
+}
+
 pub fn get_object_disk_read_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT,
@@ -217,6 +227,27 @@ pub fn get_object_disk_read_timeout() -> Duration {
     )
 }
 
+/// Per-shard erasure write stall budget: a shard write (or shutdown) that makes
+/// no forward progress for this long is failed and its disk dropped before
+/// commit. Re-armed on every shard write, so it bounds a stall rather than the
+/// whole transfer. `0` disables the deadline (wait indefinitely).
+pub fn get_object_disk_write_stall_timeout() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+    ))
+}
+
+/// Optional absolute per-object erasure write cap (administrator slow-drip
+/// backstop). `0` (default) disables the cap; the per-shard stall timeout is the
+/// primary guarantee.
+pub fn get_object_disk_write_absolute_cap() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+    ))
+}
+
 pub fn get_drive_active_check_interval() -> Duration {
     Duration::from_secs(rustfs_utils::get_env_u64(
         rustfs_config::ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS,
@@ -1635,6 +1666,64 @@ mod tests {
         });
     }
 
+    #[test]
+    fn drive_walkdir_peek_timeout_uses_wider_default_when_unset() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                    temp_env::with_var_unset(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, || {
+                        assert_eq!(
+                            get_drive_walkdir_peek_timeout(),
+                            Duration::from_secs(rustfs_config::DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS)
+                        );
+                    });
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_walkdir_peek_timeout_is_never_stricter_than_stall_timeout() {
+        temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, Some("3"), || {
+            temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, Some("13"), || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                    assert_eq!(get_drive_walkdir_peek_timeout(), Duration::from_secs(13));
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_walkdir_peek_timeout_prefers_canonical_over_legacy() {
+        temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, Some("23"), || {
+            temp_env::with_var(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("17"), || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, || {
+                    assert_eq!(get_drive_walkdir_peek_timeout(), Duration::from_secs(23));
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_walkdir_peek_timeout_uses_high_latency_profile_default() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                    temp_env::with_var(
+                        rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE,
+                        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY),
+                        || {
+                            assert_eq!(
+                                get_drive_walkdir_peek_timeout(),
+                                Duration::from_secs(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS * 2)
+                            );
+                        },
+                    );
+                });
+            });
+        });
+    }
+
     #[test]
     fn drive_walkdir_timeout_prefers_canonical_over_legacy() {
         temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("11"), || {
@@ -1653,6 +1742,32 @@ mod tests {
         });
     }
 
+    #[test]
+    fn object_disk_write_stall_timeout_default_and_override() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, || {
+            assert_eq!(
+                get_object_disk_write_stall_timeout(),
+                Duration::from_secs(rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT)
+            );
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("9"), || {
+            assert_eq!(get_object_disk_write_stall_timeout(), Duration::from_secs(9));
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("0"), || {
+            assert!(get_object_disk_write_stall_timeout().is_zero(), "0 disables the stall deadline");
+        });
+    }
+
+    #[test]
+    fn object_disk_write_absolute_cap_defaults_disabled() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, || {
+            assert!(get_object_disk_write_absolute_cap().is_zero(), "absolute cap is disabled by default");
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, Some("120"), || {
+            assert_eq!(get_object_disk_write_absolute_cap(), Duration::from_secs(120));
+        });
+    }
+
     #[test]
     fn object_disk_read_timeout_uses_default_when_unset() {
         temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT, || {

crates/ecstore/src/disk/error.rs

@@ -141,7 +141,7 @@ pub enum DiskError {
     ErasureReadQuorum,
 
     #[error("io error {0}")]
-    Io(io::Error),
+    Io(#[source] io::Error),
 
     #[error("source stalled")]
     SourceStalled,
@@ -153,6 +153,12 @@ pub enum DiskError {
     InvalidPath,
 }
 
+impl From<crate::erasure::coding::ErasureConstructionError> for DiskError {
+    fn from(error: crate::erasure::coding::ErasureConstructionError) -> Self {
+        Self::Io(error.into_io_error())
+    }
+}
+
 impl DiskError {
     pub fn other<E>(error: E) -> Self
     where
@@ -601,6 +607,26 @@ mod tests {
     use super::*;
     use std::collections::HashMap;
 
+    #[test]
+    fn other_preserves_erasure_construction_source_chain() {
+        use crate::erasure::coding::ErasureConstructionError;
+        use std::error::Error as _;
+
+        let error = DiskError::from(ErasureConstructionError::ModernEncoder {
+            source: reed_solomon_erasure::Error::TooManyShards,
+        });
+        let io_source = error.source().expect("DiskError::Io must expose its io::Error source");
+        assert!(io_source.is::<io::Error>());
+        let construction_source = io_source
+            .source()
+            .expect("io::Error must expose the erasure construction error");
+        assert!(construction_source.is::<ErasureConstructionError>());
+        let encoder_source = construction_source
+            .source()
+            .expect("construction error must expose the encoder error");
+        assert!(encoder_source.is::<reed_solomon_erasure::Error>());
+    }
+
     #[test]
     fn test_disk_error_variants() {
         let errors = vec![

crates/ecstore/src/disk/os.rs

@@ -645,6 +645,34 @@ pub fn file_exists(path: impl AsRef<Path>) -> bool {
     std::fs::metadata(path.as_ref()).map(|_| true).unwrap_or(false)
 }
 
+/// Whether an [`io::Error`] means "the directory is not empty".
+///
+/// POSIX lets `rmdir`/`rename` report a non-empty directory as either
+/// `ENOTEMPTY` or `EEXIST`. Linux uses `ENOTEMPTY` (which Rust surfaces as
+/// [`io::ErrorKind::DirectoryNotEmpty`]); illumos/Solaris return `EEXIST`
+/// (errno 17), which Rust surfaces as [`io::ErrorKind::AlreadyExists`] and
+/// which the `DirectoryNotEmpty` kind therefore never catches. Matching only on
+/// the kind silently misclassifies the Solaris case as a hard failure, so
+/// callers that must treat a still-populated directory as benign (deleting the
+/// object metadata while a rollback-staging dir remains, non-force
+/// `DeleteBucket` on a populated bucket) have to test the raw errno as well.
+/// Mirrors MinIO's `isSysErrNotEmpty`.
+pub fn is_dir_not_empty_error(err: &io::Error) -> bool {
+    // Linux/Windows: ENOTEMPTY / ERROR_DIR_NOT_EMPTY -> DirectoryNotEmpty.
+    if err.kind() == io::ErrorKind::DirectoryNotEmpty {
+        return true;
+    }
+    // illumos/Solaris report a non-empty `rmdir`/`rename` as EEXIST (errno 17),
+    // which Rust surfaces as `AlreadyExists` (so the `DirectoryNotEmpty` kind
+    // never catches it). Confirm against the raw errno directly so the
+    // classification holds regardless of how the platform std maps it.
+    #[cfg(unix)]
+    if matches!(err.raw_os_error(), Some(libc::ENOTEMPTY) | Some(libc::EEXIST)) {
+        return true;
+    }
+    false
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -809,6 +837,62 @@ mod tests {
         assert!(!should_retry_rename(&denied, 1));
     }
 
+    #[test]
+    fn is_dir_not_empty_error_recognizes_directory_not_empty_kind() {
+        let err = io::Error::from(io::ErrorKind::DirectoryNotEmpty);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn is_dir_not_empty_error_recognizes_raw_enotempty() {
+        // Linux/BSD/macOS non-empty rmdir/rename errno.
+        let err = io::Error::from_raw_os_error(libc::ENOTEMPTY);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn is_dir_not_empty_error_recognizes_solaris_eexist() {
+        // illumos/Solaris report a non-empty rmdir/rename as EEXIST, which Rust
+        // surfaces as `AlreadyExists` (never `DirectoryNotEmpty`). This is the
+        // core of rustfs/rustfs#4978: matching only the kind misclassified this
+        // benign condition as a hard failure.
+        let err = io::Error::from_raw_os_error(libc::EEXIST);
+        assert_eq!(err.kind(), io::ErrorKind::AlreadyExists);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[test]
+    fn is_dir_not_empty_error_rejects_unrelated_errors() {
+        assert!(!is_dir_not_empty_error(&io::Error::from(io::ErrorKind::NotFound)));
+        assert!(!is_dir_not_empty_error(&io::Error::from(io::ErrorKind::PermissionDenied)));
+        #[cfg(unix)]
+        {
+            assert!(!is_dir_not_empty_error(&io::Error::from_raw_os_error(libc::EACCES)));
+            assert!(!is_dir_not_empty_error(&io::Error::from_raw_os_error(libc::ENOENT)));
+        }
+    }
+
+    #[tokio::test]
+    async fn is_dir_not_empty_error_matches_real_non_empty_rmdir() {
+        // Validate against the host's actual errno, whatever it is: Linux/macOS
+        // return ENOTEMPTY, illumos/Solaris return EEXIST. The removal must be
+        // classified as "not empty" on every platform.
+        let temp_dir = tempdir().expect("create temp dir");
+        let populated = temp_dir.path().join("populated");
+        std::fs::create_dir(&populated).expect("create dir");
+        std::fs::write(populated.join("child"), b"x").expect("write child");
+
+        let err = std::fs::remove_dir(&populated).expect_err("non-empty rmdir must fail");
+        assert!(
+            is_dir_not_empty_error(&err),
+            "non-empty rmdir must classify as not-empty, got kind {:?} errno {:?}",
+            err.kind(),
+            err.raw_os_error()
+        );
+    }
+
     #[tokio::test]
     async fn rename_all_moves_existing_directory_tree() {
         // Guards the rename_data commit path, which funnels through