feat: expose list read-dir amplification metrics (#5103)

Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer.

Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage.

Co-authored-by: heihutu <heihutu@gmail.com>

.agents/skills/adversarial-validation/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: adversarial-validation
-description: Execute the Adversarial Validation policy from the root AGENTS.md — run the six reviewer roles (correctness, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
 ---
 
 # Adversarial Validation Playbooks
@@ -53,14 +53,22 @@ shipped bug or rule that earns each probe its place.
 - Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
   - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
   - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
-- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
-  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
-  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Constant and String Usage'; Adversarial Validation section names the smaller-diff clause as a correctness-adversary finding.
 - For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
   - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
   - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
 
-Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, mid-stream reconstruct error propagation, and a minimal-diff rewrite — no break found; diff is already the minimal in-place edit."
+Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
+
+### Simplicity adversary
+
+- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
+- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
+  - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
+
+Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
 
 ### Security reviewer
 
@@ -243,7 +251,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
 - If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
   - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
   - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
-- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, and remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent). Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
+- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
   - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
   - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
 - For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.

.agents/skills/code-change-verification/SKILL.md

@@ -43,16 +43,7 @@ Use this skill to review code changes consistently before merge, before release,
 
 #### Rust-specific checks (apply to all Rust changes)
 
-- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
-- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
-- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
-- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
-- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
-- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
-- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
-- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
-- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
-- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
+Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
 
 ### 4) Findings-first output
 - Order findings by severity:

.agents/skills/rust-code-quality/SKILL.md

@@ -36,6 +36,9 @@ rg -n 'println!\|eprintln!' <changed-files> | grep -v test
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
+
+# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
+rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 ```
 
 ## Manual Review Checklist
@@ -55,8 +58,8 @@ For every Rust code change, verify:
 - [ ] No `f64 as usize` without prior clamping
 
 ### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used
-- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
+- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
+- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
 - [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
@@ -84,12 +87,19 @@ For every Rust code change, verify:
 - [ ] No camelCase statics or Hungarian notation
 - [ ] New string literals don't duplicate existing constants
 
+### Reuse and Necessity
+- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
+- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
+- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
+- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
+- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
+
 ## Severity Classification
 
 - **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
-- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
-- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
-- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
 
 ## Output Template
 

.agents/skills/security-advisory-lessons/SKILL.md

@@ -60,12 +60,14 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 ### IAM and service accounts
 - Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
 - For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
+- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
 - Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
 - Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
 
 ### STS, OIDC, and federation flows
 - Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
 - JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
+- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
 - Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
 - OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
 
@@ -137,7 +139,9 @@ Use these prompts while reviewing a diff:
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
 - Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
+- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
 - Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
+- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
 - Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
 - Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -27,14 +27,15 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### IAM import, service accounts, and privilege boundaries
 
 - `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
 - `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
 - `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
 - `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
 
 ### STS, OIDC, and federation flows
 
-- `GHSA-5qfg-mf7r-jp3w`: `AssumeRoleWithWebIdentity` was reachable without the required request authentication and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption.
-- `GHSA-ccrv-v8v9-ch9q`: service-account-controlled material could self-sign JWT session tokens with forged policy claims. Lesson: session tokens must be signed by a trusted issuer/key path and validation must reject self-signed or principal-controlled tokens.
+- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
+- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
@@ -58,7 +59,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, and `GHSA-9gf3-jx4p-4xxf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -122,6 +123,7 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/lint-fmt.mak

@@ -60,6 +60,11 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
+.PHONY: log-analyzer-rules-check
+log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
+	@echo "🩺 Checking log-analyzer rule anchors..."
+	./scripts/check_log_analyzer_rules.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -23,7 +23,7 @@ pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-gua
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check

.config/make/tests.mak

@@ -26,6 +26,9 @@ script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
 	./scripts/test_entrypoint_credentials.sh
+	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
+	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
+	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
 
 .PHONY: test
 test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)

.config/nextest.toml

@@ -42,7 +42,7 @@ e2e-reliability = { max-threads = 1 }
 
 # --- default profile (local): serialize the flaky groups, never retry --------
 [[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/))'
+filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
 
 # Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
@@ -99,7 +99,7 @@ retries = 2
 # QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
 # make_bucket into InsufficientWriteQuorum via shared global state under load.
 [[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/)'
+filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
 test-group = 'ecstore-serial-flaky'
 retries = 2
 
@@ -192,7 +192,7 @@ test-group = 'ecstore-serial-flaky'
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud)_test::|^fake_s3_target::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools)_test::|^fake_s3_target::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
@@ -286,14 +286,8 @@ path = "junit.xml"
 # ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
 # OPEN issue, and the fixing PR MUST delete the exclusion. The passing
 # negative-path siblings of each family stay in as regression guards.
-#   * rustfs#4842 — extract/snowball expand pipeline 500s (mtime=0
-#     OffsetDateTime deserialization + same-path failures).
 #   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
 #     archive even under ignore-errors semantics.
-#   * rustfs#4844 — anonymous POST-object with SSE-S3 / bucket-default SSE
-#     returns 500.
-#   * rustfs#4845 — 403 on allowed anonymous POST object-lock fields and on
-#     the list metadata=true extension.
 #   * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
 #     under parallel load (multi-node in-process clusters; natural home is
 #     ci-7's nightly cluster lane).
@@ -303,13 +297,8 @@ default-filter = """
     & !test(/^protocols::/)
     & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
     & !test(/^replication_extension_test::/)
-    & !test(/^multipart_auth_test::test_signed_put_object_extract_(accepts_compat_header|expands_tar_entries_with_prefix_headers|expands_tar_gz_archive|expands_tbz2_archive|expands_tgz_archive|expands_txz_archive|expands_tzst_archive|normalizes_prefix_header_value|preserves_directory_markers_by_default|preserves_object_lock_legal_hold|preserves_object_lock_retention|preserves_pax_metadata_and_version_id|preserves_request_metadata_on_extracted_objects|preserves_sse_c|preserves_sse_s3_and_redirect|preserves_storage_class|uses_bucket_default_sse_s3)$/)
-    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(prefers_exact_minio_prefix_over_suffix_fallback|supports_minio_prefix_and_directory_markers)$/)
     & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
     & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
-    & !test(/^multipart_auth_test::test_anonymous_post_object_(accepts_sse_s3|rejects_sse_s3_missing_from_policy_conditions|uses_bucket_default_sse_kms|uses_bucket_default_sse_s3)$/)
-    & !test(/^multipart_auth_test::test_anonymous_post_object_(accepts_object_lock_legal_hold_field|accepts_object_lock_retention_fields)$/)
-    & !test(/^list_object(s_v2|_versions)_metadata_extension_test::/)
     & !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
 """
 fail-fast = false

.docker/observability/grafana/dashboards/rustfs.json

@@ -1744,7 +1744,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "sum by (bucket) (rustfs_bucket_api_objects_total{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -1844,7 +1844,7 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "sum by (bucket) (rustfs_bucket_api_usage_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
+          "expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
           "legendFormat": "{{bucket}}",
           "range": true,
           "refId": "A"
@@ -11583,7 +11583,7 @@
           "text": "All",
           "value": "$__all"
         },
-        "definition": "label_values(rustfs_bucket_api_objects_total,bucket)",
+        "definition": "label_values(rustfs_cluster_usage_buckets_objects_count,bucket)",
         "includeAll": true,
         "label": "Bucket",
         "multi": true,
@@ -11591,7 +11591,7 @@
         "options": [],
         "query": {
           "qryType": 1,
-          "query": "label_values(rustfs_bucket_api_objects_total,bucket)",
+          "query": "label_values(rustfs_cluster_usage_buckets_objects_count,bucket)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
         "refresh": 2,

.docker/test/site-replication/run-object-flow-check.sh

@@ -18,6 +18,7 @@ set -eu
 ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
 SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
 BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
+DELETE_BUCKET="${RUSTFS_SITE_REPL_DELETE_BUCKET:-site-repl-delete-$(date +%Y%m%d-%H%M%S)-$$}"
 PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
 WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
 WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
@@ -85,17 +86,39 @@ wait_for_object() {
 
 wait_for_bucket() {
   site="$1"
+  bucket="${2:-$BUCKET}"
   attempt=1
 
   while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
-    if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
+    if mc stat "$site/$bucket" >/dev/null 2>&1; then
       return 0
     fi
     sleep "$WAIT_SLEEP_SECONDS"
     attempt=$((attempt + 1))
   done
 
-  echo "bucket was not replicated in time: $site/$BUCKET" >&2
+  echo "bucket was not replicated in time: $site/$bucket" >&2
+  return 1
+}
+
+wait_for_bucket_delete() {
+  site="$1"
+  bucket="$2"
+  attempt=1
+
+  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+    if result="$(mc stat --json "$site/$bucket" 2>&1)"; then
+      :
+    else
+      case "$result" in
+        *NoSuchBucket*) return 0 ;;
+      esac
+    fi
+    sleep "$WAIT_SLEEP_SECONDS"
+    attempt=$((attempt + 1))
+  done
+
+  echo "bucket deletion was not replicated in time: $site/$bucket" >&2
   return 1
 }
 
@@ -186,6 +209,20 @@ EOF
   echo "verified replicated downloads for $object_name"
 done
 
+echo "creating empty bucket for replicated delete check: $DELETE_BUCKET"
+mc mb "site1/$DELETE_BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket "$site" "$DELETE_BUCKET"
+done
+
+echo "deleting empty bucket on site1: $DELETE_BUCKET"
+mc rb "site1/$DELETE_BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket_delete "$site" "$DELETE_BUCKET"
+done
+
 echo "site replication object flow check passed"
 echo "bucket: $BUCKET"
 echo "prefix: $PREFIX"

.github/cla.yml

@@ -15,8 +15,8 @@
 enabled: true
 
 document:
-  version: v1
-  url: https://github.com/rustfs/cla/blob/main/cla/v1.md
+  version: v2
+  url: https://github.com/rustfs/cla/blob/main/cla/v2.md
 
 signing:
   mode: comment

.github/pull_request_template.md

@@ -33,4 +33,4 @@ documentation impact. Use N/A when there is no expected impact.
 
 ---
 
-Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
+Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v2.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.

.github/workflows/ci.yml

@@ -167,6 +167,13 @@ jobs:
           cargo nextest run --profile ci --all --exclude e2e_test
           cargo test --all --doc
 
+      # rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
+      # verbatim in the source tree (log message drifted without updating the
+      # rule). Placed here where the workspace — including the la-dump-anchors
+      # bin — is already built by the clippy/test steps above.
+      - name: Check log-analyzer rule anchors
+        run: ./scripts/check_log_analyzer_rules.sh
+
       - name: Upload test junit report
         if: always()
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -221,29 +228,23 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      # The #4877 restore self-deadlock is fixed in this PR, which re-enabled
-      # test_multipart_restore_preserves_parts_and_etag. The remaining exclusions
-      # each hit a DIFFERENT, independent issue (all tracked under
-      # rustfs/backlog#1148; they keep #[ignore] with a backlog reference):
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract). The remaining exclusions each hit a
+      # DIFFERENT, independent issue (all tracked under rustfs/backlog#1148;
+      # they keep #[ignore] with a backlog reference):
       #   - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
       #     noncurrent transition/expiry after an immediate compensation transition.
-      #   - test_transition_and_restore_flows: transition metadata is missing on
-      #     one drive (assert_transition_meta_consistent: "missing xl.meta ... on
-      #     disk2") - an EC metadata-distribution issue, not the restore lock.
-      #   - test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore:
-      #     DeleteRestoredAction sets opts.transition.expire_restored, but no
-      #     delete path reads that flag, so cleanup deletes the whole object
-      #     instead of only the local restored copy (ObjectNotFound afterwards).
-      #     The expire_restored delete semantics are unimplemented.
-      #   - restore_object_usecase_reports_ongoing_conflict_and_completion: asserts
-      #     a concurrent get_object_info observes ongoing-request=true mid-restore,
-      #     which #4877's read-vs-restore serialization rules out (see backlog#1148
-      #     ilm-8 criterion 1 - an API-semantics decision, not a bug).
       - name: Run ignored ILM integration tests serially
         run: |
           cargo nextest run -j1 --run-ignored ignored-only \
             -p rustfs-scanner -p rustfs \
-            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition) or test(restore_object_usecase_reports_ongoing_conflict_and_completion) or test(test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore))'
+            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
 
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)

.github/workflows/star-history.yml

@@ -0,0 +1,26 @@
+name: Star History
+
+on:
+  schedule:
+    - cron: "17 3 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: star-history
+  cancel-in-progress: false
+
+jobs:
+  update:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
+        with:
+          github-token: ${{ github.token }}
+          output-branch: star-history
+          output-path: .
+          chart-style: gradient
+          animate: "true"
+          contributors: "true"

AGENTS.md

@@ -14,7 +14,7 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 
 ## Execution Discipline
 
-- Read the relevant existing code, tests, and local guidance before changing behavior.
+- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write).
 - State assumptions when they affect the implementation or verification path.
 - If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
 - For multi-step work, keep the plan minimal and tied to verifiable outcomes.
@@ -41,14 +41,27 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - Do not refactor existing code only to make it easier to unit test.
 - Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
 - Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
 - Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
-## Constant and String Usage
+## Reuse Before You Write
 
-- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
-- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
-- If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
-- When changing existing behavior, keep naming and format consistency by aligning with established project constants.
+Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
+
+- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
+- **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
+- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
+- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
+
+## Necessary Code Only
+
+Net-new code — files, types, branches, comments — is cost to justify, not progress:
+
+- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
+- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
+- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
+- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success.
+- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value.
 
 ## Sources of Truth
 
@@ -141,7 +154,7 @@ Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 - **Exempt:** docs/comments/instruction-only changes, formatting, typos with
   no runtime surface. Skip this section.
 - **Mechanical:** pure renames, file moves, test-only or tooling changes —
-  correctness adversary only.
+  correctness and simplicity adversaries only.
 - **Standard (the default):** any change that affects behavior.
 - **High risk:** touches locking, erasure coding, quorum/heal, replication,
   multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`),
@@ -161,9 +174,8 @@ encode this repo's shipped bugs.
 
 - **Correctness adversary** — construct a concrete input/state/interleaving
   that yields wrong output, data loss, or a crash. Probe error paths and edge
-  values (empty, nil UUID, zero-length, quorum−1, missing version). For code
-  diffs, a materially smaller or more idiomatic diff achieving the same
-  behavior is also a finding (see Change Style for Existing Logic).
+  values (empty, nil UUID, zero-length, quorum−1, missing version).
+- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
 - **Security reviewer** — authn/authz bypass, injection, secret leakage,
   untrusted deserialization (see Serde Safety), path traversal, timing leaks.
 - **Concurrency/durability reviewer** — lock ordering, races, cancellation,
@@ -179,11 +191,12 @@ encode this repo's shipped bugs.
   wrong while all tests stay green — if one exists, coverage is insufficient.
   A missing test is a finding, not a note.
 
-Standard tier: correctness adversary + test-coverage skeptic, plus every
-role whose domain the diff touches (async or shared-state code →
-concurrency; parsing of untrusted input → security; public crate API shape
-→ compatibility; per-request or per-object hot paths → performance).
-High risk: all six roles.
+Standard tier: correctness adversary + simplicity adversary + test-coverage
+skeptic, plus every role whose domain the diff touches (async or
+shared-state code → concurrency; parsing of untrusted input → security;
+public crate API shape → compatibility; per-request or per-object hot paths
+→ performance).
+High risk: all seven roles.
 
 ### Protocol
 

ARCHITECTURE.md

@@ -41,7 +41,7 @@ The repository is a Cargo workspace with a flat `crates/` layout:
 
 ```
 rustfs/                      # Workspace root (virtual manifest)
-├── rustfs/                  # Main binary + library crate (75K lines)
+├── rustfs/                  # Main binary + library crate
 │   └── src/
 │       ├── main.rs          # Entry point, startup sequence
 │       ├── lib.rs           # Module tree root
@@ -53,7 +53,7 @@ rustfs/                      # Workspace root (virtual manifest)
 │       ├── config/          # CLI args, config parsing, workload profiles
 │       └── ...
 ├── crates/                  # library crates (authoritative list: Cargo.toml [workspace].members)
-│   ├── ecstore/             # Erasure-coded storage engine (⚠️ 87K lines)
+│   ├── ecstore/             # Erasure-coded storage engine
 │   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
 │   ├── io-core/             # Zero-copy I/O, scheduling, buffer pool
 │   ├── io-metrics/          # I/O metrics collection
@@ -83,124 +83,25 @@ A request flows **downward** through the layers. No layer should reach upward
 
 ### Crate Reference
 
-> Depth levels, line counts, and crate counts in this section are a
-> point-in-time snapshot and drift with refactors. Treat them as orders of
-> magnitude; `Cargo.toml` and `cargo tree` are the source of truth.
-
-Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
-
-```
-Depth 0 — LEAF (no internal deps):
-  appauth, checksums, config, credentials, crypto, io-metrics,
-  madmin, s3-common, workers, zip
-
-Depth 1:
-  io-core (→ io-metrics)
-  policy (→ config, credentials, crypto)
-  utils                                   (historical → config edge removed; now effectively leaf)
-
-Depth 2:
-  concurrency, filemeta, keystone, kms, lock, obs,
-  signer, targets, trusted-proxies
-
-Depth 3:
-  common                                  (historical → filemeta/madmin edges removed; now effectively leaf)
-
-Depth 4:
-  object-capacity, protos, rio
-
-Depth 5 — CORE:
-  ecstore (16 internal deps, 11 dependents — the architectural heart)
-
-Depth 6:
-  audit, heal, iam, metrics, notify, s3select-api, scanner
-
-Depth 7:
-  object-io, protocols, s3select-query
-
-Depth 8 — TOP:
-  rustfs (35 internal deps — the binary, depends on almost everything)
-```
+`Cargo.toml` is the authoritative workspace membership and `cargo tree` is the
+authoritative dependency graph. This overview deliberately avoids line-count
+and dependency-depth snapshots because both quickly become stale during
+refactors.
 
 #### By Domain
 
-**Core Infrastructure:**
+| Domain | Current workspace crates | Responsibility |
+|--------|--------------------------|----------------|
+| Foundation | `checksums`, `common`, `config`, `data-usage`, `utils` | Shared configuration, data-usage models, utilities, and checksums. |
+| I/O and storage | `concurrency`, `ecstore`, `filemeta`, `heal`, `io-core`, `io-metrics`, `lifecycle`, `lock`, `object-capacity`, `object-data-cache`, `replication`, `rio`, `rio-v2`, `scanner`, `storage-api` | Erasure-coded object storage, metadata, recovery, lifecycle, replication, locking, cache, and I/O pipelines. |
+| Security and identity | `credentials`, `crypto`, `iam`, `keystone`, `kms`, `policy`, `security-governance`, `signer`, `tls-runtime`, `trusted-proxies` | Credentials, authentication, authorization, encryption, key management, TLS, and security contracts. |
+| Protocols and contracts | `extension-schema`, `madmin`, `protos`, `protocols`, `s3-ops`, `s3-types`, `s3select-api`, `s3select-query` | Admin, inter-node, S3, S3 Select, and optional protocol contracts. |
+| Operations and integration | `audit`, `notify`, `obs`, `targets`, `zip` | Auditing, observability, event delivery, notification targets, and archive support. |
+| Test support | `e2e_test`, `test-utils` | End-to-end validation and shared test bootstrap utilities. |
 
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `config` | 3.3K | Configuration types and environment parsing |
-| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
-| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
-| `madmin` | 5.5K | Admin API request/response types |
-
-**I/O Pipeline:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
-| `io-metrics` | 4.5K | I/O operation metrics and counters |
-| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
-| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
-| `concurrency` | 0.8K | Shared concurrency contract types: workload admission snapshots, worker-slot pool, policy types (runtime control lives in `rustfs/src/storage`) |
-
-**Storage Engine:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
-| `filemeta` | 10K | File/object metadata types and versioning |
-| `checksums` | 732 | Checksum computation |
-| `lock` | 7.1K | Distributed lock manager |
-| `heal` | 5.9K | Data healing / bitrot repair |
-| `scanner` | 5.4K | Background data usage scanner |
-| `object-capacity` | 2.5K | Capacity tracking and management |
-
-**Security & Auth:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `crypto` | 1.6K | Encryption primitives |
-| `credentials` | 713 | Credential types (access key / secret key) |
-| `signer` | 1.4K | S3 v4 request signing |
-| `iam` | 9.0K | Identity and access management |
-| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
-| `kms` | 8.1K | Key management service integration |
-| `keystone` | 1.9K | OpenStack Keystone auth |
-| `appauth` | 143 | Application-level auth tokens |
-
-**Protocol & API:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
-| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
-| `s3-common` | 738 | Shared S3 types |
-| `s3select-api` | 1.9K | S3 Select interface |
-| `s3select-query` | 3.6K | S3 Select query engine |
-
-**Observability:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `metrics` | 8.4K | Prometheus metric collectors |
-| `io-metrics` | 4.5K | I/O-specific metrics |
-| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
-| `audit` | 2.4K | Audit logging |
-
-**Events:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `notify` | 5.5K | Event notification system |
-| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
-
-**Other:**
-
-| Crate | Lines | Purpose |
-|-------|-------|---------|
-| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
-| `zip` | 986 | ZIP archive support for bulk downloads |
-| `workers` | 136 | Simple worker abstraction |
+The `rustfs` binary crate composes these libraries into the running server.
+`ecstore` remains the storage engine at the architectural center; its internal
+module split is tracked under `docs/architecture/`.
 
 ## Architecture Invariants
 
@@ -212,7 +113,7 @@ Depth 8 — TOP:
    No upward imports.
 
 2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
-   `io-metrics`, `madmin`, `s3-common` should depend only on external crates.
+   `io-metrics`, and `madmin` should depend only on external crates.
    - ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin`
      edges were removed; do not reintroduce them (see Known Structural Issues).
 

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
+- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
 
 ### Added
 - **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
@@ -38,6 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
+- **Storage-class validation on startup (upgrade note)**: A persisted explicit storage class (`RUSTFS_STORAGE_CLASS_STANDARD` / `RUSTFS_STORAGE_CLASS_RRS`, for example `EC:2`) is now validated against the actual per-pool drive counts at startup and rejected when a pool cannot satisfy it. This is fail-closed and correct, but a cluster that persisted a storage class larger than a small or heterogeneous pool can hold (for example `EC:2` alongside a 2-drive pool), which earlier releases accepted and silently resolved to an invalid layout, will now refuse to start after upgrade. To recover, unset `RUSTFS_STORAGE_CLASS_STANDARD` so the server derives a valid per-pool default automatically, or set it to a value every pool can satisfy.
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
 - **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.

Cargo.toml

@@ -31,6 +31,7 @@ members = [
     "crates/lifecycle", # Lifecycle rule evaluation contracts
     "crates/kms", # Key Management Service
     "crates/lock", # Distributed locking implementation
+    "crates/log-analyzer", # Offline log fault-analysis core (rustfs diagnose)
     "crates/madmin", # Management dashboard and admin API interface
     "crates/notify", # Notification system for events
     "crates/obs", # Observability utilities
@@ -108,6 +109,7 @@ rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.10" }
 rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.10" }
 rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.10" }
 rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.10" }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.10" }
 rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.10" }
 rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.10" }
 rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.10" }
@@ -137,27 +139,27 @@ async_zip = { default-features = false, version = "0.0.18" }
 mysql_async = { default-features = false, version = "0.37" }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
-async-trait = "0.1.89"
-async-nats = "0.49.1"
+async-trait = "0.1.91"
+async-nats = "0.50.0"
 axum = "0.8.9"
-futures = "0.3.32"
-futures-core = "0.3.32"
+futures = "0.3.33"
+futures-core = "0.3.33"
 futures-lite = "2.6.1"
-futures-util = "0.3.32"
+futures-util = "0.3.33"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.8.0" }
 lapin = { default-features = false, version = "4.10.0" }
-hyper = { version = "1.10.1" }
+hyper = { version = "1.11.0" }
 hyper-rustls = { default-features = false, version = "0.27.9" }
 hyper-util = { version = "0.1.20" }
 http = "1.4.2"
 http-body = "1.1.0"
 http-body-util = "0.1.4"
 minlz = "1.2.3"
-reqwest = { default-features = false, version = "0.13.4" }
+reqwest = "0.13.4"
 rustfs-kafka-async = { version = "1.2.0" }
 socket2 = { version = "0.6.5" }
-tokio = { version = "1.52.3" }
+tokio = { version = "1.53.1" }
 tokio-rustls = { default-features = false, version = "0.26.4" }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
@@ -179,8 +181,8 @@ prost = "0.14.4"
 quick-xml = "0.41.0"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
-serde = { version = "1.0.228" }
-serde_json = { version = "1.0.150" }
+serde = { version = "1.0.229" }
+serde_json = { version = "1.0.151" }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -209,8 +211,8 @@ zeroize = { version = "1.9.0" }
 # Time and Date
 chrono = { version = "0.4.45" }
 humantime = "2.4.0"
-jiff = { version = "0.2.32" }
-time = { version = "0.3.53" }
+jiff = { version = "0.2.34" }
+time = { version = "0.3.54" }
 
 # Database
 deadpool-postgres = { version = "0.14" }
@@ -218,21 +220,21 @@ tokio-postgres = { default-features = false, version = "0.7.18" }
 tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
-anyhow = "1.0.103"
+anyhow = "1.0.104"
 arc-swap = "1.9.2"
-astral-tokio-tar = "0.6.3"
+astral-tokio-tar = "0.6.4"
 atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.9.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-s3 = { default-features = false, version = "1.138.0" }
+aws-sdk-s3 = { default-features = false, version = "1.138.1" }
 aws-smithy-http-client = { default-features = false, version = "1.2.0" }
 aws-smithy-runtime-api = { version = "1.13.0" }
 aws-smithy-types = { version = "1.6.1" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.2" }
+clap = { version = "4.6.3" }
 const-str = { version = "1.1.0" }
 convert_case = "0.11.0"
 criterion = { version = "0.8" }
@@ -242,7 +244,7 @@ crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
 datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
 derive_builder = "0.20.2"
-enumset = "1.1.13"
+enumset = "1.1.14"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
@@ -254,7 +256,7 @@ hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 ipnetwork = { version = "0.21.1" }
 lazy_static = "1.5.0"
-libc = "0.2.186"
+libc = "0.2.187"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
 memmap2 = "0.9.11"
@@ -279,12 +281,12 @@ rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.0" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
-rumqttc = { package = "rumqttc-next", version = "0.33.2" }
-redis = { version = "1.4.0" }
+rumqttc = { package = "rumqttc-next", version = "0.33.3" }
+redis = { version = "1.4.1" }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "ce69c3f10824535c7c24b2f71cdb2aaa4dffb5e0" }
+s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "a5471625975f5014f7b28eee7e4d801f1b32f529" }
 serial_test = "3.5.0"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
@@ -297,7 +299,7 @@ sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.18"
+thiserror = "2.0.19"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
@@ -308,6 +310,7 @@ url = "2.5.8"
 urlencoding = "2.1.3"
 uuid = { version = "1.24.0" }
 vaultrs = { version = "0.8.0" }
+tar = "0.4.46"
 walkdir = "2.5.0"
 windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.17" }
@@ -330,15 +333,15 @@ libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
 suppaftp = { version = "10.0.1" }
 rcgen = "0.14.8"
-russh = { version = "0.62.2" }
+russh = { version = "0.62.3" }
 russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = "0.1"
-hotpath = "0.21"
+mimalloc = "0.1.52"
+hotpath = "0.21.5"
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 
@@ -367,21 +370,3 @@ inherits = "release"
 inherits = "release"
 debug = true
 strip = "none"
-
-# Pin hyper to a revision that carries the HTTP/1 "flush buffered data before
-# shutdown" fix (hyperium/hyper#4018, commit 72046cc7). This lands as a
-# `[patch.crates-io]` entry — not on the `hyper` workspace dependency — so that
-# every consumer in the tree, including the transitive `hyper-util` server path
-# (`conn::auto` / `GracefulShutdown`) that actually drives our connections,
-# resolves to the fixed hyper rather than the buggy crates.io copy.
-#
-# hyper <= 1.10.1 can call `poll_shutdown()` on the socket while response bytes
-# are still buffered (a prior `poll_flush()` returned `Poll::Pending` and the
-# result was discarded). A backpressured / slow-reading peer then receives a
-# graceful FIN before the full Content-Length body is flushed, which standard S3
-# clients (minio-go / warp) report as `unexpected EOF` on large-object GET under
-# load. The fix is not in any crates.io release yet as of hyper 1.10.1; drop
-# this patch once a released version (> 1.10.1) contains commit 72046cc7.
-# See rustfs/backlog#1232.
-[patch.crates-io]
-hyper = { git = "https://github.com/hyperium/hyper.git", rev = "ccc1e850dc0cda3e71b0acd11f60ca3d48d09034" }

README.md

@@ -338,12 +338,18 @@ If you have any questions or need assistance:
 RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS contributors">
+</picture>
 </a>
 
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS star history chart">
+</picture>
 
 ## License
 

README_ZH.md

@@ -247,12 +247,18 @@ rustfs --help
 RustFS 是一个社区驱动的项目，我们感谢所有的贡献。请查看 [贡献者](https://github.com/rustfs/rustfs/graphs/contributors) 页面，看看那些让 RustFS 变得更好的了不起的人们。
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS 贡献者">
+</picture>
 </a>
 
 ## Star 历史
 
-[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
+  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS Star 历史图表">
+</picture>
 
 ## 许可证
 

crates/common/src/globals.rs

@@ -54,6 +54,15 @@ pub async fn get_global_local_node_name() -> String {
     GLOBAL_LOCAL_NODE_NAME.read().await.clone()
 }
 
+/// Read the local node name without waiting for initialization or a writer.
+pub fn try_get_global_local_node_name() -> Option<String> {
+    GLOBAL_LOCAL_NODE_NAME
+        .try_read()
+        .ok()
+        .map(|name| name.clone())
+        .filter(|name| !name.is_empty())
+}
+
 /// Set the global RustFS initialization time to the current UTC time.
 pub async fn set_global_init_time_now() {
     let now = Utc::now();

crates/common/src/heal_channel.rs

@@ -243,6 +243,19 @@ pub enum HealAdmissionResult {
     Dropped(HealAdmissionDropReason),
 }
 
+/// Admission decision together with the canonical task identifier.
+///
+/// A merged request must return the identifier of the task that already owns
+/// the work instead of exposing the discarded request identifier as a new
+/// client token.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct HealAdmissionReceipt {
+    /// Admission decision for the submitted request.
+    pub result: HealAdmissionResult,
+    /// Canonical identifier of the accepted or merged task.
+    pub task_id: String,
+}
+
 impl HealAdmissionResult {
     pub fn result_label(self) -> &'static str {
         match self {
@@ -382,8 +395,25 @@ pub type HealChannelSender = mpsc::UnboundedSender<HealChannelCommand>;
 /// Heal channel receiver
 pub type HealChannelReceiver = mpsc::UnboundedReceiver<HealChannelCommand>;
 
+/// Canonical-receipt start command kept separate from the legacy public enum.
+#[derive(Debug)]
+pub struct HealReceiptCommand {
+    /// Heal request to admit.
+    pub request: HealChannelRequest,
+    /// Completion channel for the admission receipt.
+    pub response_tx: oneshot::Sender<Result<HealAdmissionReceipt, String>>,
+}
+
+/// Canonical-receipt command receiver.
+pub type HealReceiptReceiver = mpsc::UnboundedReceiver<HealReceiptCommand>;
+
+struct HealChannelSenders {
+    command: HealChannelSender,
+    receipt: mpsc::UnboundedSender<HealReceiptCommand>,
+}
+
 /// Global heal channel sender
-static GLOBAL_HEAL_CHANNEL_SENDER: OnceLock<HealChannelSender> = OnceLock::new();
+static GLOBAL_HEAL_CHANNEL_SENDERS: OnceLock<HealChannelSenders> = OnceLock::new();
 
 type HealResponseSender = broadcast::Sender<HealChannelResponse>;
 
@@ -392,17 +422,24 @@ static GLOBAL_HEAL_RESPONSE_SENDER: OnceLock<HealResponseSender> = OnceLock::new
 
 /// Initialize global heal channel
 pub fn init_heal_channel() -> Result<HealChannelReceiver, &'static str> {
-    let (tx, rx) = mpsc::unbounded_channel();
-    if GLOBAL_HEAL_CHANNEL_SENDER.set(tx).is_ok() {
-        Ok(rx)
-    } else {
-        Err("Heal channel sender already initialized")
-    }
+    let (receiver, receipt_receiver) = init_heal_channels()?;
+    drop(receipt_receiver);
+    Ok(receiver)
+}
+
+/// Initialize the legacy command and canonical-receipt channels atomically.
+pub fn init_heal_channels() -> Result<(HealChannelReceiver, HealReceiptReceiver), &'static str> {
+    let (command, command_receiver) = mpsc::unbounded_channel();
+    let (receipt, receipt_receiver) = mpsc::unbounded_channel();
+    GLOBAL_HEAL_CHANNEL_SENDERS
+        .set(HealChannelSenders { command, receipt })
+        .map_err(|_| "Heal channel sender already initialized")?;
+    Ok((command_receiver, receipt_receiver))
 }
 
 /// Get global heal channel sender
 pub fn get_heal_channel_sender() -> Option<&'static HealChannelSender> {
-    GLOBAL_HEAL_CHANNEL_SENDER.get()
+    GLOBAL_HEAL_CHANNEL_SENDERS.get().map(|senders| &senders.command)
 }
 
 /// Send heal command through global channel
@@ -436,6 +473,21 @@ pub fn subscribe_heal_responses() -> broadcast::Receiver<HealChannelResponse> {
     heal_response_sender().subscribe()
 }
 
+/// Send heal start request and wait for structured admission feedback.
+pub async fn send_heal_request_with_receipt(request: HealChannelRequest) -> Result<HealAdmissionReceipt, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    let senders = GLOBAL_HEAL_CHANNEL_SENDERS
+        .get()
+        .ok_or_else(|| "Heal channel not initialized".to_string())?;
+    senders
+        .receipt
+        .send(HealReceiptCommand { request, response_tx })
+        .map_err(|err| format!("Failed to send heal receipt command: {err}"))?;
+    response_rx
+        .await
+        .map_err(|e| format!("Failed to receive heal admission response: {e}"))?
+}
+
 /// Send heal start request and wait for structured admission feedback.
 pub async fn send_heal_request_with_admission(request: HealChannelRequest) -> Result<HealAdmissionResult, String> {
     let (response_tx, response_rx) = oneshot::channel();

crates/common/src/metrics.rs

@@ -2434,6 +2434,18 @@ impl Metrics {
         self.current_scan_cycle_work_active.store(false, Ordering::Relaxed);
     }
 
+    pub fn current_scan_cycle_has_unresolved_heal_work(&self) -> bool {
+        if !self.current_scan_cycle_work_active.load(Ordering::Relaxed) {
+            return false;
+        }
+
+        let source_work = self.scanner_source_work_since(&self.current_scan_cycle_source_work_start_values());
+        [ScannerWorkSource::Heal, ScannerWorkSource::Bitrot]
+            .into_iter()
+            .filter_map(|source| source_work.get(source.index()))
+            .any(|work| work.queued > 0 || work.skipped > 0 || work.failed > 0 || work.missed > 0)
+    }
+
     fn scan_cycle_work_snapshot(&self) -> ScanCycleWorkSnapshot {
         ScanCycleWorkSnapshot {
             objects_scanned: self.lifetime(Metric::ScanObject),
@@ -3361,6 +3373,40 @@ mod tests {
         metrics.finish_scan_cycle_work(start);
     }
 
+    #[test]
+    fn unresolved_heal_work_only_reflects_the_active_cycle() {
+        let metrics = Metrics::new();
+        assert!(!metrics.current_scan_cycle_has_unresolved_heal_work());
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_missed(ScannerWorkSource::Heal, 1);
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+
+        metrics.finish_scan_cycle_work(start);
+        assert!(!metrics.current_scan_cycle_has_unresolved_heal_work());
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_queued(ScannerWorkSource::Heal, 1);
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+        metrics.finish_scan_cycle_work(start);
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_work(
+            ScannerWorkSource::Bitrot,
+            ScannerSourceWorkUpdate {
+                skipped: 1,
+                ..Default::default()
+            },
+        );
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+        metrics.finish_scan_cycle_work(start);
+
+        let start = metrics.start_scan_cycle_work();
+        metrics.record_scanner_source_failed(ScannerWorkSource::Bitrot, 1);
+        assert!(metrics.current_scan_cycle_has_unresolved_heal_work());
+        metrics.finish_scan_cycle_work(start);
+    }
+
     #[tokio::test]
     async fn report_marks_transition_failures_as_blocked_lifecycle_control() {
         let metrics = Metrics::new();

crates/config/src/constants/api.rs

@@ -50,3 +50,43 @@ pub const ENV_API_RATE_LIMIT_BURST: &str = "RUSTFS_API_RATE_LIMIT_BURST";
 
 /// Default for `RUSTFS_API_RATE_LIMIT_BURST` (`0` = same as RPM).
 pub const DEFAULT_API_RATE_LIMIT_BURST: u32 = 0;
+
+/// Sustained S3 API request budget per addressed bucket, in requests per
+/// minute — a collective ceiling shared by all clients of that bucket.
+///
+/// Complements the per-client-IP dimension: it protects the server from one
+/// hot bucket regardless of how many client IPs the traffic comes from. `0`
+/// disables the bucket dimension. Requires `RUSTFS_API_RATE_LIMIT_ENABLE`.
+/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_RPM
+/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_RPM=60000
+pub const ENV_API_RATE_LIMIT_BUCKET_RPM: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_RPM";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_RPM` (`0` = dimension disabled).
+pub const DEFAULT_API_RATE_LIMIT_BUCKET_RPM: u32 = 0;
+
+/// Burst capacity per bucket (maximum tokens in the bucket-dimension bucket).
+///
+/// `0` means "same as `RUSTFS_API_RATE_LIMIT_BUCKET_RPM`".
+/// Environment variable: RUSTFS_API_RATE_LIMIT_BUCKET_BURST
+/// Example: RUSTFS_API_RATE_LIMIT_BUCKET_BURST=2000
+pub const ENV_API_RATE_LIMIT_BUCKET_BURST: &str = "RUSTFS_API_RATE_LIMIT_BUCKET_BURST";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_BUCKET_BURST` (`0` = same as bucket RPM).
+pub const DEFAULT_API_RATE_LIMIT_BUCKET_BURST: u32 = 0;
+
+/// Maximum concurrently served connections on the main API listener.
+///
+/// `0` (the default) means unlimited. When set, the accept loop stops
+/// accepting once the cap is reached and lets the kernel backlog absorb
+/// bursts, releasing capacity as connections close. This bounds file
+/// descriptor and memory usage under a connection flood.
+///
+/// The cap covers everything on the main listener — S3, admin, console,
+/// and internode gRPC — so size it well above peer-node count plus the
+/// expected client concurrency.
+/// Environment variable: RUSTFS_API_MAX_CONNECTIONS
+/// Example: RUSTFS_API_MAX_CONNECTIONS=10000
+pub const ENV_API_MAX_CONNECTIONS: &str = "RUSTFS_API_MAX_CONNECTIONS";
+
+/// Default for `RUSTFS_API_MAX_CONNECTIONS` (`0` = unlimited).
+pub const DEFAULT_API_MAX_CONNECTIONS: usize = 0;

crates/config/src/constants/drive.rs

@@ -39,6 +39,11 @@ pub const DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS: u64 = 5;
 pub const ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS";
 pub const DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: u64 = 5;
 
+/// Maximum time the metacache merge consumer waits for the next visible
+/// `walk_dir()` entry from a reader before detaching it from the merge.
+pub const ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS";
+pub const DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: u64 = 10;
+
 /// Interval in seconds between active health probes for local and remote drives.
 pub const ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_INTERVAL_SECS";
 pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;

crates/credentials/src/credentials.rs

@@ -260,13 +260,10 @@ fn resolve_rpc_secret(env_secret: Option<&str>, global_access: Option<&str>, glo
 
     match (global_access, global_secret) {
         (Some(access_key), Some(secret_key)) => {
-            // Fail closed: never derive the RPC secret while the default secret
-            // key is in effect. The derivation uses `secret_key` as the HMAC key,
-            // so a public default secret yields a publicly computable RPC secret
-            // that any network peer can use to forge internode RPC signatures.
-            // Operators running with default credentials must configure
-            // RUSTFS_RPC_SECRET (or set a non-default RUSTFS_SECRET_KEY) instead.
-            if secret_key.trim() == DEFAULT_SECRET_KEY {
+            // Fail closed when either half of the active credential pair still
+            // uses the public default. Operators must configure both custom
+            // credentials or provide RUSTFS_RPC_SECRET explicitly.
+            if access_key.trim() == DEFAULT_ACCESS_KEY || secret_key.trim() == DEFAULT_SECRET_KEY {
                 return None;
             }
             derive_rpc_secret(access_key, secret_key)
@@ -589,18 +586,11 @@ mod tests {
     fn test_resolve_rpc_secret_rejects_default_credentials_for_derivation() {
         assert!(resolve_rpc_secret(None, None, None).is_none());
 
-        // Fail closed: the default secret key must not yield a derivable RPC
-        // secret, otherwise the derived value is publicly computable and any
-        // network peer can forge internode RPC signatures.
+        // Fail closed when either half of the credential pair uses the public
+        // default.
         assert!(resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).is_none());
-
-        // A default access key paired with a non-default secret key is still
-        // safe to derive: the HMAC key (the secret key) is not public.
-        let expected = derive_rpc_secret(DEFAULT_ACCESS_KEY, "custom-global-secret").expect("secret should derive");
-        assert_eq!(
-            resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some("custom-global-secret")).as_deref(),
-            Some(expected.as_str())
-        );
+        assert!(resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some("custom-global-secret")).is_none());
+        assert!(resolve_rpc_secret(None, Some("custom-access"), Some(DEFAULT_SECRET_KEY)).is_none());
 
         assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-access"), Some("custom-global-secret")).is_none());
     }
@@ -635,6 +625,10 @@ mod tests {
             resolve_rpc_secret(Some("custom-rpc-secret"), None, None).as_deref(),
             Some("custom-rpc-secret")
         );
+        assert_eq!(
+            resolve_rpc_secret(Some("custom-rpc-secret"), Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).as_deref(),
+            Some("custom-rpc-secret")
+        );
         let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
         assert_eq!(
             resolve_rpc_secret(None, Some("custom-access"), Some("custom-global-secret")).as_deref(),

crates/data-usage/src/data_usage.rs

@@ -18,9 +18,27 @@ use std::{
     collections::{HashMap, HashSet},
     hash::{DefaultHasher, Hash, Hasher},
     path::Path,
-    time::SystemTime,
+    time::{Duration, SystemTime},
 };
 
+/// Maximum amount a persisted `last_update` may lead the local wall clock before the
+/// persisted timestamp is treated as untrustworthy.
+///
+/// Invariant: the "skip stale usage update" monotonicity check (incoming `last_update`
+/// <= existing `last_update` => skip persisting) is only valid while the existing
+/// timestamp could plausibly have been produced by a healthy clock. If the on-disk
+/// snapshot is future-dated beyond this tolerance (NTP step-back, or scanner
+/// leadership moving to a node with a slower clock), the comparison would skip every
+/// save forever and freeze admin usage stats; callers must bypass the skip instead.
+pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 * 60);
+
+/// Returns true when `existing_last_update` is ahead of `now` by more than
+/// [`USAGE_LAST_UPDATE_FUTURE_TOLERANCE`], i.e. the persisted timestamp cannot be
+/// trusted for staleness comparisons and a fresh snapshot save must be allowed.
+pub fn usage_last_update_is_untrusted_future(existing_last_update: SystemTime, now: SystemTime) -> bool {
+    existing_last_update > now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE
+}
+
 #[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq)]
 pub struct TierStats {
     pub total_size: u64,
@@ -1297,6 +1315,22 @@ pub struct CompressionTotalInfo {
 mod tests {
     use super::*;
 
+    #[test]
+    fn test_usage_last_update_future_tolerance_boundary() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+
+        // Within tolerance (including the exact boundary) the timestamp is trusted.
+        assert!(!usage_last_update_is_untrusted_future(now, now));
+        assert!(!usage_last_update_is_untrusted_future(now - Duration::from_secs(60), now));
+        assert!(!usage_last_update_is_untrusted_future(now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE, now));
+
+        // Beyond tolerance the persisted timestamp is untrustworthy.
+        assert!(usage_last_update_is_untrusted_future(
+            now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE + Duration::from_secs(1),
+            now
+        ));
+    }
+
     #[test]
     fn test_data_usage_info_creation() {
         let mut info = DataUsageInfo::new();

crates/e2e_test/Cargo.toml

@@ -57,7 +57,7 @@ http.workspace = true
 http-body-util.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
-reqwest = { workspace = true, default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "multipart"] }
+reqwest = { workspace = true, features = ["json", "multipart", "stream"] }
 rustfs-signer.workspace = true
 tracing = { workspace = true }
 tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }

crates/e2e_test/src/admin_pools_test.rs

@@ -0,0 +1,79 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use http::header::HOST;
+use reqwest::StatusCode;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serde::Deserialize;
+use std::error::Error;
+
+#[derive(Debug, Deserialize)]
+struct PoolListItem {
+    id: usize,
+    cmdline: String,
+    status: String,
+}
+
+async fn signed_admin_get(env: &RustFSTestEnvironment, path: &str) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}{path}", env.url);
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
+    let request = http::Request::builder()
+        .method(http::Method::GET)
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+        .body(Body::empty())?;
+    let signed = sign_v4(request, 0, &env.access_key, &env.secret_key, "", "us-east-1");
+
+    let mut request = local_http_client().get(&url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    Ok(request.send().await?)
+}
+
+#[tokio::test]
+async fn single_drive_pools_list_succeeds_without_enabling_decommission_status() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let response = signed_admin_get(&env, "/rustfs/admin/v3/pools/list").await?;
+    let status = response.status();
+    let body = response.bytes().await?;
+
+    assert_eq!(status, StatusCode::OK, "pools list failed: {}", String::from_utf8_lossy(&body));
+    let pools: Vec<PoolListItem> = serde_json::from_slice(&body)?;
+    assert_eq!(pools.len(), 1);
+    assert_eq!(pools[0].id, 0);
+    assert_eq!(pools[0].cmdline, env.temp_dir);
+    assert_eq!(pools[0].status, "active");
+
+    let response = signed_admin_get(&env, "/rustfs/admin/v3/decommission/status").await?;
+    let status = response.status();
+    let body = response.text().await?;
+    assert_eq!(
+        status,
+        StatusCode::NOT_IMPLEMENTED,
+        "decommission status changed for a single pool: {body}"
+    );
+    assert!(body.contains("NotImplemented"), "unexpected decommission error body: {body}");
+
+    Ok(())
+}

crates/e2e_test/src/api_rate_limit_test.rs

@@ -86,6 +86,52 @@ async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResu
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn api_rate_limit_bucket_dimension_throttles_per_bucket() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    // Bucket dimension only: the readiness-poll ListBuckets calls hit "/"
+    // (no bucket) and therefore do not consume any budget.
+    env.start_rustfs_server_with_env(
+        vec![],
+        &[
+            ("RUSTFS_API_RATE_LIMIT_ENABLE", "true"),
+            ("RUSTFS_API_RATE_LIMIT_BUCKET_RPM", "60"),
+            ("RUSTFS_API_RATE_LIMIT_BUCKET_BURST", "5"),
+        ],
+    )
+    .await?;
+
+    let client = local_http_client();
+
+    // Unauthenticated GETs are still counted arrivals (403, not 429, while
+    // within budget); the sixth rapid hit on the same bucket must throttle.
+    let mut throttled = false;
+    for i in 0..6 {
+        let response = client.get(format!("{}/hot-bucket/object-{i}", env.url)).send().await?;
+        if response.status() == reqwest::StatusCode::TOO_MANY_REQUESTS {
+            throttled = true;
+            assert!(
+                response.headers().contains_key(reqwest::header::RETRY_AFTER),
+                "bucket-dimension 429 must carry Retry-After"
+            );
+            break;
+        }
+    }
+    assert!(throttled, "6 rapid requests against burst 5 must trip the bucket dimension");
+
+    // A different bucket has its own budget.
+    let other = client.get(format!("{}/cold-bucket/object", env.url)).send().await?;
+    assert_ne!(
+        other.status(),
+        reqwest::StatusCode::TOO_MANY_REQUESTS,
+        "an unrelated bucket must not be throttled"
+    );
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn api_rate_limit_stays_inert_by_default() -> TestResult {

crates/e2e_test/src/common.rs

@@ -979,6 +979,7 @@ impl RustFSTestClusterEnvironment {
         }
 
         let mut extra_env = Vec::new();
+        extra_env.push(("RUSTFS_RPC_SECRET".to_string(), String::new()));
         if multidrive {
             extra_env.push(("RUSTFS_UNSAFE_BYPASS_DISK_CHECK".to_string(), "true".to_string()));
         }
@@ -986,8 +987,8 @@ impl RustFSTestClusterEnvironment {
         Ok(Self {
             nodes,
             temp_dir,
-            access_key: DEFAULT_ACCESS_KEY.to_string(),
-            secret_key: DEFAULT_SECRET_KEY.to_string(),
+            access_key: "rustfs-cluster-test-access".to_string(),
+            secret_key: "rustfs-cluster-test-secret".to_string(),
             extra_env,
             topology,
         })

crates/e2e_test/src/connection_cap_test.rs

@@ -0,0 +1,127 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! E2E coverage for the opt-in global connection cap on the main API listener
+//! (backlog#1191 follow-up, `RUSTFS_API_MAX_CONNECTIONS`): permits must be
+//! released when connections close (no leak), and the cap must actually bound
+//! concurrency — a queued connection is served only after a held one closes.
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use serial_test::serial;
+use std::time::Duration;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::TcpStream;
+use tokio::time::timeout;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+/// Open a TCP connection and write one unauthenticated `GET /` (any response,
+/// e.g. 403, proves the connection was accepted and served).
+async fn open_and_request(addr: &str, connection: &str) -> std::io::Result<TcpStream> {
+    let mut stream = TcpStream::connect(addr).await?;
+    let request = format!("GET / HTTP/1.1\r\nHost: {addr}\r\nConnection: {connection}\r\n\r\n");
+    stream.write_all(request.as_bytes()).await?;
+    Ok(stream)
+}
+
+/// Read until the response head is complete, or `None` on timeout/close —
+/// a `None` on an open socket means the connection sits unaccepted in the
+/// kernel backlog behind the cap.
+async fn read_response_head(stream: &mut TcpStream, dur: Duration) -> Option<String> {
+    let deadline = tokio::time::Instant::now() + dur;
+    let mut buf = vec![0u8; 4096];
+    let mut collected = String::new();
+    loop {
+        let remaining = deadline.checked_duration_since(tokio::time::Instant::now())?;
+        match timeout(remaining, stream.read(&mut buf)).await {
+            Ok(Ok(0)) | Ok(Err(_)) | Err(_) => return None,
+            Ok(Ok(n)) => {
+                collected.push_str(&String::from_utf8_lossy(&buf[..n]));
+                if collected.contains("\r\n\r\n") {
+                    return Some(collected);
+                }
+            }
+        }
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn connection_cap_releases_permits_on_close() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_API_MAX_CONNECTIONS", "2")])
+        .await?;
+
+    // Ten sequential connections against cap 2: if permits leaked, the third
+    // request would already hang in the backlog and time out.
+    for i in 0..10 {
+        let mut stream = open_and_request(&env.address, "close").await?;
+        let head = read_response_head(&mut stream, Duration::from_secs(10))
+            .await
+            .unwrap_or_else(|| panic!("request {i} got no response — a connection permit leaked"));
+        assert!(head.starts_with("HTTP/1.1"), "request {i} unexpected response: {head}");
+    }
+
+    Ok(())
+}
+
+/// Open a TCP connection and send an INCOMPLETE request head. Once accepted
+/// it pins a connection permit: hyper waits for the rest of the head (75s
+/// default header timeout) until we close the socket.
+async fn open_and_stall(addr: &str) -> std::io::Result<TcpStream> {
+    let mut stream = TcpStream::connect(addr).await?;
+    stream
+        .write_all(format!("GET / HTTP/1.1\r\nHost: {addr}\r\n").as_bytes())
+        .await?;
+    Ok(stream)
+}
+
+#[tokio::test]
+#[serial]
+async fn connection_cap_blocks_excess_connections_until_permits_free() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_API_MAX_CONNECTIONS", "2")])
+        .await?;
+    // Let the readiness poller's pooled connection close and free its permit.
+    tokio::time::sleep(Duration::from_secs(1)).await;
+
+    // Two stalled connections saturate cap 2 (a served-and-closed connection
+    // would release its permit immediately, so stalling is what makes the
+    // occupancy deterministic).
+    let stalled_a = open_and_stall(&env.address).await?;
+    let stalled_b = open_and_stall(&env.address).await?;
+    tokio::time::sleep(Duration::from_millis(300)).await;
+
+    // A complete request now sits in the kernel backlog: connect() succeeds
+    // but no permit is available, so no response arrives.
+    let mut blocked = open_and_request(&env.address, "close").await?;
+    assert!(
+        read_response_head(&mut blocked, Duration::from_secs(3)).await.is_none(),
+        "cap 2 with two stalled connections must leave the third unserved"
+    );
+
+    // Dropping the stalled connections releases their permits (hyper sees
+    // EOF while reading the head); the queued request's bytes already sit in
+    // the socket buffer, so it must now be accepted and served.
+    drop(stalled_a);
+    drop(stalled_b);
+    let head = read_response_head(&mut blocked, Duration::from_secs(10))
+        .await
+        .expect("queued connection must be served after permits are released");
+    assert!(head.starts_with("HTTP/1.1"), "unexpected response: {head}");
+
+    Ok(())
+}

crates/e2e_test/src/copy_object_checksum_test.rs

@@ -0,0 +1,307 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Regression test for Issue #4996: CopyObject must return the destination object's
+//! checksum in `CopyObjectResult` and persist it so a later checksum-mode HEAD/GET
+//! returns the same value. Covers both the requested-algorithm case (compute fresh)
+//! and the no-algorithm case (preserve the source object's existing checksum).
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use aws_sdk_s3::types::{BucketVersioningStatus, ChecksumAlgorithm, ChecksumMode, VersioningConfiguration};
+    use base64::Engine as _;
+    use base64::engine::general_purpose::STANDARD as BASE64;
+    use serial_test::serial;
+    use sha2::{Digest, Sha256};
+    use tracing::info;
+
+    async fn create_versioned_bucket(client: &aws_sdk_s3::Client, bucket: &str) {
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("Failed to create bucket");
+        client
+            .put_bucket_versioning()
+            .bucket(bucket)
+            .versioning_configuration(
+                VersioningConfiguration::builder()
+                    .status(BucketVersioningStatus::Enabled)
+                    .build(),
+            )
+            .send()
+            .await
+            .expect("Failed to enable versioning");
+    }
+
+    /// Requested algorithm: a CopyObject asking for SHA256 must compute it over the copied
+    /// bytes, return it in `CopyObjectResult.ChecksumSHA256`, and persist it so a checksum-mode
+    /// HEAD on the destination returns the identical value.
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_with_checksum_algorithm_returns_and_persists_sha256() {
+        init_logging();
+        info!("Issue #4996: CopyObject with ChecksumAlgorithm=SHA256 must return and persist the checksum");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-checksum-req-src";
+        let dst_bucket = "copy-checksum-req-dst";
+        let src_key = "objects/source.bin";
+        let dst_key = "objects/dest.bin";
+
+        create_versioned_bucket(&client, src_bucket).await;
+        create_versioned_bucket(&client, dst_bucket).await;
+
+        let content = b"deterministic synthetic payload for copy-object checksum #4996";
+        let expected_sha256 = BASE64.encode(Sha256::digest(content));
+
+        client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT source failed");
+
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}"))
+            .checksum_algorithm(ChecksumAlgorithm::Sha256)
+            .send()
+            .await
+            .expect("CopyObject with ChecksumAlgorithm must succeed");
+
+        // (3) The response must carry the freshly computed SHA-256 of the copied bytes.
+        let result = copy_out
+            .copy_object_result()
+            .expect("issue #4996: CopyObject must return a CopyObjectResult");
+        assert_eq!(
+            result.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: CopyObjectResult.ChecksumSHA256 must equal the SHA-256 of the copied bytes"
+        );
+
+        // (4) A checksum-mode HEAD on the destination must return the same SHA-256.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .checksum_mode(ChecksumMode::Enabled)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: destination checksum-mode HEAD must return the same SHA-256 the copy reported"
+        );
+
+        env.stop_server();
+    }
+
+    /// No algorithm requested: when the source object already carries a checksum, the copy must
+    /// preserve it on the destination (AWS default), visible via a checksum-mode HEAD.
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_without_algorithm_preserves_source_checksum() {
+        init_logging();
+        info!("Issue #4996: CopyObject without ChecksumAlgorithm must preserve the source object's checksum");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-checksum-preserve-src";
+        let dst_bucket = "copy-checksum-preserve-dst";
+        let src_key = "objects/source.bin";
+        let dst_key = "objects/dest.bin";
+
+        create_versioned_bucket(&client, src_bucket).await;
+        create_versioned_bucket(&client, dst_bucket).await;
+
+        let content = b"another deterministic payload whose source checksum must survive the copy";
+        let expected_sha256 = BASE64.encode(Sha256::digest(content));
+
+        // Store the source WITH a SHA-256 checksum so it has one to preserve.
+        let put_src = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .checksum_algorithm(ChecksumAlgorithm::Sha256)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT source with checksum failed");
+        assert_eq!(
+            put_src.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "source PUT must report the SHA-256 it stored"
+        );
+
+        // Copy WITHOUT specifying a checksum algorithm.
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}"))
+            .send()
+            .await
+            .expect("CopyObject without ChecksumAlgorithm must succeed");
+
+        // The response should echo the preserved source checksum.
+        let result = copy_out
+            .copy_object_result()
+            .expect("issue #4996: CopyObject must return a CopyObjectResult");
+        assert_eq!(
+            result.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: a no-algorithm copy must preserve and report the source object's SHA-256"
+        );
+
+        // And a checksum-mode HEAD on the destination must return that same preserved SHA-256.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .checksum_mode(ChecksumMode::Enabled)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.checksum_sha256(),
+            Some(expected_sha256.as_str()),
+            "issue #4996: destination checksum-mode HEAD must return the preserved source SHA-256"
+        );
+
+        env.stop_server();
+    }
+
+    /// Requested algorithm differs from the source's: a source stored with SHA256, copied while
+    /// requesting CRC32, must return/persist the freshly computed CRC32 and must NOT carry the
+    /// source's SHA256 through. Guards the request-over-source precedence and the destination's
+    /// checksum-not-inherited path, and exercises the CRC32 code path (a different branch of
+    /// ChecksumType::from_string than SHA256).
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_requested_algorithm_overrides_source_checksum() {
+        init_logging();
+        info!("Issue #4996: a requested CopyObject checksum algorithm must override the source object's algorithm");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-checksum-override-src";
+        let dst_bucket = "copy-checksum-override-dst";
+        let src_key = "objects/source.bin";
+        let ref_key = "objects/reference-crc32.bin";
+        let dst_key = "objects/dest.bin";
+
+        create_versioned_bucket(&client, src_bucket).await;
+        create_versioned_bucket(&client, dst_bucket).await;
+
+        let content = b"payload whose copy must be re-checksummed with a different algorithm";
+        let expected_sha256 = BASE64.encode(Sha256::digest(content));
+
+        // Source is stored WITH a SHA-256 checksum.
+        client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .checksum_algorithm(ChecksumAlgorithm::Sha256)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("PUT source with SHA256 failed");
+
+        // Establish the canonical CRC32 the server computes for this content via a reference PUT,
+        // so the copy's CRC32 can be asserted against an exact server-computed value.
+        let ref_put = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(ref_key)
+            .checksum_algorithm(ChecksumAlgorithm::Crc32)
+            .body(ByteStream::from_static(content))
+            .send()
+            .await
+            .expect("reference PUT with CRC32 failed");
+        let expected_crc32 = ref_put
+            .checksum_crc32()
+            .expect("reference PUT must report a CRC32")
+            .to_string();
+
+        // Copy the SHA256 source while requesting CRC32.
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}"))
+            .checksum_algorithm(ChecksumAlgorithm::Crc32)
+            .send()
+            .await
+            .expect("CopyObject requesting a different algorithm must succeed");
+
+        let result = copy_out
+            .copy_object_result()
+            .expect("issue #4996: CopyObject must return a CopyObjectResult");
+        // The requested CRC32 must be computed and returned.
+        assert_eq!(
+            result.checksum_crc32(),
+            Some(expected_crc32.as_str()),
+            "issue #4996: a requested CRC32 must be computed fresh over the copied bytes"
+        );
+        // The source's SHA256 must NOT leak through — the requested algorithm wins.
+        assert_eq!(
+            result.checksum_sha256(),
+            None,
+            "issue #4996: the source object's SHA256 must not be inherited when a different algorithm is requested"
+        );
+        assert_ne!(
+            result.checksum_crc32(),
+            Some(expected_sha256.as_str()),
+            "sanity: CRC32 field must not carry the SHA256 value"
+        );
+
+        // The destination must persist CRC32 (and only CRC32) for a checksum-mode HEAD.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .checksum_mode(ChecksumMode::Enabled)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.checksum_crc32(),
+            Some(expected_crc32.as_str()),
+            "issue #4996: destination checksum-mode HEAD must return the requested CRC32"
+        );
+        assert_eq!(
+            head.checksum_sha256(),
+            None,
+            "issue #4996: destination must not report the source's SHA256 after an override copy"
+        );
+
+        env.stop_server();
+    }
+}

crates/e2e_test/src/copy_object_version_restore_test.rs

@@ -160,4 +160,146 @@ mod tests {
 
         env.stop_server();
     }
+
+    /// Regression test for Issue #4976: a versioned-source CopyObject must echo the exact source
+    /// version copied via `x-amz-copy-source-version-id` (SDK `CopySourceVersionId`), kept distinct
+    /// from the newly created destination `x-amz-version-id`.
+    #[tokio::test]
+    #[serial]
+    async fn test_copy_of_non_latest_source_version_returns_copy_source_version_id() {
+        init_logging();
+        info!("Issue #4976: versioned CopyObject must return x-amz-copy-source-version-id for the exact source version");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = env.create_s3_client();
+        let src_bucket = "copy-source-version-header-src";
+        let dst_bucket = "copy-source-version-header-dst";
+        let src_key = "reports/quarantined.bin";
+        let dst_key = "reports/promoted.bin";
+
+        for bucket in [src_bucket, dst_bucket] {
+            client
+                .create_bucket()
+                .bucket(bucket)
+                .send()
+                .await
+                .expect("Failed to create bucket");
+            client
+                .put_bucket_versioning()
+                .bucket(bucket)
+                .versioning_configuration(
+                    VersioningConfiguration::builder()
+                        .status(BucketVersioningStatus::Enabled)
+                        .build(),
+                )
+                .send()
+                .await
+                .expect("Failed to enable versioning");
+        }
+
+        // Source version 1: the exact (non-latest) version we will copy.
+        let v1_content = b"quarantined payload -- source version one (target of the copy)";
+        let put_v1 = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .body(ByteStream::from_static(v1_content))
+            .send()
+            .await
+            .expect("PUT source v1 failed");
+        let v1_id = put_v1.version_id().expect("source v1 must have a version id").to_string();
+
+        // Source version 2: becomes the latest, so v1 is deliberately NOT the current version.
+        let v2_content = b"quarantined payload -- source version two (now current, must be ignored)";
+        let put_v2 = client
+            .put_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .body(ByteStream::from_static(v2_content))
+            .send()
+            .await
+            .expect("PUT source v2 failed");
+        let v2_id = put_v2.version_id().expect("source v2 must have a version id").to_string();
+        assert_ne!(v1_id, v2_id, "the two source puts must produce distinct versions");
+
+        // Copy the exact NON-LATEST source version into the destination bucket.
+        let copy_out = client
+            .copy_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .copy_source(format!("{src_bucket}/{src_key}?versionId={v1_id}"))
+            .send()
+            .await
+            .expect("CopyObject of a versioned source must succeed");
+
+        // (3) The response must echo the exact source version copied.
+        let copy_source_version_id = copy_out
+            .copy_source_version_id()
+            .expect("issue #4976: response must include x-amz-copy-source-version-id for a versioned source");
+        assert_eq!(
+            copy_source_version_id, v1_id,
+            "x-amz-copy-source-version-id must equal the exact source version requested, not the latest source version"
+        );
+        assert_ne!(
+            copy_source_version_id, v2_id,
+            "x-amz-copy-source-version-id must not be the latest source version"
+        );
+
+        // (4) The destination header must identify a distinct, newly created version.
+        let dst_version_id = copy_out
+            .version_id()
+            .expect("destination copy must create a new version id")
+            .to_string();
+        assert!(!dst_version_id.is_empty(), "destination version id must be present");
+        assert_ne!(dst_version_id, v1_id, "destination version must be distinct from the source version");
+        assert_ne!(
+            dst_version_id, v2_id,
+            "destination version must be distinct from the source latest version"
+        );
+
+        // (5) The destination must hold the exact bytes/size of the copied (v1) source version.
+        let head = client
+            .head_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .send()
+            .await
+            .expect("HEAD destination failed");
+        assert_eq!(
+            head.content_length(),
+            Some(v1_content.len() as i64),
+            "destination size must equal the copied source version (v1)"
+        );
+
+        let get_dst = client
+            .get_object()
+            .bucket(dst_bucket)
+            .key(dst_key)
+            .version_id(&dst_version_id)
+            .send()
+            .await
+            .expect("GET destination failed");
+        let dst_body = get_dst.body.collect().await.expect("collect destination body").into_bytes();
+        assert_eq!(
+            dst_body.as_ref(),
+            v1_content,
+            "destination bytes must exactly equal the copied source version (v1), not the latest (v2)"
+        );
+
+        // (6) The source version copied from must remain present and independently readable.
+        let get_src_v1 = client
+            .get_object()
+            .bucket(src_bucket)
+            .key(src_key)
+            .version_id(&v1_id)
+            .send()
+            .await
+            .expect("GET source v1 failed after copy");
+        let src_v1_body = get_src_v1.body.collect().await.expect("collect source v1 body").into_bytes();
+        assert_eq!(src_v1_body.as_ref(), v1_content, "source v1 must remain intact after the copy");
+
+        env.stop_server();
+    }
 }

crates/e2e_test/src/lib.rs

@@ -83,6 +83,10 @@ mod security_boundary_test;
 #[cfg(test)]
 mod api_rate_limit_test;
 
+// Opt-in global connection cap on the main listener (backlog#1191 follow-up)
+#[cfg(test)]
+mod connection_cap_test;
+
 // Admin authorization gate: non-admin denial + root-credential lifecycle (backlog#1151 sec-4)
 #[cfg(test)]
 mod admin_auth_test;
@@ -186,6 +190,9 @@ mod copy_object_metadata_test;
 #[cfg(test)]
 mod copy_object_version_restore_test;
 
+#[cfg(test)]
+mod copy_object_checksum_test;
+
 // S3 dummy-compat bucket API tests
 #[cfg(test)]
 mod bucket_logging_test;
@@ -235,6 +242,9 @@ mod console_smoke_test;
 #[cfg(test)]
 mod admin_iam_crud_test;
 
+#[cfg(test)]
+mod admin_pools_test;
+
 // Replication extension end-to-end regression tests
 #[cfg(test)]
 mod replication_extension_test;

crates/e2e_test/src/multipart_auth_test.rs

@@ -53,6 +53,17 @@ fn sse_customer_key_md5_base64(key: &str) -> String {
     base64::engine::general_purpose::STANDARD.encode(md5::compute(key).0)
 }
 
+/// Env var consumed by the local SSE-S3 DEK provider when KMS is not configured.
+///
+/// Since rustfs#3564 the server fails closed on managed SSE (SSE-S3 or
+/// bucket-default encryption) unless KMS is configured or this master key is
+/// provided, so tests exercising managed SSE on a bare server must seed it.
+const LOCAL_SSE_MASTER_KEY_ENV: &str = "RUSTFS_SSE_S3_MASTER_KEY";
+
+fn local_sse_master_key_value() -> String {
+    base64::engine::general_purpose::STANDARD.encode([0x42u8; 32])
+}
+
 async fn make_tar(files: &[(&str, &[u8])], dirs: &[&str]) -> Vec<u8> {
     let buf = Cursor::new(Vec::new());
     let mut builder = tokio_tar::Builder::new(buf);
@@ -105,7 +116,11 @@ async fn make_tar_with_pax_entry(path: &str, data: &[u8], mtime: Option<u64>, pa
             pax_payload.extend(build_pax_record(key, value));
         }
 
-        let mut pax_header = tokio_tar::Header::new_gnu();
+        // Pax extension entries must carry a POSIX ustar header — this is what real
+        // tar writers emit, and the server-side reader rejects an XHeader typeflag on
+        // GNU-format headers ("extension typeflag is not permitted on an unrecognized
+        // header").
+        let mut pax_header = tokio_tar::Header::new_ustar();
         pax_header.set_entry_type(tokio_tar::EntryType::XHeader);
         pax_header.set_size(pax_payload.len() as u64);
         pax_header.set_mode(0o644);
@@ -926,7 +941,9 @@ async fn test_anonymous_post_object_accepts_sse_s3() -> Result<(), Box<dyn std::
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-sse-s3";
     let object_key = "post-sse-s3-object.txt";
@@ -982,7 +999,9 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_s3() -> Result<(), B
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-default-sse-s3";
     let object_key = "post-default-sse-s3-object.txt";
@@ -1053,7 +1072,9 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_kms() -> Result<(),
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-default-sse-kms";
     let object_key = "post-default-sse-kms-object.txt";
@@ -1172,15 +1193,24 @@ async fn test_anonymous_post_object_rejects_sse_s3_policy_mismatch() -> Result<(
 
 #[tokio::test]
 #[serial]
-async fn test_anonymous_post_object_rejects_sse_s3_missing_from_policy_conditions()
+async fn test_anonymous_post_object_accepts_sse_s3_missing_from_policy_conditions()
 -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
 
+    // MinIO-compatible POST-policy validation (s3s-project/s3s#608) exempts the
+    // x-amz-server-side-encryption* form fields from the "every form field must
+    // appear in the policy conditions" rule, so an SSE-S3 field that the policy
+    // does not mention is accepted and encryption is applied. When the policy
+    // does cover the field, a value mismatch is still rejected — see
+    // test_anonymous_post_object_rejects_sse_s3_policy_mismatch.
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let master_key = local_sse_master_key_value();
+    env.start_rustfs_server_with_env(vec![], &[(LOCAL_SSE_MASTER_KEY_ENV, master_key.as_str())])
+        .await?;
 
     let bucket = "anon-post-sse-s3-missing";
     let object_key = "post-sse-s3-missing-object.txt";
+    let expected_body = b"post-sse-s3-missing".to_vec();
 
     let admin_client = env.create_s3_client();
     admin_client.create_bucket().bucket(bucket).send().await?;
@@ -1198,7 +1228,7 @@ async fn test_anonymous_post_object_rejects_sse_s3_missing_from_policy_condition
         .text("x-amz-server-side-encryption", "AES256")
         .part(
             "file",
-            reqwest::multipart::Part::bytes(b"post-sse-s3-missing".to_vec())
+            reqwest::multipart::Part::bytes(expected_body.clone())
                 .file_name("upload.txt")
                 .mime_str("text/plain")?,
         );
@@ -1212,11 +1242,15 @@ async fn test_anonymous_post_object_rejects_sse_s3_missing_from_policy_condition
     let status = post_resp.status();
     let response_body = post_resp.text().await?;
 
-    assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
-    assert!(
-        response_body.contains("<Code>AccessDenied</Code>"),
-        "response should contain AccessDenied code, got: {response_body}"
-    );
+    assert_eq!(status, reqwest::StatusCode::NO_CONTENT);
+    assert!(response_body.is_empty(), "204 response should not contain a body, got: {response_body}");
+
+    let head = admin_client.head_object().bucket(bucket).key(object_key).send().await?;
+    assert_eq!(head.server_side_encryption().map(|value| value.as_str()), Some("AES256"));
+
+    let uploaded = admin_client.get_object().bucket(bucket).key(object_key).send().await?;
+    let uploaded = uploaded.body.collect().await?.into_bytes();
+    assert_eq!(uploaded.as_ref(), expected_body.as_slice());
 
     Ok(())
 }
@@ -5041,7 +5075,9 @@ async fn test_signed_put_object_extract_preserves_sse_s3_and_redirect() -> Resul
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let sse_master_key = base64::engine::general_purpose::STANDARD.encode([0x42u8; 32]);
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_SSE_S3_MASTER_KEY", sse_master_key.as_str())])
+        .await?;
 
     let bucket = "signed-extract-sse-s3-redirect";
     let archive_key = "encrypted-metadata.tar";
@@ -5318,7 +5354,9 @@ async fn test_signed_put_object_extract_uses_bucket_default_sse_s3() -> Result<(
     init_logging();
 
     let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
+    let sse_master_key = base64::engine::general_purpose::STANDARD.encode([0x42u8; 32]);
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_SSE_S3_MASTER_KEY", sse_master_key.as_str())])
+        .await?;
 
     let bucket = "signed-extract-default-sse-s3";
     let archive_key = "default-encryption.tar";

crates/e2e_test/src/notification_webhook_test.rs

@@ -43,6 +43,12 @@ use s3s::Body;
 use serde_json::Value;
 use serial_test::serial;
 use std::error::Error;
+use std::path::Path;
+use std::sync::{
+    Arc, Once,
+    atomic::{AtomicBool, Ordering},
+};
+use std::thread;
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::TcpListener;
 use tokio::sync::mpsc;
@@ -150,6 +156,83 @@ async fn spawn_event_collector() -> Result<(String, mpsc::UnboundedReceiver<Valu
     Ok((format!("http://{endpoint_ip}.nip.io:{port}/events"), rx, handle))
 }
 
+fn spawn_https_event_collector(ca_path: &Path) -> Result<(String, Arc<AtomicBool>, thread::JoinHandle<()>), BoxError> {
+    use rustls::{
+        ServerConfig,
+        pki_types::{PrivateKeyDer, PrivatePkcs8KeyDer},
+    };
+    use std::io::ErrorKind;
+    use std::net::TcpListener as StdTcpListener;
+
+    static INSTALL_CRYPTO_PROVIDER: Once = Once::new();
+    INSTALL_CRYPTO_PROVIDER.call_once(|| {
+        let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+    });
+
+    let listener = StdTcpListener::bind("0.0.0.0:0")?;
+    listener.set_nonblocking(true)?;
+    let addr = listener.local_addr()?;
+    let endpoint_ip = local_ip()?;
+    let endpoint_host = format!("{endpoint_ip}.nip.io");
+
+    let rcgen::CertifiedKey { cert, signing_key } = rcgen::generate_simple_self_signed(vec![endpoint_host.clone()])?;
+    std::fs::write(ca_path, cert.pem())?;
+
+    let cert_chain = vec![cert.der().clone()];
+    let key_der = PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(signing_key.serialize_der()));
+    let server_config = Arc::new(
+        ServerConfig::builder()
+            .with_no_client_auth()
+            .with_single_cert(cert_chain, key_der)?,
+    );
+
+    let running = Arc::new(AtomicBool::new(true));
+    let server_running = Arc::clone(&running);
+    let handle = thread::spawn(move || {
+        while server_running.load(Ordering::Relaxed) {
+            match listener.accept() {
+                Ok((stream, _)) => {
+                    let config = Arc::clone(&server_config);
+                    handle_https_probe(stream, config);
+                }
+                Err(err) if err.kind() == ErrorKind::WouldBlock => thread::sleep(Duration::from_millis(20)),
+                Err(_) => break,
+            }
+        }
+    });
+
+    Ok((format!("https://{endpoint_host}:{}/events", addr.port()), running, handle))
+}
+
+fn handle_https_probe(stream: std::net::TcpStream, server_config: Arc<rustls::ServerConfig>) {
+    use std::io::{Read, Write};
+
+    let _ = stream.set_read_timeout(Some(Duration::from_secs(5)));
+    let _ = stream.set_write_timeout(Some(Duration::from_secs(5)));
+    let Ok(connection) = rustls::ServerConnection::new(server_config) else {
+        return;
+    };
+    let mut tls_stream = rustls::StreamOwned::new(connection, stream);
+    let mut buf = [0u8; 1024];
+    if tls_stream.read(&mut buf).is_err() {
+        return;
+    }
+    let response = "HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n";
+    let _ = tls_stream.write_all(response.as_bytes());
+    let _ = tls_stream.flush();
+}
+
+fn stop_https_event_collector(endpoint: &str, running: Arc<AtomicBool>, handle: thread::JoinHandle<()>) -> TestResult {
+    running.store(false, Ordering::Relaxed);
+    if let Ok(parsed) = endpoint.parse::<reqwest::Url>()
+        && let Some(port) = parsed.port()
+    {
+        let _ = std::net::TcpStream::connect(("127.0.0.1", port));
+    }
+    handle.join().map_err(|_| "https event collector thread panicked")?;
+    Ok(())
+}
+
 /// Decoded object key of the first record in an event envelope.
 fn event_key(envelope: &Value) -> Option<String> {
     let raw = envelope["Records"][0]["s3"]["object"]["key"].as_str()?;
@@ -284,13 +367,24 @@ async fn enable_notify_module(env: &RustFSTestEnvironment) -> TestResult {
 /// Registers a webhook notification target with a persistent queue directory, so
 /// delivery goes through the durable store-and-forward path.
 async fn configure_webhook_target(env: &RustFSTestEnvironment, target_name: &str, endpoint: &str) -> TestResult {
+    configure_webhook_target_with_key_values(env, target_name, vec![("endpoint", endpoint.to_string())]).await
+}
+
+async fn configure_webhook_target_with_key_values(
+    env: &RustFSTestEnvironment,
+    target_name: &str,
+    mut key_values: Vec<(&str, String)>,
+) -> TestResult {
     let queue_dir = format!("{}/notify-queue-{target_name}", env.temp_dir);
     tokio::fs::create_dir_all(&queue_dir).await?;
+    if !key_values.iter().any(|(key, _)| *key == "queue_dir") {
+        key_values.push(("queue_dir", queue_dir));
+    }
     let payload = serde_json::json!({
-        "key_values": [
-            { "key": "endpoint", "value": endpoint },
-            { "key": "queue_dir", "value": queue_dir },
-        ]
+        "key_values": key_values
+            .into_iter()
+            .map(|(key, value)| serde_json::json!({ "key": key, "value": value }))
+            .collect::<Vec<_>>(),
     });
     let url = format!("{}/rustfs/admin/v3/target/notify_webhook/{target_name}", env.url);
     let response = signed_admin_request(env, http::Method::PUT, &url, Some(payload.to_string().into_bytes())).await?;
@@ -321,6 +415,28 @@ async fn wait_for_target_registered(env: &RustFSTestEnvironment, target_name: &s
     Err(format!("target {target_name} was not registered in admin ARNs").into())
 }
 
+async fn wait_for_target_listed(env: &RustFSTestEnvironment, target_name: &str) -> TestResult {
+    let url = format!("{}/rustfs/admin/v3/target/list", env.url);
+    for _ in 0..40 {
+        let response = signed_admin_request(env, http::Method::GET, &url, None).await?;
+        if response.status() == StatusCode::OK {
+            let body: Value = serde_json::from_slice(&response.bytes().await?)?;
+            let listed = body["notification_endpoints"].as_array().is_some_and(|endpoints| {
+                endpoints.iter().any(|endpoint| {
+                    endpoint["account_id"].as_str() == Some(target_name)
+                        && endpoint["service"].as_str() == Some("webhook")
+                        && endpoint["status"].as_str().is_some()
+                })
+            });
+            if listed {
+                return Ok(());
+            }
+        }
+        tokio::time::sleep(Duration::from_millis(250)).await;
+    }
+    Err(format!("target {target_name} was not listed in admin targets").into())
+}
+
 /// Binds a bucket to a webhook target for ObjectCreated:*/ObjectRemoved:* events,
 /// filtered to `prefix` + `suffix`.
 async fn put_notification_config(client: &Client, bucket: &str, target_name: &str, prefix: &str, suffix: &str) -> TestResult {
@@ -367,6 +483,38 @@ fn trimmed_etag(value: Option<&str>) -> Option<String> {
 // Tests
 // ---------------------------------------------------------------------------
 
+/// Regression for rustfs#5052: with the notify module enabled through
+/// RUSTFS_NOTIFY_ENABLE, an HTTPS webhook using a configured CA must be accepted
+/// and remain visible in the admin target list.
+#[tokio::test]
+#[serial]
+async fn test_https_webhook_target_lists_with_notify_env_enabled() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_NOTIFY_ENABLE", "true")])
+        .await?;
+
+    let ca_path = Path::new(&env.temp_dir).join("https-webhook-ca.pem");
+    let (endpoint, running, handle) = spawn_https_event_collector(&ca_path)?;
+    let target = "peri1https";
+
+    configure_webhook_target_with_key_values(
+        &env,
+        target,
+        vec![
+            ("endpoint", endpoint.clone()),
+            ("client_ca", ca_path.to_string_lossy().into_owned()),
+        ],
+    )
+    .await?;
+    wait_for_target_listed(&env, target).await?;
+
+    env.stop_server();
+    stop_https_event_collector(&endpoint, running, handle)?;
+    Ok(())
+}
+
 /// PUT / multipart-complete / DELETE each deliver one event with correct fields,
 /// and the prefix/suffix filter drops non-matching keys.
 #[tokio::test]

crates/e2e_test/src/quota_test.rs

@@ -277,6 +277,64 @@ mod integration_tests {
         Ok(())
     }
 
+    /// backlog#1336 regression: a PUT that merely declares `Content-Encoding: aws-chunked`
+    /// (no SigV4 streaming payload, so no `x-amz-decoded-content-length`) carries an unframed
+    /// body whose wire Content-Length is the real object size. Quota admission must use that
+    /// length — with and without a hard quota configured — instead of rejecting the request
+    /// with 400 UnexpectedContent, and an over-quota aws-chunked PUT must still get the quota
+    /// rejection.
+    #[tokio::test]
+    #[serial]
+    async fn test_quota_admission_aws_chunked_declared_encoding() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        init_logging();
+        if skip_without_awscurl() {
+            return Ok(());
+        }
+        let env = QuotaTestEnv::new().await?;
+        env.create_bucket().await?;
+
+        let put_aws_chunked = |key: &'static str, size_bytes: usize| {
+            env.client
+                .put_object()
+                .bucket(&env.bucket_name)
+                .key(key)
+                .content_encoding("aws-chunked")
+                .body(aws_sdk_s3::primitives::ByteStream::from(vec![0u8; size_bytes]))
+                .send()
+        };
+
+        // No quota configured: the declared aws-chunked PUT must be admitted.
+        put_aws_chunked("no-quota.bin", 512)
+            .await
+            .expect("declared aws-chunked PUT without quota must succeed");
+        assert!(env.object_exists("no-quota.bin").await?);
+
+        // Hard quota configured: a within-quota declared aws-chunked PUT is admitted
+        // against its wire Content-Length.
+        env.set_bucket_quota(4 * 1024).await?;
+        put_aws_chunked("within-quota.bin", 1024)
+            .await
+            .expect("declared aws-chunked PUT within quota must succeed");
+        assert!(env.object_exists("within-quota.bin").await?);
+
+        // An over-quota declared aws-chunked PUT is rejected by quota admission —
+        // not with UnexpectedContent.
+        let err = put_aws_chunked("over-quota.bin", 16 * 1024)
+            .await
+            .expect_err("declared aws-chunked PUT over quota must be rejected");
+        let err_debug = format!("{err:?}");
+        assert!(
+            !err_debug.contains("UnexpectedContent"),
+            "over-quota rejection must be the quota error, not UnexpectedContent: {err_debug}"
+        );
+        assert!(!env.object_exists("over-quota.bin").await?);
+
+        env.clear_bucket_quota().await?;
+        env.cleanup_bucket().await?;
+
+        Ok(())
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_quota_update_and_clear() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {

crates/e2e_test/src/reliant/grpc_lock_server.rs

@@ -798,6 +798,13 @@ impl NodeService for MinimalLockNodeService {
         Err(Status::unimplemented("lock-only test server"))
     }
 
+    async fn scanner_activity(
+        &self,
+        _request: Request<rustfs_protos::proto_gen::node_service::ScannerActivityRequest>,
+    ) -> Result<Response<rustfs_protos::proto_gen::node_service::ScannerActivityResponse>, Status> {
+        Err(Status::unimplemented("lock-only test server"))
+    }
+
     async fn background_heal_status(
         &self,
         _request: Request<rustfs_protos::proto_gen::node_service::BackgroundHealStatusRequest>,

crates/e2e_test/src/reliant/tiering.rs

@@ -64,7 +64,7 @@ const TIER_NAME: &str = "COLDTIER";
 const TIER_BUCKET: &str = "ilm7-cold-tier";
 const TIER_PREFIX: &str = "tiered";
 const SOURCE_BUCKET: &str = "ilm7-hot";
-const OBJECT_KEY: &str = "tier/report.bin";
+const OBJECT_KEY: &str = "tier/鲁A12345/report.bin";
 const CONTENT_TYPE: &str = "application/x-ilm7";
 const USER_META_KEY: &str = "ilm7-origin";
 const USER_META_VAL: &str = "hermetic-transition";

crates/e2e_test/src/replication_extension_test.rs

@@ -1331,6 +1331,42 @@ async fn assert_managed_sse_replication_fails_explicitly(label: &str, kms: bool)
     Ok(())
 }
 
+async fn wait_for_source_delete_marker_replication_failed(
+    env: &RustFSTestEnvironment,
+    bucket: &str,
+    key: &str,
+) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
+    let url = format!(
+        "{}/rustfs/admin/v3/replication/diff?bucket={}&prefix={}",
+        env.url,
+        urlencoding::encode(bucket),
+        urlencoding::encode(key)
+    );
+
+    loop {
+        let response = signed_request(http::Method::POST, &url, &env.access_key, &env.secret_key, None, None).await?;
+        if response.status() != StatusCode::OK {
+            return Err(format!("replication diff failed with status {}", response.status()).into());
+        }
+        let diff: serde_json::Value = response.json().await?;
+        let failed = diff["Entries"].as_array().is_some_and(|entries| {
+            entries.iter().any(|entry| {
+                entry["Object"].as_str() == Some(key)
+                    && entry["IsDeleteMarker"].as_bool() == Some(true)
+                    && entry["ReplicationStatus"].as_str() == Some("FAILED")
+            })
+        });
+        if failed {
+            return Ok(());
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!("source delete marker {key} never reported FAILED; last diff={diff}").into());
+        }
+        sleep(Duration::from_millis(200)).await;
+    }
+}
+
 /// Return the `LastModified` of the (single) delete marker for `key`, if present.
 async fn delete_marker_last_modified(
     client: &Client,
@@ -1935,7 +1971,10 @@ async fn wait_for_site_replication_enabled(
 ) -> Result<SiteReplicationInfo, Box<dyn Error + Send + Sync>> {
     for _ in 0..40 {
         let info = site_replication_info(env).await?;
-        if info.enabled && info.sites.len() == expected_sites {
+        if info.enabled
+            && info.sites.len() == expected_sites
+            && info.sites.iter().all(|peer| peer.sync_state == SyncStatus::Enable)
+        {
             return Ok(info);
         }
         sleep(Duration::from_millis(250)).await;
@@ -3691,23 +3730,9 @@ async fn test_bucket_replication_replays_failed_entries_after_source_restart() -
     Ok(())
 }
 
-/// backlog#1147 repl-5, scenario (c) — replayed delete marker keeps the source
-/// mtime (mirrors backlog#867).
-///
-/// A delete marker is created while the target is down; the source is then
-/// restarted (data preserved) and the target brought back, so the marker
-/// replicates through the failure-replay path. The replayed marker must carry
-/// the SOURCE's `LastModified`, not the replay time. A deliberate gap before
-/// recovery makes any regression (replay-time stamping) obvious.
-///
-/// The source restart is load-bearing, not just paranoia: on a live
-/// (never-restarted) source, the failed delete-marker replication wedges the
-/// per-object `/[replicate]/<key>` namespace lock and the marker never
-/// replicates even after the target recovers — tracked as backlog#1278. Once
-/// that is fixed, a restart-free variant of this scenario should be added.
 #[tokio::test]
 #[serial]
-async fn test_bucket_replication_replayed_delete_marker_preserves_source_mtime() -> TestResult {
+async fn test_bucket_replication_replayed_delete_marker_preserves_source_mtime_without_source_restart() -> TestResult {
     init_logging();
 
     let mut source_env = RustFSTestEnvironment::new().await?;
@@ -3759,13 +3784,11 @@ async fn test_bucket_replication_replayed_delete_marker_preserves_source_mtime()
         .await?
         .ok_or("source has no delete marker after DELETE")?;
 
+    wait_for_source_delete_marker_replication_failed(&source_env, source_bucket, object_key).await?;
+
     // Widen the gap so a replay-time-stamping regression is unmistakable.
     sleep(Duration::from_secs(3)).await;
 
-    // Restart the source (see the doc comment: live-source replay is wedged by
-    // backlog#1278), then bring the target back; the restarted source's scanner
-    // heal pass replays the failed delete marker.
-    source_env.restart_server_preserving_data(vec![], &source_env_vars).await?;
     target_env.restart_server_preserving_data(vec![], &[]).await?;
 
     let target_mtime = wait_for_target_delete_marker(&target_client, target_bucket, object_key).await?;

crates/ecstore/Cargo.toml

@@ -61,6 +61,7 @@ rustfs-kms.workspace = true
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
 rustfs-object-capacity.workspace = true
+arc-swap.workspace = true
 async-trait.workspace = true
 bytes = { workspace = true, features = ["serde"] }
 byteorder = { workspace = true }
@@ -122,7 +123,7 @@ libc.workspace = true
 # (backlog#1178); "fs" comes from the workspace default.
 rustix = { workspace = true, features = ["process", "fs"] }
 rustfs-madmin.workspace = true
-reqwest = { workspace = true, default-features = false, features = ["rustls", "http2", "system-proxy"] }
+reqwest = { workspace = true }
 aes-gcm = { workspace = true, features = ["rand_core"] }
 chacha20poly1305.workspace = true
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }

crates/ecstore/src/api/mod.rs

@@ -67,7 +67,11 @@ pub mod bucket {
         }
 
         pub mod tier_delete_journal {
-            pub use crate::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry;
+            #[cfg(feature = "test-util")]
+            pub use crate::bucket::lifecycle::tier_delete_journal::recover_tier_delete_journal_entries;
+            pub use crate::bucket::lifecycle::tier_delete_journal::{
+                persist_tier_delete_journal_entry, record_tier_delete_journal_backend_identity,
+            };
         }
 
         pub mod tier_last_day_stats {
@@ -247,7 +251,8 @@ pub mod config {
             CLASS_RRS, CLASS_STANDARD, Config, DEEP_ARCHIVE, DEFAULT_INLINE_BLOCK, DEFAULT_KVS, DEFAULT_RRS_PARITY,
             EXPRESS_ONEZONE, GLACIER, GLACIER_IR, INLINE_BLOCK, INLINE_BLOCK_ENV, INTELLIGENT_TIERING, MIN_PARITY_DRIVES,
             ONEZONE_IA, OPTIMIZE, OPTIMIZE_ENV, OUTPOSTS, RRS, RRS_ENV, SCHEME_PREFIX, SNOW, STANDARD, STANDARD_ENV, STANDARD_IA,
-            StorageClass, default_parity_count, lookup_config, parse_storage_class, validate_parity, validate_parity_inner,
+            StorageClass, default_parity_count, lookup_config, lookup_config_for_pools, parse_storage_class, validate_parity,
+            validate_parity_inner,
         };
     }
 
@@ -258,12 +263,14 @@ pub mod config {
 
 pub mod data_usage {
     pub use crate::data_usage::{
-        DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, init_compression_total_memory_from_backend,
-        load_compression_total_from_memory, load_data_usage_from_backend, record_bucket_delete_marker_memory,
+        DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
+        init_compression_total_memory_from_backend, live_bucket_usage_computations, load_compression_total_from_memory,
+        load_data_usage_from_backend, load_data_usage_from_backend_cached, record_bucket_delete_marker_memory,
         record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
         record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
         refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
         remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
+        store_data_usage_in_backend,
     };
 }
 
@@ -308,8 +315,8 @@ pub mod error {
 
 pub mod erasure {
     pub use crate::erasure::coding::{
-        BitrotReader, BitrotWriter, BitrotWriterWrapper, CustomWriter, Erasure, ReedSolomonEncoder, calc_shard_size,
-        calc_shard_size_legacy,
+        BitrotReader, BitrotWriter, BitrotWriterWrapper, CustomWriter, Erasure, ErasureConstructionError, ReedSolomonEncoder,
+        calc_shard_size, calc_shard_size_legacy,
     };
 }
 
@@ -351,9 +358,12 @@ pub mod notification {
 
 pub mod object {
     pub use crate::object_api::{
-        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions,
-        PutObjReader, RangedDecompressReader, StreamConsumer, register_get_object_body_cache_hook, register_object_mutation_hook,
+        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, GetObjectBodySource,
+        GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, StreamConsumer,
+        get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
+        register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
     };
+    pub use crate::store::PreparedGetObjectReader;
 }
 
 pub mod rebalance {
@@ -374,8 +384,11 @@ pub mod rio {
 pub mod rpc {
     pub use crate::cluster::rpc::{
         LocalPeerS3Client, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, SERVICE_SIGNAL_REFRESH_CONFIG,
-        SERVICE_SIGNAL_RELOAD_DYNAMIC, TONIC_RPC_PREFIX, TonicInterceptor, gen_tonic_signature_interceptor,
-        node_service_time_out_client, node_service_time_out_client_no_auth, verify_rpc_signature,
+        SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers,
+        gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
+        node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
+        sign_tonic_rpc_response_proof, verify_rpc_signature, verify_tonic_canonical_body_digest, verify_tonic_rpc_response_proof,
+        verify_tonic_rpc_signature,
     };
 }
 
@@ -402,7 +415,7 @@ pub mod tier {
         pub use crate::services::tier::tier::{
             ERR_TIER_BACKEND_IN_USE, ERR_TIER_BACKEND_NOT_EMPTY, ERR_TIER_INVALID_CONFIG, ERR_TIER_MISSING_CREDENTIALS,
             ERR_TIER_TYPE_UNSUPPORTED, TIER_CONFIG_FILE, TIER_CONFIG_FORMAT, TIER_CONFIG_V1, TIER_CONFIG_VERSION, TierConfigMgr,
-            is_err_config_not_found, try_migrate_tiering_config,
+            TierConfigUpdateError, is_err_config_not_found, try_migrate_tiering_config,
         };
     }
 
@@ -413,7 +426,7 @@ pub mod tier {
     pub mod tier_config {
         pub use crate::services::tier::tier_config::{
             ServicePrincipalAuth, TierAliyun, TierAzure, TierConfig, TierGCS, TierHuaweicloud, TierMinIO, TierR2, TierRustFS,
-            TierS3, TierTencent, TierType,
+            TierS3, TierTencent, TierType, TierWasabi,
         };
     }
 
@@ -424,6 +437,13 @@ pub mod tier {
         };
     }
 
+    pub mod tier_mutation_peer {
+        pub use crate::services::tier::tier_mutation_peer::{
+            MAX_TIER_MUTATION_PEER_COMMIT_ETAG_SIZE, TierMutationPeerError, TierMutationPeerOutcome, TierMutationPeerResult,
+            TierMutationPeerState, handle_tier_mutation_peer_request,
+        };
+    }
+
     pub mod warm_backend {
         pub use crate::services::tier::warm_backend::{
             WarmBackend, WarmBackendGetOpts, WarmBackendImpl, build_transition_put_options, check_warm_backend, new_warm_backend,
@@ -433,9 +453,9 @@ pub mod tier {
     #[cfg(feature = "test-util")]
     pub mod test_util {
         pub use crate::services::tier::test_util::{
-            FaultConfig, MockStoredObject, MockWarmBackend, MockWarmOp, TransitionMeta, assert_transition_meta_consistent,
-            free_version_count, read_transition_meta, register_mock_tier, register_mock_tier_backend,
-            wait_for_free_version_absence,
+            FaultConfig, MockStoredObject, MockWarmBackend, MockWarmOp, TransitionCleanupStoreBarrier, TransitionMeta,
+            assert_transition_meta_consistent, free_version_count, read_transition_meta, register_mock_tier,
+            register_mock_tier_backend, wait_for_free_version_absence,
         };
     }
 }

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -280,6 +280,7 @@ pub struct BucketTargetSys {
     pub hc_client: Arc<HttpClient>,
     pub a_mutex: Arc<Mutex<HashMap<String, ArnErrs>>>,
     pub arn_errs_map: Arc<RwLock<HashMap<String, ArnErrs>>>,
+    heartbeat_started: OnceLock<()>,
 }
 
 impl BucketTargetSys {
@@ -295,9 +296,20 @@ impl BucketTargetSys {
             hc_client: Arc::new(HttpClient::new()),
             a_mutex: Arc::new(Mutex::new(HashMap::new())),
             arn_errs_map: Arc::new(RwLock::new(HashMap::new())),
+            heartbeat_started: OnceLock::new(),
         }
     }
 
+    pub(crate) fn start_heartbeat(&'static self) {
+        if self.heartbeat_started.set(()).is_err() {
+            return;
+        }
+
+        tokio::spawn(async move {
+            self.heartbeat().await;
+        });
+    }
+
     pub async fn is_offline(&self, url: &Url) -> bool {
         let key = endpoint_health_key(url);
         {
@@ -367,7 +379,7 @@ impl BucketTargetSys {
     async fn check_endpoint_health(&self, endpoint: &str, scheme: &str) -> bool {
         let scheme = if scheme.is_empty() { "https" } else { scheme };
         let url = format!("{scheme}://{endpoint}/");
-        match self.hc_client.head(url).timeout(Duration::from_secs(3)).send().await {
+        match self.hc_client.get(url).timeout(Duration::from_secs(3)).send().await {
             Ok(response) => response.status().as_u16() < 500,
             Err(_) => false,
         }

crates/ecstore/src/bucket/lifecycle/mod.rs

@@ -28,3 +28,4 @@ pub mod tier_delete_journal;
 pub mod tier_free_version_recovery;
 pub mod tier_last_day_stats;
 pub mod tier_sweeper;
+pub mod transition_transaction;

crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use std::sync::Arc;
+use std::{future::Future, sync::Arc, time::Duration};
 
 use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
@@ -20,10 +20,11 @@ use tokio_util::sync::CancellationToken;
 use tracing::{debug, warn};
 
 use crate::bucket::lifecycle::config_boundary;
-use crate::bucket::lifecycle::tier_sweeper::{Jentry, delete_object_from_remote_tier_idempotent};
+use crate::bucket::lifecycle::tier_sweeper::{Jentry, delete_object_from_remote_tier_idempotent_with_manager_and_identity};
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::{Error, Result};
 use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
+use crate::services::tier::tier::tier_destination_id_from_metadata;
 use crate::storage_api_contracts::{
     list::ListOperations as _,
     object::{DeletedObject, ObjectIO, ObjectOperations, ObjectToDelete},
@@ -37,8 +38,11 @@ const LOG_SUBSYSTEM_LIFECYCLE: &str = "lifecycle";
 const EVENT_LIFECYCLE_TIER_DELETE_JOURNAL: &str = "lifecycle_tier_delete_journal";
 
 pub const DEFAULT_TIER_DELETE_JOURNAL_RECOVERY_LIMIT: usize = 1_000;
-const TIER_DELETE_JOURNAL_VERSION: u8 = 1;
-const TIER_DELETE_JOURNAL_PREFIX: &str = "ilm/tier-delete-journal/";
+const TIER_DELETE_JOURNAL_RECOVERY_INTERVAL: Duration = Duration::from_secs(60);
+const TIER_DELETE_JOURNAL_RECOVERY_TIMEOUT: Duration = Duration::from_secs(300);
+const TIER_DELETE_JOURNAL_VERSION: u8 = 2;
+const TIER_DELETE_JOURNAL_EXACT_VERSION: u8 = 3;
+pub(crate) const TIER_DELETE_JOURNAL_PREFIX: &str = "ilm/tier-delete-journal/";
 
 #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
 #[serde(deny_unknown_fields)]
@@ -47,22 +51,31 @@ struct PersistedTierDeleteJournalEntry {
     obj_name: String,
     version_id: String,
     tier_name: String,
+    #[serde(default)]
+    backend_identity: Option<[u8; 32]>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    version_id_exact: Option<bool>,
 }
 
 impl PersistedTierDeleteJournalEntry {
     fn from_jentry(je: &Jentry) -> Self {
         Self {
-            version: TIER_DELETE_JOURNAL_VERSION,
+            version: if je.version_id_exact {
+                TIER_DELETE_JOURNAL_EXACT_VERSION
+            } else if je.backend_identity.is_some() {
+                TIER_DELETE_JOURNAL_VERSION
+            } else {
+                1
+            },
             obj_name: je.obj_name.clone(),
             version_id: je.version_id.clone(),
             tier_name: je.tier_name.clone(),
+            backend_identity: je.backend_identity,
+            version_id_exact: je.version_id_exact.then_some(true),
         }
     }
 
     fn into_jentry(self) -> Result<Jentry> {
-        if self.version != TIER_DELETE_JOURNAL_VERSION {
-            return Err(Error::other(format!("unsupported tier delete journal version {}", self.version)));
-        }
         // Empty `version_id` is a legal sentinel for objects transitioned to an
         // unversioned remote tier (see CLAUDE.md: a tier version of `None`/`""`
         // means the tier bucket is unversioned, so the remote delete is issued
@@ -71,10 +84,40 @@ impl PersistedTierDeleteJournalEntry {
         if self.obj_name.is_empty() || self.tier_name.is_empty() {
             return Err(Error::other("tier delete journal entry is incomplete"));
         }
+        if self.version != TIER_DELETE_JOURNAL_EXACT_VERSION && self.version_id_exact.unwrap_or(false) {
+            return Err(Error::other(
+                "legacy tier delete journal entry has an unsupported exact version constraint",
+            ));
+        }
+        let (backend_identity, version_id_exact) = match self.version {
+            1 => (None, false),
+            TIER_DELETE_JOURNAL_VERSION => (
+                Some(
+                    self.backend_identity
+                        .ok_or_else(|| Error::other("tier delete journal v2 entry is missing its backend identity"))?,
+                ),
+                false,
+            ),
+            TIER_DELETE_JOURNAL_EXACT_VERSION => {
+                if self.version_id.is_empty() || self.version_id_exact != Some(true) {
+                    return Err(Error::other("tier delete journal v3 entry is missing its exact version constraint"));
+                }
+                (
+                    Some(
+                        self.backend_identity
+                            .ok_or_else(|| Error::other("tier delete journal v3 entry is missing its backend identity"))?,
+                    ),
+                    true,
+                )
+            }
+            version => return Err(Error::other(format!("unsupported tier delete journal version {version}"))),
+        };
         Ok(Jentry {
             obj_name: self.obj_name,
             version_id: self.version_id,
             tier_name: self.tier_name,
+            backend_identity,
+            version_id_exact,
         })
     }
 }
@@ -95,6 +138,14 @@ pub(crate) fn tier_delete_journal_object_name(je: &Jentry) -> String {
     hasher.update(je.obj_name.as_bytes());
     hasher.update([0]);
     hasher.update(je.version_id.as_bytes());
+    if let Some(backend_identity) = je.backend_identity {
+        hasher.update([0]);
+        hasher.update(backend_identity);
+    }
+    if je.version_id_exact {
+        hasher.update([0]);
+        hasher.update(b"exact-version-id");
+    }
     format!(
         "{TIER_DELETE_JOURNAL_PREFIX}{}.json",
         rustfs_utils::crypto::hex(hasher.finalize().as_slice())
@@ -112,6 +163,16 @@ fn encode_tier_delete_journal_entry(je: &Jentry) -> Result<Vec<u8>> {
         .map_err(|err| Error::other(format!("encode tier delete journal failed: {err}")))
 }
 
+pub fn record_tier_delete_journal_backend_identity(
+    je: &mut Jentry,
+    metadata: &std::collections::HashMap<String, String>,
+) -> std::io::Result<()> {
+    if let Some(identity) = tier_destination_id_from_metadata(metadata)? {
+        je.backend_identity = Some(identity);
+    }
+    Ok(())
+}
+
 pub async fn persist_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
 where
     S: ObjectIO<
@@ -148,7 +209,18 @@ where
 }
 
 pub async fn process_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
-    delete_object_from_remote_tier_idempotent(&je.obj_name, &je.version_id, &je.tier_name).await?;
+    let backend_identity = je
+        .backend_identity
+        .ok_or_else(|| std::io::Error::other("legacy tier delete journal has no durable backend identity"))?;
+    delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+        &je.obj_name,
+        &je.version_id,
+        &je.tier_name,
+        backend_identity,
+        &api.tier_config_mgr(),
+        je.version_id_exact,
+    )
+    .await?;
     remove_tier_delete_journal_entry(api, je).await
 }
 
@@ -218,6 +290,21 @@ pub async fn recover_tier_delete_journal_entries(
             }
         };
 
+        if je.backend_identity.is_none() {
+            stats.failed += 1;
+            warn!(
+                event = EVENT_LIFECYCLE_TIER_DELETE_JOURNAL,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_LIFECYCLE,
+                journal_object = %object.name,
+                remote_object = %je.obj_name,
+                remote_version_id = %je.version_id,
+                tier = %je.tier_name,
+                "Legacy tier delete journal entry has no durable backend identity and will be retained"
+            );
+            continue;
+        }
+
         match process_tier_delete_journal_entry(api.clone(), &je).await {
             Ok(()) => stats.deleted += 1,
             Err(err) => {
@@ -241,16 +328,33 @@ pub async fn recover_tier_delete_journal_entries(
 }
 
 pub async fn run_tier_delete_journal_recovery_loop(api: Arc<ECStore>, cancel_token: CancellationToken) {
-    let mut interval = tokio::time::interval(std::time::Duration::from_secs(60));
+    let mut interval = tokio::time::interval(TIER_DELETE_JOURNAL_RECOVERY_INTERVAL);
+    interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
     let mut marker: Option<String> = None;
 
     loop {
+        #[cfg(test)]
         tokio::select! {
+            biased;
             _ = cancel_token.cancelled() => return,
-            _ = interval.tick() => {}
+            _ = interval.tick() => {},
+            _ = api.ctx.wait_for_tier_delete_journal_recovery() => {},
+        }
+        #[cfg(not(test))]
+        tokio::select! {
+            biased;
+            _ = cancel_token.cancelled() => return,
+            _ = interval.tick() => {},
         }
 
-        match recover_tier_delete_journal_entries(api.clone(), DEFAULT_TIER_DELETE_JOURNAL_RECOVERY_LIMIT, marker.clone()).await {
+        let recovery =
+            recover_tier_delete_journal_entries(api.clone(), DEFAULT_TIER_DELETE_JOURNAL_RECOVERY_LIMIT, marker.clone());
+        let Some(result) =
+            await_tier_delete_journal_recovery(&cancel_token, TIER_DELETE_JOURNAL_RECOVERY_TIMEOUT, recovery).await
+        else {
+            return;
+        };
+        match result {
             Ok(stats) => {
                 marker = stats.next_marker;
                 debug!(
@@ -279,16 +383,44 @@ pub async fn run_tier_delete_journal_recovery_loop(api: Arc<ECStore>, cancel_tok
     }
 }
 
+async fn await_tier_delete_journal_recovery<T, F>(
+    cancel_token: &CancellationToken,
+    timeout: Duration,
+    recovery: F,
+) -> Option<Result<T>>
+where
+    F: Future<Output = Result<T>>,
+{
+    tokio::select! {
+        _ = cancel_token.cancelled() => None,
+        result = tokio::time::timeout(timeout, recovery) => Some(match result {
+            Ok(result) => result,
+            Err(_) => Err(Error::other(format!(
+                "tier delete journal recovery timed out after {} seconds",
+                timeout.as_secs()
+            ))),
+        }),
+    }
+}
+
 #[cfg(test)]
 mod tests {
-    use super::{decode_tier_delete_journal_entry, encode_tier_delete_journal_entry, tier_delete_journal_object_name};
+    use super::{
+        TIER_DELETE_JOURNAL_EXACT_VERSION, await_tier_delete_journal_recovery, decode_tier_delete_journal_entry,
+        encode_tier_delete_journal_entry, record_tier_delete_journal_backend_identity, tier_delete_journal_object_name,
+    };
     use crate::bucket::lifecycle::tier_sweeper::Jentry;
+    use crate::error::Result;
+    use std::time::Duration;
+    use tokio_util::sync::CancellationToken;
 
     fn journal_entry() -> Jentry {
         Jentry {
             obj_name: "remote/object".to_string(),
             version_id: "remote-version".to_string(),
             tier_name: "WARM".to_string(),
+            backend_identity: Some([7; 32]),
+            version_id_exact: false,
         }
     }
 
@@ -302,6 +434,83 @@ mod tests {
         assert_eq!(decoded.obj_name, je.obj_name);
         assert_eq!(decoded.version_id, je.version_id);
         assert_eq!(decoded.tier_name, je.tier_name);
+        assert_eq!(decoded.backend_identity, je.backend_identity);
+        assert_eq!(decoded.version_id_exact, je.version_id_exact);
+    }
+
+    #[test]
+    fn tier_delete_journal_roundtrips_exact_put_response_constraint() {
+        let mut exact = journal_entry();
+        exact.version_id = uuid::Uuid::nil().to_string();
+        exact.version_id_exact = true;
+        let mut normalized = exact.clone();
+        normalized.version_id_exact = false;
+
+        let encoded = encode_tier_delete_journal_entry(&exact).expect("exact journal entry should encode");
+        let persisted: serde_json::Value = serde_json::from_slice(&encoded).expect("exact journal JSON should decode");
+        let decoded = decode_tier_delete_journal_entry(&encoded).expect("exact journal entry should decode");
+
+        assert_eq!(persisted["version"], TIER_DELETE_JOURNAL_EXACT_VERSION);
+        assert_eq!(persisted["version_id_exact"], true);
+        assert!(decoded.version_id_exact);
+        assert_ne!(tier_delete_journal_object_name(&exact), tier_delete_journal_object_name(&normalized));
+    }
+
+    #[test]
+    fn tier_delete_journal_rejects_invalid_exact_version_constraints() {
+        let identity = vec![7_u8; 32];
+        let invalid = [
+            serde_json::json!({
+                "version": 1,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "version_id_exact": true,
+            }),
+            serde_json::json!({
+                "version": 2,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+                "version_id_exact": true,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+                "version_id_exact": true,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "backend_identity": identity,
+                "version_id_exact": false,
+            }),
+            serde_json::json!({
+                "version": TIER_DELETE_JOURNAL_EXACT_VERSION,
+                "obj_name": "remote/object",
+                "version_id": "exact-version",
+                "tier_name": "WARM",
+                "version_id_exact": true,
+            }),
+        ];
+
+        for persisted in invalid {
+            let encoded = serde_json::to_vec(&persisted).expect("invalid journal fixture should encode");
+            decode_tier_delete_journal_entry(&encoded).expect_err("invalid exact journal constraint must fail closed");
+        }
     }
 
     #[test]
@@ -317,6 +526,63 @@ mod tests {
         assert!(!first.contains("remote/object"));
     }
 
+    #[test]
+    fn tier_delete_journal_paths_separate_legacy_and_backend_identities() {
+        let mut legacy = journal_entry();
+        legacy.backend_identity = None;
+        let mut backend_a = journal_entry();
+        backend_a.backend_identity = Some([1; 32]);
+        let mut backend_b = journal_entry();
+        backend_b.backend_identity = Some([2; 32]);
+
+        assert_eq!(
+            tier_delete_journal_object_name(&legacy),
+            "ilm/tier-delete-journal/5ba6a7eb6338412b771613a6845a42ae5b8e26b5d201323eb01b38c5b42ff300.json"
+        );
+        assert_ne!(tier_delete_journal_object_name(&legacy), tier_delete_journal_object_name(&backend_a));
+        assert_ne!(tier_delete_journal_object_name(&backend_a), tier_delete_journal_object_name(&backend_b));
+    }
+
+    #[test]
+    fn tier_delete_journal_v2_requires_backend_identity() {
+        let payload = br#"{"version":2,"obj_name":"remote/object","version_id":"v1","tier_name":"WARM"}"#;
+
+        let err = decode_tier_delete_journal_entry(payload).expect_err("v2 entry without identity must fail closed");
+
+        assert!(err.to_string().contains("backend identity"));
+    }
+
+    #[test]
+    fn tier_delete_journal_uses_persisted_transition_destination_identity() {
+        let mut je = journal_entry();
+        je.backend_identity = None;
+        let identity = [9_u8; 32];
+        let mut metadata = std::collections::HashMap::new();
+        rustfs_utils::http::metadata_compat::insert_str(
+            &mut metadata,
+            rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
+            rustfs_utils::crypto::hex(identity),
+        );
+
+        record_tier_delete_journal_backend_identity(&mut je, &metadata).expect("persisted transition identity should decode");
+        let encoded = encode_tier_delete_journal_entry(&je).expect("identity-bound journal should encode");
+        let decoded = decode_tier_delete_journal_entry(&encoded).expect("identity-bound journal should decode");
+
+        assert_eq!(decoded.backend_identity, Some(identity));
+    }
+
+    #[test]
+    fn tier_delete_journal_without_transition_identity_stays_legacy() {
+        let mut je = journal_entry();
+        je.backend_identity = None;
+
+        let encoded = encode_tier_delete_journal_entry(&je).expect("legacy journal should remain encodable");
+        let persisted: serde_json::Value = serde_json::from_slice(&encoded).expect("journal JSON should decode");
+
+        assert_eq!(persisted["version"], 1);
+        assert!(persisted["backend_identity"].is_null());
+    }
+
     #[test]
     fn tier_delete_journal_rejects_incomplete_entry() {
         let payload = br#"{"version":1,"obj_name":"","version_id":"v1","tier_name":"WARM"}"#;
@@ -339,6 +605,7 @@ mod tests {
         assert_eq!(decoded.obj_name, "remote/object");
         assert!(decoded.version_id.is_empty());
         assert_eq!(decoded.tier_name, "WARM");
+        assert_eq!(decoded.backend_identity, None);
     }
 
     #[test]
@@ -360,4 +627,29 @@ mod tests {
 
         assert!(err.to_string().contains("decode tier delete journal failed"));
     }
+
+    #[tokio::test]
+    async fn tier_delete_journal_recovery_has_a_hard_outer_timeout() {
+        let result = await_tier_delete_journal_recovery(
+            &CancellationToken::new(),
+            Duration::from_millis(10),
+            std::future::pending::<Result<()>>(),
+        )
+        .await
+        .expect("an elapsed timeout should return a recovery error")
+        .expect_err("a permanently pending recovery must time out");
+
+        assert!(result.to_string().contains("recovery timed out"), "{result}");
+    }
+
+    #[tokio::test]
+    async fn tier_delete_journal_recovery_drops_in_flight_work_on_shutdown() {
+        let cancel = CancellationToken::new();
+        cancel.cancel();
+
+        let result =
+            await_tier_delete_journal_recovery(&cancel, Duration::from_secs(30), std::future::pending::<Result<()>>()).await;
+
+        assert!(result.is_none(), "shutdown must cancel the in-flight recovery future");
+    }
 }

crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs

@@ -13,11 +13,14 @@
 // limitations under the License.
 
 use std::sync::Arc;
+#[cfg(test)]
+use std::sync::Mutex;
 use std::time::Duration;
 
 use tokio::sync::mpsc;
 use tokio_util::sync::CancellationToken;
 
+use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version;
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::Result;
 use crate::object_api::ObjectInfo;
@@ -31,10 +34,78 @@ use rustfs_filemeta::FileInfo;
 pub const DEFAULT_FREE_VERSION_RECOVERY_LIMIT: usize = 1_000;
 const DEFAULT_FREE_VERSION_RECOVERY_SCAN_LIMIT: usize = 10_000;
 const BACKGROUND_WALKDIR_TIMEOUT: Duration = Duration::from_secs(60);
+#[cfg(not(test))]
+const BACKGROUND_WALK_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(5);
+#[cfg(test)]
+const BACKGROUND_WALK_SHUTDOWN_TIMEOUT: Duration = Duration::from_millis(100);
 
 type ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, crate::error::Error>;
 type WalkOptions = StorageWalkOptions<fn(&FileInfo) -> bool>;
 
+#[cfg(test)]
+pub(super) enum RecoveryWalkTestAction {
+    SendItemsThenError(Vec<ObjectInfo>, crate::error::Error),
+    SendItemsThenHang(Vec<ObjectInfo>, Arc<tokio::sync::Notify>),
+    SendItemsUntilReceiverCloses(Arc<tokio::sync::Notify>),
+    ReturnError(crate::error::Error),
+    WaitForCancellation(Arc<tokio::sync::Notify>),
+}
+
+#[cfg(test)]
+type RecoveryWalkTestHook = Box<dyn Fn(&str) -> Option<RecoveryWalkTestAction> + Send + Sync>;
+
+#[cfg(test)]
+static RECOVERY_WALK_TEST_HOOK: Mutex<Option<RecoveryWalkTestHook>> = Mutex::new(None);
+
+#[cfg(test)]
+static RECOVERY_BUCKET_LIST_WAIT_HOOK: Mutex<Option<Arc<tokio::sync::Notify>>> = Mutex::new(None);
+
+#[cfg(test)]
+pub(super) struct RecoveryWalkHookGuard;
+
+#[cfg(test)]
+impl Drop for RecoveryWalkHookGuard {
+    fn drop(&mut self) {
+        let mut hook = RECOVERY_WALK_TEST_HOOK
+            .lock()
+            .expect("recovery walk test hook lock should not poison");
+        *hook = None;
+    }
+}
+
+#[cfg(test)]
+pub(super) fn set_recovery_walk_test_hook(
+    hook_fn: impl Fn(&str) -> Option<RecoveryWalkTestAction> + Send + Sync + 'static,
+) -> RecoveryWalkHookGuard {
+    let mut hook = RECOVERY_WALK_TEST_HOOK
+        .lock()
+        .expect("recovery walk test hook lock should not poison");
+    *hook = Some(Box::new(hook_fn));
+    RecoveryWalkHookGuard
+}
+
+#[cfg(test)]
+pub(super) struct RecoveryBucketListWaitHookGuard;
+
+#[cfg(test)]
+impl Drop for RecoveryBucketListWaitHookGuard {
+    fn drop(&mut self) {
+        let mut hook = RECOVERY_BUCKET_LIST_WAIT_HOOK
+            .lock()
+            .expect("recovery bucket-list test hook lock should not poison");
+        *hook = None;
+    }
+}
+
+#[cfg(test)]
+pub(super) fn set_recovery_bucket_list_wait_hook(started: Arc<tokio::sync::Notify>) -> RecoveryBucketListWaitHookGuard {
+    let mut hook = RECOVERY_BUCKET_LIST_WAIT_HOOK
+        .lock()
+        .expect("recovery bucket-list test hook lock should not poison");
+    *hook = Some(started);
+    RecoveryBucketListWaitHookGuard
+}
+
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct FreeVersionRecoveryStats {
     pub scanned: usize,
@@ -68,12 +139,22 @@ pub async fn recover_tier_free_versions(
     limit: usize,
     bucket_marker: Option<String>,
     object_marker: Option<String>,
+) -> Result<FreeVersionRecoveryStats> {
+    recover_tier_free_versions_with_cancel(api, limit, bucket_marker, object_marker, CancellationToken::new()).await
+}
+
+pub(super) async fn recover_tier_free_versions_with_cancel(
+    api: Arc<ECStore>,
+    limit: usize,
+    bucket_marker: Option<String>,
+    object_marker: Option<String>,
+    cancel_token: CancellationToken,
 ) -> Result<FreeVersionRecoveryStats> {
     if limit == 0 {
         return Err(std::io::Error::other("free-version recovery limit must be greater than zero").into());
     }
 
-    let page = list_tier_free_versions(api, limit, bucket_marker.clone(), object_marker.clone()).await?;
+    let page = list_tier_free_versions(api, limit, bucket_marker.clone(), object_marker.clone(), cancel_token.clone()).await?;
     let mut stats = FreeVersionRecoveryStats {
         scanned: 0,
         enqueued: 0,
@@ -87,8 +168,11 @@ pub async fn recover_tier_free_versions(
 
     let mut retry_cursor = RetryCursor::new(bucket_marker, object_marker);
     for oi in page.items {
+        if cancel_token.is_cancelled() {
+            return Err(tier_free_version_recovery_cancelled());
+        }
         retry_cursor.visit(&oi);
-        if !record_recovered_free_version_enqueue(&mut stats, queue_recovered_free_version(oi).await) {
+        if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(oi).await) {
             let (bucket_marker, object_marker) = retry_cursor.retry_markers();
             stats.truncated = true;
             stats.next_bucket_marker = bucket_marker;
@@ -100,6 +184,18 @@ pub async fn recover_tier_free_versions(
     Ok(stats)
 }
 
+fn tier_free_version_recovery_cancelled() -> crate::error::Error {
+    std::io::Error::new(std::io::ErrorKind::Interrupted, "tier free-version recovery cancelled").into()
+}
+
+fn tier_free_version_recovery_walk_shutdown_timed_out() -> crate::error::Error {
+    std::io::Error::new(
+        std::io::ErrorKind::TimedOut,
+        "tier free-version recovery walk did not stop after cancellation",
+    )
+    .into()
+}
+
 fn record_recovered_free_version_enqueue(stats: &mut FreeVersionRecoveryStats, queued: bool) -> bool {
     stats.scanned += 1;
     if queued {
@@ -111,10 +207,6 @@ fn record_recovered_free_version_enqueue(stats: &mut FreeVersionRecoveryStats, q
     }
 }
 
-async fn queue_recovered_free_version(oi: ObjectInfo) -> bool {
-    crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version(oi).await
-}
-
 #[derive(Debug, Clone, PartialEq, Eq)]
 struct RetryCursor {
     input_bucket_marker: Option<String>,
@@ -160,11 +252,12 @@ impl RetryCursor {
     }
 }
 
-async fn list_tier_free_versions(
+pub(super) async fn list_tier_free_versions(
     api: Arc<ECStore>,
     limit: usize,
     bucket_marker: Option<String>,
     object_marker: Option<String>,
+    cancel_token: CancellationToken,
 ) -> Result<FreeVersionRecoveryPage> {
     let mut page = FreeVersionRecoveryPage {
         items: Vec::new(),
@@ -179,21 +272,42 @@ async fn list_tier_free_versions(
         return Ok(page);
     }
 
-    let buckets = api.list_bucket(&BucketOptions::default()).await?;
+    let bucket_options = BucketOptions::default();
+    let list_buckets = async {
+        #[cfg(test)]
+        let wait_hook = RECOVERY_BUCKET_LIST_WAIT_HOOK
+            .lock()
+            .expect("recovery bucket-list test hook lock should not poison")
+            .clone();
+        #[cfg(test)]
+        if let Some(started) = wait_hook {
+            started.notify_one();
+            std::future::pending::<()>().await;
+        }
+        api.list_bucket(&bucket_options).await
+    };
+    tokio::pin!(list_buckets);
+    let buckets = tokio::select! {
+        biased;
+        _ = cancel_token.cancelled() => return Err(tier_free_version_recovery_cancelled()),
+        result = &mut list_buckets => result?,
+    };
     let mut bucket_seen = bucket_marker.is_none();
     let mut truncated_after: Option<RecoveryCursor> = None;
     let walk_scan_limit = recovery_walk_scan_limit(limit);
 
     for bucket in buckets {
+        if cancel_token.is_cancelled() {
+            return Err(tier_free_version_recovery_cancelled());
+        }
         if bucket.name == RUSTFS_META_BUCKET {
             continue;
         }
         if !bucket_seen {
-            if bucket_marker.as_deref() == Some(bucket.name.as_str()) {
-                bucket_seen = true;
-            } else {
+            if bucket_marker.as_deref().is_some_and(|marker| bucket.name.as_str() < marker) {
                 continue;
             }
+            bucket_seen = true;
         }
 
         page.buckets_scanned += 1;
@@ -204,16 +318,94 @@ async fn list_tier_free_versions(
         };
 
         let (tx, mut rx) = mpsc::channel::<ObjectInfoOrErr>(100);
-        let cancel = CancellationToken::new();
+        let cancel = cancel_token.child_token();
         let mut draining_after_truncation = false;
+        let mut drain_deadline = None;
         let mut last_seen_object: Option<String> = None;
         let mut scanned_objects = 0usize;
-        let walk = tokio::spawn({
+        let mut walk = tokio::spawn({
             let api = api.clone();
             let bucket_name = bucket.name.clone();
             let object_marker = bucket_object_marker.clone();
             let cancel = cancel.clone();
             async move {
+                #[cfg(test)]
+                let test_action = {
+                    let hook = RECOVERY_WALK_TEST_HOOK
+                        .lock()
+                        .expect("recovery walk test hook lock should not poison");
+                    hook.as_ref().and_then(|hook| hook(&bucket_name))
+                };
+                #[cfg(test)]
+                if let Some(action) = test_action {
+                    match action {
+                        RecoveryWalkTestAction::SendItemsThenError(items, err) => {
+                            for item in items {
+                                if tx
+                                    .send(ObjectInfoOrErr {
+                                        item: Some(item),
+                                        err: None,
+                                    })
+                                    .await
+                                    .is_err()
+                                {
+                                    return Ok(());
+                                }
+                            }
+                            let _ = tx
+                                .send(ObjectInfoOrErr {
+                                    item: None,
+                                    err: Some(err),
+                                })
+                                .await;
+                            return Ok(());
+                        }
+                        RecoveryWalkTestAction::SendItemsThenHang(items, started) => {
+                            for item in items {
+                                if tx
+                                    .send(ObjectInfoOrErr {
+                                        item: Some(item),
+                                        err: None,
+                                    })
+                                    .await
+                                    .is_err()
+                                {
+                                    return Ok(());
+                                }
+                            }
+                            started.notify_one();
+                            return std::future::pending().await;
+                        }
+                        RecoveryWalkTestAction::SendItemsUntilReceiverCloses(started) => {
+                            started.notify_one();
+                            let mut index = 0usize;
+                            loop {
+                                if tx
+                                    .send(ObjectInfoOrErr {
+                                        item: Some(ObjectInfo {
+                                            bucket: bucket_name.clone(),
+                                            name: format!("nonrecoverable-{index:08}"),
+                                            ..Default::default()
+                                        }),
+                                        err: None,
+                                    })
+                                    .await
+                                    .is_err()
+                                {
+                                    return Ok(());
+                                }
+                                index = index.saturating_add(1);
+                            }
+                        }
+                        RecoveryWalkTestAction::ReturnError(err) => return Err(err),
+                        RecoveryWalkTestAction::WaitForCancellation(started) => {
+                            started.notify_one();
+                            cancel.cancelled().await;
+                            return Err(tier_free_version_recovery_cancelled());
+                        }
+                    }
+                }
+
                 api.walk(
                     cancel,
                     &bucket_name,
@@ -231,47 +423,88 @@ async fn list_tier_free_versions(
             }
         });
 
-        while let Some(item) = rx.recv().await {
+        let mut receive_error = None;
+        loop {
+            let item = tokio::select! {
+                biased;
+                _ = cancel_token.cancelled() => {
+                    cancel.cancel();
+                    receive_error = Some(tier_free_version_recovery_cancelled());
+                    break;
+                }
+                _ = async {
+                    if let Some(deadline) = drain_deadline {
+                        tokio::time::sleep_until(deadline).await;
+                    } else {
+                        std::future::pending::<()>().await;
+                    }
+                }, if drain_deadline.is_some() => {
+                    receive_error = Some(tier_free_version_recovery_walk_shutdown_timed_out());
+                    break;
+                }
+                item = rx.recv() => match item {
+                    Some(item) => item,
+                    None => break,
+                },
+            };
             page.scanned_entries += 1;
-            if draining_after_truncation {
-                continue;
-            }
             if let Some(err) = item.err {
                 cancel.cancel();
-                walk.await.map_err(|err| std::io::Error::other(err.to_string()))??;
-                return Err(err);
+                receive_error = Some(err);
+                break;
+            }
+            if draining_after_truncation {
+                continue;
             }
             let Some(oi) = item.item else {
                 continue;
             };
             record_scanned_object(&mut last_seen_object, &mut scanned_objects, &oi.name);
             if let Some(cursor) = &truncated_after
-                && cursor.object != oi.name
+                && (cursor.bucket.as_str() != bucket.name.as_str() || cursor.object.as_str() != oi.name.as_str())
             {
                 page.truncated = true;
                 cancel.cancel();
                 draining_after_truncation = true;
+                drain_deadline = Some(tokio::time::Instant::now() + BACKGROUND_WALK_SHUTDOWN_TIMEOUT);
                 continue;
             }
             if is_recoverable_tier_free_version(&oi) {
-                let cursor = RecoveryCursor {
+                let current_cursor = RecoveryCursor {
                     bucket: bucket.name.clone(),
                     object: oi.name.clone(),
                 };
                 page.items.push(oi);
-                page.next_bucket_marker = Some(cursor.bucket.clone());
-                page.next_object_marker = Some(cursor.object.clone());
+                page.next_bucket_marker = Some(current_cursor.bucket.clone());
+                page.next_object_marker = Some(current_cursor.object.clone());
                 if page.items.len() >= limit && truncated_after.is_none() {
-                    truncated_after = Some(cursor);
+                    truncated_after = Some(current_cursor);
                 }
             }
         }
 
-        walk.await.map_err(|err| std::io::Error::other(err.to_string()))??;
+        drop(rx);
+        let walk_shutdown_timeout = drain_deadline
+            .map(|deadline| deadline.saturating_duration_since(tokio::time::Instant::now()))
+            .unwrap_or(BACKGROUND_WALK_SHUTDOWN_TIMEOUT);
+        let walk_result = match tokio::time::timeout(walk_shutdown_timeout, &mut walk).await {
+            Ok(result) => result.map_err(|err| std::io::Error::other(err.to_string()))?,
+            Err(_) => {
+                walk.abort();
+                let _ = walk.await;
+                if let Some(err) = receive_error {
+                    return Err(err);
+                }
+                return Err(tier_free_version_recovery_walk_shutdown_timed_out());
+            }
+        };
+        if let Some(err) = receive_error {
+            return Err(err);
+        }
+        walk_result?;
         mark_scan_truncated_if_needed(&mut page, scanned_objects, walk_scan_limit, &bucket.name, last_seen_object.as_deref());
 
         if page.truncated {
-            page.next_bucket_marker = Some(bucket.name.clone());
             break;
         }
     }

crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs

@@ -23,6 +23,7 @@ use crate::bucket::lifecycle::bucket_lifecycle_ops::ExpiryOp;
 use crate::bucket::lifecycle::lifecycle::{self, ObjectOpts};
 use crate::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry;
 use crate::client::signer_error::error_chain_contains_signer_header_marker;
+use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease};
 use crate::storage_api_contracts::lifecycle::TransitionedObject;
 use crate::store::ECStore;
 use rustfs_utils::get_env_usize;
@@ -247,6 +248,8 @@ impl ObjSweeper {
                 obj_name: self.remote_object.clone(),
                 version_id: self.transition_version_id.clone(),
                 tier_name: self.transition_tier.clone(),
+                backend_identity: None,
+                version_id_exact: false,
             });
         }
         None
@@ -281,6 +284,8 @@ pub struct Jentry {
     pub(crate) obj_name: String,
     pub(crate) version_id: String,
     pub(crate) tier_name: String,
+    pub(crate) backend_identity: Option<TierDestinationId>,
+    pub(crate) version_id_exact: bool,
 }
 
 impl ExpiryOp for Jentry {
@@ -312,6 +317,28 @@ async fn delete_object_from_remote_tier_raw(obj_name: &str, rv_id: &str, tier_na
         return result;
     }
 
+    let tier_config_mgr = runtime_sources::tier_config_mgr_handle();
+    delete_object_from_remote_tier_raw_with_manager(obj_name, rv_id, tier_name, &tier_config_mgr).await
+}
+
+async fn delete_object_from_remote_tier_raw_with_manager(
+    obj_name: &str,
+    rv_id: &str,
+    tier_name: &str,
+    tier_config_mgr: &Arc<tokio::sync::RwLock<TierConfigMgr>>,
+) -> Result<(), std::io::Error> {
+    let lease = TierConfigMgr::acquire_operation_lease(&tier_config_mgr, tier_name)
+        .await
+        .map_err(std::io::Error::other)?;
+    delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, &lease, false).await
+}
+
+async fn delete_object_from_remote_tier_raw_with_lease(
+    obj_name: &str,
+    rv_id: &str,
+    lease: &TierOperationLease,
+    version_id_exact: bool,
+) -> Result<(), std::io::Error> {
     if remote_delete_breaker_is_open(Instant::now()).await {
         metrics::counter!(METRIC_DELETE_REMOTE_BREAKER_TOTAL).increment(1);
         return Err(std::io::Error::other(ERR_REMOTE_DELETE_BREAKER_OPEN));
@@ -323,13 +350,11 @@ async fn delete_object_from_remote_tier_raw(obj_name: &str, rv_id: &str, tier_na
         .map_err(|_| std::io::Error::other(ERR_REMOTE_DELETE_LIMITER_CLOSED))?;
     let _inflight = RemoteDeleteInflightGuard::new();
 
-    let tier_config_mgr = runtime_sources::tier_config_mgr_handle();
-    let mut config_mgr = tier_config_mgr.write().await;
-    let w = match config_mgr.get_driver(tier_name).await {
-        Ok(w) => w,
-        Err(e) => return Err(std::io::Error::other(e)),
-    };
-    w.remove(obj_name, rv_id).await
+    if version_id_exact {
+        lease.remove_exact(obj_name, rv_id).await
+    } else {
+        lease.remove(obj_name, rv_id).await
+    }
 }
 
 #[cfg(test)]
@@ -341,6 +366,30 @@ fn run_remote_tier_delete_test_hook(obj_name: &str, rv_id: &str, tier_name: &str
         .map(|hook| hook(obj_name, rv_id, tier_name))
 }
 
+#[cfg(test)]
+pub(super) struct RemoteTierDeleteHookGuard;
+
+#[cfg(test)]
+impl Drop for RemoteTierDeleteHookGuard {
+    fn drop(&mut self) {
+        let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
+            .lock()
+            .expect("remote tier delete test hook lock should not poison");
+        *hook = None;
+    }
+}
+
+#[cfg(test)]
+pub(super) fn set_remote_tier_delete_test_hook(
+    hook_fn: impl Fn(&str, &str, &str) -> std::io::Result<()> + Send + Sync + 'static,
+) -> RemoteTierDeleteHookGuard {
+    let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
+        .lock()
+        .expect("remote tier delete test hook lock should not poison");
+    *hook = Some(Box::new(hook_fn));
+    RemoteTierDeleteHookGuard
+}
+
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum RemoteTierDeleteOutcome {
     Deleted,
@@ -364,6 +413,38 @@ pub async fn delete_object_from_remote_tier_idempotent(
     }
 }
 
+pub(crate) async fn delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+    obj_name: &str,
+    rv_id: &str,
+    tier_name: &str,
+    backend_identity: TierDestinationId,
+    tier_config_mgr: &Arc<tokio::sync::RwLock<TierConfigMgr>>,
+    version_id_exact: bool,
+) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
+    let lease = TierConfigMgr::acquire_operation_lease_for_backend_identity(tier_config_mgr, tier_name, backend_identity)
+        .await
+        .map_err(std::io::Error::other)?;
+    delete_object_from_remote_tier_with_lease_idempotent(obj_name, rv_id, &lease, version_id_exact).await
+}
+
+pub(crate) async fn delete_object_from_remote_tier_with_lease_idempotent(
+    obj_name: &str,
+    rv_id: &str,
+    lease: &TierOperationLease,
+    version_id_exact: bool,
+) -> Result<RemoteTierDeleteOutcome, std::io::Error> {
+    match delete_object_from_remote_tier_raw_with_lease(obj_name, rv_id, lease, version_id_exact).await {
+        Ok(()) => Ok(RemoteTierDeleteOutcome::Deleted),
+        Err(err) if is_remote_tier_not_found_error(&err) => Ok(RemoteTierDeleteOutcome::AlreadyRemoved),
+        Err(err) => {
+            if should_record_remote_delete_failure(&err) {
+                record_remote_delete_failure(&err, Instant::now()).await;
+            }
+            Err(err)
+        }
+    }
+}
+
 pub(crate) fn is_remote_tier_not_found_error(err: &std::io::Error) -> bool {
     let message = err.to_string();
     message.contains("NoSuchKey")
@@ -401,6 +482,8 @@ pub fn transitioned_force_delete_journal_entry(transitioned: &TransitionedObject
         obj_name: transitioned.name.clone(),
         version_id: transitioned.version_id.clone(),
         tier_name: transitioned.tier.clone(),
+        backend_identity: None,
+        version_id_exact: false,
     })
 }
 
@@ -409,34 +492,14 @@ mod test {
     use crate::client::signer_error::invalid_utf8_header_error;
 
     use super::{
-        ERR_REMOTE_DELETE_BREAKER_OPEN, ERR_REMOTE_DELETE_LIMITER_CLOSED, REMOTE_TIER_DELETE_TEST_HOOK, RemoteDeleteBreaker,
-        RemoteTierDeleteOutcome, delete_object_from_remote_tier_idempotent, is_remote_tier_not_found_error,
-        is_signer_header_error, should_record_remote_delete_failure,
+        ERR_REMOTE_DELETE_BREAKER_OPEN, ERR_REMOTE_DELETE_LIMITER_CLOSED, RemoteDeleteBreaker, RemoteTierDeleteOutcome,
+        delete_object_from_remote_tier_idempotent, delete_object_from_remote_tier_idempotent_with_manager_and_identity,
+        is_remote_tier_not_found_error, is_signer_header_error, set_remote_tier_delete_test_hook,
+        should_record_remote_delete_failure,
     };
     use std::io::{Error, ErrorKind};
     use std::time::{Duration, Instant};
 
-    struct RemoteTierDeleteHookGuard;
-
-    impl Drop for RemoteTierDeleteHookGuard {
-        fn drop(&mut self) {
-            let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
-                .lock()
-                .expect("remote tier delete test hook lock should not poison");
-            *hook = None;
-        }
-    }
-
-    fn set_remote_tier_delete_test_hook(
-        hook_fn: impl Fn(&str, &str, &str) -> std::io::Result<()> + Send + Sync + 'static,
-    ) -> RemoteTierDeleteHookGuard {
-        let mut hook = REMOTE_TIER_DELETE_TEST_HOOK
-            .lock()
-            .expect("remote tier delete test hook lock should not poison");
-        *hook = Some(Box::new(hook_fn));
-        RemoteTierDeleteHookGuard
-    }
-
     #[test]
     fn signer_header_error_detection_matches_utf8_failures() {
         let err = Error::new(
@@ -506,6 +569,59 @@ mod test {
         assert!(err.to_string().contains("driver not found"));
     }
 
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    async fn journal_delete_rejects_backend_identity_mismatch() {
+        let manager = crate::services::tier::tier::TierConfigMgr::new();
+        crate::services::tier::test_util::register_mock_tier(&manager, "WARM").await;
+        let lease = crate::services::tier::tier::TierConfigMgr::acquire_operation_lease(&manager, "WARM")
+            .await
+            .expect("test tier lease should be available");
+        let mut mismatched = lease.backend_identity();
+        mismatched[0] ^= 1;
+        drop(lease);
+
+        let err = delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+            "remote/object",
+            "remote-version",
+            "WARM",
+            mismatched,
+            &manager,
+            false,
+        )
+        .await
+        .expect_err("journal recovery must fail closed when the tier name was rebound");
+
+        assert!(err.to_string().contains("identity no longer matches"));
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    async fn journal_delete_dispatches_an_exact_version_constraint() {
+        let manager = crate::services::tier::tier::TierConfigMgr::new();
+        let backend = crate::services::tier::test_util::register_mock_tier(&manager, "WARM").await;
+        let lease = crate::services::tier::tier::TierConfigMgr::acquire_operation_lease(&manager, "WARM")
+            .await
+            .expect("test tier lease should be available");
+        let identity = lease.backend_identity();
+        drop(lease);
+
+        let outcome = delete_object_from_remote_tier_idempotent_with_manager_and_identity(
+            "remote/object",
+            "exact-version",
+            "WARM",
+            identity,
+            &manager,
+            true,
+        )
+        .await
+        .expect("an exact journal delete should reach the backend");
+
+        assert_eq!(outcome, RemoteTierDeleteOutcome::Deleted);
+        assert_eq!(backend.exact_remove_count(), 1);
+        assert_eq!(backend.remove_count().await, 1);
+    }
+
     #[test]
     fn breaker_opens_at_threshold_and_recovers_after_window() {
         let mut breaker = RemoteDeleteBreaker::new(3, Duration::from_secs(30));

crates/ecstore/src/bucket/metadata_sys.rs

@@ -412,6 +412,10 @@ impl BucketMetadataSys {
         }
     }
 
+    pub(crate) fn object_store(&self) -> Arc<ECStore> {
+        self.api.clone()
+    }
+
     pub async fn init(&mut self, buckets: Vec<String>) {
         let _ = self.init_internal(buckets).await;
     }

crates/ecstore/src/bucket/utils.rs

@@ -183,6 +183,16 @@ pub fn is_valid_object_name(object: &str) -> bool {
     is_valid_object_prefix(object)
 }
 
+/// Client-facing reason attached to rejections of object keys that Win32/NTFS
+/// cannot represent as file paths (issue #3299). Deployments on Linux/macOS
+/// accept the full S3 key character set.
+pub const WINDOWS_RESERVED_CHARACTERS_REASON: &str =
+    "object key contains characters unsupported on Windows hosts (one of ':', '*', '?', '\"', '|', '<', '>')";
+
+/// Client-facing reason for path segments Windows can store but not address
+/// afterwards (issue #3449): trailing dot/space or reserved DOS device names.
+pub const WINDOWS_RESERVED_SEGMENT_REASON: &str = "object key contains a path segment unsupported on Windows hosts (trailing dot or space, or a reserved device name such as NUL/CON/COM1)";
+
 /// Reserved DOS device names that shadow regular files on Windows, even when
 /// an extension is appended (e.g. `NUL.txt` resolves to the `NUL` device).
 const WINDOWS_RESERVED_NAMES: &[&str] = &[
@@ -226,13 +236,21 @@ pub fn check_object_name_for_length_and_slash(bucket: &str, object: &str) -> Res
             || object.contains('>')
         // || object.contains('\\')
         {
-            return Err(StorageError::ObjectNameInvalid(bucket.to_owned(), object.to_owned()));
+            return Err(StorageError::InvalidArgument(
+                bucket.to_owned(),
+                object.to_owned(),
+                WINDOWS_RESERVED_CHARACTERS_REASON.to_owned(),
+            ));
         }
 
         // Reject names that NTFS would happily create but the Win32 path
         // layer cannot read back (os error 3), e.g. `baddir.` or `NUL.txt`.
         if object_name_has_windows_incompatible_segment(object) {
-            return Err(StorageError::ObjectNameInvalid(bucket.to_owned(), object.to_owned()));
+            return Err(StorageError::InvalidArgument(
+                bucket.to_owned(),
+                object.to_owned(),
+                WINDOWS_RESERVED_SEGMENT_REASON.to_owned(),
+            ));
         }
     }
 
@@ -537,8 +555,8 @@ mod tests {
             let result = check_object_name_for_length_and_slash("test-bucket", object);
             if cfg!(target_os = "windows") {
                 assert!(
-                    matches!(result, Err(StorageError::ObjectNameInvalid(..))),
-                    "object name must be rejected on Windows: {object:?}"
+                    matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_SEGMENT_REASON),
+                    "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}"
                 );
             } else {
                 assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}");
@@ -554,6 +572,23 @@ mod tests {
         }
     }
 
+    #[test]
+    fn test_check_object_name_windows_reserved_characters() {
+        // Keys containing Win32-reserved characters are rejected on Windows
+        // with a descriptive reason (issue #3299); valid elsewhere.
+        for object in ["path/*sUt*mykey", "a:b", "what?", "pipe|name", "quote\"d", "lt<gt>"] {
+            let result = check_object_name_for_length_and_slash("test-bucket", object);
+            if cfg!(target_os = "windows") {
+                assert!(
+                    matches!(&result, Err(StorageError::InvalidArgument(_, _, reason)) if reason == WINDOWS_RESERVED_CHARACTERS_REASON),
+                    "object name must be rejected on Windows with a descriptive reason: {object:?}, got {result:?}"
+                );
+            } else {
+                assert!(result.is_ok(), "object name must remain valid on non-Windows: {object:?}");
+            }
+        }
+    }
+
     #[test]
     fn test_check_bucket_and_object_names() {
         // Valid names

crates/ecstore/src/cache_value/metacache_set.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::disk::disk_store::get_drive_walkdir_stall_timeout;
+use crate::disk::disk_store::{get_drive_walkdir_peek_timeout, get_drive_walkdir_stall_timeout};
 use crate::disk::error::DiskError;
 use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
 use metrics::counter;
@@ -175,6 +175,10 @@ pub(crate) enum TestReaderBehavior {
     ProducerError(DiskError),
     PrimaryErrorThenFallback(DiskError),
     PartialThenTimeout(Vec<MetaCacheEntry>),
+    DelayedEntries {
+        delay: Duration,
+        entries: Vec<MetaCacheEntry>,
+    },
 }
 
 #[cfg(test)]
@@ -339,6 +343,13 @@ async fn list_path_raw_inner(
                         drop(out);
                         Some(err)
                     }
+                    TestReaderBehavior::DelayedEntries { delay, entries } => {
+                        tokio::time::sleep(delay).await;
+                        let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
+                        out.write(&entries).await.expect("delayed test entries should be written");
+                        out.close().await.expect("delayed test entries should close");
+                        return Ok(());
+                    }
                 }
             } else {
                 None
@@ -473,6 +484,17 @@ async fn list_path_raw_inner(
                             record_producer_error(&producer_errs_clone, disk_idx, &err);
                             return Err(err);
                         }
+                        TestReaderBehavior::DelayedEntries { delay, entries } => {
+                            tokio::time::sleep(delay).await;
+                            let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
+                            out.write(&entries)
+                                .await
+                                .expect("delayed test fallback entries should be written");
+                            out.close().await.expect("delayed test fallback entries should close");
+                            need_fallback = false;
+                            last_err = None;
+                            continue;
+                        }
                     }
                 }
 
@@ -609,47 +631,23 @@ async fn list_path_raw_inner(
         // Consumer-side peek timeout, and a caveat worth understanding
         // (rustfs/backlog#1217).
         //
-        // This budget is the SAME SOURCE and SAME VALUE as the producer-side
-        // walk stall budget: both default to `walkdir_stall_timeout` and fall
-        // back to `get_drive_walkdir_stall_timeout()` (5s). But the two measure
-        // different things:
+        // This budget must not be stricter than the producer-side walk stall
+        // budget. The two measure different things:
         //   * producer stall: bounds a single drive READ inside the walk (see
         //     `with_walk_stall_deadline` in `disk/local.rs`);
         //   * this consumer peek: bounds the interval between two ADJACENT
         //     entries arriving from a drive's reader (`peek_with_timeout`).
         //
-        // Because they are coupled to the same value, the consumer cannot wait
-        // meaningfully longer for the next entry than the producer is allowed to
-        // spend producing one. When a drive walks a region dense with
-        // non-listable internal items (many entries the producer filters out
-        // before emitting the next visible one), a HEALTHY drive can take longer
-        // than one budget to hand the consumer its next entry. The consumer then
-        // classifies it as `PeekOutcome::TimedOut` and DETACHES that reader (it
-        // is replaced by a drained duplex below), dropping a good drive from the
-        // merge. That caps the "large prefix always succeeds" guarantee: a wide
-        // enough non-listable stretch can knock healthy drives out of quorum.
-        //
-        // This is left documented, not decoupled. Giving the consumer peek an
-        // independent, strictly-larger budget would reduce these false detaches,
-        // but it also delays detaching a genuinely dead drive by the same amount
-        // and shifts listing tail-latency semantics; that trade-off wants soak
-        // data before it changes the default, so it is deferred to a follow-up.
-        // The constraint for any such change: the consumer peek must be >= the
-        // producer stall (never stricter), so it can never declare a drive
-        // stalled before the producer itself would have failed.
-        let peek_timeout = opts
-            .walkdir_stall_timeout
-            .or({
-                #[cfg(test)]
-                {
-                    opts.peek_timeout
-                }
-                #[cfg(not(test))]
-                {
-                    None
-                }
-            })
-            .unwrap_or_else(get_drive_walkdir_stall_timeout);
+        // A drive can spend one stall budget walking a dense non-listable region
+        // before it can publish the next visible entry. Use an independent,
+        // profile-aware consumer budget so the merge does not detach that reader
+        // before the producer itself would have failed.
+        let producer_stall_timeout = opts.walkdir_stall_timeout.unwrap_or_else(get_drive_walkdir_stall_timeout);
+        let configured_peek_timeout = get_drive_walkdir_peek_timeout().max(producer_stall_timeout);
+        #[cfg(not(test))]
+        let peek_timeout = configured_peek_timeout;
+        #[cfg(test)]
+        let peek_timeout = opts.peek_timeout.unwrap_or(configured_peek_timeout);
         let mut errs: Vec<Option<DiskError>> = Vec::with_capacity(readers.len());
         for _ in 0..readers.len() {
             errs.push(None);
@@ -1246,6 +1244,57 @@ mod tests {
         assert_eq!(err, DiskError::Timeout);
     }
 
+    #[tokio::test]
+    async fn list_path_raw_waits_past_producer_stall_for_slow_progressing_reader() {
+        let entry = MetaCacheEntry {
+            name: "bucket/visible-object".to_string(),
+            metadata: vec![1, 2, 3],
+            cached: None,
+            reusable: false,
+        };
+        let seen = Arc::new(Mutex::new(Vec::new()));
+        let seen_clone = seen.clone();
+
+        timeout(
+            Duration::from_secs(1),
+            list_path_raw(
+                CancellationToken::new(),
+                ListPathRawOptions {
+                    disks: vec![None, None],
+                    min_disks: 2,
+                    walkdir_stall_timeout: Some(Duration::from_millis(20)),
+                    test_reader_behaviors: vec![
+                        TestReaderBehavior::DelayedEntries {
+                            delay: Duration::from_millis(60),
+                            entries: vec![entry.clone()],
+                        },
+                        TestReaderBehavior::Entries(vec![entry]),
+                    ],
+                    partial: Some(Box::new(move |entries: MetaCacheEntries, _: &[Option<DiskError>]| {
+                        let seen = seen_clone.clone();
+                        Box::pin(async move {
+                            let mut names = entries.0.iter().flatten().map(|entry| entry.name.as_str());
+                            if let (Some(first), Some(second), None) = (names.next(), names.next(), names.next())
+                                && first == second
+                            {
+                                seen.lock().expect("seen mutex poisoned").push(first.to_owned());
+                            }
+                        })
+                    })),
+                    ..Default::default()
+                },
+            ),
+        )
+        .await
+        .expect("slow-progressing reader should complete inside the consumer peek budget")
+        .expect("slow-progressing reader should not be detached by the producer stall budget");
+
+        assert_eq!(
+            seen.lock().expect("seen mutex poisoned").as_slice(),
+            &["bucket/visible-object".to_string()]
+        );
+    }
+
     #[tokio::test]
     async fn list_path_raw_prefers_timeout_for_mixed_errors_beyond_quorum_budget() {
         let err = list_path_raw(

crates/ecstore/src/client/api_error_response.rs

@@ -103,7 +103,7 @@ pub fn http_resp_to_error_response(
     object_name: &str,
 ) -> ErrorResponse {
     let err_body = String::from_utf8_lossy(&b).to_string();
-    if h.is_empty() || resp_status.is_client_error() || resp_status.is_server_error() {
+    if h.is_empty() || !(resp_status.is_client_error() || resp_status.is_server_error()) {
         return ErrorResponse {
             status_code: resp_status,
             code: S3ErrorCode::ResponseInterrupted,
@@ -329,4 +329,21 @@ mod tests {
         assert_eq!(value["RequestId"], Value::String("req-xml-123".to_string()));
         assert!(value.get("request_id").is_none(), "external error contract must not expose request_id");
     }
+
+    #[test]
+    fn parses_s3_error_code_from_client_error_response() {
+        let mut headers = HeaderMap::new();
+        headers.insert("x-amz-request-id", "request-id".parse().expect("request ID header should parse"));
+
+        let response = http_resp_to_error_response(
+            StatusCode::NOT_FOUND,
+            &headers,
+            b"<Error><Code>NoSuchVersion</Code><Message>remote detail</Message></Error>".to_vec(),
+            "bucket",
+            "object",
+        );
+
+        assert_eq!(response.code, S3ErrorCode::NoSuchVersion);
+        assert_eq!(response.status_code, StatusCode::NOT_FOUND);
+    }
 }

crates/ecstore/src/client/api_get_object.rs

@@ -30,7 +30,7 @@ use tokio_util::io::StreamReader;
 use crate::client::{
     api_error_response::err_invalid_argument,
     api_get_options::GetObjectOptions,
-    transition_api::{ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, to_object_info},
+    transition_api::{ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, to_object_info_for_provider},
 };
 use futures_util::StreamExt;
 use http_body_util::BodyExt;
@@ -77,7 +77,8 @@ impl TransitionClient {
             )
             .await?;
 
-        let object_stat = to_object_info(bucket_name, object_name, resp.headers())?;
+        let object_stat =
+            to_object_info_for_provider(bucket_name, object_name, resp.headers(), self.provider_version_capabilities())?;
 
         let h = resp.headers().clone();
 

crates/ecstore/src/client/api_put_object_multipart.rs

@@ -41,7 +41,7 @@ use crate::client::{
     transition_api::{ReaderImpl, RequestMetadata, TransitionClient, UploadInfo},
 };
 use rustfs_utils::path::trim_etag;
-use s3s::header::{X_AMZ_EXPIRATION, X_AMZ_VERSION_ID};
+use s3s::header::X_AMZ_EXPIRATION;
 
 impl TransitionClient {
     pub async fn put_object_multipart(
@@ -417,11 +417,7 @@ impl TransitionClient {
             bucket: complete_multipart_upload_result.bucket,
             key: complete_multipart_upload_result.key,
             etag: trim_etag(&complete_multipart_upload_result.etag),
-            version_id: if let Some(h_x_amz_version_id) = h.get(X_AMZ_VERSION_ID) {
-                h_x_amz_version_id.to_str().unwrap_or("").to_string()
-            } else {
-                "".to_string()
-            },
+            version_id: self.raw_version_id(&h)?.unwrap_or_default().to_string(),
             location: complete_multipart_upload_result.location,
             expiration: exp_time,
             expiration_rule_id: rule_id,

crates/ecstore/src/client/api_put_object_streaming.rs

@@ -44,7 +44,7 @@ use crate::client::{
 
 use crate::client::utils::base64_encode;
 use rustfs_utils::path::trim_etag;
-use s3s::header::{X_AMZ_EXPIRATION, X_AMZ_VERSION_ID};
+use s3s::header::X_AMZ_EXPIRATION;
 
 /// Read exactly `want` bytes for a single multipart part, or fewer if the reader
 /// reaches EOF first. Advances the reader so the next call returns the following
@@ -567,11 +567,7 @@ impl TransitionClient {
             key: object_name.to_string(),
             etag: trim_etag(h.get("ETag").and_then(|v| v.to_str().ok()).unwrap_or("")),
 
-            version_id: if let Some(h_x_amz_version_id) = h.get(X_AMZ_VERSION_ID) {
-                h_x_amz_version_id.to_str().unwrap_or("").to_string()
-            } else {
-                "".to_string()
-            },
+            version_id: self.raw_version_id(h)?.unwrap_or_default().to_string(),
             size,
             expiration: exp_time,
             expiration_rule_id: rule_id,

crates/ecstore/src/client/api_remove.rs

@@ -201,12 +201,7 @@ impl TransitionClient {
             object_name: object_name.to_string(),
             object_version_id: opts.version_id,
             delete_marker: resp.headers().get(X_AMZ_DELETE_MARKER).map_or(false, |v| v == "true"),
-            delete_marker_version_id: resp
-                .headers()
-                .get(X_AMZ_VERSION_ID)
-                .and_then(|v| v.to_str().ok())
-                .unwrap_or_default()
-                .to_string(),
+            delete_marker_version_id: self.raw_version_id(resp.headers())?.unwrap_or_default().to_string(),
             ..Default::default()
         })
     }

crates/ecstore/src/client/api_stat.rs

@@ -18,7 +18,7 @@
 #![allow(unused_must_use)]
 #![allow(clippy::all)]
 
-use http::{HeaderMap, HeaderValue};
+use http::{HeaderMap, HeaderValue, StatusCode};
 use http_body_util::BodyExt;
 use hyper::body::Body;
 use hyper::body::Bytes;
@@ -31,13 +31,62 @@ use uuid::Uuid;
 use crate::client::{
     api_error_response::{ErrorResponse, err_invalid_argument, http_resp_to_error_response},
     api_get_options::GetObjectOptions,
-    transition_api::{ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, to_object_info},
+    transition_api::{
+        ObjectInfo, ReadCloser, ReaderImpl, RequestMetadata, TransitionClient, collect_response_body, to_object_info_for_provider,
+    },
 };
 use s3s::{
-    dto::VersioningConfiguration,
+    dto::{BucketVersioningStatus, MFADelete, VersioningConfiguration},
     header::{X_AMZ_DELETE_MARKER, X_AMZ_VERSION_ID},
 };
 
+const S3_XML_NAMESPACE: &str = "http://s3.amazonaws.com/doc/2006-03-01/";
+
+#[derive(serde::Deserialize)]
+#[serde(deny_unknown_fields)]
+struct StrictVersioningConfiguration {
+    #[serde(rename = "@xmlns")]
+    namespace: Option<String>,
+    #[serde(rename = "MfaDelete")]
+    mfa_delete: Option<String>,
+    #[serde(rename = "Status")]
+    status: Option<String>,
+}
+
+#[derive(serde::Deserialize)]
+enum StrictVersioningResponse {
+    VersioningConfiguration(StrictVersioningConfiguration),
+}
+
+fn parse_bucket_versioning_response(
+    status: StatusCode,
+    headers: &HeaderMap,
+    body: Vec<u8>,
+    bucket_name: &str,
+) -> Result<VersioningConfiguration, std::io::Error> {
+    if status != StatusCode::OK {
+        return Err(std::io::Error::other(http_resp_to_error_response(status, headers, body, bucket_name, "")));
+    }
+
+    let StrictVersioningResponse::VersioningConfiguration(parsed) =
+        quick_xml::de::from_reader(body.as_slice()).map_err(|err| std::io::Error::new(std::io::ErrorKind::InvalidData, err))?;
+    if parsed
+        .namespace
+        .as_deref()
+        .is_some_and(|namespace| namespace != S3_XML_NAMESPACE)
+    {
+        return Err(std::io::Error::new(
+            std::io::ErrorKind::InvalidData,
+            "unexpected GetBucketVersioning XML namespace",
+        ));
+    }
+    Ok(VersioningConfiguration {
+        mfa_delete: parsed.mfa_delete.map(MFADelete::from),
+        status: parsed.status.map(BucketVersioningStatus::from),
+        ..Default::default()
+    })
+}
+
 impl TransitionClient {
     pub async fn bucket_exists(&self, bucket_name: &str) -> Result<bool, std::io::Error> {
         let resp = self
@@ -123,19 +172,8 @@ impl TransitionClient {
                 let resp_status = resp.status();
                 let h = resp.headers().clone();
 
-                let mut body_vec = Vec::new();
-                let mut body = resp.into_body();
-                while let Some(frame) = body.frame().await {
-                    let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-                    if let Some(data) = frame.data_ref() {
-                        body_vec.extend_from_slice(data);
-                    }
-                }
-                let resperr = http_resp_to_error_response(resp_status, &h, body_vec, bucket_name, "");
-
-                warn!("get bucket versioning, resperr: {:?}", resperr);
-
-                Ok(VersioningConfiguration::default())
+                let body_vec = collect_response_body(resp.into_body(), rustfs_config::MAX_S3_CLIENT_RESPONSE_SIZE).await?;
+                parse_bucket_versioning_response(resp_status, &h, body_vec, bucket_name)
             }
 
             Err(err) => Err(std::io::Error::other(err)),
@@ -206,20 +244,20 @@ impl TransitionClient {
                             ..Default::default()
                         };
                         return Ok(ObjectInfo {
-                            version_id: h
-                                .get(X_AMZ_VERSION_ID)
-                                .and_then(|v| v.to_str().ok())
-                                .and_then(|s| Uuid::from_str(s).ok()),
+                            version_id: self
+                                .raw_version_id(h)?
+                                .and_then(|s| Uuid::from_str(s).ok())
+                                .filter(|v| !v.is_nil()),
                             is_delete_marker: delete_marker,
                             ..Default::default()
                         });
                         //err_resp
                     }
                     return Ok(ObjectInfo {
-                        version_id: h
-                            .get(X_AMZ_VERSION_ID)
-                            .and_then(|v| v.to_str().ok())
-                            .and_then(|s| Uuid::from_str(s).ok()),
+                        version_id: self
+                            .raw_version_id(h)?
+                            .and_then(|s| Uuid::from_str(s).ok())
+                            .filter(|v| !v.is_nil()),
                         is_delete_marker: delete_marker,
                         replication_ready: replication_ready,
                         ..Default::default()
@@ -227,7 +265,7 @@ impl TransitionClient {
                     //http_resp_to_error_response(resp, bucket_name, object_name)
                 }
 
-                Ok(to_object_info(bucket_name, object_name, h).expect("operation should succeed"))
+                to_object_info_for_provider(bucket_name, object_name, h, self.provider_version_capabilities())
             }
             Err(err) => {
                 return Err(std::io::Error::other(err));
@@ -235,3 +273,72 @@ impl TransitionClient {
         }
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::parse_bucket_versioning_response;
+    use http::{HeaderMap, StatusCode};
+    use s3s::dto::BucketVersioningStatus;
+
+    #[test]
+    fn parses_bucket_versioning_statuses_mfa_delete_and_unversioned_state() {
+        for (xml, expected) in [
+            (
+                br#"<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Enabled</Status></VersioningConfiguration>"#
+                    .as_slice(),
+                Some(BucketVersioningStatus::ENABLED),
+            ),
+            (
+                br#"<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Status>Suspended</Status></VersioningConfiguration>"#
+                    .as_slice(),
+                Some(BucketVersioningStatus::SUSPENDED),
+            ),
+            (
+                br#"<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>"#.as_slice(),
+                None,
+            ),
+        ] {
+            let config = parse_bucket_versioning_response(StatusCode::OK, &HeaderMap::new(), xml.to_vec(), "tier-bucket")
+                .expect("valid GetBucketVersioning response should parse");
+            assert_eq!(config.status.as_ref().map(|status| status.as_str()), expected);
+        }
+
+        let config = parse_bucket_versioning_response(
+            StatusCode::OK,
+            &HeaderMap::new(),
+            br#"<VersioningConfiguration><MfaDelete>Enabled</MfaDelete></VersioningConfiguration>"#.to_vec(),
+            "tier-bucket",
+        )
+        .expect("valid MfaDelete should parse");
+        assert_eq!(config.mfa_delete.as_ref().map(|status| status.as_str()), Some("Enabled"));
+    }
+
+    #[test]
+    fn rejects_non_ok_and_malformed_bucket_versioning_responses() {
+        let valid = br#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#.to_vec();
+        for status in [StatusCode::NO_CONTENT, StatusCode::PARTIAL_CONTENT] {
+            let status_err = parse_bucket_versioning_response(status, &HeaderMap::new(), valid.clone(), "tier-bucket")
+                .expect_err("GetBucketVersioning must return exactly HTTP 200");
+            assert_eq!(status_err.kind(), std::io::ErrorKind::Other);
+        }
+
+        let parse_err = parse_bucket_versioning_response(
+            StatusCode::OK,
+            &HeaderMap::new(),
+            b"<VersioningConfiguration>".to_vec(),
+            "tier-bucket",
+        )
+        .expect_err("malformed GetBucketVersioning XML must fail closed");
+        assert_eq!(parse_err.kind(), std::io::ErrorKind::InvalidData);
+
+        for xml in [
+            b"<VersioningConfiguration><Statuz>Enabled</Statuz></VersioningConfiguration>".as_slice(),
+            b"<WrongRoot><Status>Enabled</Status></WrongRoot>".as_slice(),
+            b"<VersioningConfiguration xmlns=\"https://example.invalid\"/>".as_slice(),
+        ] {
+            let strict_err = parse_bucket_versioning_response(StatusCode::OK, &HeaderMap::new(), xml.to_vec(), "tier-bucket")
+                .expect_err("unknown GetBucketVersioning XML must fail closed");
+            assert_eq!(strict_err.kind(), std::io::ErrorKind::InvalidData);
+        }
+    }
+}

crates/ecstore/src/client/mod.rs

@@ -38,6 +38,7 @@ pub mod constants;
 pub mod credentials;
 pub mod object_api_utils;
 pub mod object_handlers_common;
+pub(crate) mod provider_versions;
 pub(crate) mod runtime_sources;
 pub mod signer_error;
 pub mod transition_api;

crates/ecstore/src/client/provider_versions.rs

@@ -0,0 +1,236 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use std::io::{Error, ErrorKind};
+
+use http::HeaderMap;
+
+const X_AMZ_VERSION_ID: &str = "x-amz-version-id";
+const X_OSS_VERSION_ID: &str = "x-oss-version-id";
+const X_COS_VERSION_ID: &str = "x-cos-version-id";
+const X_OBS_VERSION_ID: &str = "x-obs-version-id";
+const MAX_REMOTE_VERSION_ID_LEN: usize = 1024;
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(crate) enum BucketVersioningState {
+    Unknown,
+    Disabled,
+    Suspended,
+    Enabled,
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub(crate) enum RemoteVersion {
+    Unknown,
+    Disabled,
+    SuspendedNull,
+    Exact(String),
+}
+
+impl RemoteVersion {
+    pub(crate) fn exact_id(&self) -> Option<&str> {
+        match self {
+            Self::SuspendedNull => Some("null"),
+            Self::Exact(version_id) => Some(version_id),
+            Self::Unknown | Self::Disabled => None,
+        }
+    }
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub(crate) struct ProviderVersionCapabilities {
+    raw_version_header: Option<&'static str>,
+    pub(crate) exact_get_delete: bool,
+}
+
+impl ProviderVersionCapabilities {
+    pub(crate) fn for_tier_type(tier_type: &str) -> Self {
+        if tier_type.eq_ignore_ascii_case("s3")
+            || tier_type.eq_ignore_ascii_case("rustfs")
+            || tier_type.eq_ignore_ascii_case("minio")
+            || tier_type.eq_ignore_ascii_case("r2")
+            || tier_type.eq_ignore_ascii_case("wasabi")
+        {
+            Self {
+                raw_version_header: Some(X_AMZ_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else if tier_type.eq_ignore_ascii_case("aliyun") {
+            Self {
+                raw_version_header: Some(X_OSS_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else if tier_type.eq_ignore_ascii_case("tencent") {
+            Self {
+                raw_version_header: Some(X_COS_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else if tier_type.eq_ignore_ascii_case("huaweicloud") {
+            Self {
+                raw_version_header: Some(X_OBS_VERSION_ID),
+                exact_get_delete: true,
+            }
+        } else {
+            Self {
+                raw_version_header: None,
+                exact_get_delete: false,
+            }
+        }
+    }
+
+    pub(crate) fn raw_version_id(self, headers: &HeaderMap) -> Result<Option<&str>, Error> {
+        let Some(header_name) = self.raw_version_header else {
+            return Ok(None);
+        };
+        let Some(value) = headers.get(header_name) else {
+            return Ok(None);
+        };
+        let value = value
+            .to_str()
+            .map_err(|_| Error::new(ErrorKind::InvalidData, "remote object version id is not valid ASCII"))?;
+        validate_remote_version_id(value)?;
+        Ok(Some(value))
+    }
+
+    pub(crate) fn remote_version(self, headers: &HeaderMap, versioning: BucketVersioningState) -> Result<RemoteVersion, Error> {
+        let Some(value) = self.raw_version_id(headers)? else {
+            return Ok(match versioning {
+                BucketVersioningState::Disabled => RemoteVersion::Disabled,
+                BucketVersioningState::Unknown | BucketVersioningState::Suspended | BucketVersioningState::Enabled => {
+                    RemoteVersion::Unknown
+                }
+            });
+        };
+        if value == "null" {
+            return Ok(RemoteVersion::SuspendedNull);
+        }
+        Ok(RemoteVersion::Exact(value.to_string()))
+    }
+}
+
+fn validate_remote_version_id(version_id: &str) -> Result<(), Error> {
+    if version_id.is_empty() {
+        return Err(Error::new(
+            ErrorKind::InvalidData,
+            "remote tier returned an empty object version id header",
+        ));
+    }
+    if version_id.len() > MAX_REMOTE_VERSION_ID_LEN {
+        return Err(Error::new(
+            ErrorKind::InvalidData,
+            "remote tier returned an oversized object version id header",
+        ));
+    }
+    if version_id.chars().any(char::is_control) {
+        return Err(Error::new(
+            ErrorKind::InvalidData,
+            "remote tier returned an object version id containing control characters",
+        ));
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{BucketVersioningState, ProviderVersionCapabilities, RemoteVersion};
+    use http::{HeaderMap, HeaderValue};
+
+    #[test]
+    fn provider_version_header_matrix_preserves_opaque_versions() {
+        for (tier_type, header_name) in [
+            ("s3", "x-amz-version-id"),
+            ("S3", "x-amz-version-id"),
+            ("rustfs", "x-amz-version-id"),
+            ("RustFS", "x-amz-version-id"),
+            ("minio", "x-amz-version-id"),
+            ("MinIO", "x-amz-version-id"),
+            ("r2", "x-amz-version-id"),
+            ("R2", "x-amz-version-id"),
+            ("wasabi", "x-amz-version-id"),
+            ("Wasabi", "x-amz-version-id"),
+            ("aliyun", "x-oss-version-id"),
+            ("Aliyun", "x-oss-version-id"),
+            ("tencent", "x-cos-version-id"),
+            ("Tencent", "x-cos-version-id"),
+            ("huaweicloud", "x-obs-version-id"),
+            ("Huaweicloud", "x-obs-version-id"),
+        ] {
+            let mut headers = HeaderMap::new();
+            headers.insert(header_name, HeaderValue::from_static("opaque.version_01"));
+            let capabilities = ProviderVersionCapabilities::for_tier_type(tier_type);
+
+            assert_eq!(capabilities.raw_version_id(&headers).expect("raw version"), Some("opaque.version_01"));
+            assert_eq!(
+                capabilities
+                    .remote_version(&headers, BucketVersioningState::Enabled)
+                    .expect("remote version"),
+                RemoteVersion::Exact("opaque.version_01".to_string())
+            );
+        }
+    }
+
+    #[test]
+    fn provider_version_header_matrix_does_not_cross_read_sibling_headers() {
+        let mut headers = HeaderMap::new();
+        headers.insert("x-amz-version-id", HeaderValue::from_static("aws-version"));
+        headers.insert("x-cos-version-id", HeaderValue::from_static("cos-version"));
+
+        assert_eq!(
+            ProviderVersionCapabilities::for_tier_type("s3")
+                .raw_version_id(&headers)
+                .expect("aws raw version"),
+            Some("aws-version")
+        );
+        assert_eq!(
+            ProviderVersionCapabilities::for_tier_type("tencent")
+                .raw_version_id(&headers)
+                .expect("cos raw version"),
+            Some("cos-version")
+        );
+    }
+
+    #[test]
+    fn provider_version_missing_header_is_unknown_until_bucket_state_is_known() {
+        let headers = HeaderMap::new();
+        let capabilities = ProviderVersionCapabilities::for_tier_type("aliyun");
+
+        assert_eq!(
+            capabilities
+                .remote_version(&headers, BucketVersioningState::Unknown)
+                .expect("unknown versioning"),
+            RemoteVersion::Unknown
+        );
+        assert_eq!(
+            capabilities
+                .remote_version(&headers, BucketVersioningState::Disabled)
+                .expect("disabled versioning"),
+            RemoteVersion::Disabled
+        );
+    }
+
+    #[test]
+    fn provider_version_rejects_empty_or_oversized_headers() {
+        let oversized = "v".repeat(1025);
+        for bad in ["", oversized.as_str()] {
+            let mut headers = HeaderMap::new();
+            headers.insert("x-oss-version-id", HeaderValue::from_str(bad).expect("test header value"));
+
+            assert!(
+                ProviderVersionCapabilities::for_tier_type("aliyun")
+                    .raw_version_id(&headers)
+                    .is_err()
+            );
+        }
+    }
+}

crates/ecstore/src/client/transition_api.rs

@@ -31,6 +31,7 @@ use crate::client::{
     },
     constants::{UNSIGNED_PAYLOAD, UNSIGNED_PAYLOAD_TRAILER},
     credentials::{CredContext, Credentials, SignatureType, Static},
+    provider_versions::{BucketVersioningState, ProviderVersionCapabilities},
     signer_error,
 };
 use crate::{client::checksum::ChecksumMode, object_api::GetObjectReader};
@@ -41,7 +42,7 @@ use http::{
     request::{Builder, Request},
 };
 use http_body::Body;
-use http_body_util::BodyExt;
+use http_body_util::{BodyExt, LengthLimitError, Limited};
 use hyper::body::Bytes;
 use hyper::body::Incoming;
 use hyper_rustls::{ConfigBuilderExt, HttpsConnector};
@@ -54,9 +55,7 @@ use rustfs_rio::HashReader;
 use rustfs_utils::HashAlgorithm;
 use rustfs_utils::{
     net::get_endpoint_url,
-    retry::{
-        DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, MAX_RETRY, RetryTimer, is_http_status_retryable, is_s3code_retryable,
-    },
+    retry::{DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, MAX_RETRY, RetryTimer},
 };
 use rustls_pki_types::PrivateKeyDer;
 use rustls_pki_types::pem::PemObject;
@@ -81,9 +80,25 @@ use url::{Url, form_urlencoded};
 use uuid::Uuid;
 
 const C_USER_AGENT: &str = "RustFS (linux; x86)";
+pub(crate) const MAX_S3_ERROR_RESPONSE_SIZE: usize = 64 * 1024;
 
 const SUCCESS_STATUS: [StatusCode; 3] = [StatusCode::OK, StatusCode::NO_CONTENT, StatusCode::PARTIAL_CONTENT];
 
+pub(crate) async fn collect_response_body<B>(body: B, limit: usize) -> Result<Vec<u8>, std::io::Error>
+where
+    B: Body<Data = Bytes>,
+    B::Error: Into<Box<dyn std::error::Error + Send + Sync>>,
+{
+    let body = Limited::new(body, limit).collect().await.map_err(|err| {
+        if err.is::<LengthLimitError>() {
+            std::io::Error::new(std::io::ErrorKind::InvalidData, "remote tier response body exceeds limit")
+        } else {
+            std::io::Error::other(err)
+        }
+    })?;
+    Ok(body.to_bytes().to_vec())
+}
+
 const C_UNKNOWN: i32 = -1;
 const C_OFFLINE: i32 = 0;
 const C_ONLINE: i32 = 1;
@@ -92,9 +107,18 @@ fn invalid_utf8_header_error(scope: &str, header_name: &str) -> std::io::Error {
     signer_error::invalid_utf8_header_error(scope, header_name)
 }
 
-fn validate_header_values(headers: &HeaderMap, scope: &str) -> Result<(), std::io::Error> {
+fn validate_header_values(headers: &HeaderMap, scope: &str, signer_type: &SignatureType) -> Result<(), std::io::Error> {
     for (name, value) in headers {
-        value.to_str().map_err(|_| invalid_utf8_header_error(scope, name.as_str()))?;
+        if signer_type == &SignatureType::SignatureV2 {
+            // The SigV2 canonicalizer only supports visible ASCII values. Keep rejecting
+            // non-ASCII here so it cannot silently omit a value that is sent on the wire.
+            value.to_str().map_err(|_| invalid_utf8_header_error(scope, name.as_str()))?;
+        } else {
+            let value = std::str::from_utf8(value.as_bytes()).map_err(|_| invalid_utf8_header_error(scope, name.as_str()))?;
+            if value.chars().any(|ch| !ch.is_ascii() && ch.is_whitespace()) {
+                return Err(invalid_utf8_header_error(scope, name.as_str()));
+            }
+        }
     }
     Ok(())
 }
@@ -300,6 +324,14 @@ impl TransitionClient {
         self.endpoint_url.clone()
     }
 
+    pub(crate) fn provider_version_capabilities(&self) -> ProviderVersionCapabilities {
+        ProviderVersionCapabilities::for_tier_type(&self.tier_type)
+    }
+
+    pub(crate) fn raw_version_id<'a>(&self, headers: &'a HeaderMap) -> Result<Option<&'a str>, std::io::Error> {
+        self.provider_version_capabilities().raw_version_id(headers)
+    }
+
     fn trace_errors_only_off(&self) {
         if let Ok(mut trace_errors_only) = self.trace_errors_only.lock() {
             *trace_errors_only = false;
@@ -379,20 +411,17 @@ impl TransitionClient {
     pub async fn doit(&self, req: Request<s3s::Body>) -> Result<Response<Incoming>, std::io::Error> {
         let req_method;
         let req_uri;
-        let req_headers;
         let resp;
         let http_client = self.http_client.clone();
         {
             req_method = req.method().clone();
             req_uri = req.uri().clone();
-            req_headers = req.headers().clone();
 
             debug!("endpoint_url: {}", self.endpoint_url.as_str().to_string());
             resp = http_client.request(req);
         }
         let resp = resp.await;
         debug!("http_client url: {} {}", req_method, req_uri);
-        debug!("http_client headers: {:?}", req_headers);
         if let Err(err) = resp {
             error!("http_client call error: {:?}", err);
             return Err(std::io::Error::other(err));
@@ -402,28 +431,23 @@ impl TransitionClient {
             Ok(r) => r,
             Err(_) => return Err(std::io::Error::other("Unexpected error in response")),
         };
-        debug!("http_resp: {:?}", resp);
+        debug!(status = %resp.status(), "remote tier response received");
 
         //let b = resp.body_mut().store_all_unlimited().await.unwrap().to_vec();
         //debug!("http_resp_body: {}", String::from_utf8(b).unwrap());
 
         //if self.is_trace_enabled && !(self.trace_errors_only && resp.status() == StatusCode::OK) {
         if !resp.status().is_success() {
-            //self.dump_http(&cloned_req, &resp)?;
-            let mut body_vec = Vec::new();
-            let mut body = resp.into_body();
-            while let Some(frame) = body.frame().await {
-                let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-                if let Some(data) = frame.data_ref() {
-                    body_vec.extend_from_slice(data);
-                }
-            }
-            let body_str = String::from_utf8_lossy(&body_vec);
-            warn!("err_body: {}", body_str);
-            Err(std::io::Error::other(format!("http_client call error: {}", body_str)))
-        } else {
-            Ok(resp)
+            let status = resp.status();
+            let request_id = resp
+                .headers()
+                .get("x-amz-request-id")
+                .and_then(|value| value.to_str().ok())
+                .unwrap_or_default()
+                .to_string();
+            warn!(status = %status, request_id, "remote tier request rejected");
         }
+        Ok(resp)
     }
 
     pub async fn execute_method(
@@ -465,17 +489,22 @@ impl TransitionClient {
             let resp_status = resp.status();
             let h = resp.headers().clone();
 
-            let mut body_vec = Vec::new();
-            let mut body = resp.into_body();
-            while let Some(frame) = body.frame().await {
-                let frame = frame.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))?;
-                if let Some(data) = frame.data_ref() {
-                    body_vec.extend_from_slice(data);
-                }
-            }
-            let mut err_response =
-                http_resp_to_error_response(resp_status, &h, body_vec.clone(), &metadata.bucket_name, &metadata.object_name);
-            err_response.message = format!("remote tier error: {}", err_response.message);
+            let body_vec = collect_response_body(resp.into_body(), MAX_S3_ERROR_RESPONSE_SIZE).await?;
+            let parsed_error =
+                http_resp_to_error_response(resp_status, &h, body_vec, &metadata.bucket_name, &metadata.object_name);
+            let routing_region = parsed_error.region;
+            let code = match parsed_error.code {
+                S3ErrorCode::Custom(_) => S3ErrorCode::ResponseInterrupted,
+                code => code,
+            };
+            let err_response = ErrorResponse {
+                status_code: resp_status,
+                message: format!("remote tier request failed with status {resp_status}: {}", code.as_str()),
+                code,
+                bucket_name: metadata.bucket_name.clone(),
+                key: metadata.object_name.clone(),
+                ..Default::default()
+            };
 
             if self.region == "" {
                 return match err_response.code {
@@ -484,20 +513,20 @@ impl TransitionClient {
                         Err(std::io::Error::other(err_response))
                     }
                     S3ErrorCode::AccessDenied => {
-                        if err_response.region == "" {
+                        if routing_region.is_empty() {
                             return Err(std::io::Error::other(err_response));
                         }
                         if metadata.bucket_name != "" {
                             if let Ok(mut bucket_loc_cache) = self.bucket_loc_cache.lock() {
                                 if let Some(location) = bucket_loc_cache.get(&metadata.bucket_name) {
-                                    if location != err_response.region {
-                                        bucket_loc_cache.set(&metadata.bucket_name, &err_response.region);
+                                    if location != routing_region {
+                                        bucket_loc_cache.set(&metadata.bucket_name, &routing_region);
                                         //continue;
                                     }
                                 }
                             }
-                        } else if err_response.region != metadata.bucket_location {
-                            metadata.bucket_location = err_response.region.clone();
+                        } else if routing_region != metadata.bucket_location {
+                            metadata.bucket_location = routing_region;
                             //continue;
                         }
                         Err(std::io::Error::other(err_response))
@@ -508,18 +537,10 @@ impl TransitionClient {
                 };
             }
 
-            if is_s3code_retryable(err_response.code.as_str()) {
-                continue;
-            }
-
-            if is_http_status_retryable(&resp_status) {
-                continue;
-            }
-
-            break;
+            return Err(std::io::Error::other(err_response));
         }
 
-        Err(std::io::Error::other("resp err"))
+        Err(std::io::Error::other("remote tier request did not produce a response"))
     }
 
     async fn new_request(
@@ -586,7 +607,7 @@ impl TransitionClient {
                     )));
                 }
                 if let Some(extra_headers) = metadata.extra_pre_sign_header.as_ref() {
-                    validate_header_values(extra_headers, "presign extra header")?;
+                    validate_header_values(extra_headers, "presign extra header", &signer_type)?;
                     let headers = req.headers_mut();
                     for (k, v) in extra_headers {
                         headers.insert(k, v.clone());
@@ -611,7 +632,7 @@ impl TransitionClient {
         }
 
         self.set_user_agent(&mut req);
-        validate_header_values(&metadata.custom_header, "request custom header")?;
+        validate_header_values(&metadata.custom_header, "request custom header", &signer_type)?;
 
         for (k, v) in metadata.custom_header.clone() {
             if let Some(key) = k {
@@ -1144,6 +1165,15 @@ impl Default for UploadInfo {
 /// This function parses various S3 response headers to construct an ObjectInfo struct
 /// containing metadata about an S3 object.
 pub fn to_object_info(bucket_name: &str, object_name: &str, h: &HeaderMap) -> Result<ObjectInfo, std::io::Error> {
+    to_object_info_for_provider(bucket_name, object_name, h, ProviderVersionCapabilities::for_tier_type("s3"))
+}
+
+pub(crate) fn to_object_info_for_provider(
+    bucket_name: &str,
+    object_name: &str,
+    h: &HeaderMap,
+    version_capabilities: ProviderVersionCapabilities,
+) -> Result<ObjectInfo, std::io::Error> {
     // Helper function to get header value as string
     let get_header = |name: &str| -> String { h.get(name).and_then(|val| val.to_str().ok()).unwrap_or("").to_string() };
 
@@ -1261,14 +1291,11 @@ pub fn to_object_info(bucket_name: &str, object_name: &str, h: &HeaderMap) -> Re
     };
 
     // Extract version ID
-    let version_id = {
-        let version_id_str = get_header("x-amz-version-id");
-        if !version_id_str.is_empty() {
-            Some(Uuid::parse_str(&version_id_str).unwrap_or_else(|_| Uuid::nil()))
-        } else {
-            None
-        }
-    };
+    let version_id = version_capabilities
+        .remote_version(h, BucketVersioningState::Unknown)?
+        .exact_id()
+        .and_then(|version_id| Uuid::parse_str(version_id).ok())
+        .filter(|version_id| !version_id.is_nil());
 
     // Check if it's a delete marker
     let is_delete_marker = get_header("x-amz-delete-marker") == "true";
@@ -1383,8 +1410,43 @@ pub struct CreateBucketConfiguration {
 
 #[cfg(test)]
 mod tests {
-    use super::{build_tls_config, signer_error_to_io_error, validate_header_values, with_rustls_init_guard};
+    use super::{
+        MAX_S3_CLIENT_RESPONSE_SIZE, MAX_S3_ERROR_RESPONSE_SIZE, SignatureType, build_tls_config, collect_response_body,
+        signer_error_to_io_error, to_object_info_for_provider, validate_header_values, with_rustls_init_guard,
+    };
+    use crate::client::provider_versions::ProviderVersionCapabilities;
     use http::{HeaderMap, HeaderValue};
+    use http_body_util::Full;
+    use hyper::body::Bytes;
+    use uuid::Uuid;
+
+    #[tokio::test]
+    async fn response_body_limit_rejects_the_first_byte_past_the_boundary() {
+        let exact = collect_response_body(
+            Full::new(Bytes::from(vec![0_u8; MAX_S3_CLIENT_RESPONSE_SIZE])),
+            MAX_S3_CLIENT_RESPONSE_SIZE,
+        )
+        .await
+        .expect("a response exactly at the limit should be accepted");
+        assert_eq!(exact.len(), MAX_S3_CLIENT_RESPONSE_SIZE);
+
+        let err = collect_response_body(
+            Full::new(Bytes::from(vec![0_u8; MAX_S3_CLIENT_RESPONSE_SIZE + 1])),
+            MAX_S3_CLIENT_RESPONSE_SIZE,
+        )
+        .await
+        .expect_err("oversized remote response must fail closed");
+
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
+
+        let err = collect_response_body(
+            Full::new(Bytes::from(vec![0_u8; MAX_S3_ERROR_RESPONSE_SIZE + 1])),
+            MAX_S3_ERROR_RESPONSE_SIZE,
+        )
+        .await
+        .expect_err("oversized S3 error response must use the smaller error limit");
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
+    }
 
     #[test]
     fn rustls_guard_converts_panics_to_io_errors() {
@@ -1427,11 +1489,34 @@ mod tests {
             HeaderValue::from_bytes(&[0xFF]).expect("invalid utf8 bytes should be accepted by HeaderValue"),
         );
 
-        let err =
-            validate_header_values(&headers, "request custom header").expect_err("invalid header value should fail validation");
+        let err = validate_header_values(&headers, "request custom header", &SignatureType::SignatureV4)
+            .expect_err("invalid header value should fail validation");
         assert!(err.to_string().contains("x-amz-meta-invalid"));
     }
 
+    #[test]
+    fn validate_header_values_accepts_utf8_for_v4_but_not_v2() {
+        let mut headers = HeaderMap::new();
+        headers.insert(
+            "x-amz-meta-name",
+            HeaderValue::from_bytes("20260715/鲁A12345/object".as_bytes()).expect("valid utf8 metadata header"),
+        );
+
+        assert!(validate_header_values(&headers, "request custom header", &SignatureType::SignatureV4).is_ok());
+        assert!(validate_header_values(&headers, "request custom header", &SignatureType::SignatureV2).is_err());
+    }
+
+    #[test]
+    fn validate_header_values_rejects_non_ascii_whitespace_for_v4() {
+        let mut headers = HeaderMap::new();
+        headers.insert(
+            "x-amz-meta-name",
+            HeaderValue::from_bytes("tier/a\u{00a0}b/object".as_bytes()).expect("valid utf8 metadata header"),
+        );
+
+        assert!(validate_header_values(&headers, "request custom header", &SignatureType::SignatureV4).is_err());
+    }
+
     #[test]
     fn signer_error_mapping_preserves_header_name() {
         let err = signer_error_to_io_error(
@@ -1443,4 +1528,39 @@ mod tests {
         assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
         assert!(err.to_string().contains("x-amz-meta-invalid"));
     }
+
+    #[test]
+    fn object_info_uses_provider_version_header_without_uuid_coercion() {
+        let mut headers = HeaderMap::new();
+        headers.insert("x-cos-version-id", HeaderValue::from_static("opaque.version_01"));
+
+        let info =
+            to_object_info_for_provider("bucket", "object", &headers, ProviderVersionCapabilities::for_tier_type("tencent"))
+                .expect("opaque provider version should parse");
+
+        assert_eq!(info.version_id, None);
+        assert_eq!(
+            info.metadata.get("x-cos-version-id").and_then(|value| value.to_str().ok()),
+            Some("opaque.version_01")
+        );
+    }
+
+    #[test]
+    fn object_info_keeps_s3_uuid_version_id_and_filters_nil() {
+        let version_id = Uuid::new_v4();
+        let mut headers = HeaderMap::new();
+        headers.insert(
+            "x-amz-version-id",
+            HeaderValue::from_str(&version_id.to_string()).expect("uuid header should be valid"),
+        );
+
+        let info = to_object_info_for_provider("bucket", "object", &headers, ProviderVersionCapabilities::for_tier_type("s3"))
+            .expect("s3 uuid version should parse");
+        assert_eq!(info.version_id, Some(version_id));
+
+        headers.insert("x-amz-version-id", HeaderValue::from_static("00000000-0000-0000-0000-000000000000"));
+        let info = to_object_info_for_provider("bucket", "object", &headers, ProviderVersionCapabilities::for_tier_type("s3"))
+            .expect("nil s3 uuid version should parse");
+        assert_eq!(info.version_id, None);
+    }
 }

crates/ecstore/src/cluster/control_plane.rs

@@ -32,6 +32,9 @@ const FAILURE_DOMAIN_NOT_REPORTED: &str = "failure domain labels not reported by
 const NUMA_NOT_WIRED: &str = "NUMA topology not wired into runtime";
 const PROFILING_NOT_WIRED: &str = "profiling capability not wired into ECStore";
 const PEER_HEALTH_NOT_REPORTED: &str = "peer health not reported by endpoints";
+const PEER_HEALTH_LOCAL_NODE: &str = "local node does not require peer health probing";
+const PEER_HEALTH_REACHABLE: &str = "peer marked reachable by internode health tracker";
+const PEER_HEALTH_UNREACHABLE: &str = "peer marked unreachable by internode health tracker";
 const CONTROL_RPC_SEPARATED: &str = "control RPC remains on the gRPC control plane";
 const DATA_STREAM_RPC_SEPARATED: &str = "remote disk data streams remain on the internode data transport";
 
@@ -341,12 +344,24 @@ pub fn peer_health_snapshot_from_membership(membership: &ClusterMembershipSnapsh
             .map(|node| ClusterPeerHealth {
                 node_id: node.node_id.clone(),
                 is_local: node.is_local,
-                status: CapabilityStatus::unknown().with_reason(PEER_HEALTH_NOT_REPORTED),
+                status: peer_health_status_for_node(node),
             })
             .collect(),
     }
 }
 
+fn peer_health_status_for_node(node: &ClusterNodeMembership) -> CapabilityStatus {
+    if node.is_local {
+        return CapabilityStatus::supported().with_reason(PEER_HEALTH_LOCAL_NODE);
+    }
+
+    match rustfs_io_metrics::internode_metrics::cluster_peer_observed_online_status(&node.grid_host) {
+        Some(true) => CapabilityStatus::supported().with_reason(PEER_HEALTH_REACHABLE),
+        Some(false) => CapabilityStatus::unknown().with_reason(PEER_HEALTH_UNREACHABLE),
+        None => CapabilityStatus::disabled().with_reason(PEER_HEALTH_NOT_REPORTED),
+    }
+}
+
 pub fn rpc_boundary_snapshot() -> ClusterRpcBoundarySnapshot {
     ClusterRpcBoundarySnapshot {
         control_channels: ["metadata", "lock", "health", "administrative"]
@@ -498,6 +513,7 @@ mod tests {
     use std::collections::BTreeSet;
 
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
+    use crate::storage_api_contracts::topology::CapabilityState;
 
     #[test]
     fn topology_snapshot_maps_endpoint_sets_without_local_paths() {
@@ -588,16 +604,55 @@ mod tests {
     }
 
     #[test]
-    fn peer_health_snapshot_reports_static_unknown_status() {
-        let membership = membership_snapshot_from_endpoint_pools(&sample_url_endpoint_pools());
+    fn peer_health_snapshot_reports_observed_peer_status() {
+        let online_host = "http://cluster-control-plane-peer-online-test:9000";
+        let offline_host = "http://cluster-control-plane-peer-offline-test:9000";
+        rustfs_io_metrics::internode_metrics::record_peer_reachable(online_host);
+        rustfs_io_metrics::internode_metrics::record_peer_unreachable(offline_host, 1);
+        let membership = ClusterMembershipSnapshot {
+            nodes: vec![
+                ClusterNodeMembership {
+                    node_id: LOCAL_NODE_ID.to_string(),
+                    grid_host: String::new(),
+                    is_local: true,
+                    pools: vec![0],
+                },
+                ClusterNodeMembership {
+                    node_id: "cluster-control-plane-peer-online-test:9000".to_string(),
+                    grid_host: online_host.to_string(),
+                    is_local: false,
+                    pools: vec![0],
+                },
+                ClusterNodeMembership {
+                    node_id: "cluster-control-plane-peer-offline-test:9000".to_string(),
+                    grid_host: offline_host.to_string(),
+                    is_local: false,
+                    pools: vec![0],
+                },
+                ClusterNodeMembership {
+                    node_id: "cluster-control-plane-peer-unknown-test:9000".to_string(),
+                    grid_host: "http://cluster-control-plane-peer-unknown-test:9000".to_string(),
+                    is_local: false,
+                    pools: vec![0],
+                },
+            ],
+            drives: Vec::new(),
+        };
         let snapshot = peer_health_snapshot_from_membership(&membership);
 
-        assert_eq!(snapshot.peers.len(), 2);
-        assert_eq!(snapshot.peers[0].node_id, "node1.example:9000");
+        assert_eq!(snapshot.peers.len(), 4);
+        assert_eq!(snapshot.peers[0].node_id, LOCAL_NODE_ID);
         assert!(snapshot.peers[0].is_local);
-        assert_eq!(snapshot.peers[0].status.reason.as_deref(), Some(PEER_HEALTH_NOT_REPORTED));
-        assert_eq!(snapshot.peers[1].node_id, "node2.example:9000");
+        assert!(snapshot.peers[0].status.state.is_supported());
+        assert_eq!(snapshot.peers[0].status.reason.as_deref(), Some(PEER_HEALTH_LOCAL_NODE));
+        assert_eq!(snapshot.peers[1].node_id, "cluster-control-plane-peer-online-test:9000");
         assert!(!snapshot.peers[1].is_local);
+        assert!(snapshot.peers[1].status.state.is_supported());
+        assert_eq!(snapshot.peers[1].status.reason.as_deref(), Some(PEER_HEALTH_REACHABLE));
+        assert_eq!(snapshot.peers[2].status.reason.as_deref(), Some(PEER_HEALTH_UNREACHABLE));
+        assert!(!snapshot.peers[2].status.state.is_supported());
+        assert_eq!(snapshot.peers[3].status.reason.as_deref(), Some(PEER_HEALTH_NOT_REPORTED));
+        assert_eq!(snapshot.peers[3].status.state, CapabilityState::Disabled);
     }
 
     #[test]

crates/ecstore/src/cluster/rpc/client.rs

@@ -12,12 +12,17 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::cluster::rpc::{TONIC_RPC_PREFIX, gen_signature_headers};
+use crate::cluster::rpc::http_auth::RPC_CONTENT_SHA256_HEADER;
+use crate::cluster::rpc::{gen_tonic_signature_headers, normalize_tonic_rpc_audience};
 use crate::disk::error::{DiskError, Error as DiskErrorType};
 use crate::runtime::sources as runtime_sources;
-use http::Method;
+use http::Uri;
 use rustfs_protos::{
-    ChannelClass, create_new_channel, get_channel_for_class, proto_gen::node_service::node_service_client::NodeServiceClient,
+    ChannelClass, create_new_channel, get_channel_for_class,
+    proto_gen::node_service::{
+        heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
+        tier_mutation_control_service_client::TierMutationControlServiceClient,
+    },
 };
 use std::{error::Error, io::ErrorKind};
 use tonic::{service::interceptor::InterceptedService, transport::Channel};
@@ -36,6 +41,36 @@ pub async fn node_service_time_out_client(
     node_service_time_out_client_for_class(addr, interceptor, ChannelClass::Control).await
 }
 
+pub async fn heal_control_time_out_client(
+    addr: &str,
+    interceptor: TonicInterceptor,
+) -> Result<HealControlServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
+    let interceptor = interceptor.with_rpc_audience(addr)?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
+    let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
+    Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
+        .max_decoding_message_size(max_message_size)
+        .max_encoding_message_size(max_message_size))
+}
+
+pub async fn tier_mutation_control_time_out_client(
+    addr: &str,
+    interceptor: TonicInterceptor,
+) -> Result<TierMutationControlServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
+    let interceptor = interceptor.with_rpc_audience(addr)?;
+    let channel = match runtime_sources::cached_node_channel(addr).await {
+        Some(channel) => channel,
+        None => create_new_channel(addr).await?,
+    };
+    let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
+    Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
+        .max_decoding_message_size(max_message_size)
+        .max_encoding_message_size(max_message_size))
+}
+
 /// Build a `NodeServiceClient` bound to the [`ChannelClass`]-appropriate channel for `addr`.
 ///
 /// Bulk `bytes`-carrying RPCs (ReadAll/WriteAll/ReadMultiple/BatchReadVersion) pass
@@ -47,6 +82,7 @@ pub async fn node_service_time_out_client_for_class(
     interceptor: TonicInterceptor,
     class: ChannelClass,
 ) -> Result<NodeServiceClient<InterceptedService<Channel, TonicInterceptor>>, Box<dyn Error>> {
+    let interceptor = interceptor.with_rpc_audience(addr)?;
     let channel = match class {
         ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
             Some(channel) => {
@@ -111,11 +147,25 @@ pub(crate) fn is_network_like_disk_error(err: &DiskErrorType) -> bool {
     }
 }
 
-pub struct TonicSignatureInterceptor;
+pub struct TonicSignatureInterceptor {
+    audience: Option<String>,
+}
 
 impl tonic::service::Interceptor for TonicSignatureInterceptor {
     fn call(&mut self, mut req: tonic::Request<()>) -> Result<tonic::Request<()>, tonic::Status> {
-        let headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET)
+        let method = req
+            .extensions()
+            .get::<tonic::GrpcMethod<'_>>()
+            .ok_or_else(|| tonic::Status::unauthenticated("Missing gRPC method metadata"))?;
+        let audience = self
+            .audience
+            .as_deref()
+            .ok_or_else(|| tonic::Status::unauthenticated("Missing gRPC audience"))?;
+        let content_sha256 = req
+            .metadata()
+            .get(RPC_CONTENT_SHA256_HEADER)
+            .and_then(|value| value.to_str().ok());
+        let headers = gen_tonic_signature_headers(audience, method.service(), method.method(), content_sha256)
             .map_err(|_| tonic::Status::unauthenticated("No valid auth token"))?;
         req.metadata_mut().as_mut().extend(headers);
         inject_trace_context_into_metadata(req.metadata_mut());
@@ -125,7 +175,7 @@ impl tonic::service::Interceptor for TonicSignatureInterceptor {
 }
 
 pub fn gen_tonic_signature_interceptor() -> TonicSignatureInterceptor {
-    TonicSignatureInterceptor
+    TonicSignatureInterceptor { audience: None }
 }
 
 pub struct NoOpInterceptor;
@@ -141,6 +191,22 @@ pub enum TonicInterceptor {
     NoOp(NoOpInterceptor),
 }
 
+impl TonicInterceptor {
+    fn with_rpc_audience(mut self, addr: &str) -> std::io::Result<Self> {
+        if let Self::Signature(interceptor) = &mut self {
+            let uri = addr
+                .parse::<Uri>()
+                .map_err(|_| std::io::Error::other("Invalid gRPC peer URI"))?;
+            let audience = uri
+                .authority()
+                .map(|authority| normalize_tonic_rpc_audience(authority.as_str()))
+                .ok_or_else(|| std::io::Error::other("Missing gRPC peer authority"))?;
+            interceptor.audience = Some(audience?);
+        }
+        Ok(self)
+    }
+}
+
 impl tonic::service::Interceptor for TonicInterceptor {
     fn call(&mut self, req: tonic::Request<()>) -> Result<tonic::Request<()>, tonic::Status> {
         match self {
@@ -165,6 +231,20 @@ mod tests {
         runtime_sources::ensure_test_rpc_secret();
     }
 
+    fn test_request() -> tonic::Request<()> {
+        let mut request = tonic::Request::new(());
+        request
+            .extensions_mut()
+            .insert(tonic::GrpcMethod::new("node_service.NodeService", "Ping"));
+        request
+    }
+
+    fn test_interceptor() -> TonicSignatureInterceptor {
+        TonicSignatureInterceptor {
+            audience: Some("node-a:9000".to_string()),
+        }
+    }
+
     fn with_trace_parent<F>(trace_id_hex: &str, f: F)
     where
         F: FnOnce(),
@@ -192,20 +272,55 @@ mod tests {
     #[test]
     fn test_signature_interceptor_keeps_auth_headers() {
         ensure_test_rpc_secret();
-        let mut interceptor = TonicSignatureInterceptor;
-        let req = tonic::Request::new(());
+        let mut interceptor = test_interceptor();
+        let req = test_request();
 
         let req = interceptor.call(req).expect("interceptor call should succeed");
 
         assert!(req.metadata().contains_key("x-rustfs-signature"));
         assert!(req.metadata().contains_key("x-rustfs-timestamp"));
+        assert!(req.metadata().contains_key("x-rustfs-rpc-signature-v2"));
+        assert!(req.metadata().contains_key("x-rustfs-rpc-nonce"));
+        assert!(
+            crate::cluster::rpc::verify_tonic_rpc_signature(
+                "node-a:9000",
+                "/node_service.NodeService/Ping",
+                req.metadata().as_ref(),
+            )
+            .is_ok(),
+            "interceptor signature should bind the configured peer audience and generated method"
+        );
+    }
+
+    #[test]
+    fn test_signature_interceptor_binds_audience_from_peer_uri() {
+        let interceptor = TonicInterceptor::Signature(gen_tonic_signature_interceptor())
+            .with_rpc_audience("http://node-a:9000")
+            .expect("peer URI should provide an audience");
+        let TonicInterceptor::Signature(interceptor) = interceptor else {
+            panic!("signature interceptor variant should be preserved");
+        };
+
+        assert_eq!(interceptor.audience.as_deref(), Some("node-a:9000"));
+    }
+
+    #[test]
+    fn test_signature_interceptor_requires_generated_method_metadata() {
+        ensure_test_rpc_secret();
+        let mut interceptor = test_interceptor();
+        let error = interceptor
+            .call(tonic::Request::new(()))
+            .expect_err("requests without an exact generated method must fail closed");
+
+        assert_eq!(error.code(), tonic::Code::Unauthenticated);
+        assert_eq!(error.message(), "Missing gRPC method metadata");
     }
 
     #[test]
     fn test_signature_interceptor_may_inject_request_id() {
         ensure_test_rpc_secret();
-        let mut interceptor = TonicSignatureInterceptor;
-        let req = tonic::Request::new(());
+        let mut interceptor = test_interceptor();
+        let req = test_request();
 
         let span = tracing::info_span!("grpc-rpc-test-span");
         let _guard = span.enter();
@@ -219,8 +334,8 @@ mod tests {
     #[test]
     fn test_signature_interceptor_injects_traceparent_metadata() {
         ensure_test_rpc_secret();
-        let mut interceptor = TonicSignatureInterceptor;
-        let req = tonic::Request::new(());
+        let mut interceptor = test_interceptor();
+        let req = test_request();
 
         with_trace_parent("4bf92f3577b34da6a3ce929d0e0e4736", || {
             let req = interceptor.call(req).expect("interceptor call should succeed");

crates/ecstore/src/cluster/rpc/http_auth.rs

@@ -19,8 +19,9 @@
 //! GHSA-r5qv-rc46-hv8q (internode RPC authentication must fail closed, fixed in
 //! rustfs/rustfs#4402) is anchored by the `ghsa_r5qv_*` tests in the module
 //! below, plus the broader negative-signature suite. The advisory class is: a
-//! node must never accept an RPC whose auth is missing, malformed, replayed, or
-//! signed with the default/empty shared secret. See
+//! node must never accept an RPC whose auth is missing, malformed, or signed
+//! with the default/empty shared secret. Body-bound v2 requests additionally
+//! receive process-local replay protection. See
 //! `docs/testing/security-regressions.md` for the full advisory -> test map.
 //!
 //! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-r5qv-rc46-hv8q>
@@ -29,23 +30,90 @@ use crate::cluster::rpc::context_propagation::{inject_request_id_into_http_heade
 use base64::Engine as _;
 use base64::engine::general_purpose;
 use hmac::{Hmac, KeyInit, Mac};
+use http::uri::Authority;
 use http::{HeaderMap, HeaderValue, Method, Uri};
 #[cfg(test)]
 use rustfs_credentials::{DEFAULT_SECRET_KEY, RPC_SECRET_REQUIRED_MESSAGE};
 use rustfs_credentials::{RPC_SECRET_REQUIRED_OPERATOR_MESSAGE, try_get_rpc_token};
+use sha2::Digest as _;
 use sha2::Sha256;
-use std::sync::Once;
+use std::collections::{HashSet, VecDeque};
+use std::sync::{LazyLock, Mutex, Once};
+use std::time::{Duration, Instant};
 use time::OffsetDateTime;
 use tracing::error;
+use uuid::Uuid;
 
 type HmacSha256 = Hmac<Sha256>;
 
 const SIGNATURE_HEADER: &str = "x-rustfs-signature";
 const TIMESTAMP_HEADER: &str = "x-rustfs-timestamp";
+const RPC_AUTH_VERSION_HEADER: &str = "x-rustfs-rpc-auth-version";
+const RPC_SIGNATURE_V2_HEADER: &str = "x-rustfs-rpc-signature-v2";
+const RPC_NONCE_HEADER: &str = "x-rustfs-rpc-nonce";
+pub(crate) const RPC_CONTENT_SHA256_HEADER: &str = "x-rustfs-content-sha256";
+const RPC_AUTH_VERSION_V2: &str = "2";
+const RPC_RESPONSE_PROOF_DOMAIN: &[u8] = b"rustfs-rpc-response-proof-v1\0";
+const UNSIGNED_PAYLOAD: &str = "UNSIGNED-PAYLOAD";
+const UNSIGNED_PAYLOAD_NONCE: &str = "unsigned";
 const SIGNATURE_VALID_DURATION: i64 = 300; // 5 minutes
+const REPLAY_CACHE_RETENTION: Duration = Duration::from_secs(601);
+const MAX_REPLAY_PROTECTED_NONCES: usize = 65_536;
 pub const TONIC_RPC_PREFIX: &str = "/node_service.NodeService";
 static RPC_SECRET_RESOLUTION_LOG_ONCE: Once = Once::new();
 
+#[derive(Default)]
+struct RpcNonceCache {
+    nonces: HashSet<Uuid>,
+    expirations: VecDeque<(Instant, i64, Uuid)>,
+    max_wall_time: i64,
+}
+
+impl RpcNonceCache {
+    fn remove_expired(&mut self, now: Instant, wall_time: i64) {
+        while matches!(
+            self.expirations.front(),
+            Some((expires_at, valid_until, _)) if *expires_at < now && *valid_until < wall_time
+        ) {
+            let Some((_, _, nonce)) = self.expirations.pop_front() else {
+                break;
+            };
+            self.nonces.remove(&nonce);
+        }
+    }
+
+    fn check_and_record(
+        &mut self,
+        nonce: Uuid,
+        signed_at: i64,
+        now: Instant,
+        wall_time: i64,
+        expires_at: Instant,
+        capacity: usize,
+    ) -> std::io::Result<()> {
+        self.max_wall_time = self.max_wall_time.max(wall_time);
+        if self.max_wall_time.saturating_sub(signed_at) > SIGNATURE_VALID_DURATION {
+            return Err(std::io::Error::other("RPC request timestamp expired after clock regression"));
+        }
+        self.remove_expired(now, self.max_wall_time);
+        if self.nonces.contains(&nonce) {
+            return Err(std::io::Error::other("RPC request replay detected"));
+        }
+        if self.nonces.len() >= capacity {
+            return Err(std::io::Error::other("RPC replay cache capacity exceeded"));
+        }
+        self.nonces.insert(nonce);
+        self.expirations
+            .push_back((expires_at, signed_at.saturating_add(SIGNATURE_VALID_DURATION), nonce));
+        Ok(())
+    }
+}
+
+// This cache is a process-local wire-replay defense only. Mutation handlers
+// still need a stable operation ID and coordinator-owned idempotency across
+// retries, node failover, and restart.
+static LOCAL_RPC_NONCE_CACHE: LazyLock<Mutex<RpcNonceCache>> = LazyLock::new(|| Mutex::new(RpcNonceCache::default()));
+
 /// Get the shared secret for HMAC signing
 #[cfg(test)]
 fn resolve_shared_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> std::io::Result<String> {
@@ -71,6 +139,30 @@ fn get_shared_secret() -> std::io::Result<String> {
     })
 }
 
+fn rpc_response_proof_mac(canonical_body: &[u8]) -> std::io::Result<HmacSha256> {
+    let secret = get_shared_secret()?;
+    let mut mac =
+        <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
+    mac.update(RPC_RESPONSE_PROOF_DOMAIN);
+    mac.update(
+        &u64::try_from(canonical_body.len())
+            .map_err(|_| std::io::Error::other("RPC response proof length cannot be represented"))?
+            .to_be_bytes(),
+    );
+    mac.update(canonical_body);
+    Ok(mac)
+}
+
+pub fn sign_tonic_rpc_response_proof(canonical_body: &[u8]) -> std::io::Result<Vec<u8>> {
+    Ok(rpc_response_proof_mac(canonical_body)?.finalize().into_bytes().to_vec())
+}
+
+pub fn verify_tonic_rpc_response_proof(canonical_body: &[u8], proof: &[u8]) -> std::io::Result<()> {
+    rpc_response_proof_mac(canonical_body)?
+        .verify_slice(proof)
+        .map_err(|_| std::io::Error::other("Invalid RPC response proof"))
+}
+
 /// Build the canonical payload covered by the RPC HMAC.
 fn signature_payload(url: &str, method: &Method, timestamp: i64) -> String {
     let uri: Uri = url.parse().expect("Invalid URL");
@@ -109,6 +201,96 @@ fn verify_signature(secret: &str, url: &str, method: &Method, timestamp: i64, si
     mac.verify_slice(&signature).is_ok()
 }
 
+#[derive(Clone, Copy)]
+struct SignatureV2Scope<'a> {
+    audience: &'a str,
+    service: &'a str,
+    rpc_method: &'a str,
+    timestamp: &'a str,
+    nonce: &'a str,
+    content_sha256: &'a str,
+}
+
+fn update_signature_v2(mac: &mut HmacSha256, scope: SignatureV2Scope<'_>) {
+    for part in [
+        b"rustfs-rpc-auth-v2|".as_slice(),
+        scope.audience.as_bytes(),
+        b"|/",
+        scope.service.as_bytes(),
+        b"/",
+        scope.rpc_method.as_bytes(),
+        b"|POST|",
+        scope.timestamp.as_bytes(),
+        b"|",
+        scope.nonce.as_bytes(),
+        b"|",
+        scope.content_sha256.as_bytes(),
+    ] {
+        mac.update(part);
+    }
+}
+
+fn generate_signature_v2(secret: &str, scope: SignatureV2Scope<'_>) -> std::io::Result<String> {
+    let mut mac =
+        <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
+    update_signature_v2(&mut mac, scope);
+    Ok(general_purpose::STANDARD.encode(mac.finalize().into_bytes()))
+}
+
+fn verify_signature_v2(secret: &str, scope: SignatureV2Scope<'_>, signature: &str) -> bool {
+    let Ok(signature) = general_purpose::STANDARD.decode(signature) else {
+        return false;
+    };
+    let Ok(mut mac) = <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()) else {
+        return false;
+    };
+    update_signature_v2(&mut mac, scope);
+    mac.verify_slice(&signature).is_ok()
+}
+
+fn valid_content_sha256(value: &str) -> bool {
+    value == UNSIGNED_PAYLOAD
+        || (value.len() == 64
+            && value
+                .bytes()
+                .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)))
+}
+
+fn header_value(value: &str, name: &str) -> std::io::Result<HeaderValue> {
+    HeaderValue::from_str(value).map_err(|_| std::io::Error::other(format!("Invalid {name} header value")))
+}
+
+pub fn normalize_tonic_rpc_audience(value: &str) -> std::io::Result<String> {
+    let authority = value
+        .parse::<Authority>()
+        .map_err(|_| std::io::Error::other("Invalid gRPC peer authority"))?;
+    Ok(authority.as_str().to_ascii_lowercase())
+}
+
+fn check_timestamp(timestamp: i64) -> std::io::Result<()> {
+    let current_time = OffsetDateTime::now_utc().unix_timestamp();
+    if current_time.saturating_sub(timestamp) > SIGNATURE_VALID_DURATION
+        || timestamp.saturating_sub(current_time) > SIGNATURE_VALID_DURATION
+    {
+        return Err(std::io::Error::other("Request timestamp expired"));
+    }
+    Ok(())
+}
+
+fn check_and_record_nonce(nonce: Uuid, signed_at: i64) -> std::io::Result<()> {
+    let wall_time = OffsetDateTime::now_utc().unix_timestamp();
+    let mut cache = LOCAL_RPC_NONCE_CACHE
+        .lock()
+        .map_err(|_| std::io::Error::other("RPC replay cache unavailable"))?;
+    // Take the monotonic timestamp after acquiring the lock so expiration
+    // entries remain ordered by the same serialization point as insertion.
+    let now = Instant::now();
+    let expires_at = now
+        .checked_add(REPLAY_CACHE_RETENTION)
+        .ok_or_else(|| std::io::Error::other("RPC replay expiry overflow"))?;
+    cache.check_and_record(nonce, signed_at, now, wall_time, expires_at, MAX_REPLAY_PROTECTED_NONCES)
+}
+
 /// Build headers with authentication signature
 pub fn build_auth_headers(url: &str, method: &Method, headers: &mut HeaderMap) -> std::io::Result<()> {
     let auth_headers = gen_signature_headers(url, method)?;
@@ -135,6 +317,184 @@ pub fn gen_signature_headers(url: &str, method: &Method) -> std::io::Result<Head
     Ok(headers)
 }
 
+/// Generate rolling-upgrade-safe gRPC auth metadata.
+///
+/// The legacy signature remains present for old servers. New servers prefer the
+/// v2 signature and bind it to the destination authority and exact generated
+/// gRPC method. A versioned canonical mutation payload can opt into the
+/// additional body-digest capability.
+pub fn gen_tonic_signature_headers(
+    audience: &str,
+    service: &str,
+    rpc_method: &str,
+    content_sha256: Option<&str>,
+) -> std::io::Result<HeaderMap> {
+    if audience.is_empty() || service.is_empty() || rpc_method.is_empty() || service.contains('/') || rpc_method.contains('/') {
+        return Err(std::io::Error::other("Invalid RPC v2 signing scope"));
+    }
+    let content_sha256 = content_sha256.unwrap_or(UNSIGNED_PAYLOAD);
+    if !valid_content_sha256(content_sha256) {
+        return Err(std::io::Error::other("Invalid RPC content SHA-256"));
+    }
+
+    let secret = get_shared_secret()?;
+    let timestamp = OffsetDateTime::now_utc().unix_timestamp();
+    let timestamp_header = timestamp.to_string();
+    let body_nonce = (content_sha256 != UNSIGNED_PAYLOAD).then(|| Uuid::new_v4().to_string());
+    let nonce = body_nonce.as_deref().unwrap_or(UNSIGNED_PAYLOAD_NONCE);
+    let legacy_signature = generate_signature(&secret, TONIC_RPC_PREFIX, &Method::GET, timestamp);
+    let signature_v2 = generate_signature_v2(
+        &secret,
+        SignatureV2Scope {
+            audience,
+            service,
+            rpc_method,
+            timestamp: &timestamp_header,
+            nonce,
+            content_sha256,
+        },
+    )?;
+
+    let mut headers = HeaderMap::new();
+    headers.insert(SIGNATURE_HEADER, header_value(&legacy_signature, SIGNATURE_HEADER)?);
+    headers.insert(TIMESTAMP_HEADER, header_value(&timestamp_header, TIMESTAMP_HEADER)?);
+    headers.insert(RPC_AUTH_VERSION_HEADER, HeaderValue::from_static(RPC_AUTH_VERSION_V2));
+    headers.insert(RPC_SIGNATURE_V2_HEADER, header_value(&signature_v2, RPC_SIGNATURE_V2_HEADER)?);
+    headers.insert(RPC_NONCE_HEADER, header_value(nonce, RPC_NONCE_HEADER)?);
+    headers.insert(RPC_CONTENT_SHA256_HEADER, header_value(content_sha256, RPC_CONTENT_SHA256_HEADER)?);
+    Ok(headers)
+}
+
+/// Bind a mutation to a versioned, deterministic canonical payload.
+///
+/// Do not pass a protobuf re-encoding here: unknown fields and map ordering are
+/// not a stable mixed-version contract.
+pub fn set_tonic_canonical_body_digest<T>(request: &mut tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
+    let digest = hex_simd::encode_to_string(Sha256::digest(canonical_body), hex_simd::AsciiCase::Lower);
+    request
+        .metadata_mut()
+        .as_mut()
+        .insert(RPC_CONTENT_SHA256_HEADER, header_value(&digest, RPC_CONTENT_SHA256_HEADER)?);
+    Ok(())
+}
+
+pub fn verify_tonic_canonical_body_digest<T>(request: &tonic::Request<T>, canonical_body: &[u8]) -> std::io::Result<()> {
+    let version = request
+        .metadata()
+        .get(RPC_AUTH_VERSION_HEADER)
+        .and_then(|value| value.to_str().ok());
+    if version != Some(RPC_AUTH_VERSION_V2) {
+        return Err(std::io::Error::other("RPC mutation requires v2 authentication"));
+    }
+    let expected = request
+        .metadata()
+        .get(RPC_CONTENT_SHA256_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC content SHA-256"))?;
+    if expected == UNSIGNED_PAYLOAD || !valid_content_sha256(expected) {
+        return Err(std::io::Error::other("RPC body is not bound to the signature"));
+    }
+    let actual = hex_simd::encode_to_string(Sha256::digest(canonical_body), hex_simd::AsciiCase::Lower);
+    if actual != expected {
+        return Err(std::io::Error::other("RPC content SHA-256 mismatch"));
+    }
+    Ok(())
+}
+
+fn has_v2_auth_headers(headers: &HeaderMap) -> bool {
+    [
+        RPC_AUTH_VERSION_HEADER,
+        RPC_SIGNATURE_V2_HEADER,
+        RPC_NONCE_HEADER,
+        RPC_CONTENT_SHA256_HEADER,
+    ]
+    .iter()
+    .any(|name| headers.contains_key(*name))
+}
+
+/// Verify gRPC authentication, preferring v2 without downgrade on malformed v2 metadata.
+pub fn verify_tonic_rpc_signature(audience: &str, path: &str, headers: &HeaderMap) -> std::io::Result<()> {
+    if !has_v2_auth_headers(headers) {
+        // RUSTFS_COMPAT_TODO(heal-rpc-auth-v2): accept old peers during rolling upgrades. Remove after the minimum
+        // supported RustFS peer version sends v2 authentication on every internode gRPC request.
+        return verify_rpc_signature(TONIC_RPC_PREFIX, &Method::GET, headers);
+    }
+
+    let path = path
+        .strip_prefix('/')
+        .ok_or_else(|| std::io::Error::other("Invalid RPC request path"))?;
+    let (service, rpc_method) = path
+        .split_once('/')
+        .filter(|(service, rpc_method)| !service.is_empty() && !rpc_method.is_empty() && !rpc_method.contains('/'))
+        .ok_or_else(|| std::io::Error::other("Invalid RPC request path"))?;
+    if audience.is_empty() {
+        return Err(std::io::Error::other("Missing RPC audience"));
+    }
+
+    let version = headers
+        .get(RPC_AUTH_VERSION_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC auth version"))?;
+    if version != RPC_AUTH_VERSION_V2 {
+        return Err(std::io::Error::other("Unsupported RPC auth version"));
+    }
+    let signature = headers
+        .get(RPC_SIGNATURE_V2_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC v2 signature"))?;
+    let timestamp_header = headers
+        .get(TIMESTAMP_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing timestamp header"))?;
+    let timestamp = timestamp_header
+        .parse::<i64>()
+        .map_err(|_| std::io::Error::other("Invalid timestamp format"))?;
+    check_timestamp(timestamp)?;
+    let nonce = headers
+        .get(RPC_NONCE_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC nonce"))?;
+    let content_sha256 = headers
+        .get(RPC_CONTENT_SHA256_HEADER)
+        .and_then(|value| value.to_str().ok())
+        .ok_or_else(|| std::io::Error::other("Missing RPC content SHA-256"))?;
+    if !valid_content_sha256(content_sha256) {
+        return Err(std::io::Error::other("Invalid RPC content SHA-256"));
+    }
+    let parsed_nonce = if content_sha256 == UNSIGNED_PAYLOAD {
+        if nonce != UNSIGNED_PAYLOAD_NONCE {
+            return Err(std::io::Error::other("Invalid unsigned RPC nonce"));
+        }
+        None
+    } else {
+        let parsed_nonce = Uuid::parse_str(nonce).map_err(|_| std::io::Error::other("Invalid RPC nonce"))?;
+        if parsed_nonce.is_nil() {
+            return Err(std::io::Error::other("Invalid RPC nonce"));
+        }
+        Some(parsed_nonce)
+    };
+
+    let secret = get_shared_secret()?;
+    if !verify_signature_v2(
+        &secret,
+        SignatureV2Scope {
+            audience,
+            service,
+            rpc_method,
+            timestamp: timestamp_header,
+            nonce,
+            content_sha256,
+        },
+        signature,
+    ) {
+        return Err(std::io::Error::other("Invalid RPC v2 signature"));
+    }
+    if let Some(nonce) = parsed_nonce {
+        check_and_record_nonce(nonce, timestamp)?;
+    }
+    Ok(())
+}
+
 /// Verify the request signature for RPC requests
 pub fn verify_rpc_signature(url: &str, method: &Method, headers: &HeaderMap) -> std::io::Result<()> {
     // Get signature from header
@@ -153,14 +513,7 @@ pub fn verify_rpc_signature(url: &str, method: &Method, headers: &HeaderMap) ->
         .parse()
         .map_err(|_| std::io::Error::other("Invalid timestamp format"))?;
 
-    // Check timestamp validity (prevent replay attacks)
-    let current_time = OffsetDateTime::now_utc().unix_timestamp();
-
-    if current_time.saturating_sub(timestamp) > SIGNATURE_VALID_DURATION
-        || timestamp.saturating_sub(current_time) > SIGNATURE_VALID_DURATION
-    {
-        return Err(std::io::Error::other("Request timestamp expired"));
-    }
+    check_timestamp(timestamp)?;
 
     // Verify signature with constant-time HMAC comparison.
     let secret = get_shared_secret()?;
@@ -707,4 +1060,203 @@ mod tests {
             assert!(result.is_ok(), "Round-trip test failed for {method} {url}");
         }
     }
+
+    #[test]
+    fn tonic_v2_signature_is_bound_to_exact_method() {
+        ensure_test_rpc_secret();
+        let headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+
+        assert!(verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers).is_ok());
+        let error = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/SignalService", &headers)
+            .expect_err("signature replayed to a different method must fail");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn tonic_v2_signature_is_bound_to_exact_service() {
+        ensure_test_rpc_secret();
+        let headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+
+        let error = verify_tonic_rpc_signature("node-a:9000", "/other.NodeService/Ping", &headers)
+            .expect_err("signature replayed to a different service must fail");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn tonic_v2_signature_is_bound_to_destination_audience() {
+        ensure_test_rpc_secret();
+        let headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+
+        let error = verify_tonic_rpc_signature("node-b:9000", "/node_service.NodeService/Ping", &headers)
+            .expect_err("signature replayed to a different node must fail");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn malformed_v2_auth_does_not_downgrade_to_valid_legacy_signature() {
+        ensure_test_rpc_secret();
+        let mut headers = gen_tonic_signature_headers("node-a:9000", "node_service.NodeService", "Ping", None)
+            .expect("tonic auth headers should build");
+        headers.insert(RPC_SIGNATURE_V2_HEADER, HeaderValue::from_static("invalid"));
+
+        assert!(
+            verify_rpc_signature(TONIC_RPC_PREFIX, &Method::GET, &headers).is_ok(),
+            "the compatibility signature should remain valid for old servers"
+        );
+        let error = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers)
+            .expect_err("new servers must not downgrade malformed v2 auth");
+        assert_eq!(error.to_string(), "Invalid RPC v2 signature");
+    }
+
+    #[test]
+    fn legacy_tonic_signature_remains_accepted_during_rolling_upgrade() {
+        ensure_test_rpc_secret();
+        let headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
+
+        assert!(verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers).is_ok());
+    }
+
+    #[test]
+    fn body_bound_tonic_request_rejects_replay_and_body_tampering() {
+        ensure_test_rpc_secret();
+        let body = b"heal-control-request";
+        let mut request = tonic::Request::new(());
+        set_tonic_canonical_body_digest(&mut request, body).expect("canonical body digest should be attached");
+        let content_sha256 = request
+            .metadata()
+            .get(RPC_CONTENT_SHA256_HEADER)
+            .and_then(|value| value.to_str().ok());
+        let headers =
+            gen_tonic_signature_headers("node-a:9000", "node_service.HealControlService", "HealControl", content_sha256)
+                .expect("body-bound auth headers should build");
+        request.metadata_mut().as_mut().extend(headers.clone());
+
+        assert!(verify_tonic_rpc_signature("node-a:9000", "/node_service.HealControlService/HealControl", &headers).is_ok());
+        let replay = verify_tonic_rpc_signature("node-a:9000", "/node_service.HealControlService/HealControl", &headers)
+            .expect_err("reusing a body-bound nonce must fail");
+        assert_eq!(replay.to_string(), "RPC request replay detected");
+        assert!(verify_tonic_canonical_body_digest(&request, body).is_ok());
+        let tampered = verify_tonic_canonical_body_digest(&request, b"different-body")
+            .expect_err("a different canonical request body must fail");
+        assert_eq!(tampered.to_string(), "RPC content SHA-256 mismatch");
+    }
+
+    #[test]
+    fn tier_mutation_rpc_contract_requires_method_bound_v2_body_digest() {
+        ensure_test_rpc_secret();
+        let mutation_id = uuid::uuid!("12345678-1234-5678-9abc-def012345678");
+        let body = rustfs_protos::canonical_tier_mutation_rpc_body(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            rustfs_protos::TierMutationRpcPhase::Prepare,
+            mutation_id,
+            b"canonical-tier-mutation-prepare",
+        )
+        .expect("small tier mutation body should encode");
+        let mut request = tonic::Request::new(());
+        set_tonic_canonical_body_digest(&mut request, &body).expect("canonical body digest should be attached");
+        let content_sha256 = request
+            .metadata()
+            .get(RPC_CONTENT_SHA256_HEADER)
+            .and_then(|value| value.to_str().ok());
+        let headers = gen_tonic_signature_headers(
+            "node-a:9000",
+            "node_service.TierMutationControlService",
+            "PrepareTierMutation",
+            content_sha256,
+        )
+        .expect("body-bound tier mutation auth headers should build");
+        request.metadata_mut().as_mut().extend(headers.clone());
+
+        assert!(
+            verify_tonic_rpc_signature("node-a:9000", "/node_service.TierMutationControlService/PrepareTierMutation", &headers)
+                .is_ok(),
+            "tier mutation RPC signature must bind destination, service, method, nonce, and body digest"
+        );
+        let method_replay =
+            verify_tonic_rpc_signature("node-a:9000", "/node_service.TierMutationControlService/CommitTierMutation", &headers)
+                .expect_err("prepare auth must not replay to commit");
+        assert_eq!(method_replay.to_string(), "Invalid RPC v2 signature");
+        let service_replay = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/PrepareTierMutation", &headers)
+            .expect_err("tier mutation auth must not replay to the legacy node service path");
+        assert_eq!(service_replay.to_string(), "Invalid RPC v2 signature");
+        let tampered_body = rustfs_protos::canonical_tier_mutation_rpc_body(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            rustfs_protos::TierMutationRpcPhase::Commit,
+            mutation_id,
+            b"canonical-tier-mutation-prepare",
+        )
+        .expect("small tier mutation body should encode");
+        let tampered =
+            verify_tonic_canonical_body_digest(&request, &tampered_body).expect_err("commit body must not match prepare digest");
+        assert_eq!(tampered.to_string(), "RPC content SHA-256 mismatch");
+    }
+
+    #[test]
+    fn partial_v2_metadata_fails_closed() {
+        ensure_test_rpc_secret();
+        let mut headers = gen_signature_headers(TONIC_RPC_PREFIX, &Method::GET).expect("legacy auth headers should build");
+        headers.insert(RPC_AUTH_VERSION_HEADER, HeaderValue::from_static(RPC_AUTH_VERSION_V2));
+
+        let error = verify_tonic_rpc_signature("node-a:9000", "/node_service.NodeService/Ping", &headers)
+            .expect_err("partial v2 metadata must not fall back to legacy auth");
+        assert_eq!(error.to_string(), "Missing RPC v2 signature");
+    }
+
+    #[test]
+    fn canonical_mutation_digest_rejects_legacy_only_auth() {
+        let mut request = tonic::Request::new(());
+        set_tonic_canonical_body_digest(&mut request, b"heal-control-v1\0start").expect("canonical digest should be attached");
+
+        let error = verify_tonic_canonical_body_digest(&request, b"heal-control-v1\0start")
+            .expect_err("mutation body verification must also require v2 auth");
+        assert_eq!(error.to_string(), "RPC mutation requires v2 authentication");
+    }
+
+    #[test]
+    fn nonce_cache_expires_by_monotonic_deadline_and_fails_closed_at_capacity() {
+        let now = Instant::now();
+        let expiry = now.checked_add(REPLAY_CACHE_RETENTION).expect("test expiry should fit");
+        let after_expiry = expiry.checked_add(Duration::from_secs(1)).expect("test expiry should fit");
+        let nonce_a = Uuid::new_v4();
+        let nonce_b = Uuid::new_v4();
+        let mut cache = RpcNonceCache::default();
+
+        cache
+            .check_and_record(nonce_a, 100, now, 100, expiry, 1)
+            .expect("first nonce should be recorded");
+        let capacity = cache
+            .check_and_record(nonce_b, 100, now, 100, expiry, 1)
+            .expect_err("a full replay cache must fail closed");
+        assert_eq!(capacity.to_string(), "RPC replay cache capacity exceeded");
+        cache
+            .check_and_record(nonce_b, 702, after_expiry, 702, after_expiry, 1)
+            .expect("expired nonce should release capacity");
+        assert!(!cache.nonces.contains(&nonce_a));
+        assert!(cache.nonces.contains(&nonce_b));
+    }
+
+    #[test]
+    fn nonce_cache_rejects_replay_after_wall_clock_regression() {
+        let now = Instant::now();
+        let expiry = now.checked_add(REPLAY_CACHE_RETENTION).expect("test expiry should fit");
+        let after_expiry = expiry.checked_add(Duration::from_secs(1)).expect("test expiry should fit");
+        let nonce = Uuid::new_v4();
+        let mut cache = RpcNonceCache::default();
+
+        cache
+            .check_and_record(nonce, 1_000, now, 1_000, expiry, 2)
+            .expect("first nonce should be recorded");
+        let replay = cache
+            .check_and_record(nonce, 1_000, after_expiry, 900, after_expiry, 2)
+            .expect_err("wall clock regression must not make an old signature reusable");
+        assert_eq!(replay.to_string(), "RPC request replay detected");
+
+        let stale = cache
+            .check_and_record(Uuid::new_v4(), 600, after_expiry, 900, after_expiry, 2)
+            .expect_err("the monotonic wall-clock high-water mark must fail closed");
+        assert_eq!(stale.to_string(), "RPC request timestamp expired after clock regression");
+    }
 }

crates/ecstore/src/cluster/rpc/mod.rs

@@ -28,12 +28,17 @@ pub(crate) use background_monitor::spawn_background_monitor;
 pub use client::{
     TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
 };
-pub use http_auth::{TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, verify_rpc_signature};
+pub use http_auth::{
+    TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, gen_tonic_signature_headers, normalize_tonic_rpc_audience,
+    set_tonic_canonical_body_digest, sign_tonic_rpc_response_proof, verify_rpc_signature, verify_tonic_canonical_body_digest,
+    verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
+};
 #[cfg(test)]
 pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
 pub use internode_data_transport::build_internode_data_transport_from_env;
 pub use peer_rest_client::{
     PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
+    ScannerPeerActivity,
 };
 pub(crate) use peer_s3_client::heal_bucket_local_on_disks;
 pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, S3PeerSys};

crates/ecstore/src/cluster/rpc/peer_rest_client.rs

@@ -12,7 +12,11 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::cluster::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
+use crate::cluster::rpc::client::{
+    TonicInterceptor, gen_tonic_signature_interceptor, heal_control_time_out_client, node_service_time_out_client,
+    tier_mutation_control_time_out_client,
+};
+use crate::cluster::rpc::{set_tonic_canonical_body_digest, verify_tonic_rpc_response_proof};
 use crate::error::{Error, Result};
 use crate::{
     disk::disk_store::{get_drive_active_check_interval, get_drive_active_check_timeout},
@@ -20,6 +24,7 @@ use crate::{
     runtime::sources as runtime_sources,
     services::metrics_realtime::{CollectMetricsOpts, MetricType},
 };
+use bytes::Bytes;
 use rmp_serde::{Deserializer, Serializer};
 use rustfs_madmin::{
     ServerProperties,
@@ -27,17 +32,19 @@ use rustfs_madmin::{
     metrics::RealtimeMetrics,
     net::NetInfo,
 };
-use rustfs_protos::evict_failed_connection;
 use rustfs_protos::proto_gen::node_service::{
-    CancelDecommissionRequest, ClearDecommissionRequest, DeleteBucketMetadataRequest, DeletePolicyRequest,
-    DeleteServiceAccountRequest, DeleteUserRequest, GetCpusRequest, GetLiveEventsRequest, GetMemInfoRequest, GetMetricsRequest,
-    GetNetInfoRequest, GetOsInfoRequest, GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest, GetSysConfigRequest,
-    GetSysErrorsRequest, LoadBucketMetadataRequest, LoadGroupRequest, LoadPolicyMappingRequest, LoadPolicyRequest,
-    LoadRebalanceMetaRequest, LoadServiceAccountRequest, LoadTransitionTierConfigRequest, LoadUserRequest,
-    LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest, ReloadSiteReplicationConfigRequest, ServerInfoRequest,
-    SignalServiceRequest, StartDecommissionRequest, StartProfilingRequest, StopRebalanceRequest,
-    node_service_client::NodeServiceClient,
+    BackgroundHealStatusRequest, CancelDecommissionRequest, ClearDecommissionRequest, DeleteBucketMetadataRequest,
+    DeletePolicyRequest, DeleteServiceAccountRequest, DeleteUserRequest, GetCpusRequest, GetLiveEventsRequest, GetMemInfoRequest,
+    GetMetricsRequest, GetNetInfoRequest, GetOsInfoRequest, GetPartitionsRequest, GetProcInfoRequest, GetSeLinuxInfoRequest,
+    GetSysConfigRequest, GetSysErrorsRequest, HealControlRequest, LoadBucketMetadataRequest, LoadGroupRequest,
+    LoadPolicyMappingRequest, LoadPolicyRequest, LoadRebalanceMetaRequest, LoadServiceAccountRequest,
+    LoadTransitionTierConfigRequest, LoadUserRequest, LocalStorageInfoRequest, Mss, ReloadPoolMetaRequest,
+    ReloadSiteReplicationConfigRequest, ScannerActivityRequest, ScannerActivityResponse, ServerInfoRequest, SignalServiceRequest,
+    StartDecommissionRequest, StartProfilingRequest, StopRebalanceRequest, TierMutationAbortRequest, TierMutationCommitRequest,
+    TierMutationControlResponse, TierMutationPeerState, TierMutationPrepareRequest, node_service_client::NodeServiceClient,
+    tier_mutation_control_service_client::TierMutationControlServiceClient,
 };
+use rustfs_protos::{TierMutationRpcPhase, evict_failed_connection};
 use rustfs_utils::XHost;
 use serde::{Deserialize, Serialize as _};
 use std::{
@@ -54,14 +61,53 @@ use tonic::Request;
 use tonic::service::interceptor::InterceptedService;
 use tonic::transport::Channel;
 use tracing::{debug, info, warn};
+use uuid::Uuid;
 
 pub const PEER_RESTSIGNAL: &str = "signal";
 pub const PEER_RESTSUB_SYS: &str = "sub-sys";
 pub const PEER_RESTDRY_RUN: &str = "dry-run";
 pub const SERVICE_SIGNAL_REFRESH_CONFIG: u64 = 1;
 pub const SERVICE_SIGNAL_RELOAD_DYNAMIC: u64 = 2;
+const BACKGROUND_HEAL_STATUS_MAX_MESSAGE_SIZE: usize = 64 * 1024;
+const HEAL_CONTROL_FINGERPRINT_MAX_SIZE: usize = 256;
+const HEAL_CONTROL_PAYLOAD_MAX_SIZE: usize = 64 * 1024;
 const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
 const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
+const SCANNER_ACTIVITY_MAX_MESSAGE_SIZE: usize = 1024;
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ScannerPeerActivity {
+    pub instance_id: String,
+    pub namespace_generation: u64,
+    pub maintenance_generation: u64,
+}
+
+fn decode_scanner_activity(response: ScannerActivityResponse) -> Result<ScannerPeerActivity> {
+    let instance_id = response.instance_id;
+    if instance_id.len() != 32
+        || !instance_id
+            .as_bytes()
+            .iter()
+            .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(byte))
+    {
+        return Err(Error::other("peer returned an invalid scanner activity instance ID"));
+    }
+    Ok(ScannerPeerActivity {
+        instance_id,
+        namespace_generation: response.namespace_generation,
+        maintenance_generation: response.maintenance_generation,
+    })
+}
+
+fn validate_heal_control_capability_proof(canonical_ack: &[u8], proof: &[u8]) -> Result<()> {
+    verify_tonic_rpc_response_proof(canonical_ack, proof)
+        .map_err(|_| Error::other("peer returned an invalid heal control capability proof"))
+}
+
+fn validate_heal_control_response_proof(canonical_response: &[u8], proof: &[u8]) -> Result<()> {
+    verify_tonic_rpc_response_proof(canonical_response, proof)
+        .map_err(|_| Error::other("peer returned an invalid heal control response proof"))
+}
 
 #[derive(Clone, Debug)]
 pub struct PeerLiveEventsBatch {
@@ -78,6 +124,82 @@ pub struct PeerRestClient {
     recovery_running: Arc<AtomicBool>,
 }
 
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum PeerTierMutationState {
+    Prepared,
+    Committed,
+    Aborted,
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct PeerTierMutationOutcome {
+    pub state: PeerTierMutationState,
+    pub applied: bool,
+}
+
+fn validate_tier_mutation_response_proof(
+    version: u32,
+    phase: TierMutationRpcPhase,
+    mutation_id: Uuid,
+    canonical_payload: &[u8],
+    response: &TierMutationControlResponse,
+) -> Result<()> {
+    let canonical_response =
+        rustfs_protos::canonical_tier_mutation_rpc_response_body(rustfs_protos::TierMutationRpcResponseProofInput {
+            version,
+            phase,
+            mutation_id,
+            canonical_payload,
+            success: response.success,
+            state: response.state,
+            applied: response.applied,
+            error_info: response.error_info.as_deref(),
+        })
+        .map_err(|_| Error::other("tier mutation response length cannot be represented"))?;
+    verify_tonic_rpc_response_proof(&canonical_response, &response.response_proof)
+        .map_err(|_| Error::other("peer returned an invalid tier mutation response proof"))
+}
+
+fn decode_tier_mutation_peer_state(state: i32) -> Result<PeerTierMutationState> {
+    match TierMutationPeerState::try_from(state).map_err(|_| Error::other("peer returned an invalid tier mutation state"))? {
+        TierMutationPeerState::Prepared => Ok(PeerTierMutationState::Prepared),
+        TierMutationPeerState::Committed => Ok(PeerTierMutationState::Committed),
+        TierMutationPeerState::Aborted => Ok(PeerTierMutationState::Aborted),
+        TierMutationPeerState::Unspecified => Err(Error::other("peer returned an unspecified tier mutation state")),
+    }
+}
+
+fn validate_tier_mutation_payload_len(phase: TierMutationRpcPhase, payload_len: usize) -> Result<()> {
+    let limit = match phase {
+        TierMutationRpcPhase::Prepare => rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE,
+        TierMutationRpcPhase::Commit => rustfs_protos::TIER_MUTATION_RPC_MAX_COMMIT_PAYLOAD_SIZE,
+        TierMutationRpcPhase::Abort => {
+            if payload_len == 0 {
+                return Ok(());
+            }
+            return Err(Error::other("tier mutation abort payload must be empty"));
+        }
+        _ => return Err(Error::other("tier mutation rpc phase is unsupported")),
+    };
+    if payload_len > limit {
+        return Err(Error::other("tier mutation payload exceeds size limit"));
+    }
+    Ok(())
+}
+
+fn tier_mutation_phase_label(phase: TierMutationRpcPhase) -> &'static str {
+    match phase {
+        TierMutationRpcPhase::Prepare => "prepare",
+        TierMutationRpcPhase::Commit => "commit",
+        TierMutationRpcPhase::Abort => "abort",
+        _ => "unknown",
+    }
+}
+
+fn tier_mutation_control_status_error(phase: TierMutationRpcPhase, status: tonic::Status) -> Error {
+    Error::other(format!("peer tier mutation {} RPC failed: {status}", tier_mutation_phase_label(phase)))
+}
+
 impl PeerRestClient {
     fn recovery_monitor_span(grid_host: &str) -> tracing::Span {
         tracing::info_span!(
@@ -141,6 +263,48 @@ impl PeerRestClient {
             })
     }
 
+    async fn get_heal_control_client(
+        &self,
+    ) -> Result<
+        rustfs_protos::proto_gen::node_service::heal_control_service_client::HealControlServiceClient<
+            InterceptedService<Channel, TonicInterceptor>,
+        >,
+    > {
+        if self.offline.load(Ordering::Acquire) {
+            self.mark_offline_and_spawn_recovery();
+            return Err(Error::other(format!("peer {} is temporarily offline", self.grid_host)));
+        }
+
+        heal_control_time_out_client(&self.grid_host, TonicInterceptor::Signature(gen_tonic_signature_interceptor()))
+            .await
+            .map_err(|err| {
+                let storage_err = Error::other(format!("can not get heal control client, err: {err}"));
+                if Self::is_network_like_error(&storage_err) {
+                    self.mark_offline_and_spawn_recovery();
+                }
+                storage_err
+            })
+    }
+
+    async fn get_tier_mutation_control_client(
+        &self,
+    ) -> Result<TierMutationControlServiceClient<InterceptedService<Channel, TonicInterceptor>>> {
+        if self.offline.load(Ordering::Acquire) {
+            self.mark_offline_and_spawn_recovery();
+            return Err(Error::other(format!("peer {} is temporarily offline", self.grid_host)));
+        }
+
+        tier_mutation_control_time_out_client(&self.grid_host, TonicInterceptor::Signature(gen_tonic_signature_interceptor()))
+            .await
+            .map_err(|err| {
+                let storage_err = Error::other(format!("can not get tier mutation control client, err: {err}"));
+                if Self::is_network_like_error(&storage_err) {
+                    self.mark_offline_and_spawn_recovery();
+                }
+                storage_err
+            })
+    }
+
     /// Evict the connection to this peer from the global cache.
     /// This should be called when communication with this peer fails.
     pub async fn evict_connection(&self) {
@@ -639,12 +803,208 @@ impl PeerRestClient {
         Err(Error::NotImplemented)
     }
 
-    pub async fn load_bucket_metadata(&self, bucket: &str) -> Result<()> {
+    pub async fn background_heal_status(&self) -> Result<Option<Vec<u8>>> {
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_client()
+                    .await?
+                    .max_decoding_message_size(BACKGROUND_HEAL_STATUS_MAX_MESSAGE_SIZE);
+                let response = match client
+                    .background_heal_status(Request::new(BackgroundHealStatusRequest::default()))
+                    .await
+                {
+                    Ok(response) => response.into_inner(),
+                    Err(status) if status.code() == tonic::Code::Unimplemented => {
+                        // RUSTFS_COMPAT_TODO(heal-status-rpc-v1): accept old peers without node heal snapshots. Remove after the minimum supported RustFS peer version implements BackgroundHealStatus.
+                        return Ok(None);
+                    }
+                    Err(status) => return Err(status.into()),
+                };
+                if !response.success {
+                    return Err(Error::other(
+                        response
+                            .error_info
+                            .unwrap_or_else(|| "peer background heal status failed without an error".to_string()),
+                    ));
+                }
+                Ok(Some(response.bg_heal_state.to_vec()))
+            }
+            .await,
+        )
+        .await
+    }
+
+    pub async fn prepare_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Result<PeerTierMutationOutcome> {
+        self.tier_mutation_control(TierMutationRpcPhase::Prepare, mutation_id, canonical_payload)
+            .await
+    }
+
+    pub async fn commit_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Result<PeerTierMutationOutcome> {
+        self.tier_mutation_control(TierMutationRpcPhase::Commit, mutation_id, canonical_payload)
+            .await
+    }
+
+    pub async fn abort_tier_mutation(&self, mutation_id: Uuid) -> Result<PeerTierMutationOutcome> {
+        self.tier_mutation_control(TierMutationRpcPhase::Abort, mutation_id, Bytes::new())
+            .await
+    }
+
+    async fn tier_mutation_control(
+        &self,
+        phase: TierMutationRpcPhase,
+        mutation_id: Uuid,
+        canonical_payload: Bytes,
+    ) -> Result<PeerTierMutationOutcome> {
+        validate_tier_mutation_payload_len(phase, canonical_payload.len())?;
+        let version = rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION;
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_tier_mutation_control_client()
+                    .await?
+                    .max_encoding_message_size(rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE)
+                    .max_decoding_message_size(rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE);
+                let canonical_body =
+                    rustfs_protos::canonical_tier_mutation_rpc_body(version, phase, mutation_id, canonical_payload.as_ref())
+                        .map_err(|_| Error::other("tier mutation request length cannot be represented"))?;
+                let mutation_id_text = mutation_id.to_string();
+                let response = match phase {
+                    TierMutationRpcPhase::Prepare => {
+                        let mut request = Request::new(TierMutationPrepareRequest {
+                            version,
+                            mutation_id: mutation_id_text.clone(),
+                            canonical_payload: canonical_payload.clone(),
+                        });
+                        set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                        client
+                            .prepare_tier_mutation(request)
+                            .await
+                            .map_err(|status| tier_mutation_control_status_error(phase, status))?
+                            .into_inner()
+                    }
+                    TierMutationRpcPhase::Commit => {
+                        let mut request = Request::new(TierMutationCommitRequest {
+                            version,
+                            mutation_id: mutation_id_text.clone(),
+                            canonical_payload: canonical_payload.clone(),
+                        });
+                        set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                        client
+                            .commit_tier_mutation(request)
+                            .await
+                            .map_err(|status| tier_mutation_control_status_error(phase, status))?
+                            .into_inner()
+                    }
+                    TierMutationRpcPhase::Abort => {
+                        let mut request = Request::new(TierMutationAbortRequest {
+                            version,
+                            mutation_id: mutation_id_text,
+                            canonical_payload: canonical_payload.clone(),
+                        });
+                        set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                        client
+                            .abort_tier_mutation(request)
+                            .await
+                            .map_err(|status| tier_mutation_control_status_error(phase, status))?
+                            .into_inner()
+                    }
+                    _ => return Err(Error::other("tier mutation rpc phase is unsupported")),
+                };
+                validate_tier_mutation_response_proof(version, phase, mutation_id, &canonical_payload, &response)?;
+                if !response.success {
+                    return Err(Error::other(
+                        response
+                            .error_info
+                            .unwrap_or_else(|| "peer tier mutation failed without an error".to_string()),
+                    ));
+                }
+                let state = decode_tier_mutation_peer_state(response.state)?;
+                Ok(PeerTierMutationOutcome {
+                    state,
+                    applied: response.applied,
+                })
+            }
+            .await,
+        )
+        .await
+    }
+
+    pub async fn heal_control(&self, version: u32, topology_fingerprint: String, command: Vec<u8>) -> Result<Vec<u8>> {
+        if topology_fingerprint.len() > HEAL_CONTROL_FINGERPRINT_MAX_SIZE {
+            return Err(Error::other("heal control topology fingerprint exceeds size limit"));
+        }
+        if command.len() > HEAL_CONTROL_PAYLOAD_MAX_SIZE {
+            return Err(Error::other("heal control command exceeds size limit"));
+        }
+        let capability_probe = rustfs_protos::is_heal_control_capability_probe(&command);
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_heal_control_client()
+                    .await?
+                    .max_encoding_message_size(rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE)
+                    .max_decoding_message_size(rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE);
+                let canonical_body = rustfs_protos::canonical_heal_control_request_body(version, &topology_fingerprint, &command)
+                    .map_err(|_| Error::other("heal control request length cannot be represented"))?;
+                let mut request = Request::new(HealControlRequest {
+                    version,
+                    topology_fingerprint: topology_fingerprint.clone(),
+                    command: command.clone().into(),
+                });
+                request.set_timeout(rustfs_protos::heal_control_execution_timeout());
+                set_tonic_canonical_body_digest(&mut request, &canonical_body)?;
+                let response = client.heal_control(request).await?.into_inner();
+                if !response.success {
+                    return Err(Error::other(
+                        response
+                            .error_info
+                            .unwrap_or_else(|| "peer heal control failed without an error".to_string()),
+                    ));
+                }
+                if !capability_probe {
+                    let canonical_response = rustfs_protos::canonical_heal_control_response_body(
+                        version,
+                        &topology_fingerprint,
+                        &command,
+                        &response.result,
+                    )
+                    .map_err(|_| Error::other("heal control response length cannot be represented"))?;
+                    validate_heal_control_response_proof(&canonical_response, &response.response_proof)?;
+                }
+                Ok(response.result.to_vec())
+            }
+            .await,
+        )
+        .await
+    }
+
+    /// Confirms that a peer supports the current heal-control coordination
+    /// contract and has the same storage
+    /// topology. Every non-success response is an error so old or divergent
+    /// peers cannot be mistaken for compatible ones.
+    pub async fn probe_heal_control(&self, topology_fingerprint: String) -> Result<()> {
+        let nonce = uuid::Uuid::new_v4();
+        let probe = rustfs_protos::heal_control_capability_probe(nonce.as_bytes());
+        let canonical_ack = rustfs_protos::canonical_heal_control_capability_ack(
+            rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION,
+            &topology_fingerprint,
+            &probe,
+        )
+        .map_err(|_| Error::other("heal control capability acknowledgement length cannot be represented"))?;
+        let proof = self
+            .heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
+            .await?;
+        validate_heal_control_capability_proof(&canonical_ack, &proof)
+    }
+
+    pub async fn load_bucket_metadata(&self, bucket: &str, scanner_maintenance_change: bool) -> Result<()> {
         self.finalize_result(
             async {
                 let mut client = self.get_client().await?;
                 let request = Request::new(LoadBucketMetadataRequest {
                     bucket: bucket.to_string(),
+                    scanner_maintenance_change,
                 });
 
                 let response = client.load_bucket_metadata(request).await?.into_inner();
@@ -908,6 +1268,25 @@ impl PeerRestClient {
         .await
     }
 
+    pub async fn scanner_activity(&self) -> Result<ScannerPeerActivity> {
+        self.finalize_result(
+            async {
+                let mut client = self
+                    .get_client()
+                    .await?
+                    .max_decoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE)
+                    .max_encoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE);
+                let response = client
+                    .scanner_activity(Request::new(ScannerActivityRequest {}))
+                    .await?
+                    .into_inner();
+                decode_scanner_activity(response)
+            }
+            .await,
+        )
+        .await
+    }
+
     pub async fn get_metacache_listing(&self) -> Result<()> {
         warn!("get_metacache_listing is not implemented in PeerRestClient");
         Err(Error::NotImplemented)
@@ -1157,6 +1536,48 @@ mod tests {
         )
     }
 
+    #[test]
+    fn scanner_activity_requires_restart_safe_peer_identity() {
+        let missing_instance = ScannerActivityResponse {
+            instance_id: String::new(),
+            namespace_generation: 7,
+            maintenance_generation: 3,
+        };
+        assert!(
+            decode_scanner_activity(missing_instance)
+                .expect_err("an empty instance ID is not restart safe")
+                .to_string()
+                .contains("instance ID")
+        );
+
+        let malformed_instance = ScannerActivityResponse {
+            instance_id: "ABCDEF0123456789ABCDEF0123456789".to_string(),
+            namespace_generation: 7,
+            maintenance_generation: 3,
+        };
+        assert!(
+            decode_scanner_activity(malformed_instance)
+                .expect_err("activity instance IDs must use the canonical lowercase hex form")
+                .to_string()
+                .contains("instance ID")
+        );
+
+        let activity = decode_scanner_activity(ScannerActivityResponse {
+            instance_id: "0123456789abcdef0123456789abcdef".to_string(),
+            namespace_generation: 7,
+            maintenance_generation: 3,
+        })
+        .expect("complete activity responses should be accepted");
+        assert_eq!(
+            activity,
+            ScannerPeerActivity {
+                instance_id: "0123456789abcdef0123456789abcdef".to_string(),
+                namespace_generation: 7,
+                maintenance_generation: 3,
+            }
+        );
+    }
+
     #[test]
     fn peer_rest_client_marks_network_like_errors() {
         assert!(PeerRestClient::is_network_like_error(&Error::other("transport error")));
@@ -1177,6 +1598,230 @@ mod tests {
         assert!(err.to_string().contains("temporarily offline"));
     }
 
+    #[tokio::test]
+    async fn peer_rest_client_rejects_oversized_heal_control_before_dialing() {
+        let client = test_peer_client();
+        let err = client
+            .heal_control(1, "fingerprint".to_string(), vec![0; HEAL_CONTROL_PAYLOAD_MAX_SIZE + 1])
+            .await
+            .expect_err("oversized heal control payload must fail locally");
+
+        assert!(err.to_string().contains("exceeds size limit"));
+        assert!(!client.offline.load(Ordering::Acquire));
+    }
+
+    #[test]
+    fn heal_control_capability_proof_must_authenticate_exact_ack() {
+        runtime_sources::ensure_test_rpc_secret();
+        let proof = crate::cluster::rpc::sign_tonic_rpc_response_proof(b"expected").expect("test proof should sign");
+        assert!(validate_heal_control_capability_proof(b"expected", &proof).is_ok());
+        let err = validate_heal_control_capability_proof(b"different", &proof)
+            .expect_err("a proof for a different acknowledgement must fail closed");
+        assert!(err.to_string().contains("invalid heal control capability proof"));
+    }
+
+    #[test]
+    fn heal_control_response_proof_binds_command_and_result() {
+        runtime_sources::ensure_test_rpc_secret();
+        let canonical = rustfs_protos::canonical_heal_control_response_body(2, "fingerprint", b"query", b"result")
+            .expect("small response should encode");
+        let proof = crate::cluster::rpc::sign_tonic_rpc_response_proof(&canonical).expect("test proof should sign");
+        assert!(validate_heal_control_response_proof(&canonical, &proof).is_ok());
+
+        for tampered in [
+            rustfs_protos::canonical_heal_control_response_body(2, "fingerprint", b"cancel", b"result").unwrap(),
+            rustfs_protos::canonical_heal_control_response_body(2, "fingerprint", b"query", b"tampered").unwrap(),
+        ] {
+            let err = validate_heal_control_response_proof(&tampered, &proof)
+                .expect_err("proof must not authenticate a different command or result");
+            assert!(err.to_string().contains("invalid heal control response proof"));
+        }
+    }
+
+    struct TierMutationResponseFixture<'a> {
+        version: u32,
+        phase: TierMutationRpcPhase,
+        mutation_id: Uuid,
+        canonical_payload: &'a [u8],
+        success: bool,
+        state: i32,
+        applied: bool,
+        error_info: Option<&'a str>,
+    }
+
+    fn signed_tier_mutation_response(input: TierMutationResponseFixture<'_>) -> TierMutationControlResponse {
+        let canonical =
+            rustfs_protos::canonical_tier_mutation_rpc_response_body(rustfs_protos::TierMutationRpcResponseProofInput {
+                version: input.version,
+                phase: input.phase,
+                mutation_id: input.mutation_id,
+                canonical_payload: input.canonical_payload,
+                success: input.success,
+                state: input.state,
+                applied: input.applied,
+                error_info: input.error_info,
+            })
+            .expect("small tier mutation response should encode");
+        let response_proof =
+            crate::cluster::rpc::sign_tonic_rpc_response_proof(&canonical).expect("tier mutation response should sign");
+        TierMutationControlResponse {
+            success: input.success,
+            state: input.state,
+            applied: input.applied,
+            error_info: input.error_info.map(str::to_string),
+            response_proof: response_proof.into(),
+        }
+    }
+
+    #[test]
+    fn tier_mutation_response_proof_binds_phase_payload_state_and_error() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mutation_id = Uuid::new_v4();
+        let payload = b"tier-mutation-prepare";
+        let response = signed_tier_mutation_response(TierMutationResponseFixture {
+            version: rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            phase: TierMutationRpcPhase::Prepare,
+            mutation_id,
+            canonical_payload: payload,
+            success: true,
+            state: TierMutationPeerState::Prepared as i32,
+            applied: true,
+            error_info: None,
+        });
+        validate_tier_mutation_response_proof(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            TierMutationRpcPhase::Prepare,
+            mutation_id,
+            payload,
+            &response,
+        )
+        .expect("matching tier mutation response proof should verify");
+
+        for tampered in [
+            TierMutationControlResponse {
+                success: false,
+                error_info: Some("peer failed".to_string()),
+                ..response.clone()
+            },
+            TierMutationControlResponse {
+                state: TierMutationPeerState::Committed as i32,
+                ..response.clone()
+            },
+            TierMutationControlResponse {
+                applied: false,
+                ..response.clone()
+            },
+        ] {
+            let err = validate_tier_mutation_response_proof(
+                rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+                TierMutationRpcPhase::Prepare,
+                mutation_id,
+                payload,
+                &tampered,
+            )
+            .expect_err("tampered tier mutation response proof must fail");
+            assert!(err.to_string().contains("invalid tier mutation response proof"));
+        }
+
+        let err = validate_tier_mutation_response_proof(
+            rustfs_protos::TIER_MUTATION_RPC_PROTOCOL_VERSION,
+            TierMutationRpcPhase::Commit,
+            mutation_id,
+            payload,
+            &response,
+        )
+        .expect_err("response proof must bind request phase");
+        assert!(err.to_string().contains("invalid tier mutation response proof"));
+    }
+
+    #[test]
+    fn tier_mutation_peer_state_decode_fails_closed() {
+        assert_eq!(
+            decode_tier_mutation_peer_state(TierMutationPeerState::Prepared as i32).expect("prepared state should decode"),
+            PeerTierMutationState::Prepared
+        );
+        assert_eq!(
+            decode_tier_mutation_peer_state(TierMutationPeerState::Committed as i32).expect("committed state should decode"),
+            PeerTierMutationState::Committed
+        );
+        assert_eq!(
+            decode_tier_mutation_peer_state(TierMutationPeerState::Aborted as i32).expect("aborted state should decode"),
+            PeerTierMutationState::Aborted
+        );
+        assert!(decode_tier_mutation_peer_state(TierMutationPeerState::Unspecified as i32).is_err());
+        assert!(decode_tier_mutation_peer_state(99).is_err());
+    }
+
+    #[test]
+    fn tier_mutation_payload_guard_rejects_invalid_lengths() {
+        validate_tier_mutation_payload_len(
+            TierMutationRpcPhase::Prepare,
+            rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE,
+        )
+        .expect("max prepare payload should fit");
+        assert!(
+            validate_tier_mutation_payload_len(
+                TierMutationRpcPhase::Prepare,
+                rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE + 1,
+            )
+            .is_err()
+        );
+        validate_tier_mutation_payload_len(
+            TierMutationRpcPhase::Commit,
+            rustfs_protos::TIER_MUTATION_RPC_MAX_COMMIT_PAYLOAD_SIZE,
+        )
+        .expect("max commit payload should fit");
+        assert!(
+            validate_tier_mutation_payload_len(
+                TierMutationRpcPhase::Commit,
+                rustfs_protos::TIER_MUTATION_RPC_MAX_COMMIT_PAYLOAD_SIZE + 1,
+            )
+            .is_err()
+        );
+        validate_tier_mutation_payload_len(TierMutationRpcPhase::Abort, 0).expect("empty abort payload should fit");
+        assert!(validate_tier_mutation_payload_len(TierMutationRpcPhase::Abort, 1).is_err());
+    }
+
+    #[test]
+    fn tier_mutation_rpc_status_matrix_fails_closed_for_old_or_unresponsive_peers() {
+        for (phase, label) in [
+            (TierMutationRpcPhase::Prepare, "prepare"),
+            (TierMutationRpcPhase::Commit, "commit"),
+            (TierMutationRpcPhase::Abort, "abort"),
+        ] {
+            for status in [
+                tonic::Status::unimplemented("old peer has no tier mutation control service"),
+                tonic::Status::deadline_exceeded("peer tier mutation control timed out"),
+                tonic::Status::unavailable("peer tier mutation control unavailable"),
+            ] {
+                let err = tier_mutation_control_status_error(phase, status);
+                let rendered = err.to_string();
+                assert!(rendered.contains(&format!("peer tier mutation {label} RPC failed")), "{rendered}");
+                assert!(
+                    rendered.contains("old peer")
+                        || rendered.contains("timed out")
+                        || rendered.contains("unavailable")
+                        || rendered.contains("Unavailable"),
+                    "{rendered}"
+                );
+            }
+        }
+    }
+
+    #[tokio::test]
+    async fn peer_rest_client_rejects_oversized_tier_prepare_before_dialing() {
+        let client = test_peer_client();
+        let err = client
+            .prepare_tier_mutation(
+                Uuid::new_v4(),
+                Bytes::from(vec![0; rustfs_protos::TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE + 1]),
+            )
+            .await
+            .expect_err("oversized tier prepare should fail before dialing");
+        assert!(err.to_string().contains("tier mutation payload exceeds size limit"));
+        assert!(!client.offline.load(Ordering::Acquire));
+    }
+
     #[tokio::test]
     async fn peer_rest_client_prepare_retry_clears_offline_gate() {
         // finalize_result sets the offline gate on a network error; without

crates/ecstore/src/cluster/rpc/peer_s3_client.rs

@@ -23,7 +23,7 @@ use crate::disk::{DiskAPI, DiskStore, disk_store::get_max_timeout_duration};
 use crate::runtime::instance::{InstanceContext, bootstrap_ctx};
 use crate::runtime::sources as runtime_sources;
 use crate::storage_api_contracts::bucket::{BucketInfo, BucketOptions, DeleteBucketOptions, MakeBucketOptions};
-use crate::store::utils::is_reserved_or_invalid_bucket;
+use crate::store::{has_xlmeta_files, utils::is_reserved_or_invalid_bucket};
 use crate::{
     disk::{
         self, VolumeInfo,
@@ -614,13 +614,24 @@ impl PeerS3Client for LocalPeerS3Client {
             return Err(Error::ErasureWriteQuorum);
         }
 
+        let force = if opts.force_if_empty && !opts.force {
+            for disk in local_disks.iter() {
+                if has_xlmeta_files(&disk.path().join(bucket)).await.map_err(Error::Io)? {
+                    return Err(Error::VolumeNotEmpty);
+                }
+            }
+            true
+        } else {
+            opts.force
+        };
+
         let mut futures = Vec::with_capacity(local_disks.len());
 
         for disk in local_disks.iter() {
             // Non-force delete refuses a non-empty bucket (VolumeNotEmpty), which
             // the recreate loop below turns into BucketNotEmpty; only an explicit
             // force delete removes recursively (backlog#799 B1).
-            futures.push(disk.delete_volume(bucket, opts.force));
+            futures.push(disk.delete_volume(bucket, force));
         }
 
         let results = join_all(futures).await;
@@ -988,13 +999,15 @@ impl PeerS3Client for RemotePeerS3Client {
         .await
     }
 
-    async fn delete_bucket(&self, bucket: &str, _opts: &DeleteBucketOptions) -> Result<()> {
+    async fn delete_bucket(&self, bucket: &str, opts: &DeleteBucketOptions) -> Result<()> {
         self.execute_with_timeout(
             || async {
+                let options = serde_json::to_string(opts)?;
                 let mut client = self.get_client().await?;
 
                 let request = Request::new(DeleteBucketRequest {
                     bucket: bucket.to_string(),
+                    options,
                 });
                 let response = client.delete_bucket(request).await?.into_inner();
                 if !response.success {

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -1129,6 +1129,9 @@ fn decode_batch_read_version_response_items(
             let resp = decode_msgpack_or_json::<BatchReadVersionResp>(buf, "", "BatchReadVersionResp").map_err(|err| {
                 Error::other(format!("decode BatchReadVersionResp msgpack item {index} from {endpoint} failed: {err}"))
             })?;
+            if resp.success {
+                validate_decoded_file_info(&resp.file_info)?;
+            }
             batch_read_version_resps.push(resp);
         }
         return Ok(batch_read_version_resps);
@@ -1143,12 +1146,19 @@ fn decode_batch_read_version_response_items(
         let resp = serde_json::from_str::<BatchReadVersionResp>(json_str).map_err(|err| {
             Error::other(format!("decode BatchReadVersionResp json item {index} from {endpoint} failed: {err}"))
         })?;
+        if resp.success {
+            validate_decoded_file_info(&resp.file_info)?;
+        }
         batch_read_version_resps.push(resp);
     }
 
     Ok(batch_read_version_resps)
 }
 
+fn validate_decoded_file_info(file_info: &FileInfo) -> Result<()> {
+    file_info.validate_for_metadata_read().map_err(Into::into)
+}
+
 #[async_trait::async_trait]
 impl DiskAPI for RemoteDisk {
     #[tracing::instrument(level = "trace", skip_all)]
@@ -1825,6 +1835,7 @@ impl DiskAPI for RemoteDisk {
                 }
 
                 let file_info = decode_msgpack_or_json::<FileInfo>(&response.file_info_bin, &response.file_info, "FileInfo")?;
+                validate_decoded_file_info(&file_info)?;
 
                 Ok(file_info)
             },
@@ -2712,6 +2723,25 @@ mod tests {
 
     static INIT: Once = Once::new();
 
+    #[test]
+    fn decoded_remote_metadata_rejects_default_like_delete_marker() {
+        let forged = FileInfo {
+            deleted: true,
+            ..Default::default()
+        };
+        assert!(matches!(validate_decoded_file_info(&forged), Err(DiskError::FileCorrupt)));
+
+        let marker = FileInfo {
+            volume: "bucket".to_string(),
+            name: "object".to_string(),
+            version_id: Some(Uuid::new_v4()),
+            deleted: true,
+            mod_time: Some(::time::OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+        validate_decoded_file_info(&marker).expect("canonical remote delete marker should validate");
+    }
+
     #[derive(Clone, Default)]
     struct CapturedLogs {
         buffer: Arc<Mutex<Vec<u8>>>,
@@ -3021,17 +3051,19 @@ mod tests {
     }
 
     fn sample_batch_read_version_resp(index: usize, path: &str, success: bool) -> BatchReadVersionResp {
+        let mut file_info = FileInfo::new(path, 1, 0);
+        file_info.erasure.index = 1;
         BatchReadVersionResp {
             index,
             path: path.to_string(),
             version_id: "version-a".to_string(),
             success,
+            file_info,
             error: if success {
                 String::new()
             } else {
                 "file version not found".to_string()
             },
-            ..Default::default()
         }
     }
 
@@ -3055,6 +3087,25 @@ mod tests {
         assert!(decoded[0].success);
     }
 
+    #[test]
+    fn batch_read_version_response_rejects_invalid_success_metadata() {
+        let endpoint = sample_remote_endpoint();
+        let mut response_item = sample_batch_read_version_resp(0, "invalid-object", true);
+        response_item.file_info.erasure.data_blocks = 0;
+        response_item.file_info.erasure.parity_blocks = 2;
+        let response = BatchReadVersionResponse {
+            success: true,
+            batch_read_version_resps: Vec::new(),
+            batch_read_version_resps_bin: vec![encode_msgpack(&response_item).expect("msgpack response should encode").into()],
+            error: None,
+        };
+
+        let err = decode_batch_read_version_response_items(response, &endpoint)
+            .expect_err("successful remote response with invalid metadata must fail closed");
+
+        assert_eq!(err, DiskError::FileCorrupt);
+    }
+
     #[test]
     fn batch_read_version_response_decode_reports_corrupt_msgpack_item() {
         let endpoint = sample_remote_endpoint();

crates/ecstore/src/config/com.rs

@@ -279,6 +279,17 @@ where
     Ok(data)
 }
 
+/// Read an existing config object without treating an empty payload as absent.
+/// Callers that validate their own payload format need to distinguish corruption
+/// from `ConfigNotFound`.
+pub(crate) async fn read_config_preserve_empty<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
+where
+    S: EcstoreObjectIO,
+{
+    let (data, _obj) = read_config_with_metadata_inner(api, file, &ObjectOptions::default(), true).await?;
+    Ok(data)
+}
+
 pub async fn read_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
 where
     S: ObjectIO<
@@ -304,6 +315,26 @@ where
 }
 
 pub async fn read_config_with_metadata<S>(api: Arc<S>, file: &str, opts: &ObjectOptions) -> Result<(Vec<u8>, ObjectInfo)>
+where
+    S: ObjectIO<
+            Error = Error,
+            RangeSpec = HTTPRangeSpec,
+            HeaderMap = HeaderMap,
+            ObjectOptions = ObjectOptions,
+            ObjectInfo = ObjectInfo,
+            GetObjectReader = GetObjectReader,
+            PutObjectReader = PutObjReader,
+        >,
+{
+    read_config_with_metadata_inner(api, file, opts, false).await
+}
+
+async fn read_config_with_metadata_inner<S>(
+    api: Arc<S>,
+    file: &str,
+    opts: &ObjectOptions,
+    preserve_empty: bool,
+) -> Result<(Vec<u8>, ObjectInfo)>
 where
     S: ObjectIO<
             Error = Error,
@@ -330,7 +361,7 @@ where
 
     let data = rd.read_all().await?;
 
-    if data.is_empty() {
+    if data.is_empty() && !preserve_empty {
         return Err(Error::ConfigNotFound);
     }
 
@@ -425,8 +456,7 @@ async fn new_and_save_server_config<S>(api: Arc<S>) -> Result<Config>
 where
     S: EcstoreObjectIO + StorageAdminApi,
 {
-    let mut cfg = new_server_config();
-    lookup_configs(&mut cfg, api.clone()).await;
+    let cfg = new_server_config();
     save_server_config(api, &cfg).await?;
 
     Ok(cfg)
@@ -1245,9 +1275,7 @@ where
     let cfg = if runtime_sources::first_cluster_node_is_local().await {
         new_and_save_server_config(api.clone()).await?
     } else {
-        let mut cfg = new_server_config();
-        lookup_configs(&mut cfg, api).await;
-        cfg
+        new_server_config()
     };
     warn!("Configuration initialization complete ({})", context);
     Ok(cfg)
@@ -1545,14 +1573,11 @@ where
     save_config(api, &config_file, data).await
 }
 
-pub async fn lookup_configs<S>(cfg: &mut Config, api: Arc<S>)
+pub async fn lookup_configs<S>(cfg: &mut Config, api: Arc<S>) -> Result<()>
 where
     S: StorageAdminApi,
 {
-    // TODO: from etcd
-    if let Err(err) = apply_dynamic_config(cfg, api).await {
-        error!("apply_dynamic_config err {:?}", &err);
-    }
+    apply_dynamic_config(cfg, api).await
 }
 
 async fn apply_dynamic_config<S>(cfg: &mut Config, api: Arc<S>) -> Result<()>
@@ -1569,24 +1594,34 @@ where
 async fn apply_dynamic_config_for_sub_sys<S>(cfg: &mut Config, api: Arc<S>, subsys: &str) -> Result<()>
 where
     S: StorageAdminApi,
+{
+    apply_dynamic_config_for_sub_sys_with(
+        cfg,
+        api,
+        subsys,
+        storageclass::lookup_config_for_pools,
+        runtime_sources::set_storage_class_config,
+    )
+    .await
+}
+
+async fn apply_dynamic_config_for_sub_sys_with<S, R, F>(
+    cfg: &mut Config,
+    api: Arc<S>,
+    subsys: &str,
+    resolve_storage_class: R,
+    publish_storage_class: F,
+) -> Result<()>
+where
+    S: StorageAdminApi,
+    R: FnOnce(&KVS, &[usize]) -> Result<storageclass::Config>,
+    F: FnOnce(storageclass::Config),
 {
     let set_drive_counts = StorageAdminApi::set_drive_counts(api.as_ref());
     if subsys == STORAGE_CLASS_SUB_SYS {
         let kvs = cfg.get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER).unwrap_or_default();
-
-        for (i, count) in set_drive_counts.iter().enumerate() {
-            match storageclass::lookup_config(&kvs, *count) {
-                Ok(res) => {
-                    if i == 0 {
-                        runtime_sources::set_storage_class_config(res);
-                    }
-                }
-                Err(err) => {
-                    error!("init storage class err:{:?}", &err);
-                    break;
-                }
-            }
-        }
+        let candidate = resolve_storage_class(&kvs, &set_drive_counts)?;
+        publish_storage_class(candidate);
     }
 
     Ok(())
@@ -1595,8 +1630,9 @@ where
 #[cfg(test)]
 mod tests {
     use super::{
-        configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
-        read_config_with_metadata, storage_class_kvs_mut,
+        apply_dynamic_config_for_sub_sys_with, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
+        is_standard_object_server_config, lookup_configs, read_config, read_config_preserve_empty, read_config_with_metadata,
+        storage_class_kvs_mut,
     };
     use crate::config::{audit, notify, oidc};
     use crate::disk::endpoint::Endpoint;
@@ -2977,6 +3013,7 @@ mod tests {
         state: Mutex<RecoveryReadState>,
         heal_replacement: Option<Vec<u8>>,
         heal_calls: AtomicUsize,
+        drive_counts: Vec<usize>,
     }
 
     impl RecoveryMockStore {
@@ -2985,8 +3022,14 @@ mod tests {
                 state: Mutex::new(state),
                 heal_replacement,
                 heal_calls: AtomicUsize::new(0),
+                drive_counts: vec![2],
             }
         }
+
+        fn with_drive_counts(mut self, drive_counts: Vec<usize>) -> Self {
+            self.drive_counts = drive_counts;
+            self
+        }
     }
 
     struct ServerConfigDecryptHookGuard {
@@ -3059,6 +3102,21 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn read_config_preserve_empty_distinguishes_empty_object() {
+        let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None));
+
+        let err = read_config(store.clone(), "config/empty.json")
+            .await
+            .expect_err("the existing config contract treats empty objects as missing");
+        assert!(matches!(err, Error::ConfigNotFound));
+
+        let data = read_config_preserve_empty(store, "config/empty.json")
+            .await
+            .expect("payload-validating callers must observe the empty object");
+        assert!(data.is_empty());
+    }
+
     #[async_trait::async_trait]
     impl StorageAdminApi for RecoveryMockStore {
         type BackendInfo = rustfs_madmin::BackendInfo;
@@ -3086,7 +3144,7 @@ mod tests {
         }
 
         fn set_drive_counts(&self) -> Vec<usize> {
-            vec![2]
+            self.drive_counts.clone()
         }
     }
 
@@ -3127,6 +3185,68 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn invalid_later_pool_does_not_partially_publish_storage_class() {
+        temp_env::async_with_vars(
+            [
+                (crate::config::storageclass::STANDARD_ENV, None::<&str>),
+                (crate::config::storageclass::RRS_ENV, None::<&str>),
+                (crate::config::storageclass::OPTIMIZE_ENV, None::<&str>),
+                (crate::config::storageclass::INLINE_BLOCK_ENV, None::<&str>),
+            ],
+            async {
+                let mut cfg = Config::new();
+                storage_class_kvs_mut(&mut cfg)
+                    .insert(crate::config::storageclass::CLASS_STANDARD.to_string(), "EC:2".to_string());
+                let store =
+                    Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None).with_drive_counts(vec![4, 2]));
+
+                let result = lookup_configs(&mut cfg, store.clone()).await;
+                assert!(result.is_err(), "public lookup entry must propagate an invalid later pool");
+
+                let mut publish_count = 0;
+                let result = apply_dynamic_config_for_sub_sys_with(
+                    &mut cfg,
+                    store,
+                    STORAGE_CLASS_SUB_SYS,
+                    crate::config::storageclass::lookup_config_for_pools_without_env,
+                    |_| {
+                        publish_count += 1;
+                    },
+                )
+                .await;
+                assert!(result.is_err(), "invalid later pool must fail the complete candidate");
+                assert_eq!(publish_count, 0, "failed reload must not publish any candidate");
+            },
+        )
+        .await;
+    }
+
+    #[tokio::test]
+    async fn dynamic_storage_class_publishes_one_multi_pool_snapshot() {
+        let mut cfg = Config::new();
+        let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(Vec::new()), None).with_drive_counts(vec![4, 2]));
+
+        let mut published = None;
+        let result = apply_dynamic_config_for_sub_sys_with(
+            &mut cfg,
+            store,
+            STORAGE_CLASS_SUB_SYS,
+            crate::config::storageclass::lookup_config_for_pools_without_env,
+            |candidate| {
+                published = Some(candidate);
+            },
+        )
+        .await;
+
+        result.expect("valid multi-pool candidate should publish");
+        assert_eq!(
+            published.and_then(|cfg| cfg.parities_for_sc(crate::config::storageclass::STANDARD)),
+            Some(vec![2, 1])
+        );
+    }
+
     fn encrypted_current_server_config_blob() -> Vec<u8> {
         let mut cfg = Config::new();
         let kvs = storage_class_kvs_mut(&mut cfg);

crates/ecstore/src/config/mod.rs

@@ -26,6 +26,7 @@ pub mod storageclass;
 
 use crate::error::Result;
 use crate::store::ECStore;
+use arc_swap::ArcSwap;
 use com::{STORAGE_CLASS_SUB_SYS, lookup_configs, read_config_without_migrate_with_recovery};
 use rustfs_config::HEAL_SUB_SYS;
 use rustfs_config::audit::{
@@ -39,11 +40,12 @@ use rustfs_config::notify::{
 use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
 use rustfs_config::server_config::{register_default_kvs, set_global_server_config};
 use std::collections::HashMap;
+use std::sync::Arc;
 use std::sync::LazyLock;
-use std::sync::{Arc, RwLock};
+use tracing::warn;
 
-pub static GLOBAL_STORAGE_CLASS: LazyLock<RwLock<storageclass::Config>> =
-    LazyLock::new(|| RwLock::new(storageclass::Config::default()));
+pub static GLOBAL_STORAGE_CLASS: LazyLock<ArcSwap<storageclass::Config>> =
+    LazyLock::new(|| ArcSwap::from_pointee(storageclass::Config::default()));
 pub static GLOBAL_CONFIG_SYS: LazyLock<ConfigSys> = LazyLock::new(ConfigSys::new);
 
 pub static RUSTFS_CONFIG_PREFIX: &str = "config";
@@ -63,7 +65,7 @@ impl ConfigSys {
     pub async fn init(&self, api: Arc<ECStore>) -> Result<()> {
         let mut cfg = read_config_without_migrate_with_recovery(api.clone()).await?;
 
-        lookup_configs(&mut cfg, api).await;
+        lookup_configs(&mut cfg, api).await?;
 
         set_global_server_config(cfg);
 
@@ -72,12 +74,166 @@ impl ConfigSys {
 }
 
 pub fn get_global_storage_class() -> Option<storageclass::Config> {
-    GLOBAL_STORAGE_CLASS.read().ok().map(|guard| (*guard).clone())
+    Some(GLOBAL_STORAGE_CLASS.load().as_ref().clone())
+}
+
+pub(crate) fn get_global_storage_class_snapshot() -> Arc<storageclass::Config> {
+    GLOBAL_STORAGE_CLASS.load_full()
+}
+
+fn publish_storage_class_config(target: &ArcSwap<storageclass::Config>, cfg: storageclass::Config) {
+    let cfg = Arc::new(cfg);
+    target.store(cfg.clone());
+
+    for (pool_index, drives_per_set) in cfg.automatic_zero_parity_pools() {
+        warn!(
+            event = "storage_class_zero_redundancy",
+            component = "ecstore",
+            subsystem = "storage_class",
+            state = "degraded",
+            pool_index,
+            drives_per_set,
+            parity = 0,
+            "automatic storage class has no parity"
+        );
+    }
 }
 
 pub fn set_global_storage_class(cfg: storageclass::Config) {
-    if let Ok(mut guard) = GLOBAL_STORAGE_CLASS.write() {
-        *guard = cfg;
+    publish_storage_class_config(&GLOBAL_STORAGE_CLASS, cfg);
+}
+
+#[cfg(test)]
+mod storage_class_publish_tests {
+    use super::publish_storage_class_config;
+    use crate::config::storageclass::{self, CLASS_STANDARD, STANDARD, lookup_config_for_pools_without_env};
+    use arc_swap::ArcSwap;
+    use rustfs_config::server_config::KVS;
+    use std::io::{self, Write};
+    use std::sync::{Arc, Mutex};
+    use tracing_subscriber::fmt::MakeWriter;
+
+    #[derive(Clone, Default)]
+    struct CapturedLogs {
+        output: Arc<Mutex<Vec<u8>>>,
+        published: Option<Arc<ArcSwap<storageclass::Config>>>,
+    }
+
+    struct CapturedWriter(CapturedLogs);
+
+    impl Write for CapturedWriter {
+        fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+            if let Some(published) = &self.0.published {
+                assert_eq!(
+                    published.load_full().parities_for_sc(STANDARD),
+                    Some(vec![2, 0]),
+                    "warning must be emitted after the new snapshot is visible"
+                );
+            }
+            self.0.output.lock().expect("log buffer lock poisoned").extend_from_slice(buf);
+            Ok(buf.len())
+        }
+
+        fn flush(&mut self) -> io::Result<()> {
+            Ok(())
+        }
+    }
+
+    impl<'a> MakeWriter<'a> for CapturedLogs {
+        type Writer = CapturedWriter;
+
+        fn make_writer(&'a self) -> Self::Writer {
+            CapturedWriter(self.clone())
+        }
+    }
+
+    impl CapturedLogs {
+        fn observing(published: Arc<ArcSwap<storageclass::Config>>) -> Self {
+            Self {
+                published: Some(published),
+                ..Default::default()
+            }
+        }
+
+        fn output(&self) -> String {
+            String::from_utf8(self.output.lock().expect("log buffer lock poisoned").clone()).expect("logs should be UTF-8")
+        }
+    }
+
+    #[test]
+    fn automatic_single_disk_publish_emits_structured_warning() {
+        let cfg = lookup_config_for_pools_without_env(&KVS::new(), &[4, 1]).expect("automatic config should resolve");
+        let target = Arc::new(ArcSwap::from_pointee(Default::default()));
+        let logs = CapturedLogs::observing(target.clone());
+        let subscriber = tracing_subscriber::fmt().json().with_writer(logs.clone()).finish();
+
+        tracing::subscriber::with_default(subscriber, || publish_storage_class_config(&target, cfg));
+
+        assert_eq!(target.load_full().parities_for_sc(STANDARD), Some(vec![2, 0]));
+        let output = logs.output();
+        assert_eq!(
+            output.matches("storage_class_zero_redundancy").count(),
+            1,
+            "unexpected warning count: {output}"
+        );
+        assert!(output.contains("\"level\":\"WARN\""), "unexpected warning level: {output}");
+        assert!(output.contains("\"pool_index\":1"), "missing pool index: {output}");
+        assert!(output.contains("\"drives_per_set\":1"), "missing drive count: {output}");
+        assert!(output.contains("\"parity\":0"), "missing parity: {output}");
+    }
+
+    #[test]
+    fn explicit_zero_parity_publish_does_not_emit_automatic_warning() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:0".to_string());
+        let cfg = lookup_config_for_pools_without_env(&kvs, &[1]).expect("explicit EC:0 should resolve");
+        let target = ArcSwap::from_pointee(Default::default());
+        let logs = CapturedLogs::default();
+        let subscriber = tracing_subscriber::fmt().json().with_writer(logs.clone()).finish();
+
+        tracing::subscriber::with_default(subscriber, || publish_storage_class_config(&target, cfg));
+
+        assert_eq!(target.load_full().parities_for_sc(STANDARD), Some(vec![0]));
+        assert!(!logs.output().contains("storage_class_zero_redundancy"));
+    }
+
+    #[test]
+    fn storage_class_arc_swap_never_exposes_mixed_pool_state() {
+        let automatic = lookup_config_for_pools_without_env(&KVS::new(), &[4, 6]).expect("automatic config should resolve");
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:1".to_string());
+        let explicit = lookup_config_for_pools_without_env(&kvs, &[4, 6]).expect("explicit config should resolve");
+        let target = Arc::new(ArcSwap::from_pointee(automatic.clone()));
+        let old_snapshot = target.load_full();
+
+        let writer_target = target.clone();
+        let writer = std::thread::spawn(move || {
+            for index in 0..2_000 {
+                let next = if index % 2 == 0 { automatic.clone() } else { explicit.clone() };
+                writer_target.store(Arc::new(next));
+            }
+        });
+
+        let readers: Vec<_> = (0..4)
+            .map(|_| {
+                let reader_target = target.clone();
+                std::thread::spawn(move || {
+                    for _ in 0..2_000 {
+                        let pair = reader_target
+                            .load_full()
+                            .parities_for_sc(STANDARD)
+                            .expect("published config must have pool topology");
+                        assert!(pair == [2, 3] || pair == [1, 1], "mixed pool snapshot: {pair:?}");
+                    }
+                })
+            })
+            .collect();
+
+        writer.join().expect("writer thread should complete");
+        for reader in readers {
+            reader.join().expect("reader thread should complete");
+        }
+        assert_eq!(old_snapshot.parities_for_sc(STANDARD), Some(vec![2, 3]));
     }
 }
 

crates/ecstore/src/config/storageclass.rs

@@ -15,7 +15,7 @@
 use crate::error::{Error, Result};
 use rustfs_config::server_config::{KV, KVS};
 use serde::{Deserialize, Serialize};
-use std::env;
+use std::env::{self, VarError};
 use std::sync::LazyLock;
 use tracing::warn;
 
@@ -69,6 +69,8 @@ pub const MIN_PARITY_DRIVES: usize = 0;
 
 // Default RRS parity is always minimum parity.
 pub const DEFAULT_RRS_PARITY: usize = 1;
+const DEFAULT_RRS_STORAGE_CLASS: &str = "EC:1";
+const ZERO_SET_DRIVE_COUNT_ERROR: &str = "set drive count must be greater than zero";
 
 pub static DEFAULT_INLINE_BLOCK: usize = 128 * 1024;
 
@@ -81,7 +83,7 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
         },
         KV {
             key: CLASS_RRS.to_owned(),
-            value: "EC:1".to_owned(),
+            value: DEFAULT_RRS_STORAGE_CLASS.to_owned(),
             hidden_if_empty: false,
         },
         KV {
@@ -105,6 +107,13 @@ pub struct StorageClass {
     parity: usize,
 }
 
+#[derive(Debug, Clone)]
+struct PoolParity {
+    drives_per_set: usize,
+    parity: usize,
+    automatic: bool,
+}
+
 // Config storage class configuration
 #[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Config {
@@ -113,37 +122,89 @@ pub struct Config {
     optimize: Option<String>,
     inline_block: usize,
     initialized: bool,
+    #[serde(skip)]
+    standard_parities: Vec<PoolParity>,
+    #[serde(skip)]
+    rrs_parities: Vec<PoolParity>,
 }
 
 impl Config {
-    pub fn get_parity_for_sc(&self, sc: &str) -> Option<usize> {
+    fn storage_class(&self, sc: &str) -> (&StorageClass, &[PoolParity]) {
         match sc.trim() {
-            RRS => {
-                if self.initialized {
-                    Some(self.rrs.parity)
-                } else {
-                    None
-                }
-            }
-            // All these storage classes use standard parity configuration
-            STANDARD | DEEP_ARCHIVE | EXPRESS_ONEZONE | GLACIER | GLACIER_IR | INTELLIGENT_TIERING | ONEZONE_IA | OUTPOSTS
-            | SNOW | STANDARD_IA => {
-                if self.initialized {
-                    Some(self.standard.parity)
-                } else {
-                    None
-                }
-            }
-            _ => {
-                if self.initialized {
-                    Some(self.standard.parity)
-                } else {
-                    None
-                }
-            }
+            RRS => (&self.rrs, &self.rrs_parities),
+            _ => (&self.standard, &self.standard_parities),
         }
     }
 
+    pub(crate) fn is_initialized(&self) -> bool {
+        self.initialized
+    }
+
+    pub fn get_parity_for_sc(&self, sc: &str) -> Option<usize> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (storage_class, pools) = self.storage_class(sc);
+        let Some(first) = pools.first() else {
+            return Some(storage_class.parity);
+        };
+
+        pools.iter().all(|pool| pool.parity == first.parity).then_some(first.parity)
+    }
+
+    /// Returns the resolved parity for a pool topology.
+    ///
+    /// A topology-bound lookup fails closed for unknown drive counts and for
+    /// deserialized legacy configurations that have no pool topology. Legacy
+    /// callers retain scalar compatibility through [`Self::get_parity_for_sc`].
+    pub(crate) fn parity_for_sc(&self, sc: &str, drives_per_set: usize) -> Option<usize> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (_, pools) = self.storage_class(sc);
+
+        pools
+            .iter()
+            .find(|pool| pool.drives_per_set == drives_per_set)
+            .map(|pool| pool.parity)
+    }
+
+    pub(crate) fn parity_for_pool(&self, sc: &str, pool_index: usize, drives_per_set: usize) -> Option<usize> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (_, pools) = self.storage_class(sc);
+
+        pools
+            .get(pool_index)
+            .filter(|pool| pool.drives_per_set == drives_per_set)
+            .map(|pool| pool.parity)
+    }
+
+    pub fn parities_for_sc(&self, sc: &str) -> Option<Vec<usize>> {
+        if !self.initialized {
+            return None;
+        }
+
+        let (_, pools) = self.storage_class(sc);
+        if pools.is_empty() {
+            return None;
+        }
+
+        Some(pools.iter().map(|pool| pool.parity).collect())
+    }
+
+    pub(crate) fn automatic_zero_parity_pools(&self) -> impl Iterator<Item = (usize, usize)> + '_ {
+        self.standard_parities
+            .iter()
+            .enumerate()
+            .filter(|(_, pool)| pool.automatic && pool.parity == 0)
+            .map(|(pool_index, pool)| (pool_index, pool.drives_per_set))
+    }
+
     pub fn should_inline(&self, shard_size: i64, versioned: bool) -> bool {
         if shard_size < 0 {
             return false;
@@ -180,74 +241,175 @@ impl Config {
     }
 }
 
-pub fn lookup_config(kvs: &KVS, set_drive_count: usize) -> Result<Config> {
-    let standard = {
-        let ssc_str = {
-            if let Ok(ssc_str) = env::var(STANDARD_ENV) {
-                ssc_str
-            } else {
-                kvs.get(CLASS_STANDARD)
-            }
-        };
+#[derive(Debug, Default)]
+struct StorageClassEnvOverrides {
+    standard: Option<String>,
+    rrs: Option<String>,
+    optimize: Option<String>,
+    inline_block: Option<String>,
+}
 
-        if !ssc_str.is_empty() {
-            parse_storage_class(&ssc_str)?
-        } else {
-            StorageClass {
-                parity: default_parity_count(set_drive_count),
-            }
-        }
-    };
+impl StorageClassEnvOverrides {
+    fn from_process() -> Result<Self> {
+        Ok(Self {
+            standard: read_optional_env(STANDARD_ENV)?,
+            rrs: read_optional_env(RRS_ENV)?,
+            optimize: read_optional_env(OPTIMIZE_ENV)?,
+            inline_block: read_optional_env(INLINE_BLOCK_ENV)?,
+        })
+    }
+}
 
-    let rrs = {
-        let ssc_str = {
-            if let Ok(ssc_str) = env::var(RRS_ENV) {
-                ssc_str
-            } else {
-                kvs.get(CLASS_RRS)
-            }
-        };
+fn read_optional_env(name: &str) -> Result<Option<String>> {
+    parse_optional_env(name, env::var(name))
+}
 
-        if !ssc_str.is_empty() {
-            parse_storage_class(&ssc_str)?
-        } else {
-            StorageClass {
-                parity: { if set_drive_count == 1 { 0 } else { DEFAULT_RRS_PARITY } },
-            }
-        }
-    };
+fn parse_optional_env(name: &str, value: std::result::Result<String, VarError>) -> Result<Option<String>> {
+    match value {
+        Ok(value) => Ok(Some(value)),
+        Err(VarError::NotPresent) => Ok(None),
+        Err(VarError::NotUnicode(_)) => Err(Error::other(format!("{name} contains non-Unicode data"))),
+    }
+}
 
-    validate_parity_inner(standard.parity, rrs.parity, set_drive_count)?;
+#[derive(Debug, Clone, Copy)]
+enum ParityPolicy {
+    AutomaticStandard,
+    LegacyRrs,
+    Explicit(usize),
+}
 
-    let optimize = { env::var(OPTIMIZE_ENV).ok() };
-
-    let inline_block = {
-        if let Ok(ev) = env::var(INLINE_BLOCK_ENV) {
-            if let Ok(block) = ev.parse::<bytesize::ByteSize>() {
-                if block.as_u64() as usize > DEFAULT_INLINE_BLOCK {
-                    warn!(
-                        "inline block value bigger than recommended max of 128KiB -> {}, performance may degrade for PUT please benchmark the changes",
-                        block
-                    );
+impl ParityPolicy {
+    fn resolve(self, drives_per_set: usize) -> usize {
+        match self {
+            Self::AutomaticStandard => default_parity_count(drives_per_set),
+            Self::LegacyRrs => {
+                if drives_per_set == 1 {
+                    0
+                } else {
+                    DEFAULT_RRS_PARITY
                 }
-                block.as_u64() as usize
-            } else {
-                return Err(Error::other(format!("parse {INLINE_BLOCK_ENV} format failed")));
             }
-        } else {
-            DEFAULT_INLINE_BLOCK
+            Self::Explicit(parity) => parity,
         }
+    }
+}
+
+fn standard_policy(kvs: &KVS, value: Option<String>) -> Result<ParityPolicy> {
+    match value {
+        Some(value) if value.is_empty() => Ok(ParityPolicy::AutomaticStandard),
+        Some(value) => Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity)),
+        None => {
+            let value = kvs.get(CLASS_STANDARD);
+            if value.is_empty() {
+                Ok(ParityPolicy::AutomaticStandard)
+            } else {
+                Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity))
+            }
+        }
+    }
+}
+
+fn rrs_policy(kvs: &KVS, value: Option<String>) -> Result<ParityPolicy> {
+    match value {
+        Some(value) if value.is_empty() => Ok(ParityPolicy::LegacyRrs),
+        Some(value) => Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity)),
+        None => {
+            let value = kvs.get(CLASS_RRS);
+            if value.is_empty() || value == DEFAULT_RRS_STORAGE_CLASS {
+                Ok(ParityPolicy::LegacyRrs)
+            } else {
+                Ok(ParityPolicy::Explicit(parse_storage_class(&value)?.parity))
+            }
+        }
+    }
+}
+
+pub fn lookup_config_for_pools(kvs: &KVS, set_drive_counts: &[usize]) -> Result<Config> {
+    lookup_config_for_pools_with_env(kvs, set_drive_counts, StorageClassEnvOverrides::from_process()?)
+}
+
+fn lookup_config_for_pools_with_env(
+    kvs: &KVS,
+    set_drive_counts: &[usize],
+    overrides: StorageClassEnvOverrides,
+) -> Result<Config> {
+    if set_drive_counts.is_empty() {
+        return Err(Error::other("storage class requires at least one pool"));
+    }
+
+    let standard_policy = standard_policy(kvs, overrides.standard)?;
+    let rrs_policy = rrs_policy(kvs, overrides.rrs)?;
+    let mut standard_parities = Vec::with_capacity(set_drive_counts.len());
+    let mut rrs_parities = Vec::with_capacity(set_drive_counts.len());
+
+    for (pool_index, &drives_per_set) in set_drive_counts.iter().enumerate() {
+        let standard_parity = standard_policy.resolve(drives_per_set);
+        let rrs_parity = rrs_policy.resolve(drives_per_set);
+        validate_parity_inner(standard_parity, rrs_parity, drives_per_set).map_err(|err| {
+            Error::other(format!(
+                "storage class validation failed for pool {pool_index} ({drives_per_set} drives): {err}"
+            ))
+        })?;
+        standard_parities.push(PoolParity {
+            drives_per_set,
+            parity: standard_parity,
+            automatic: matches!(standard_policy, ParityPolicy::AutomaticStandard),
+        });
+        rrs_parities.push(PoolParity {
+            drives_per_set,
+            parity: rrs_parity,
+            automatic: matches!(rrs_policy, ParityPolicy::LegacyRrs),
+        });
+    }
+
+    let optimize = overrides.optimize;
+    let inline_block = if let Some(value) = overrides.inline_block {
+        let block = value
+            .parse::<bytesize::ByteSize>()
+            .map_err(|_| Error::other(format!("parse {INLINE_BLOCK_ENV} format failed")))?;
+        let inline_block = usize::try_from(block.as_u64())
+            .map_err(|_| Error::other(format!("{INLINE_BLOCK_ENV} exceeds the platform size limit")))?;
+        if inline_block > DEFAULT_INLINE_BLOCK {
+            warn!(
+                event = "storage_class_inline_block_large",
+                component = "ecstore",
+                subsystem = "storage_class",
+                state = "configured",
+                inline_block,
+                recommended_max = DEFAULT_INLINE_BLOCK,
+                "storage class inline block exceeds recommendation"
+            );
+        }
+        inline_block
+    } else {
+        DEFAULT_INLINE_BLOCK
     };
 
     Ok(Config {
-        standard,
-        rrs,
+        standard: StorageClass {
+            parity: standard_parities[0].parity,
+        },
+        rrs: StorageClass {
+            parity: rrs_parities[0].parity,
+        },
         optimize,
         inline_block,
         initialized: true,
+        standard_parities,
+        rrs_parities,
     })
 }
 
+pub fn lookup_config(kvs: &KVS, set_drive_count: usize) -> Result<Config> {
+    lookup_config_for_pools(kvs, &[set_drive_count])
+}
+
+#[cfg(test)]
+pub(crate) fn lookup_config_for_pools_without_env(kvs: &KVS, set_drive_counts: &[usize]) -> Result<Config> {
+    lookup_config_for_pools_with_env(kvs, set_drive_counts, StorageClassEnvOverrides::default())
+}
+
 pub fn parse_storage_class(env: &str) -> Result<StorageClass> {
     let s: Vec<&str> = env.split(':').collect();
 
@@ -281,6 +443,10 @@ pub fn validate_parity(ss_parity: usize, set_drive_count: usize) -> Result<()> {
     //     )));
     // }
 
+    if set_drive_count == 0 {
+        return Err(Error::other(ZERO_SET_DRIVE_COUNT_ERROR));
+    }
+
     if ss_parity > set_drive_count / 2 {
         return Err(Error::other(format!(
             "parity {} should be less than or equal to {}",
@@ -310,22 +476,24 @@ pub fn validate_parity_inner(ss_parity: usize, rrs_parity: usize, set_drive_coun
     //     )));
     // }
 
-    if set_drive_count > 2 {
-        if ss_parity > set_drive_count / 2 {
-            return Err(Error::other(format!(
-                "Standard storage class parity {} should be less than or equal to {}",
-                ss_parity,
-                set_drive_count / 2
-            )));
-        }
+    if set_drive_count == 0 {
+        return Err(Error::other(ZERO_SET_DRIVE_COUNT_ERROR));
+    }
 
-        if rrs_parity > set_drive_count / 2 {
-            return Err(Error::other(format!(
-                "Reduced redundancy storage class parity {} should be less than or equal to {}",
-                rrs_parity,
-                set_drive_count / 2
-            )));
-        }
+    if ss_parity > set_drive_count / 2 {
+        return Err(Error::other(format!(
+            "Standard storage class parity {} should be less than or equal to {}",
+            ss_parity,
+            set_drive_count / 2
+        )));
+    }
+
+    if rrs_parity > set_drive_count / 2 {
+        return Err(Error::other(format!(
+            "Reduced redundancy storage class parity {} should be less than or equal to {}",
+            rrs_parity,
+            set_drive_count / 2
+        )));
     }
 
     if ss_parity > 0 && rrs_parity > 0 && ss_parity < rrs_parity {
@@ -340,6 +508,171 @@ pub fn validate_parity_inner(ss_parity: usize, rrs_parity: usize, set_drive_coun
 mod tests {
     use super::*;
 
+    fn no_env_overrides() -> StorageClassEnvOverrides {
+        StorageClassEnvOverrides::default()
+    }
+
+    #[test]
+    fn automatic_parity_is_resolved_per_pool() {
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], no_env_overrides())
+            .expect("automatic storage class should support heterogeneous pools");
+
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(2));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 2), Some(1));
+        assert_eq!(cfg.parity_for_sc(RRS, 4), Some(1));
+        assert_eq!(cfg.parity_for_sc(RRS, 2), Some(1));
+        assert_eq!(cfg.get_parity_for_sc(STANDARD), None, "heterogeneous parity has no truthful scalar");
+
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[4, 6], no_env_overrides())
+            .expect("automatic parity should be valid for both pools");
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(2));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 6), Some(3));
+    }
+
+    #[test]
+    fn automatic_single_disk_pool_uses_zero_parity() {
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[4, 1], no_env_overrides())
+            .expect("automatic single-disk storage should remain supported");
+
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(2));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 1), Some(0));
+        assert_eq!(cfg.parity_for_sc(RRS, 1), Some(0));
+    }
+
+    #[test]
+    fn explicit_standard_parity_is_validated_against_every_pool() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:2".to_string());
+
+        let err = lookup_config_for_pools_with_env(&kvs, &[4, 2], no_env_overrides())
+            .expect_err("EC:2 must be rejected by the two-drive pool");
+        assert!(
+            err.to_string().contains("pool 1") && err.to_string().contains("2 drives"),
+            "error must identify the rejecting pool: {err}"
+        );
+
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:1".to_string());
+        let cfg = lookup_config_for_pools_with_env(&kvs, &[4, 2], no_env_overrides()).expect("EC:1 is valid for both pools");
+        assert_eq!(cfg.parity_for_sc(STANDARD, 4), Some(1));
+        assert_eq!(cfg.parity_for_sc(STANDARD, 2), Some(1));
+        assert_eq!(cfg.get_parity_for_sc(STANDARD), Some(1));
+    }
+
+    #[test]
+    fn environment_rrs_value_is_explicit_but_persisted_default_is_legacy() {
+        let persisted_default = lookup_config_for_pools_with_env(&DEFAULT_KVS, &[1], no_env_overrides())
+            .expect("persisted default RRS EC:1 should retain single-disk compatibility");
+        assert_eq!(persisted_default.parity_for_sc(RRS, 1), Some(0));
+
+        let env = StorageClassEnvOverrides {
+            rrs: Some("EC:1".to_string()),
+            ..Default::default()
+        };
+        let err = lookup_config_for_pools_with_env(&DEFAULT_KVS, &[1], env)
+            .expect_err("environment RRS EC:1 is explicit and must fail on a single disk");
+        assert!(err.to_string().contains("pool 0") && err.to_string().contains("1 drives"));
+    }
+
+    #[test]
+    fn explicit_environment_standard_parity_is_not_clamped() {
+        let env = StorageClassEnvOverrides {
+            standard: Some("EC:2".to_string()),
+            ..Default::default()
+        };
+        let err = lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], env)
+            .expect_err("explicit environment parity must not be clamped for a smaller pool");
+        assert!(err.to_string().contains("pool 1") && err.to_string().contains("2 drives"));
+    }
+
+    #[test]
+    fn empty_environment_values_retain_automatic_semantics() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:2".to_string());
+        kvs.insert(CLASS_RRS.to_string(), "EC:2".to_string());
+        let env = StorageClassEnvOverrides {
+            standard: Some(String::new()),
+            rrs: Some(String::new()),
+            ..Default::default()
+        };
+
+        let cfg = lookup_config_for_pools_with_env(&kvs, &[4, 2], env)
+            .expect("empty environment values should override persisted parity with automatic defaults");
+        assert_eq!(cfg.parities_for_sc(STANDARD), Some(vec![2, 1]));
+        assert_eq!(cfg.parities_for_sc(RRS), Some(vec![1, 1]));
+    }
+
+    #[test]
+    fn explicit_zero_parity_is_not_reported_as_automatic() {
+        let env = StorageClassEnvOverrides {
+            standard: Some("EC:0".to_string()),
+            ..Default::default()
+        };
+
+        let cfg = lookup_config_for_pools_with_env(&KVS::new(), &[1], env)
+            .expect("explicit zero parity should remain valid for a single-drive pool");
+        assert_eq!(cfg.parities_for_sc(STANDARD), Some(vec![0]));
+        assert_eq!(cfg.automatic_zero_parity_pools().count(), 0);
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn non_unicode_environment_value_fails_closed() {
+        use std::ffi::OsString;
+        use std::os::unix::ffi::OsStringExt;
+
+        let err = parse_optional_env(STANDARD_ENV, Err(VarError::NotUnicode(OsString::from_vec(vec![0xff]))))
+            .expect_err("non-Unicode parity must fail closed");
+        assert!(err.to_string().contains(STANDARD_ENV));
+    }
+
+    #[test]
+    fn environment_standard_override_can_rescue_persisted_config() {
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:2".to_string());
+        let env = StorageClassEnvOverrides {
+            standard: Some("EC:1".to_string()),
+            ..Default::default()
+        };
+
+        let cfg = lookup_config_for_pools_with_env(&kvs, &[4, 2], env)
+            .expect("environment override should replace the persisted parity before validation");
+        assert_eq!(cfg.parity_for_pool(STANDARD, 0, 4), Some(1));
+        assert_eq!(cfg.parity_for_pool(STANDARD, 1, 2), Some(1));
+    }
+
+    #[test]
+    fn invalid_or_unknown_pool_topology_fails_closed() {
+        assert!(lookup_config_for_pools_with_env(&KVS::new(), &[], no_env_overrides()).is_err());
+
+        let err = lookup_config_for_pools_with_env(&KVS::new(), &[4, 0], no_env_overrides())
+            .expect_err("zero-drive pools must be rejected");
+        assert!(err.to_string().contains("pool 1") && err.to_string().contains("0 drives"));
+
+        let cfg =
+            lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], no_env_overrides()).expect("valid topology should resolve");
+        assert_eq!(cfg.parity_for_pool(STANDARD, 2, 6), None);
+        assert_eq!(cfg.parity_for_pool(STANDARD, 1, 6), None);
+        assert_eq!(cfg.parity_for_pool(STANDARD, 0, 2), None);
+        assert_eq!(cfg.parity_for_pool(STANDARD, 1, 4), None);
+        assert_eq!(cfg.parity_for_sc(STANDARD, 6), None);
+    }
+
+    #[test]
+    fn resolved_pool_state_is_not_serialized() {
+        let cfg =
+            lookup_config_for_pools_with_env(&KVS::new(), &[4, 2], no_env_overrides()).expect("valid topology should resolve");
+        let encoded = serde_json::to_string(&cfg).expect("config should serialize");
+        assert!(!encoded.contains("standard_parities"));
+        assert!(!encoded.contains("rrs_parities"));
+
+        let decoded: Config = serde_json::from_str(&encoded).expect("legacy scalar config should deserialize");
+        assert_eq!(decoded.get_parity_for_sc(STANDARD), Some(2));
+        assert_eq!(decoded.parity_for_pool(STANDARD, 0, 4), None);
+        assert_eq!(decoded.parity_for_sc(STANDARD, 4), None);
+        assert_eq!(decoded.parities_for_sc(STANDARD), None);
+        assert!(validate_parity(0, 0).is_err());
+    }
+
     #[test]
     fn lookup_config_reads_rrs_from_class_rrs_key() {
         // Regression: kvs.get(RRS) used RRS="REDUCED_REDUNDANCY" instead of
@@ -348,7 +681,7 @@ mod tests {
         kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
         kvs.insert(CLASS_RRS.to_string(), "EC:2".to_string());
 
-        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        let cfg = lookup_config_for_pools_without_env(&kvs, &[8]).expect("lookup should succeed");
         assert_eq!(cfg.standard.parity, 4, "standard parity should be 4");
         assert_eq!(cfg.rrs.parity, 2, "rrs parity should be 2");
     }
@@ -360,7 +693,7 @@ mod tests {
         kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
         kvs.insert(RRS.to_string(), "EC:2".to_string());
 
-        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        let cfg = lookup_config_for_pools_without_env(&kvs, &[8]).expect("lookup should succeed");
         assert_eq!(cfg.standard.parity, 4);
         assert_eq!(
             cfg.rrs.parity, DEFAULT_RRS_PARITY,

crates/ecstore/src/core/pools_test.rs

@@ -87,50 +87,57 @@ mod capacity_dedup_tests {
     }
 
     #[test]
-    fn test_four_disk_erasure_coding() {
-        // 4-disk erasure coding: 2 data disks + 2 parity disks
+    fn test_four_node_ec_2_2_reports_stable_usable_capacity() {
+        const TIB: u64 = 1 << 40;
+        const DISK_TOTAL: u64 = 2 * TIB;
+        const DISK_FREE: u64 = TIB / 5;
+
         let disks = vec![
             rustfs_madmin::Disk {
                 endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk1".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 0,
-                total_space: 1_000_000_000_000, // 1TB
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
             rustfs_madmin::Disk {
-                endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk2".to_string(),
+                endpoint: "node2".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 1,
-                total_space: 1_000_000_000_000,
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
             rustfs_madmin::Disk {
-                endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk3".to_string(),
+                endpoint: "node3".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 2,
-                total_space: 1_000_000_000_000,
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
             rustfs_madmin::Disk {
-                endpoint: "node1".to_string(),
-                drive_path: "/mnt/disk4".to_string(),
+                endpoint: "node4".to_string(),
+                drive_path: "/media/rustfs-01".to_string(),
                 pool_index: 0,
                 set_index: 0,
                 disk_index: 3,
-                total_space: 1_000_000_000_000,
-                available_space: 250_000_000_000,
+                total_space: DISK_TOTAL,
+                available_space: DISK_FREE,
+                used_space: DISK_TOTAL - DISK_FREE,
                 state: "ok".to_string(),
                 ..Default::default()
             },
@@ -146,9 +153,17 @@ mod capacity_dedup_tests {
         };
 
         let total = get_total_usable_capacity(&disks, &info);
+        let free = get_total_usable_capacity_free(&disks, &info);
+        let used = total.saturating_sub(free);
+        let expected_total = usize::try_from(4 * TIB).expect("4 TiB must fit the supported platform's usize");
+        let expected_free = usize::try_from(2 * DISK_FREE).expect("usable free capacity must fit usize");
+        let single_node_used = usize::try_from(DISK_TOTAL - DISK_FREE).expect("single-node used capacity must fit usize");
 
-        // Only count data disks (disk_index < 2)
-        assert_eq!(total, 2_000_000_000_000, "Should count only data disks (2 × 1TB)");
+        assert_eq!(total, expected_total, "usable total must count the two data disks");
+        assert_eq!(free, expected_free, "usable free must count the two data disks");
+        assert_eq!(used, 2 * single_node_used, "usable used must be approximately 3.6 TiB");
+        assert_ne!(used, single_node_used, "used must not collapse to one node's approximately 1.8 TiB");
+        assert_ne!(used, 4 * single_node_used, "used must not report the approximately 7.2 TiB raw aggregate");
     }
 
     #[test]

crates/ecstore/src/core/sets.rs

@@ -35,7 +35,7 @@ use crate::{
     object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader},
     runtime::instance::{InstanceContext, bootstrap_ctx},
     runtime::sources as runtime_sources,
-    set_disk::SetDisks,
+    set_disk::{PreparedGetObjectMetadata, SetDisks},
     store::init_format::{check_format_erasure_values, get_format_erasure_in_quorum, load_format_erasure_all, save_format_file},
 };
 use futures::{
@@ -403,6 +403,31 @@ impl crate::storage_api_contracts::object::ObjectIO for Sets {
 }
 
 impl Sets {
+    pub(crate) async fn prepare_get_object_reader_metadata(
+        &self,
+        bucket: &str,
+        object: &str,
+        opts: &ObjectOptions,
+    ) -> Result<PreparedGetObjectMetadata> {
+        self.get_disks_by_key(object)
+            .prepare_get_object_metadata(bucket, object, opts)
+            .await
+    }
+
+    pub(crate) async fn get_object_reader_with_prepared_metadata(
+        &self,
+        bucket: &str,
+        object: &str,
+        range: Option<HTTPRangeSpec>,
+        headers: HeaderMap,
+        opts: &ObjectOptions,
+        metadata: PreparedGetObjectMetadata,
+    ) -> Result<GetObjectReader> {
+        self.get_disks_by_key(object)
+            .get_object_reader_with_prepared_metadata(bucket, object, range, headers, opts, metadata)
+            .await
+    }
+
     /// `put_object` plus the rename_data old-size backfill
     /// (rustfs/backlog#1009); see `SetDisks::put_object_with_old_current_size`.
     pub async fn put_object_with_old_current_size(

crates/ecstore/src/data_usage/mod.rs

@@ -24,7 +24,7 @@ use crate::storage_api_contracts::{
 };
 use crate::{
     bucket::{metadata_sys::get_replication_config, versioning::VersioningApi as _, versioning_sys::BucketVersioningSys},
-    config::com::read_config,
+    config::com::{read_config, read_config_preserve_empty},
     disk::DiskAPI,
     error::{Error, classify_system_path_failure_reason},
     object_api::ObjectInfo,
@@ -41,7 +41,10 @@ use rustfs_utils::path::SLASH_SEPARATOR;
 use std::{
     collections::{HashMap, HashSet, hash_map::Entry},
     future::Future,
-    sync::{Arc, LazyLock, OnceLock},
+    sync::{
+        Arc, LazyLock, OnceLock,
+        atomic::{AtomicU64, Ordering},
+    },
     time::{Duration, SystemTime},
 };
 use tokio::fs;
@@ -76,6 +79,67 @@ static USAGE_MEMORY_CACHE: OnceLock<UsageMemoryCache> = OnceLock::new();
 static USAGE_CACHE_UPDATING: OnceLock<CacheUpdating> = OnceLock::new();
 static LIVE_BUCKET_USAGE_CACHE: OnceLock<LiveBucketUsageCache> = OnceLock::new();
 
+/// Cached copy of the last persisted data usage snapshot, served to admin
+/// endpoints for up to `DATA_USAGE_CACHE_TTL_SECS` between backend reads.
+#[derive(Debug, Clone)]
+struct CachedDataUsageSnapshot {
+    info: Option<DataUsageInfo>,
+    loaded_at: tokio::time::Instant,
+}
+
+impl CachedDataUsageSnapshot {
+    fn result(&self) -> Result<DataUsageInfo, Error> {
+        self.info
+            .clone()
+            .ok_or_else(|| Error::other("data usage snapshot load recently failed"))
+    }
+}
+
+fn fresh_cached_data_usage_snapshot(
+    cache: &Option<CachedDataUsageSnapshot>,
+    now: tokio::time::Instant,
+    ttl: Duration,
+) -> Option<Result<DataUsageInfo, Error>> {
+    cache
+        .as_ref()
+        .filter(|cached| now.duration_since(cached.loaded_at) < ttl)
+        .map(CachedDataUsageSnapshot::result)
+}
+
+fn cache_data_usage_snapshot_result(
+    cache: &mut Option<CachedDataUsageSnapshot>,
+    result: Result<DataUsageInfo, Error>,
+    loaded_at: tokio::time::Instant,
+) -> Result<DataUsageInfo, Error> {
+    match result {
+        Ok(info) => {
+            *cache = Some(CachedDataUsageSnapshot {
+                info: Some(info.clone()),
+                loaded_at,
+            });
+            Ok(info)
+        }
+        Err(e) => {
+            *cache = Some(CachedDataUsageSnapshot { info: None, loaded_at });
+            Err(e)
+        }
+    }
+}
+
+type DataUsageSnapshotCache = Arc<RwLock<Option<CachedDataUsageSnapshot>>>;
+
+static DATA_USAGE_SNAPSHOT_CACHE: OnceLock<DataUsageSnapshotCache> = OnceLock::new();
+
+// Always-on revert detector for rustfs/backlog#1306: one relaxed increment per
+// full-bucket version listing is negligible and lets tests prove that admin
+// request paths never trigger live listings.
+static LIVE_BUCKET_USAGE_COMPUTATIONS: AtomicU64 = AtomicU64::new(0);
+
+/// Number of live full-bucket usage computations performed by this process.
+pub fn live_bucket_usage_computations() -> u64 {
+    LIVE_BUCKET_USAGE_COMPUTATIONS.load(Ordering::Relaxed)
+}
+
 /// Deferred persist thresholds for compression totals: persist after this many
 /// operations recorded, but no more often than the min interval.
 const COMPRESSION_PERSIST_BATCH_SIZE: u64 = 100;
@@ -115,6 +179,10 @@ fn cache_updating() -> &'static CacheUpdating {
     USAGE_CACHE_UPDATING.get_or_init(|| Arc::new(RwLock::new(false)))
 }
 
+fn data_usage_snapshot_cache() -> &'static DataUsageSnapshotCache {
+    DATA_USAGE_SNAPSHOT_CACHE.get_or_init(|| Arc::new(RwLock::new(None)))
+}
+
 fn live_bucket_usage_cache() -> &'static LiveBucketUsageCache {
     LIVE_BUCKET_USAGE_CACHE.get_or_init(|| {
         moka::future::Cache::builder()
@@ -135,6 +203,7 @@ lazy_static::lazy_static! {
         SLASH_SEPARATOR,
         DATA_USAGE_OBJ_NAME
     );
+    static ref DATA_USAGE_OBJ_BACKUP_PATH: String = format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str());
     pub static ref DATA_USAGE_BLOOM_NAME_PATH: String = format!("{}{}{}",
         crate::disk::BUCKET_META_PREFIX,
         SLASH_SEPARATOR,
@@ -147,18 +216,56 @@ lazy_static::lazy_static! {
     );
 }
 
+/// Decide whether an incoming usage snapshot must be skipped as stale, given the local
+/// wall clock `now`. Mirrors `stale_data_usage_update_reason` in
+/// `crates/scanner/src/scanner.rs` — keep the two consistent.
+///
+/// If the persisted `existing.last_update` is future-dated beyond
+/// [`rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE`] (clock step-back or a
+/// slower-clock scanner leader), it is untrustworthy: the save is allowed so usage
+/// stats cannot freeze forever.
+fn stale_data_usage_persist_reason(incoming: &DataUsageInfo, existing: &DataUsageInfo, now: SystemTime) -> Option<&'static str> {
+    match (incoming.last_update, existing.last_update) {
+        (Some(new_ts), Some(existing_ts))
+            if new_ts <= existing_ts && !rustfs_data_usage::usage_last_update_is_untrusted_future(existing_ts, now) =>
+        {
+            Some("older_or_equal_last_update")
+        }
+        _ => None,
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum UsageSnapshotSource {
+    Primary,
+    Backup,
+    Missing,
+}
+
+fn stale_data_usage_persist_reason_for_source(
+    incoming: &DataUsageInfo,
+    existing: &DataUsageInfo,
+    source: UsageSnapshotSource,
+    now: SystemTime,
+) -> Option<&'static str> {
+    let reason = stale_data_usage_persist_reason(incoming, existing, now);
+    if source == UsageSnapshotSource::Backup && incoming.last_update == existing.last_update {
+        None
+    } else {
+        reason
+    }
+}
+
 /// Store data usage info to backend storage
 #[instrument(skip(store))]
 pub async fn store_data_usage_in_backend(data_usage_info: DataUsageInfo, store: Arc<ECStore>) -> Result<(), Error> {
     // Prevent older data from overwriting newer persisted stats
-    if let Ok(buf) = read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await
-        && let Ok(existing) = serde_json::from_slice::<DataUsageInfo>(&buf)
-        && let (Some(new_ts), Some(existing_ts)) = (data_usage_info.last_update, existing.last_update)
-        && new_ts <= existing_ts
+    if let Ok((existing, source)) = load_data_usage_snapshot(store.clone()).await
+        && let Some(reason) = stale_data_usage_persist_reason_for_source(&data_usage_info, &existing, source, SystemTime::now())
     {
         info!(
-            "Skip persisting data usage: incoming last_update {:?} <= existing {:?}",
-            new_ts, existing_ts
+            "Skip persisting data usage ({reason}): incoming last_update {:?} <= existing {:?}",
+            data_usage_info.last_update, existing.last_update
         );
         return Ok(());
     }
@@ -175,6 +282,12 @@ async fn save_data_usage_in_backend(data_usage_info: DataUsageInfo, store: Arc<E
         .await
         .map_err(Error::other)?;
 
+    // Invalidate the cached snapshot so readers observe the new save on their
+    // next request instead of waiting out the remaining TTL. The next cached
+    // read reloads through `load_data_usage_from_backend`, keeping its
+    // backward-compatibility post-processing.
+    *data_usage_snapshot_cache().write().await = None;
+
     Ok(())
 }
 
@@ -224,8 +337,8 @@ pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str)
     live_bucket_usage_cache().invalidate(bucket).await;
     clear_bucket_usage_memory(bucket).await;
 
-    let data_usage_info = load_data_usage_from_backend(store.clone()).await?;
-    let existing = load_data_usage_from_backend(store.clone()).await.ok();
+    let data_usage_info = load_primary_data_usage_from_backend(store.clone()).await?;
+    let existing = load_primary_data_usage_from_backend(store.clone()).await.ok();
 
     if let Some(data_usage_info) = merge_bucket_usage_removal(data_usage_info, existing, bucket) {
         save_data_usage_in_backend(data_usage_info, store).await?;
@@ -234,47 +347,123 @@ pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str)
     Ok(())
 }
 
+fn record_usage_snapshot_failure(operation: &'static str, object: &str, e: &Error) {
+    let reason = classify_system_path_failure_reason(e);
+    record_system_path_failure("data_usage", operation, reason);
+    error!(
+        event = "data_usage_snapshot_load_failed",
+        component = "ecstore",
+        subsystem = "data_usage",
+        state = "read_failed",
+        operation,
+        reason,
+        object = %object,
+        error = %e,
+        "data usage snapshot load failed"
+    );
+}
+
+fn record_usage_snapshot_decode_failure(operation: &'static str, object: &str, e: &Error) {
+    const REASON: &str = "decode_error";
+    record_system_path_failure("data_usage", operation, REASON);
+    error!(
+        event = "data_usage_snapshot_load_failed",
+        component = "ecstore",
+        subsystem = "data_usage",
+        state = "decode_failed",
+        operation,
+        reason = REASON,
+        object = %object,
+        error = %e,
+        "data usage snapshot load failed"
+    );
+}
+
+fn parse_usage_snapshot(buf: &[u8]) -> Result<DataUsageInfo, Error> {
+    serde_json::from_slice(buf).map_err(|e| {
+        Error::other(format!(
+            "Failed to deserialize data usage info: {:?} at line {} column {}",
+            e.classify(),
+            e.line(),
+            e.column()
+        ))
+    })
+}
+
+/// Decide which usage snapshot to serve: the primary `.usage.json` or its
+/// `.bkp` backup. The backup future is polled only when the primary is unusable,
+/// so the healthy path performs a single read.
+///
+/// `Ok(DataUsageInfo::default())` is returned only when BOTH the primary and
+/// the backup are `ConfigNotFound` — a genuine "no snapshot yet". A real
+/// primary failure with a missing backup propagates as an error instead of
+/// being rendered as confirmed all-zero stats.
+async fn resolve_loaded_snapshot_with_source(
+    primary: Result<Vec<u8>, Error>,
+    backup: impl Future<Output = Result<Vec<u8>, Error>>,
+) -> Result<(DataUsageInfo, UsageSnapshotSource), Error> {
+    let primary_failure = match primary {
+        Ok(buf) => match parse_usage_snapshot(&buf) {
+            Ok(info) => return Ok((info, UsageSnapshotSource::Primary)),
+            Err(e) => {
+                record_usage_snapshot_decode_failure("parse_primary", DATA_USAGE_OBJ_NAME_PATH.as_str(), &e);
+                e
+            }
+        },
+        Err(e) => {
+            if e != Error::ConfigNotFound {
+                record_usage_snapshot_failure("read_primary", DATA_USAGE_OBJ_NAME_PATH.as_str(), &e);
+            }
+            e
+        }
+    };
+    match backup.await {
+        Ok(buf) => match parse_usage_snapshot(&buf) {
+            Ok(info) => Ok((info, UsageSnapshotSource::Backup)),
+            Err(e) => {
+                record_usage_snapshot_decode_failure("parse_backup", &DATA_USAGE_OBJ_BACKUP_PATH, &e);
+                Err(e)
+            }
+        },
+        Err(Error::ConfigNotFound) if primary_failure == Error::ConfigNotFound => {
+            Ok((DataUsageInfo::default(), UsageSnapshotSource::Missing))
+        }
+        Err(Error::ConfigNotFound) => Err(primary_failure),
+        Err(e) => {
+            record_usage_snapshot_failure("read_backup", &DATA_USAGE_OBJ_BACKUP_PATH, &e);
+            Err(e)
+        }
+    }
+}
+
+async fn resolve_loaded_snapshot(
+    primary: Result<Vec<u8>, Error>,
+    backup: impl Future<Output = Result<Vec<u8>, Error>>,
+) -> Result<DataUsageInfo, Error> {
+    Ok(resolve_loaded_snapshot_with_source(primary, backup).await?.0)
+}
+
+async fn load_data_usage_snapshot(store: Arc<ECStore>) -> Result<(DataUsageInfo, UsageSnapshotSource), Error> {
+    let primary = read_config_preserve_empty(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await;
+    resolve_loaded_snapshot_with_source(
+        primary,
+        async move { read_config_preserve_empty(store, &DATA_USAGE_OBJ_BACKUP_PATH).await },
+    )
+    .await
+}
+
+async fn load_primary_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
+    let buf = read_config_preserve_empty(store, &DATA_USAGE_OBJ_NAME_PATH).await?;
+    Ok(normalize_loaded_data_usage(parse_usage_snapshot(&buf)?).await)
+}
+
 /// Load data usage info from backend storage
 #[instrument(skip(store))]
 pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
-    let buf: Vec<u8> = match read_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH).await {
-        Ok(data) => data,
-        Err(e) => {
-            let reason = classify_system_path_failure_reason(&e);
-            record_system_path_failure("data_usage", "read_primary", reason);
-            error!(
-                path_kind = "data_usage",
-                operation = "read_primary",
-                reason,
-                object = %DATA_USAGE_OBJ_NAME_PATH.as_str(),
-                error = %e,
-                "system path read failed"
-            );
-
-            match read_config(store.clone(), format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()).as_str()).await {
-                Ok(data) => data,
-                Err(e) => {
-                    if e == Error::ConfigNotFound {
-                        return Ok(DataUsageInfo::default());
-                    }
-                    let reason = classify_system_path_failure_reason(&e);
-                    record_system_path_failure("data_usage", "read_backup", reason);
-                    error!(
-                        path_kind = "data_usage",
-                        operation = "read_backup",
-                        reason,
-                        object = %format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()),
-                        error = %e,
-                        "system path read failed"
-                    );
-                    return Err(Error::other(e));
-                }
-            }
-        }
-    };
-    let mut data_usage_info: DataUsageInfo =
-        serde_json::from_slice(&buf).map_err(|e| Error::other(format!("Failed to deserialize data usage info: {e}")))?;
+    Ok(normalize_loaded_data_usage(load_data_usage_snapshot(store).await?.0).await)
+}
 
+async fn normalize_loaded_data_usage(mut data_usage_info: DataUsageInfo) -> DataUsageInfo {
     info!("Loaded data usage info from backend with {} buckets", data_usage_info.buckets_count);
 
     // Handle backward compatibility
@@ -325,7 +514,34 @@ pub async fn load_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsa
         }
     }
 
-    Ok(data_usage_info)
+    data_usage_info
+}
+
+/// Load the persisted data usage snapshot through a small in-process cache.
+///
+/// Admin read endpoints call this on every request; the cache bounds backend
+/// reads (and the associated JSON parse and INFO log) to once per
+/// `DATA_USAGE_CACHE_TTL_SECS` per process. `save_data_usage_in_backend`
+/// invalidates the cache so a fresh scanner save is visible immediately.
+pub async fn load_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
+    let ttl = Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
+
+    {
+        let cache = data_usage_snapshot_cache().read().await;
+        if let Some(result) = fresh_cached_data_usage_snapshot(&cache, tokio::time::Instant::now(), ttl) {
+            return result;
+        }
+    }
+
+    // Re-check under the write lock so concurrent expirations trigger a single
+    // backend read instead of a stampede.
+    let mut cache = data_usage_snapshot_cache().write().await;
+    if let Some(result) = fresh_cached_data_usage_snapshot(&cache, tokio::time::Instant::now(), ttl) {
+        return result;
+    }
+
+    let result = load_data_usage_from_backend(store).await;
+    cache_data_usage_snapshot_result(&mut cache, result, tokio::time::Instant::now())
 }
 
 /// Aggregate usage information from local disk snapshots.
@@ -530,6 +746,7 @@ impl BucketUsageAccumulator {
 type UsageVersionPage = StorageListObjectVersionsInfo<ObjectInfo>;
 
 pub async fn compute_bucket_usage(store: Arc<ECStore>, bucket_name: &str) -> Result<BucketUsageInfo, Error> {
+    LIVE_BUCKET_USAGE_COMPUTATIONS.fetch_add(1, Ordering::Relaxed);
     let bucket = bucket_name.to_string();
     compute_bucket_usage_with_pages(bucket_name, move |marker, version_marker| {
         let store = Arc::clone(&store);
@@ -923,7 +1140,7 @@ async fn update_usage_cache_if_needed() {
         let updating_clone = (*cache_updating()).clone();
         tokio::spawn(async move {
             if let Some(store) = runtime_sources::object_store_handle()
-                && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
+                && let Ok(data_usage_info) = load_data_usage_from_backend(store).await
             {
                 replace_bucket_usage_memory_from_info(&data_usage_info).await;
             }
@@ -947,7 +1164,7 @@ async fn update_usage_cache_if_needed() {
     drop(updating);
 
     if let Some(store) = runtime_sources::object_store_handle()
-        && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
+        && let Ok(data_usage_info) = load_data_usage_from_backend(store).await
     {
         replace_bucket_usage_memory_from_info(&data_usage_info).await;
     }
@@ -1372,6 +1589,250 @@ mod tests {
         info
     }
 
+    fn usage_with_last_update(last_update: Option<SystemTime>) -> DataUsageInfo {
+        DataUsageInfo {
+            last_update,
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_allows_newer_incoming() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let incoming = usage_with_last_update(Some(now));
+        let existing = usage_with_last_update(Some(now - Duration::from_secs(60)));
+        assert_eq!(stale_data_usage_persist_reason(&incoming, &existing, now), None);
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_skips_older_or_equal_incoming() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing = usage_with_last_update(Some(now - Duration::from_secs(60)));
+
+        let older = usage_with_last_update(Some(now - Duration::from_secs(120)));
+        assert_eq!(
+            stale_data_usage_persist_reason(&older, &existing, now),
+            Some("older_or_equal_last_update")
+        );
+
+        let equal = usage_with_last_update(existing.last_update);
+        assert_eq!(
+            stale_data_usage_persist_reason(&equal, &existing, now),
+            Some("older_or_equal_last_update")
+        );
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_allows_save_when_existing_is_future_dated() {
+        // Existing snapshot timestamp beyond the clock tolerance is untrustworthy
+        // (clock step-back / slower-clock leader): the save must be allowed even
+        // though incoming <= existing, otherwise usage stats freeze forever.
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing =
+            usage_with_last_update(Some(now + rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE + Duration::from_secs(1)));
+        let incoming = usage_with_last_update(Some(now));
+        assert_eq!(stale_data_usage_persist_reason(&incoming, &existing, now), None);
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_skips_at_exact_tolerance_boundary() {
+        // Exactly at now + tolerance is still within the trusted window.
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing = usage_with_last_update(Some(now + rustfs_data_usage::USAGE_LAST_UPDATE_FUTURE_TOLERANCE));
+        let incoming = usage_with_last_update(Some(now));
+        assert_eq!(
+            stale_data_usage_persist_reason(&incoming, &existing, now),
+            Some("older_or_equal_last_update")
+        );
+    }
+
+    #[test]
+    fn stale_data_usage_persist_reason_preserves_none_handling() {
+        // This call site (unlike the scanner sibling) allows saves when either
+        // timestamp is missing — pin that behavior.
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+
+        let incoming_none = usage_with_last_update(None);
+        let existing_some = usage_with_last_update(Some(now - Duration::from_secs(60)));
+        assert_eq!(stale_data_usage_persist_reason(&incoming_none, &existing_some, now), None);
+
+        let incoming_some = usage_with_last_update(Some(now));
+        let existing_none = usage_with_last_update(None);
+        assert_eq!(stale_data_usage_persist_reason(&incoming_some, &existing_none, now), None);
+
+        let both_none = usage_with_last_update(None);
+        assert_eq!(stale_data_usage_persist_reason(&both_none, &usage_with_last_update(None), now), None);
+    }
+
+    #[test]
+    fn backup_stale_guard_allows_equal_snapshot_to_repair_primary() {
+        let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
+        let existing = usage_with_last_update(Some(now - Duration::from_secs(60)));
+        let equal = usage_with_last_update(existing.last_update);
+        let older = usage_with_last_update(Some(now - Duration::from_secs(120)));
+
+        assert_eq!(
+            stale_data_usage_persist_reason_for_source(&equal, &existing, UsageSnapshotSource::Backup, now),
+            None
+        );
+        assert_eq!(
+            stale_data_usage_persist_reason_for_source(&older, &existing, UsageSnapshotSource::Backup, now),
+            Some("older_or_equal_last_update")
+        );
+    }
+
+    fn snapshot_bytes(bucket: &str) -> Vec<u8> {
+        let info = data_usage_info_for_test(bucket, 3, 42, SystemTime::UNIX_EPOCH);
+        serde_json::to_vec(&info).expect("test snapshot must serialize")
+    }
+
+    #[test]
+    fn data_usage_backup_path_appends_suffix_to_primary() {
+        assert_eq!(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()));
+    }
+
+    fn assert_snapshot_bucket(info: &DataUsageInfo, bucket: &str) {
+        assert!(
+            info.buckets_usage.contains_key(bucket),
+            "expected snapshot for bucket {bucket}, got buckets {:?}",
+            info.buckets_usage.keys().collect::<Vec<_>>()
+        );
+    }
+
+    #[test]
+    fn parse_snapshot_error_does_not_include_payload_value() {
+        let err =
+            parse_usage_snapshot(br#"{"buckets_count":"secret-marker"}"#).expect_err("an invalid field type must fail decoding");
+        assert!(!err.to_string().contains("secret-marker"));
+    }
+
+    #[test]
+    fn cached_snapshot_failure_is_reused_until_ttl_expires() {
+        let loaded_at = tokio::time::Instant::now();
+        let mut cache = None;
+
+        let first = cache_data_usage_snapshot_result(&mut cache, Err(Error::ErasureReadQuorum), loaded_at);
+        assert!(matches!(first, Err(Error::ErasureReadQuorum)));
+
+        let cached = fresh_cached_data_usage_snapshot(&cache, loaded_at + Duration::from_secs(1), Duration::from_secs(30))
+            .expect("the failed load must remain cached within the TTL");
+        assert!(cached.is_err());
+
+        assert!(fresh_cached_data_usage_snapshot(&cache, loaded_at + Duration::from_secs(30), Duration::from_secs(30)).is_none());
+    }
+
+    #[test]
+    fn cached_snapshot_success_is_reused_until_ttl_expires() {
+        let loaded_at = tokio::time::Instant::now();
+        let expected = data_usage_info_for_test("bucket", 3, 42, SystemTime::UNIX_EPOCH);
+        let mut cache = None;
+
+        let first =
+            cache_data_usage_snapshot_result(&mut cache, Ok(expected), loaded_at).expect("successful load must be returned");
+        assert_snapshot_bucket(&first, "bucket");
+
+        let cached = fresh_cached_data_usage_snapshot(&cache, loaded_at + Duration::from_secs(1), Duration::from_secs(30))
+            .expect("successful load must remain cached within the TTL")
+            .expect("cached success must remain successful");
+        assert_snapshot_bucket(&cached, "bucket");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_ok_is_used_without_backup_read() {
+        let backup_read = Arc::new(std::sync::atomic::AtomicBool::new(false));
+        let backup_read_probe = Arc::clone(&backup_read);
+
+        let info = resolve_loaded_snapshot(Ok(snapshot_bytes("primary-bucket")), async move {
+            backup_read_probe.store(true, std::sync::atomic::Ordering::SeqCst);
+            Ok(snapshot_bytes("backup-bucket"))
+        })
+        .await
+        .expect("healthy primary snapshot must load");
+
+        assert_snapshot_bucket(&info, "primary-bucket");
+        assert!(
+            !backup_read.load(std::sync::atomic::Ordering::SeqCst),
+            "backup must not be read when the primary parses"
+        );
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_corrupt_falls_back_to_backup() {
+        let (info, source) =
+            resolve_loaded_snapshot_with_source(Ok(b"{not json".to_vec()), async { Ok(snapshot_bytes("backup-bucket")) })
+                .await
+                .expect("backup snapshot must be served when the primary is corrupt");
+
+        assert_snapshot_bucket(&info, "backup-bucket");
+        assert_eq!(source, UsageSnapshotSource::Backup);
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_corrupt_backup_corrupt_is_error() {
+        let err = resolve_loaded_snapshot(Ok(b"{not json".to_vec()), async { Ok(b"also not json".to_vec()) })
+            .await
+            .expect_err("two corrupt snapshots must not produce stats");
+        assert!(err.to_string().contains("deserialize"), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_corrupt_backup_missing_is_error() {
+        let err = resolve_loaded_snapshot(Ok(b"{not json".to_vec()), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect_err("a corrupt primary without a backup must not produce stats");
+        assert!(err.to_string().contains("deserialize"), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_empty_backup_missing_is_error() {
+        let err = resolve_loaded_snapshot(Ok(Vec::new()), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect_err("an empty primary object is corrupt, not an absent snapshot");
+        assert!(err.to_string().contains("deserialize"), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_read_error_falls_back_to_backup() {
+        let info = resolve_loaded_snapshot(Err(Error::ErasureReadQuorum), async { Ok(snapshot_bytes("backup-bucket")) })
+            .await
+            .expect("backup snapshot must be served when the primary read fails");
+
+        assert_snapshot_bucket(&info, "backup-bucket");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_real_error_backup_missing_is_error_not_zeros() {
+        let err = resolve_loaded_snapshot(Err(Error::ErasureReadQuorum), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect_err("primary corruption must not be rendered as all-zero stats");
+        assert!(matches!(err, Error::ErasureReadQuorum), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_primary_missing_uses_backup() {
+        let info = resolve_loaded_snapshot(Err(Error::ConfigNotFound), async { Ok(snapshot_bytes("backup-bucket")) })
+            .await
+            .expect("an existing backup must be used when the primary is missing");
+        assert_snapshot_bucket(&info, "backup-bucket");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_backup_read_error_is_preserved() {
+        let err = resolve_loaded_snapshot(Err(Error::ConfigNotFound), async { Err(Error::ErasureReadQuorum) })
+            .await
+            .expect_err("a backup read failure must be returned");
+        assert!(matches!(err, Error::ErasureReadQuorum), "unexpected error: {err}");
+    }
+
+    #[tokio::test]
+    async fn resolve_snapshot_both_missing_is_default() {
+        let info = resolve_loaded_snapshot(Err(Error::ConfigNotFound), async { Err(Error::ConfigNotFound) })
+            .await
+            .expect("no snapshot yet must resolve to empty stats");
+        assert_eq!(info.buckets_count, 0);
+        assert!(info.buckets_usage.is_empty());
+    }
+
     #[tokio::test]
     async fn compute_usage_preserves_same_object_across_1000_entry_page_boundary() {
         let first_page = UsageVersionPage {

crates/ecstore/src/diagnostics/admin_server_info.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use crate::cluster::rpc::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
-use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend};
+use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
 use crate::error::{Error, Result};
 use crate::{disk::endpoint::Endpoint, runtime::sources as runtime_sources};
 
@@ -42,6 +42,33 @@ use shadow_rs::shadow;
 shadow!(build);
 
 const SERVER_PING_TIMEOUT: Duration = Duration::from_secs(1);
+const DATA_USAGE_UNAVAILABLE_ERROR: &str = "data usage snapshot unavailable";
+
+fn apply_data_usage_result(
+    result: Result<rustfs_data_usage::DataUsageInfo>,
+    buckets: &mut rustfs_madmin::Buckets,
+    objects: &mut rustfs_madmin::Objects,
+    versions: &mut rustfs_madmin::Versions,
+    delete_markers: &mut rustfs_madmin::DeleteMarkers,
+    usage: &mut rustfs_madmin::Usage,
+) {
+    match result {
+        Ok(info) => {
+            buckets.count = info.buckets_count;
+            objects.count = info.objects_total_count;
+            versions.count = info.versions_total_count;
+            delete_markers.count = info.delete_markers_total_count;
+            usage.size = info.objects_total_size;
+        }
+        Err(_) => {
+            buckets.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            objects.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            versions.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            delete_markers.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+            usage.error = Some(DATA_USAGE_UNAVAILABLE_ERROR.to_string());
+        }
+    }
+}
 
 // pub const ITEM_OFFLINE: &str = "offline";
 // pub const ITEM_INITIALIZING: &str = "initializing";
@@ -246,22 +273,14 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
 
     if let Some(store) = runtime_sources::object_store_handle() {
         mode = ITEM_ONLINE;
-        match load_data_usage_from_backend(store.clone()).await {
-            Ok(res) => {
-                buckets.count = res.buckets_count;
-                objects.count = res.objects_total_count;
-                versions.count = res.versions_total_count;
-                delete_markers.count = res.delete_markers_total_count;
-                usage.size = res.objects_total_size;
-            }
-            Err(err) => {
-                buckets.error = Some(err.to_string());
-                objects.error = Some(err.to_string());
-                versions.error = Some(err.to_string());
-                delete_markers.error = Some(err.to_string());
-                usage.error = Some(err.to_string());
-            }
-        }
+        apply_data_usage_result(
+            load_data_usage_from_backend_cached(store.clone()).await,
+            &mut buckets,
+            &mut objects,
+            &mut versions,
+            &mut delete_markers,
+            &mut usage,
+        );
 
         let after3 = OffsetDateTime::now_utc();
 
@@ -433,7 +452,10 @@ mod tests {
     use crate::runtime::sources as runtime_sources;
     use rustfs_madmin::{Disk, ITEM_OFFLINE, ITEM_UNKNOWN};
 
-    use super::{get_local_server_property, get_online_offline_disks_stats, get_server_info};
+    use super::{
+        DATA_USAGE_UNAVAILABLE_ERROR, apply_data_usage_result, get_local_server_property, get_online_offline_disks_stats,
+        get_server_info,
+    };
 
     fn disk_with_state(endpoint: &str, state: &str) -> Disk {
         Disk {
@@ -471,6 +493,55 @@ mod tests {
         );
     }
 
+    #[test]
+    fn data_usage_errors_are_sanitized_in_server_info() {
+        let mut buckets = rustfs_madmin::Buckets::default();
+        let mut objects = rustfs_madmin::Objects::default();
+        let mut versions = rustfs_madmin::Versions::default();
+        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
+        let mut usage = rustfs_madmin::Usage::default();
+
+        apply_data_usage_result(
+            Err(crate::error::Error::other("sensitive disk path")),
+            &mut buckets,
+            &mut objects,
+            &mut versions,
+            &mut delete_markers,
+            &mut usage,
+        );
+
+        assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+        assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
+    }
+
+    #[test]
+    fn data_usage_counts_are_mapped_into_server_info() {
+        let mut buckets = rustfs_madmin::Buckets::default();
+        let mut objects = rustfs_madmin::Objects::default();
+        let mut versions = rustfs_madmin::Versions::default();
+        let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
+        let mut usage = rustfs_madmin::Usage::default();
+        let info = rustfs_data_usage::DataUsageInfo {
+            buckets_count: 2,
+            objects_total_count: 3,
+            versions_total_count: 4,
+            delete_markers_total_count: 5,
+            objects_total_size: 6,
+            ..Default::default()
+        };
+
+        apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
+
+        assert_eq!(buckets.count, 2);
+        assert_eq!(objects.count, 3);
+        assert_eq!(versions.count, 4);
+        assert_eq!(delete_markers.count, 5);
+        assert_eq!(usage.size, 6);
+    }
+
     #[serial]
     #[tokio::test]
     async fn server_info_includes_global_deployment_id() {

crates/ecstore/src/diagnostics/get.rs

@@ -137,6 +137,7 @@ pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_fou
 pub(crate) const GET_METADATA_CACHE_REASON_NOT_READ_DATA: &str = "not_read_data";
 pub(crate) const GET_METADATA_CACHE_REASON_PART_NUMBER: &str = "part_number";
 pub(crate) const GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ: &str = "raw_data_movement_read";
+pub(crate) const GET_METADATA_CACHE_REASON_STALE_PUBLICATION: &str = "stale_publication";
 pub(crate) const GET_METADATA_CACHE_REASON_USABLE: &str = "usable";
 pub(crate) const GET_METADATA_CACHE_REASON_VERSION_ID: &str = "version_id";
 pub(crate) const GET_METADATA_CACHE_REASON_VERSION_SUSPENDED: &str = "version_suspended";
@@ -386,6 +387,7 @@ mod tests {
         assert_eq!(GET_METADATA_CACHE_REASON_NOT_READ_DATA, "not_read_data");
         assert_eq!(GET_METADATA_CACHE_REASON_PART_NUMBER, "part_number");
         assert_eq!(GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ, "raw_data_movement_read");
+        assert_eq!(GET_METADATA_CACHE_REASON_STALE_PUBLICATION, "stale_publication");
         assert_eq!(GET_METADATA_CACHE_REASON_USABLE, "usable");
         assert_eq!(GET_METADATA_CACHE_REASON_VERSION_ID, "version_id");
         assert_eq!(GET_METADATA_CACHE_REASON_VERSION_SUSPENDED, "version_suspended");

crates/ecstore/src/disk/disk_store.rs

@@ -209,6 +209,16 @@ pub fn get_drive_walkdir_stall_timeout() -> Duration {
     )
 }
 
+pub fn get_drive_walkdir_peek_timeout() -> Duration {
+    let stall_timeout = get_drive_walkdir_stall_timeout();
+    let configured = get_drive_timeout_duration(
+        rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS,
+        rustfs_config::DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS.saturating_mul(2)),
+    );
+    configured.max(stall_timeout)
+}
+
 pub fn get_object_disk_read_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT,
@@ -1656,6 +1666,64 @@ mod tests {
         });
     }
 
+    #[test]
+    fn drive_walkdir_peek_timeout_uses_wider_default_when_unset() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                    temp_env::with_var_unset(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, || {
+                        assert_eq!(
+                            get_drive_walkdir_peek_timeout(),
+                            Duration::from_secs(rustfs_config::DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS)
+                        );
+                    });
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_walkdir_peek_timeout_is_never_stricter_than_stall_timeout() {
+        temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, Some("3"), || {
+            temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, Some("13"), || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                    assert_eq!(get_drive_walkdir_peek_timeout(), Duration::from_secs(13));
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_walkdir_peek_timeout_prefers_canonical_over_legacy() {
+        temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, Some("23"), || {
+            temp_env::with_var(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("17"), || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, || {
+                    assert_eq!(get_drive_walkdir_peek_timeout(), Duration::from_secs(23));
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_walkdir_peek_timeout_uses_high_latency_profile_default() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, || {
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                    temp_env::with_var(
+                        rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE,
+                        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY),
+                        || {
+                            assert_eq!(
+                                get_drive_walkdir_peek_timeout(),
+                                Duration::from_secs(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS * 2)
+                            );
+                        },
+                    );
+                });
+            });
+        });
+    }
+
     #[test]
     fn drive_walkdir_timeout_prefers_canonical_over_legacy() {
         temp_env::with_var(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("11"), || {

crates/ecstore/src/disk/error.rs

@@ -141,7 +141,7 @@ pub enum DiskError {
     ErasureReadQuorum,
 
     #[error("io error {0}")]
-    Io(io::Error),
+    Io(#[source] io::Error),
 
     #[error("source stalled")]
     SourceStalled,
@@ -153,6 +153,12 @@ pub enum DiskError {
     InvalidPath,
 }
 
+impl From<crate::erasure::coding::ErasureConstructionError> for DiskError {
+    fn from(error: crate::erasure::coding::ErasureConstructionError) -> Self {
+        Self::Io(error.into_io_error())
+    }
+}
+
 impl DiskError {
     pub fn other<E>(error: E) -> Self
     where
@@ -601,6 +607,26 @@ mod tests {
     use super::*;
     use std::collections::HashMap;
 
+    #[test]
+    fn other_preserves_erasure_construction_source_chain() {
+        use crate::erasure::coding::ErasureConstructionError;
+        use std::error::Error as _;
+
+        let error = DiskError::from(ErasureConstructionError::ModernEncoder {
+            source: reed_solomon_erasure::Error::TooManyShards,
+        });
+        let io_source = error.source().expect("DiskError::Io must expose its io::Error source");
+        assert!(io_source.is::<io::Error>());
+        let construction_source = io_source
+            .source()
+            .expect("io::Error must expose the erasure construction error");
+        assert!(construction_source.is::<ErasureConstructionError>());
+        let encoder_source = construction_source
+            .source()
+            .expect("construction error must expose the encoder error");
+        assert!(encoder_source.is::<reed_solomon_erasure::Error>());
+    }
+
     #[test]
     fn test_disk_error_variants() {
         let errors = vec![

crates/ecstore/src/disk/os.rs

@@ -645,6 +645,34 @@ pub fn file_exists(path: impl AsRef<Path>) -> bool {
     std::fs::metadata(path.as_ref()).map(|_| true).unwrap_or(false)
 }
 
+/// Whether an [`io::Error`] means "the directory is not empty".
+///
+/// POSIX lets `rmdir`/`rename` report a non-empty directory as either
+/// `ENOTEMPTY` or `EEXIST`. Linux uses `ENOTEMPTY` (which Rust surfaces as
+/// [`io::ErrorKind::DirectoryNotEmpty`]); illumos/Solaris return `EEXIST`
+/// (errno 17), which Rust surfaces as [`io::ErrorKind::AlreadyExists`] and
+/// which the `DirectoryNotEmpty` kind therefore never catches. Matching only on
+/// the kind silently misclassifies the Solaris case as a hard failure, so
+/// callers that must treat a still-populated directory as benign (deleting the
+/// object metadata while a rollback-staging dir remains, non-force
+/// `DeleteBucket` on a populated bucket) have to test the raw errno as well.
+/// Mirrors MinIO's `isSysErrNotEmpty`.
+pub fn is_dir_not_empty_error(err: &io::Error) -> bool {
+    // Linux/Windows: ENOTEMPTY / ERROR_DIR_NOT_EMPTY -> DirectoryNotEmpty.
+    if err.kind() == io::ErrorKind::DirectoryNotEmpty {
+        return true;
+    }
+    // illumos/Solaris report a non-empty `rmdir`/`rename` as EEXIST (errno 17),
+    // which Rust surfaces as `AlreadyExists` (so the `DirectoryNotEmpty` kind
+    // never catches it). Confirm against the raw errno directly so the
+    // classification holds regardless of how the platform std maps it.
+    #[cfg(unix)]
+    if matches!(err.raw_os_error(), Some(libc::ENOTEMPTY) | Some(libc::EEXIST)) {
+        return true;
+    }
+    false
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -809,6 +837,62 @@ mod tests {
         assert!(!should_retry_rename(&denied, 1));
     }
 
+    #[test]
+    fn is_dir_not_empty_error_recognizes_directory_not_empty_kind() {
+        let err = io::Error::from(io::ErrorKind::DirectoryNotEmpty);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn is_dir_not_empty_error_recognizes_raw_enotempty() {
+        // Linux/BSD/macOS non-empty rmdir/rename errno.
+        let err = io::Error::from_raw_os_error(libc::ENOTEMPTY);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn is_dir_not_empty_error_recognizes_solaris_eexist() {
+        // illumos/Solaris report a non-empty rmdir/rename as EEXIST, which Rust
+        // surfaces as `AlreadyExists` (never `DirectoryNotEmpty`). This is the
+        // core of rustfs/rustfs#4978: matching only the kind misclassified this
+        // benign condition as a hard failure.
+        let err = io::Error::from_raw_os_error(libc::EEXIST);
+        assert_eq!(err.kind(), io::ErrorKind::AlreadyExists);
+        assert!(is_dir_not_empty_error(&err));
+    }
+
+    #[test]
+    fn is_dir_not_empty_error_rejects_unrelated_errors() {
+        assert!(!is_dir_not_empty_error(&io::Error::from(io::ErrorKind::NotFound)));
+        assert!(!is_dir_not_empty_error(&io::Error::from(io::ErrorKind::PermissionDenied)));
+        #[cfg(unix)]
+        {
+            assert!(!is_dir_not_empty_error(&io::Error::from_raw_os_error(libc::EACCES)));
+            assert!(!is_dir_not_empty_error(&io::Error::from_raw_os_error(libc::ENOENT)));
+        }
+    }
+
+    #[tokio::test]
+    async fn is_dir_not_empty_error_matches_real_non_empty_rmdir() {
+        // Validate against the host's actual errno, whatever it is: Linux/macOS
+        // return ENOTEMPTY, illumos/Solaris return EEXIST. The removal must be
+        // classified as "not empty" on every platform.
+        let temp_dir = tempdir().expect("create temp dir");
+        let populated = temp_dir.path().join("populated");
+        std::fs::create_dir(&populated).expect("create dir");
+        std::fs::write(populated.join("child"), b"x").expect("write child");
+
+        let err = std::fs::remove_dir(&populated).expect_err("non-empty rmdir must fail");
+        assert!(
+            is_dir_not_empty_error(&err),
+            "non-empty rmdir must classify as not-empty, got kind {:?} errno {:?}",
+            err.kind(),
+            err.raw_os_error()
+        );
+    }
+
     #[tokio::test]
     async fn rename_all_moves_existing_directory_tree() {
         // Guards the rename_data commit path, which funnels through

crates/ecstore/src/erasure/coding/encode.rs

@@ -841,6 +841,12 @@ mod tests {
     use std::time::Duration;
     use tokio::io::{AsyncWrite, AsyncWriteExt};
 
+    fn erasure_with_zero_block_size() -> Erasure {
+        let mut erasure = Erasure::default();
+        erasure.data_shards = 1;
+        erasure
+    }
+
     #[derive(Clone, Default)]
     struct DeferredCommitWriter {
         buffered: Vec<u8>,
@@ -1500,7 +1506,7 @@ mod tests {
             HashAlgorithm::HighwayHash256S,
         ))];
 
-        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let erasure = Arc::new(erasure_with_zero_block_size());
         let reader = tokio::io::BufReader::new(Cursor::new(b"payload".to_vec()));
         let err = erasure
             .encode(reader, &mut writers, 1)
@@ -1660,7 +1666,7 @@ mod tests {
         let writer = DeferredCommitWriter::new(committed);
         let mut writers = vec![Some(bitrot_writer(writer, 16))];
 
-        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let erasure = Arc::new(erasure_with_zero_block_size());
         let reader = tokio::io::BufReader::new(Cursor::new(b"payload".to_vec()));
         let err = erasure
             .encode_batched(reader, &mut writers, 1)

crates/ecstore/src/erasure/coding/erasure.rs

@@ -25,6 +25,62 @@ use tokio::io::AsyncRead;
 use tracing::warn;
 use uuid::Uuid;
 
+const MODERN_MAX_TOTAL_SHARDS: usize = <reed_solomon_erasure::galois_8::Field as reed_solomon_erasure::Field>::ORDER;
+
+/// Errors returned when constructing an [`Erasure`] codec.
+#[derive(Debug, thiserror::Error)]
+pub enum ErasureConstructionError {
+    /// Erasure coding requires at least one data shard.
+    #[error("data_shards must be greater than zero")]
+    ZeroDataShards,
+
+    /// Erasure coding requires a non-zero logical block size.
+    #[error("block_size must be greater than zero")]
+    ZeroBlockSize,
+
+    /// The total shard count cannot be represented by `usize`.
+    #[error("data_shards ({data_shards}) + parity_shards ({parity_shards}) overflows usize")]
+    ShardCountOverflow { data_shards: usize, parity_shards: usize },
+
+    /// The modern GF(2^8) codec cannot represent this shard configuration.
+    #[error("modern codec does not support {data_shards} data shards and {parity_shards} parity shards")]
+    UnsupportedModernShardCount { data_shards: usize, parity_shards: usize },
+
+    /// The legacy SIMD codec cannot represent this shard configuration.
+    #[error("legacy codec does not support {data_shards} data shards and {parity_shards} parity shards")]
+    UnsupportedLegacyShardCount { data_shards: usize, parity_shards: usize },
+
+    /// The modern encoder rejected dimensions that passed the preflight checks.
+    #[error("failed to construct modern Reed-Solomon encoder")]
+    ModernEncoder {
+        #[source]
+        source: reed_solomon_erasure::Error,
+    },
+
+    /// The legacy encoder wrapper failed to initialize.
+    #[error("failed to construct legacy Reed-Solomon encoder")]
+    LegacyEncoder {
+        #[source]
+        source: io::Error,
+    },
+}
+
+#[derive(Debug, thiserror::Error)]
+#[error("{source}")]
+struct ErasureConstructionIoSource {
+    #[source]
+    source: ErasureConstructionError,
+}
+
+impl ErasureConstructionError {
+    pub(crate) fn into_io_error(self) -> io::Error {
+        // `io::Error::source` forwards to the wrapped error's source rather
+        // than exposing the wrapped error itself. This adapter keeps the
+        // construction error as an observable link in the source chain.
+        io::Error::other(ErasureConstructionIoSource { source: self })
+    }
+}
+
 /// Legacy calc_shard_size formula: (block_size.div_ceil(data_shards) + 1) & !1
 /// Matches main branch and filemeta::ErasureInfo for old-version files.
 pub fn calc_shard_size_legacy(block_size: usize, data_shards: usize) -> usize {
@@ -233,12 +289,9 @@ impl Clone for ReedSolomonEncoder {
 }
 
 impl ReedSolomonEncoder {
-    /// Create a new Reed-Solomon encoder with specified data and parity shards.
-    pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
+    fn try_new_typed(data_shards: usize, parity_shards: usize) -> Result<Self, reed_solomon_erasure::Error> {
         let encoder = if parity_shards > 0 {
-            ReedSolomon::new(data_shards, parity_shards)
-                .map_err(|e| io::Error::other(format!("Failed to create Reed-Solomon encoder: {e:?}")))
-                .map(Some)?
+            Some(ReedSolomon::new(data_shards, parity_shards)?)
         } else {
             None
         };
@@ -250,6 +303,12 @@ impl ReedSolomonEncoder {
         })
     }
 
+    /// Create a new Reed-Solomon encoder with specified data and parity shards.
+    pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
+        Self::try_new_typed(data_shards, parity_shards)
+            .map_err(|error| io::Error::other(format!("Failed to create Reed-Solomon encoder: {error:?}")))
+    }
+
     /// Encode data shards with parity.
     pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
         let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
@@ -464,28 +523,105 @@ impl Erasure {
     /// * `data_shards` - Number of data shards.
     /// * `parity_shards` - Number of parity shards.
     /// * `block_size` - Block size for each shard.
+    ///
+    /// # Panics
+    /// Panics when the dimensions are invalid or unsupported. RustFS production
+    /// paths use [`Self::try_new`] instead.
     pub fn new(data_shards: usize, parity_shards: usize, block_size: usize) -> Self {
-        Self::new_with_options(data_shards, parity_shards, block_size, false)
+        match Self::try_new(data_shards, parity_shards, block_size) {
+            Ok(erasure) => erasure,
+            Err(error) => panic!("invalid erasure codec configuration: {error}"),
+        }
     }
 
     /// Create a new Erasure instance with legacy format support.
     ///
     /// When `uses_legacy` is true, uses main-branch shard_size formula and reed-solomon-simd
     /// for decode/reconstruct (for reading and healing old-version files).
+    ///
+    /// # Panics
+    /// Panics when the dimensions are invalid or unsupported. RustFS production
+    /// paths use [`Self::try_new_with_options`] instead.
     pub fn new_with_options(data_shards: usize, parity_shards: usize, block_size: usize, uses_legacy: bool) -> Self {
+        match Self::try_new_with_options(data_shards, parity_shards, block_size, uses_legacy) {
+            Ok(erasure) => erasure,
+            Err(error) => panic!("invalid erasure codec configuration: {error}"),
+        }
+    }
+
+    /// Tries to create a modern Erasure codec.
+    ///
+    /// # Errors
+    /// Returns [`ErasureConstructionError`] when dimensions are zero,
+    /// overflow the shard count, exceed the modern codec's supported range, or
+    /// the underlying encoder rejects the configuration.
+    pub fn try_new(data_shards: usize, parity_shards: usize, block_size: usize) -> Result<Self, ErasureConstructionError> {
+        Self::try_new_with_options(data_shards, parity_shards, block_size, false)
+    }
+
+    /// Tries to create an Erasure codec using the selected format backend.
+    ///
+    /// # Errors
+    /// Returns [`ErasureConstructionError`] when dimensions are zero,
+    /// overflow the shard count, are unsupported by the selected codec, or the
+    /// selected encoder fails to initialize.
+    pub fn try_new_with_options(
+        data_shards: usize,
+        parity_shards: usize,
+        block_size: usize,
+        uses_legacy: bool,
+    ) -> Result<Self, ErasureConstructionError> {
+        if data_shards == 0 {
+            return Err(ErasureConstructionError::ZeroDataShards);
+        }
+        if block_size == 0 {
+            return Err(ErasureConstructionError::ZeroBlockSize);
+        }
+
+        let total_shards = data_shards
+            .checked_add(parity_shards)
+            .ok_or(ErasureConstructionError::ShardCountOverflow {
+                data_shards,
+                parity_shards,
+            })?;
+
+        if parity_shards > 0 {
+            if uses_legacy
+                && (total_shards > reed_solomon_simd::engine::GF_ORDER
+                    || !reed_solomon_simd::ReedSolomonEncoder::supports(data_shards, parity_shards))
+            {
+                return Err(ErasureConstructionError::UnsupportedLegacyShardCount {
+                    data_shards,
+                    parity_shards,
+                });
+            }
+            if !uses_legacy && total_shards > MODERN_MAX_TOTAL_SHARDS {
+                return Err(ErasureConstructionError::UnsupportedModernShardCount {
+                    data_shards,
+                    parity_shards,
+                });
+            }
+        }
+
         let encoder = if !uses_legacy && parity_shards > 0 {
-            Some(ReedSolomonEncoder::new(data_shards, parity_shards).expect("operation should succeed"))
+            Some(
+                ReedSolomonEncoder::try_new_typed(data_shards, parity_shards)
+                    .map_err(|source| ErasureConstructionError::ModernEncoder { source })?,
+            )
         } else {
             None
         };
 
         let legacy_encoder = if uses_legacy && parity_shards > 0 {
-            Some(LegacyReedSolomonEncoder::new(data_shards, parity_shards).expect("operation should succeed"))
+            Some(
+                LegacyReedSolomonEncoder::new(data_shards, parity_shards)
+                    .map_err(|source| ErasureConstructionError::LegacyEncoder { source })?,
+            )
         } else {
             None
         };
 
-        Erasure {
+        Ok(Erasure {
             data_shards,
             parity_shards,
             block_size,
@@ -493,7 +629,7 @@ impl Erasure {
             legacy_encoder,
             uses_legacy,
             _id: Uuid::nil(), // Unused in hot paths; avoid CSPRNG syscall
-        }
+        })
     }
 
     /// Encode data into data and parity shards.
@@ -946,6 +1082,15 @@ mod tests {
     use std::task::{Context, Poll};
     use tokio::io::ReadBuf;
 
+    fn erasure_with_invalid_dimensions(data_shards: usize, parity_shards: usize, block_size: usize) -> Erasure {
+        Erasure {
+            data_shards,
+            parity_shards,
+            block_size,
+            ..Default::default()
+        }
+    }
+
     fn optional_shards(shards: &[Bytes]) -> Vec<Option<Vec<u8>>> {
         shards.iter().map(|shard| Some(shard.to_vec())).collect()
     }
@@ -1010,6 +1155,156 @@ mod tests {
         }
     }
 
+    #[test]
+    fn try_new_rejects_zero_and_overflowing_dimensions_with_typed_errors() {
+        let Err(zero_data) = Erasure::try_new(0, 2, 64) else {
+            panic!("zero data shards must be rejected");
+        };
+        assert!(matches!(zero_data, ErasureConstructionError::ZeroDataShards));
+
+        let Err(zero_block) = Erasure::try_new(2, 2, 0) else {
+            panic!("zero block size must be rejected");
+        };
+        assert!(matches!(zero_block, ErasureConstructionError::ZeroBlockSize));
+
+        let Err(overflow) = Erasure::try_new(usize::MAX, 1, 64) else {
+            panic!("overflowing shard counts must be rejected");
+        };
+        assert!(matches!(
+            overflow,
+            ErasureConstructionError::ShardCountOverflow {
+                data_shards: usize::MAX,
+                parity_shards: 1,
+            }
+        ));
+    }
+
+    #[test]
+    fn try_new_rejects_backend_specific_unsupported_dimensions() {
+        let Err(modern) = Erasure::try_new(256, 1, 64) else {
+            panic!("modern dimensions beyond GF(2^8) must be rejected");
+        };
+        assert!(matches!(
+            modern,
+            ErasureConstructionError::UnsupportedModernShardCount {
+                data_shards: 256,
+                parity_shards: 1,
+            }
+        ));
+
+        let Err(legacy) = Erasure::try_new_with_options(60_000, 5_000, 64, true) else {
+            panic!("unsupported legacy SIMD dimensions must be rejected");
+        };
+        assert!(matches!(
+            legacy,
+            ErasureConstructionError::UnsupportedLegacyShardCount {
+                data_shards: 60_000,
+                parity_shards: 5_000,
+            }
+        ));
+    }
+
+    #[test]
+    fn try_new_accepts_exact_backend_shard_limits() {
+        let modern_data_shards = MODERN_MAX_TOTAL_SHARDS - 1;
+        let modern =
+            Erasure::try_new(modern_data_shards, 1, 64).expect("the modern codec's exact field-size boundary must remain valid");
+        assert_eq!(modern.total_shard_count(), MODERN_MAX_TOTAL_SHARDS);
+        assert!(modern.encoder.is_some());
+        assert!(modern.legacy_encoder.is_none());
+
+        let legacy_data_shards = reed_solomon_simd::engine::GF_ORDER / 2;
+        let legacy_parity_shards = reed_solomon_simd::engine::GF_ORDER - legacy_data_shards;
+        let legacy = Erasure::try_new_with_options(legacy_data_shards, legacy_parity_shards, 64, true)
+            .expect("the legacy codec's exact field-size boundary must remain valid");
+        assert_eq!(legacy.total_shard_count(), reed_solomon_simd::engine::GF_ORDER);
+        assert!(legacy.encoder.is_none());
+        assert!(legacy.legacy_encoder.is_some());
+    }
+
+    #[test]
+    fn try_new_accepts_zero_parity_without_backend_limits() {
+        let no_parity = Erasure::try_new(2, 0, 64).expect("zero parity is a valid generic codec configuration");
+        assert_eq!(no_parity.total_shard_count(), 2);
+        assert!(no_parity.encoder.is_none());
+        assert!(no_parity.legacy_encoder.is_none());
+
+        let large_modern = Erasure::try_new(257, 0, 64).expect("zero parity must not inherit the modern backend shard limit");
+        assert_eq!(large_modern.total_shard_count(), 257);
+        assert!(large_modern.encoder.is_none());
+        assert!(large_modern.legacy_encoder.is_none());
+
+        let legacy_data_shards = reed_solomon_simd::engine::GF_ORDER + 1;
+        let large_legacy = Erasure::try_new_with_options(legacy_data_shards, 0, 64, true)
+            .expect("zero parity must not inherit the legacy backend shard limit");
+        assert_eq!(large_legacy.total_shard_count(), legacy_data_shards);
+        assert!(large_legacy.encoder.is_none());
+        assert!(large_legacy.legacy_encoder.is_none());
+    }
+
+    #[test]
+    fn try_new_accepts_generic_layouts_above_storage_drive_limits() {
+        let erasure = Erasure::try_new(2, 3, 64).expect("generic 2+3 erasure coding must remain supported");
+        assert_eq!(erasure.total_shard_count(), 5);
+        assert!(erasure.encoder.is_some());
+        assert!(erasure.legacy_encoder.is_none());
+
+        let modern = Erasure::try_new(9, 8, 64).expect("the codec must not impose the storage layer's drive limit");
+        assert_eq!(modern.total_shard_count(), 17);
+        assert!(modern.encoder.is_some());
+        assert!(modern.legacy_encoder.is_none());
+
+        let legacy = Erasure::try_new_with_options(9, 8, 64, true)
+            .expect("the legacy codec must not impose the storage layer's drive limit");
+        assert_eq!(legacy.total_shard_count(), 17);
+        assert!(legacy.encoder.is_none());
+        assert!(legacy.legacy_encoder.is_some());
+    }
+
+    #[test]
+    fn try_new_with_options_initializes_only_the_selected_backend() {
+        let modern = Erasure::try_new_with_options(4, 2, 64, false).expect("modern codec should construct");
+        assert!(modern.encoder.is_some());
+        assert!(modern.legacy_encoder.is_none());
+
+        let legacy = Erasure::try_new_with_options(4, 2, 64, true).expect("legacy codec should construct");
+        assert!(legacy.encoder.is_none());
+        assert!(legacy.legacy_encoder.is_some());
+    }
+
+    #[test]
+    fn construction_errors_preserve_encoder_sources() {
+        let modern = ErasureConstructionError::ModernEncoder {
+            source: reed_solomon_erasure::Error::TooManyShards,
+        };
+        assert!(
+            std::error::Error::source(&modern)
+                .and_then(|source| source.downcast_ref::<reed_solomon_erasure::Error>())
+                .is_some()
+        );
+
+        let legacy = ErasureConstructionError::LegacyEncoder {
+            source: io::Error::other("legacy encoder failure"),
+        };
+        assert!(
+            std::error::Error::source(&legacy)
+                .and_then(|source| source.downcast_ref::<io::Error>())
+                .is_some()
+        );
+    }
+
+    #[test]
+    #[should_panic(expected = "invalid erasure codec configuration")]
+    fn new_panics_on_invalid_dimensions() {
+        let _ = Erasure::new(0, 1, 64);
+    }
+
+    #[test]
+    #[should_panic(expected = "invalid erasure codec configuration")]
+    fn new_with_options_panics_on_invalid_dimensions() {
+        let _ = Erasure::new_with_options(1, 1, 0, true);
+    }
+
     #[test]
     fn has_valid_dimensions_rejects_zero_block_size_or_data_shards() {
         // Well-formed erasure metadata is accepted.
@@ -1018,11 +1313,9 @@ mod tests {
 
         // Corrupted on-disk metadata with a zero block_size or zero data_shards
         // must be rejected before it reaches the shard/offset divisions.
-        // (parity_shards is kept 0 for the zero-data_shards cases so the
-        // Reed-Solomon encoder is not constructed with an invalid shard count.)
-        assert!(!Erasure::new(4, 2, 0).has_valid_dimensions());
-        assert!(!Erasure::new(0, 0, 64).has_valid_dimensions());
-        assert!(!Erasure::new(0, 0, 0).has_valid_dimensions());
+        assert!(!erasure_with_invalid_dimensions(4, 2, 0).has_valid_dimensions());
+        assert!(!erasure_with_invalid_dimensions(0, 0, 64).has_valid_dimensions());
+        assert!(!erasure_with_invalid_dimensions(0, 0, 0).has_valid_dimensions());
     }
 
     #[tokio::test]
@@ -1762,7 +2055,7 @@ mod tests {
         use std::io::Cursor;
         use std::sync::{Arc, Mutex};
 
-        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let erasure = Arc::new(erasure_with_invalid_dimensions(1, 0, 0));
         let mut reader = Cursor::new(b"payload".to_vec());
         let observed = Arc::new(Mutex::new(None));
         let observed_clone = observed.clone();

crates/ecstore/src/erasure/coding/mod.rs

@@ -20,4 +20,4 @@ pub mod erasure;
 pub mod heal;
 pub use bitrot::*;
 
-pub use erasure::{Erasure, ReedSolomonEncoder, calc_shard_size, calc_shard_size_legacy};
+pub use erasure::{Erasure, ErasureConstructionError, ReedSolomonEncoder, calc_shard_size, calc_shard_size_legacy};

crates/ecstore/src/error/mod.rs

@@ -215,11 +215,17 @@ pub enum StorageError {
     #[error("method not allowed")]
     MethodNotAllowed,
     #[error("Io error: {0}")]
-    Io(std::io::Error),
+    Io(#[source] std::io::Error),
     #[error("Lock error: {0}")]
     Lock(#[from] rustfs_lock::LockError),
 }
 
+impl From<crate::erasure::coding::ErasureConstructionError> for StorageError {
+    fn from(error: crate::erasure::coding::ErasureConstructionError) -> Self {
+        Self::Io(error.into_io_error())
+    }
+}
+
 impl StorageError {
     pub fn other<E>(error: E) -> Self
     where
@@ -1176,6 +1182,26 @@ mod tests {
     use super::*;
     use std::io::{Error as IoError, ErrorKind};
 
+    #[test]
+    fn other_preserves_erasure_construction_source_chain() {
+        use crate::erasure::coding::ErasureConstructionError;
+        use std::error::Error as _;
+
+        let error = StorageError::from(ErasureConstructionError::ModernEncoder {
+            source: reed_solomon_erasure::Error::TooManyShards,
+        });
+        let io_source = error.source().expect("StorageError::Io must expose its io::Error source");
+        assert!(io_source.is::<std::io::Error>());
+        let construction_source = io_source
+            .source()
+            .expect("io::Error must expose the erasure construction error");
+        assert!(construction_source.is::<ErasureConstructionError>());
+        let encoder_source = construction_source
+            .source()
+            .expect("construction error must expose the encoder error");
+        assert!(encoder_source.is::<reed_solomon_erasure::Error>());
+    }
+
     // Regression for #952 (ECA-11): an all-`DiskNotFound` slice (every drive in
     // every set unreachable) must NOT be classified as "all not found",
     // otherwise ListObjects silently returns an empty listing and masks a full

crates/ecstore/src/object_api/body_cache_hook.rs

@@ -20,11 +20,17 @@
 //! probing earlier would require a second metadata fan-out, and probing later
 //! (after the reader is built) means a hit no longer saves any disk I/O.
 
-use crate::object_api::ObjectInfo;
 use crate::object_api::hook_slot::HookSlot;
+use crate::object_api::{ObjectInfo, ObjectOptions};
+use crate::storage_api_contracts::range::HTTPRangeSpec;
 use bytes::Bytes;
+use std::future::Future;
 use std::sync::Arc;
 
+tokio::task_local! {
+    static SKIP_GET_OBJECT_BODY_CACHE_HOOK: bool;
+}
+
 /// Serves full-object GET bodies from a cache keyed by object identity.
 ///
 /// Implementations must validate identity (etag/version/size) against the
@@ -62,14 +68,263 @@ pub fn register_get_object_body_cache_hook(hook: Arc<dyn GetObjectBodyCacheHook>
     );
 }
 
+/// Unregister the process-wide GET body cache hook.
+///
+/// Config reloads use this when body caching becomes disabled so an adapter
+/// retained by the previous configuration cannot continue serving stale hits.
+pub fn unregister_get_object_body_cache_hook() {
+    GET_OBJECT_BODY_CACHE_HOOK.clear();
+}
+
+/// Probes the registered hook against an already resolved metadata snapshot.
+/// Staged GET callers use this once before suppressing the nested reader probe,
+/// preserving the legacy hook contract.
+#[non_exhaustive]
+pub enum GetObjectBodyCacheHookLookup {
+    Ineligible,
+    Absent,
+    Miss,
+    Hit(Bytes),
+}
+
+/// Returns the complete plaintext length only when the request can safely use
+/// the body-cache hook. Callers may use this before conditional decisions so
+/// ineligible reads retain the established reader-path error precedence.
+pub fn get_object_body_cache_plaintext_len(
+    range: &Option<HTTPRangeSpec>,
+    opts: &ObjectOptions,
+    info: &ObjectInfo,
+) -> Option<i64> {
+    crate::set_disk::body_cache_plaintext_len(range, opts, info)
+}
+
+pub async fn lookup_get_object_body_cache_hook(
+    bucket: &str,
+    object: &str,
+    range: &Option<HTTPRangeSpec>,
+    opts: &ObjectOptions,
+    info: &ObjectInfo,
+) -> GetObjectBodyCacheHookLookup {
+    let Some(plaintext_len) = get_object_body_cache_plaintext_len(range, opts, info) else {
+        return GetObjectBodyCacheHookLookup::Ineligible;
+    };
+    let Some(hook) = get_object_body_cache_hook() else {
+        return GetObjectBodyCacheHookLookup::Absent;
+    };
+    match hook.lookup(bucket, object, info).await {
+        Some(body) if i64::try_from(body.len()).is_ok_and(|body_len| body_len == plaintext_len) => {
+            GetObjectBodyCacheHookLookup::Hit(body)
+        }
+        Some(_) | None => GetObjectBodyCacheHookLookup::Miss,
+    }
+}
+
 /// The registered hook, if any.
 pub(crate) fn get_object_body_cache_hook() -> Option<Arc<dyn GetObjectBodyCacheHook>> {
     GET_OBJECT_BODY_CACHE_HOOK.get()
 }
 
+pub(crate) fn get_object_body_cache_hook_suppressed() -> bool {
+    SKIP_GET_OBJECT_BODY_CACHE_HOOK.try_with(|skip| *skip).unwrap_or(false)
+}
+
+pub(crate) async fn without_get_object_body_cache_hook<F>(future: F) -> F::Output
+where
+    F: Future,
+{
+    SKIP_GET_OBJECT_BODY_CACHE_HOOK.scope(true, future).await
+}
+
 /// Test-only: unregister the hook so tests can register and clear the slot
 /// deterministically without leaking a hook into unrelated tests.
 #[cfg(test)]
 pub(crate) fn clear_get_object_body_cache_hook() {
-    GET_OBJECT_BODY_CACHE_HOOK.clear();
+    unregister_get_object_body_cache_hook();
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+
+    struct LegacyHook {
+        calls: AtomicUsize,
+        body: Option<Bytes>,
+    }
+
+    #[async_trait::async_trait]
+    impl GetObjectBodyCacheHook for LegacyHook {
+        async fn lookup(&self, _bucket: &str, _object: &str, _info: &ObjectInfo) -> Option<Bytes> {
+            self.calls.fetch_add(1, Ordering::Relaxed);
+            self.body.clone()
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn staged_probe_preserves_legacy_hook_hit_once() {
+        clear_get_object_body_cache_hook();
+        let hook = Arc::new(LegacyHook {
+            calls: AtomicUsize::new(0),
+            body: Some(Bytes::from_static(b"legacy")),
+        });
+        register_get_object_body_cache_hook(Arc::clone(&hook) as Arc<dyn GetObjectBodyCacheHook>);
+
+        let info = ObjectInfo {
+            size: 6,
+            actual_size: 6,
+            ..Default::default()
+        };
+        let GetObjectBodyCacheHookLookup::Hit(body) =
+            lookup_get_object_body_cache_hook("bucket", "object", &None, &ObjectOptions::default(), &info).await
+        else {
+            panic!("legacy hook hit must be returned");
+        };
+        assert_eq!(body, Bytes::from_static(b"legacy"));
+        assert_eq!(hook.calls.load(Ordering::Relaxed), 1);
+        clear_get_object_body_cache_hook();
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn staged_probe_treats_wrong_length_legacy_body_as_authoritative_miss() {
+        clear_get_object_body_cache_hook();
+        let hook = Arc::new(LegacyHook {
+            calls: AtomicUsize::new(0),
+            body: Some(Bytes::from_static(b"short")),
+        });
+        register_get_object_body_cache_hook(Arc::clone(&hook) as Arc<dyn GetObjectBodyCacheHook>);
+        let info = ObjectInfo {
+            size: 6,
+            actual_size: 6,
+            ..Default::default()
+        };
+
+        assert!(matches!(
+            lookup_get_object_body_cache_hook("bucket", "object", &None, &ObjectOptions::default(), &info).await,
+            GetObjectBodyCacheHookLookup::Miss
+        ));
+        assert_eq!(hook.calls.load(Ordering::Relaxed), 1);
+        clear_get_object_body_cache_hook();
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn staged_probe_preserves_legacy_hook_miss_once() {
+        clear_get_object_body_cache_hook();
+        let hook = Arc::new(LegacyHook {
+            calls: AtomicUsize::new(0),
+            body: None,
+        });
+        register_get_object_body_cache_hook(Arc::clone(&hook) as Arc<dyn GetObjectBodyCacheHook>);
+
+        let info = ObjectInfo {
+            size: 4,
+            actual_size: 4,
+            ..Default::default()
+        };
+        assert!(matches!(
+            lookup_get_object_body_cache_hook("bucket", "object", &None, &ObjectOptions::default(), &info).await,
+            GetObjectBodyCacheHookLookup::Miss
+        ));
+        assert_eq!(hook.calls.load(Ordering::Relaxed), 1);
+        clear_get_object_body_cache_hook();
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn staged_probe_bypasses_raw_movement_and_restore_reads() {
+        clear_get_object_body_cache_hook();
+        let hook = Arc::new(LegacyHook {
+            calls: AtomicUsize::new(0),
+            body: Some(Bytes::from_static(b"body")),
+        });
+        register_get_object_body_cache_hook(Arc::clone(&hook) as Arc<dyn GetObjectBodyCacheHook>);
+        let info = ObjectInfo {
+            size: 4,
+            actual_size: 4,
+            ..Default::default()
+        };
+        let mut restore = ObjectOptions::default();
+        restore.transition.restore_request.days = Some(1);
+        let cases = [
+            ObjectOptions {
+                raw_data_movement_read: true,
+                ..Default::default()
+            },
+            ObjectOptions {
+                data_movement: true,
+                ..Default::default()
+            },
+            restore,
+        ];
+
+        for opts in &cases {
+            assert!(matches!(
+                lookup_get_object_body_cache_hook("bucket", "object", &None, opts, &info).await,
+                GetObjectBodyCacheHookLookup::Ineligible
+            ));
+        }
+        assert_eq!(hook.calls.load(Ordering::Relaxed), 0);
+        clear_get_object_body_cache_hook();
+    }
+
+    #[tokio::test]
+    #[serial_test::serial(body_cache_hook)]
+    async fn staged_probe_bypasses_pre_hook_early_return_objects() {
+        clear_get_object_body_cache_hook();
+        let hook = Arc::new(LegacyHook {
+            calls: AtomicUsize::new(0),
+            body: Some(Bytes::from_static(b"body")),
+        });
+        register_get_object_body_cache_hook(Arc::clone(&hook) as Arc<dyn GetObjectBodyCacheHook>);
+        let delete_marker = ObjectInfo {
+            delete_marker: true,
+            size: 4,
+            actual_size: 4,
+            ..Default::default()
+        };
+        let zero = ObjectInfo::default();
+        let inline = ObjectInfo {
+            inlined: true,
+            size: 4,
+            actual_size: 4,
+            parts: Arc::new(vec![rustfs_filemeta::ObjectPartInfo {
+                number: 1,
+                ..Default::default()
+            }]),
+            ..Default::default()
+        };
+        let version_only = ObjectInfo {
+            version_only: true,
+            size: 4,
+            actual_size: 4,
+            ..Default::default()
+        };
+        let metadata_only = ObjectInfo {
+            metadata_only: true,
+            size: 4,
+            actual_size: 4,
+            ..Default::default()
+        };
+
+        for info in [&delete_marker, &zero, &inline, &version_only, &metadata_only] {
+            assert!(matches!(
+                lookup_get_object_body_cache_hook("bucket", "object", &None, &ObjectOptions::default(), info).await,
+                GetObjectBodyCacheHookLookup::Ineligible
+            ));
+        }
+        assert_eq!(hook.calls.load(Ordering::Relaxed), 0);
+        clear_get_object_body_cache_hook();
+    }
+
+    #[tokio::test]
+    async fn staged_reader_scope_suppresses_only_the_nested_probe() {
+        assert!(!get_object_body_cache_hook_suppressed());
+        without_get_object_body_cache_hook(async {
+            assert!(get_object_body_cache_hook_suppressed());
+        })
+        .await;
+        assert!(!get_object_body_cache_hook_suppressed());
+    }
 }

crates/ecstore/src/object_api/hook_slot.rs

@@ -61,8 +61,7 @@ impl<T: ?Sized> HookSlot<T> {
         self.inner.read().unwrap_or_else(|poisoned| poisoned.into_inner()).clone()
     }
 
-    /// Clears the slot. Test-only: production never unregisters a hook.
-    #[cfg(test)]
+    /// Clears the slot during feature disable or test cleanup.
     pub(crate) fn clear(&self) {
         *self.inner.write().unwrap_or_else(|poisoned| poisoned.into_inner()) = None;
     }

crates/ecstore/src/object_api/mod.rs

@@ -60,9 +60,14 @@ mod types;
 
 #[cfg(test)]
 pub(crate) use body_cache_hook::clear_get_object_body_cache_hook;
-pub(crate) use body_cache_hook::get_object_body_cache_hook;
-pub use body_cache_hook::{GetObjectBodyCacheHook, register_get_object_body_cache_hook};
+pub use body_cache_hook::{
+    GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
+    register_get_object_body_cache_hook, unregister_get_object_body_cache_hook,
+};
+pub(crate) use body_cache_hook::{
+    get_object_body_cache_hook, get_object_body_cache_hook_suppressed, without_get_object_body_cache_hook,
+};
 pub(crate) use object_mutation_hook::notify_object_mutation;
-pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook};
+pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook, unregister_object_mutation_hook};
 pub use readers::*;
 pub use types::*;

crates/ecstore/src/object_api/object_mutation_hook.rs

@@ -56,6 +56,14 @@ pub fn register_object_mutation_hook(hook: Arc<dyn ObjectMutationHook>) {
     );
 }
 
+/// Unregister the process-wide object mutation hook.
+///
+/// Config reloads use this when body caching becomes disabled so the previous
+/// adapter and its cached plaintext bodies are not retained until their TTL.
+pub fn unregister_object_mutation_hook() {
+    OBJECT_MUTATION_HOOK.clear();
+}
+
 /// The registered hook, if any.
 fn object_mutation_hook() -> Option<Arc<dyn ObjectMutationHook>> {
     OBJECT_MUTATION_HOOK.get()
@@ -71,13 +79,6 @@ pub(crate) async fn notify_object_mutation(bucket: &str, object: &str) {
     }
 }
 
-/// Test-only: unregister the hook so tests can register and clear the slot
-/// deterministically without leaking a hook into unrelated tests.
-#[cfg(test)]
-pub(crate) fn clear_object_mutation_hook() {
-    OBJECT_MUTATION_HOOK.clear();
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -97,7 +98,7 @@ mod tests {
     #[tokio::test]
     #[serial_test::serial(object_mutation_hook)]
     async fn notify_invokes_registered_hook_with_identity() {
-        clear_object_mutation_hook();
+        unregister_object_mutation_hook();
         let calls = Arc::new(Mutex::new(Vec::new()));
         register_object_mutation_hook(Arc::new(RecordingHook {
             calls: Arc::clone(&calls),
@@ -106,14 +107,35 @@ mod tests {
         notify_object_mutation("bucket", "photos/a.jpg").await;
 
         assert_eq!(&*calls.lock().unwrap(), &[("bucket".to_string(), "photos/a.jpg".to_string())]);
-        clear_object_mutation_hook();
+        unregister_object_mutation_hook();
     }
 
     #[tokio::test]
     #[serial_test::serial(object_mutation_hook)]
-    async fn notify_without_registered_hook_is_noop() {
-        clear_object_mutation_hook();
-        // Must not panic when no hook is installed (the cache feature is off).
+    async fn unregister_prevents_later_notifications() {
+        unregister_object_mutation_hook();
+        let calls = Arc::new(Mutex::new(Vec::new()));
+        register_object_mutation_hook(Arc::new(RecordingHook {
+            calls: Arc::clone(&calls),
+        }));
+        unregister_object_mutation_hook();
+
         notify_object_mutation("bucket", "object").await;
+
+        assert!(calls.lock().unwrap().is_empty(), "an unregistered hook must receive no mutation callback");
+    }
+
+    #[test]
+    #[serial_test::serial(object_mutation_hook)]
+    fn unregister_releases_the_previous_hook() {
+        unregister_object_mutation_hook();
+        let calls = Arc::new(Mutex::new(Vec::new()));
+        let hook = Arc::new(RecordingHook { calls });
+        let weak = Arc::downgrade(&hook);
+        register_object_mutation_hook(hook);
+
+        assert!(weak.upgrade().is_some());
+        unregister_object_mutation_hook();
+        assert!(weak.upgrade().is_none());
     }
 }

crates/ecstore/src/object_api/readers.rs

@@ -307,6 +307,17 @@ pub struct GetObjectReader {
 }
 
 impl GetObjectReader {
+    /// Builds a fully materialized reader from a cache-coordinated body.
+    pub fn from_cache_body(mut object_info: ObjectInfo, body: Bytes) -> Result<Self> {
+        object_info.size = i64::try_from(body.len()).map_err(|_| Error::other("cached GET body length exceeds i64::MAX"))?;
+        Ok(Self {
+            stream: Box::new(std::io::Cursor::new(body.clone())),
+            object_info,
+            buffered_body: Some(body),
+            body_source: GetObjectBodySource::HookServed,
+        })
+    }
+
     /// True when `buffered_body` is the body the cache hook served. The app
     /// layer serves it as the object-data-cache source without a second lookup.
     pub fn is_cache_hook_served(&self) -> bool {
@@ -1674,6 +1685,40 @@ mod tests {
         bytes
     }
 
+    #[tokio::test]
+    async fn cache_body_uses_plaintext_length_for_compressed_metadata() {
+        let mut metadata = HashMap::new();
+        rustfs_utils::http::insert_str(
+            &mut metadata,
+            rustfs_utils::http::SUFFIX_COMPRESSION,
+            "klauspost/compress/s2".to_string(),
+        );
+        let object_info = ObjectInfo {
+            size: 3,
+            actual_size: 11,
+            user_defined: Arc::new(metadata),
+            ..Default::default()
+        };
+        assert!(object_info.is_compressed());
+
+        let body = Bytes::from_static(b"hello world");
+        let mut reader =
+            GetObjectReader::from_cache_body(object_info, body.clone()).expect("cache body length must fit in object metadata");
+
+        assert_eq!(reader.body_source, GetObjectBodySource::HookServed);
+        assert_eq!(reader.buffered_body.as_ref(), Some(&body));
+        assert_eq!(reader.object_info.size, 11);
+        assert_eq!(reader.object_info.actual_size, 11);
+        assert!(reader.object_info.is_compressed());
+        let mut restored = Vec::new();
+        reader
+            .stream
+            .read_to_end(&mut restored)
+            .await
+            .expect("cache body should stream");
+        assert_eq!(restored, body);
+    }
+
     /// Regression for the #4576 fallout: the encrypt side persists a random
     /// SSE-C nonce, and this reader-side resolver must read it back — falling
     /// back to the deterministic legacy nonce only when no IV was stored.

crates/ecstore/src/object_api/types.rs

@@ -307,9 +307,9 @@ impl ObjectInfo {
     /// The inline fast path decodes erasure-coded data entirely in memory,
     /// bypassing disk I/O, duplex pipes, and the disk-read semaphore.
     ///
-    /// The `inlined` flag is the primary signal — it is set during PUT by
-    /// `storage_class_should_inline()` which already applies the correct
-    /// version-aware threshold (128 KiB non-versioned, 16 KiB versioned).
+    /// The `inlined` flag is the primary signal — PUT sets it through the
+    /// captured storage-class snapshot's `Config::should_inline`, which applies
+    /// the correct version-aware threshold (128 KiB non-versioned, 16 KiB versioned).
     /// The size check below is a safety net using the same thresholds.
     ///
     /// Additional conditions:

crates/ecstore/src/runtime/instance.rs

@@ -50,7 +50,7 @@ use crate::services::event_notification::EventNotifier;
 use crate::services::tier::tier::TierConfigMgr;
 use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
 use s3s::region::Region;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
 use std::sync::{Arc, OnceLock};
 use tokio::sync::{OnceCell, RwLock};
 use tokio_util::sync::CancellationToken;
@@ -159,6 +159,10 @@ pub struct InstanceContext {
     /// workers (scanner/heal/tier/lifecycle) without touching another instance.
     /// Replaces the process-global cancel-token static.
     background_cancel_token: OnceLock<CancellationToken>,
+    tier_delete_journal_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
+    transition_transaction_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
+    #[cfg(test)]
+    tier_delete_journal_recovery_wakeup: tokio::sync::Notify,
 }
 
 impl InstanceContext {
@@ -193,6 +197,10 @@ impl InstanceContext {
             local_disk_set_drives: Arc::new(RwLock::new(Vec::new())),
             bucket_metadata_sys: std::sync::Mutex::new(None),
             background_cancel_token: OnceLock::new(),
+            tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()),
+            transition_transaction_recovery_stores: std::sync::Mutex::new(HashSet::new()),
+            #[cfg(test)]
+            tier_delete_journal_recovery_wakeup: tokio::sync::Notify::new(),
         }
     }
 
@@ -353,6 +361,34 @@ impl InstanceContext {
         self.background_cancel_token.get().cloned()
     }
 
+    pub(crate) fn bind_background_cancel_token(&self, token: CancellationToken) -> CancellationToken {
+        self.background_cancel_token.get_or_init(|| token).clone()
+    }
+
+    pub(crate) fn mark_tier_delete_journal_recovery_started(&self, store_id: Uuid) -> bool {
+        self.tier_delete_journal_recovery_stores
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner)
+            .insert(store_id)
+    }
+
+    pub(crate) fn mark_transition_transaction_recovery_started(&self, store_id: Uuid) -> bool {
+        self.transition_transaction_recovery_stores
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner)
+            .insert(store_id)
+    }
+
+    #[cfg(test)]
+    pub(crate) fn wake_tier_delete_journal_recovery(&self) {
+        self.tier_delete_journal_recovery_wakeup.notify_one();
+    }
+
+    #[cfg(test)]
+    pub(crate) async fn wait_for_tier_delete_journal_recovery(&self) {
+        self.tier_delete_journal_recovery_wakeup.notified().await;
+    }
+
     /// Update this instance's erasure setup type.
     pub async fn update_erasure_type(&self, setup_type: SetupType) {
         *self.erasure_kind.write().await = setup_type;
@@ -406,6 +442,20 @@ impl std::fmt::Debug for InstanceContext {
             .field("replication_stats_set", &self.replication_stats.get().is_some())
             .field("replication_pool_set", &self.replication_pool.get().is_some())
             .field("background_cancel_token_set", &self.background_cancel_token.get().is_some())
+            .field(
+                "tier_delete_journal_recovery_store_count",
+                &self
+                    .tier_delete_journal_recovery_stores
+                    .lock()
+                    .map_or(0, |stores| stores.len()),
+            )
+            .field(
+                "transition_transaction_recovery_store_count",
+                &self
+                    .transition_transaction_recovery_stores
+                    .lock()
+                    .map_or(0, |stores| stores.len()),
+            )
             .finish_non_exhaustive()
     }
 }
@@ -740,6 +790,36 @@ mod tests {
         assert!(ctx_a.background_cancel_token().unwrap().is_cancelled());
     }
 
+    #[test]
+    fn tier_delete_journal_recovery_is_deduplicated_per_store_and_instance() {
+        let ctx_a = InstanceContext::new();
+        let ctx_b = InstanceContext::new();
+        let store_a = Uuid::new_v4();
+        let store_b = Uuid::new_v4();
+
+        assert!(ctx_a.mark_tier_delete_journal_recovery_started(store_a));
+        assert!(!ctx_a.mark_tier_delete_journal_recovery_started(store_a));
+        assert!(ctx_a.mark_tier_delete_journal_recovery_started(store_b));
+        assert!(ctx_b.mark_tier_delete_journal_recovery_started(store_a));
+    }
+
+    #[test]
+    fn background_cancel_token_binds_the_provided_shutdown_token() {
+        let ctx = InstanceContext::new();
+        let shutdown = CancellationToken::new();
+        let first = ctx.bind_background_cancel_token(shutdown.clone());
+        let second = ctx.bind_background_cancel_token(CancellationToken::new());
+
+        shutdown.cancel();
+        assert!(first.is_cancelled());
+        assert!(second.is_cancelled());
+        assert!(
+            ctx.background_cancel_token()
+                .expect("shutdown token should be published")
+                .is_cancelled()
+        );
+    }
+
     // Phase 5 acceptance (backlog#939): two independent instance contexts share
     // NONE of the runtime state that used to live in process globals. This is
     // the end-to-end proof that the object-graph isolation carrier works — every

crates/ecstore/src/runtime/sources.rs

@@ -25,7 +25,7 @@ use crate::{
     bucket::lifecycle::bucket_lifecycle_ops::{ExpiryState, TransitionState},
     bucket::metadata_sys::{BucketMetadataSys, get_global_bucket_metadata_sys},
     bucket::replication::{DynReplicationPool, ReplicationStats},
-    config::{get_global_storage_class, set_global_storage_class, storageclass},
+    config::{get_global_storage_class, get_global_storage_class_snapshot, set_global_storage_class, storageclass},
     disk::{DiskAPI, DiskOption, DiskStore, new_disk},
     error::Result,
     layout::endpoints::{EndpointServerPools, SetupType},
@@ -239,23 +239,11 @@ pub(crate) fn ensure_test_rpc_secret() {
 }
 
 pub(crate) fn storage_class_parity(storage_class: Option<&str>) -> Option<usize> {
-    get_global_storage_class().and_then(|sc| sc.get_parity_for_sc(storage_class.unwrap_or_default()))
-}
-
-pub(crate) fn backend_storage_class_parities(default_standard_parity: usize) -> (Option<usize>, Option<usize>) {
-    if let Some(sc) = get_global_storage_class() {
-        let standard = sc
-            .get_parity_for_sc(storageclass::CLASS_STANDARD)
-            .or(Some(default_standard_parity));
-        let reduced_redundancy = sc.get_parity_for_sc(storageclass::RRS);
-        (standard, reduced_redundancy)
-    } else {
-        (Some(default_standard_parity), None)
-    }
+    get_global_storage_class_snapshot().get_parity_for_sc(storage_class.unwrap_or_default())
 }
 
 pub(crate) fn storage_class_should_inline(shard_size: i64, versioned: bool) -> bool {
-    get_global_storage_class().is_some_and(|sc| sc.should_inline(shard_size, versioned))
+    get_global_storage_class_snapshot().should_inline(shard_size, versioned)
 }
 
 pub(crate) fn deployment_upload_id(upload_id: &str) -> String {
@@ -330,6 +318,25 @@ pub(crate) fn storage_class_config() -> Option<storageclass::Config> {
     get_global_storage_class()
 }
 
+pub(crate) fn storage_class_config_snapshot() -> Arc<storageclass::Config> {
+    get_global_storage_class_snapshot()
+}
+
+/// Scalar STANDARD / RRS parity for backend-info reporting.
+///
+/// Retained for the rebalance/backend-info path. `get_parity_for_sc` returns
+/// `None` when the runtime config is uninitialized or (post per-pool support)
+/// when pools disagree, so STANDARD falls back to the caller's default and RRS
+/// stays `None` — matching the pre-per-pool scalar reporting.
+pub(crate) fn backend_storage_class_parities(default_standard_parity: usize) -> (Option<usize>, Option<usize>) {
+    let sc = get_global_storage_class_snapshot();
+    let standard = sc
+        .get_parity_for_sc(storageclass::CLASS_STANDARD)
+        .or(Some(default_standard_parity));
+    let reduced_redundancy = sc.get_parity_for_sc(storageclass::RRS);
+    (standard, reduced_redundancy)
+}
+
 pub(crate) fn set_storage_class_config(config: storageclass::Config) {
     set_global_storage_class(config);
 }
@@ -545,7 +552,12 @@ pub(crate) async fn initialize_local_disk_maps(
 }
 
 pub(crate) async fn init_tier_config_mgr(store: Arc<ECStore>) -> Result<()> {
-    get_global_tier_config_mgr().write().await.init(store).await
+    let handle = get_global_tier_config_mgr();
+    TierConfigMgr::reload_handle(&handle, store.clone()).await?;
+    if setup_is_dist_erasure().await {
+        tokio::spawn(TierConfigMgr::refresh_tier_config_handle(handle, store));
+    }
+    Ok(())
 }
 
 #[cfg(test)]

crates/ecstore/src/services/notification_sys.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::cluster::rpc::PeerRestClient;
+use crate::cluster::rpc::{PeerRestClient, ScannerPeerActivity};
 use crate::diagnostics::admin_server_info::get_commit_id;
 use crate::disk::DiskAPI;
 use crate::error::{Error, Result};
@@ -21,6 +21,7 @@ use crate::runtime::sources as runtime_sources;
 use crate::services::metrics_realtime::{CollectMetricsOpts, MetricType};
 use crate::services::rebalance::RebalSaveOpt;
 use crate::storage_api_contracts::admin::StorageAdminApi;
+use bytes::Bytes;
 use futures::future::join_all;
 use lazy_static::lazy_static;
 use rustfs_madmin::health::{Cpus, MemInfo, OsInfo, Partitions, ProcInfo, SysConfig, SysErrors, SysServices};
@@ -35,12 +36,14 @@ use std::sync::{Arc, Mutex, OnceLock};
 use std::time::{Duration, SystemTime};
 use tokio::time::timeout;
 use tracing::{debug, error, info, warn};
+use uuid::Uuid;
 
 /// After this many consecutive admin-call failures, mark the peer as offline.
 const CONSECUTIVE_FAILURE_THRESHOLD: u32 = 3;
 const LOG_COMPONENT_ECSTORE: &str = "ecstore";
 const LOG_SUBSYSTEM_NOTIFICATION: &str = "notification";
 const EVENT_NOTIFICATION_PEER_PROPAGATION: &str = "notification_peer_propagation";
+const SCANNER_ACTIVITY_PROBE_TIMEOUT: Duration = Duration::from_secs(5);
 
 /// Cached result from the last successful admin call to a peer.
 struct PeerAdminCache {
@@ -91,7 +94,6 @@ pub fn get_global_notification_sys() -> Option<Arc<NotificationSys>> {
 
 pub struct NotificationSys {
     pub peer_clients: Vec<Option<PeerRestClient>>,
-    #[allow(dead_code)]
     pub all_peer_clients: Vec<Option<PeerRestClient>>,
     peer_admin_caches: Vec<Mutex<PeerAdminCache>>,
 }
@@ -113,6 +115,17 @@ pub struct NotificationPeerErr {
     pub err: Option<Error>,
 }
 
+fn notification_peer_result<T>(host: String, result: Result<T>) -> NotificationPeerErr {
+    NotificationPeerErr { host, err: result.err() }
+}
+
+fn unreachable_notification_peer_err() -> NotificationPeerErr {
+    NotificationPeerErr {
+        host: String::new(),
+        err: Some(Error::other("peer is not reachable")),
+    }
+}
+
 impl NotificationSys {
     pub fn rest_client_from_hash(&self, s: &str) -> Option<PeerRestClient> {
         if self.all_peer_clients.is_empty() {
@@ -319,8 +332,8 @@ impl NotificationSys {
                 if let Some(client) = client {
                     let host = client.host.to_string();
                     match timeout(peer_timeout, client.local_storage_info()).await {
-                        Ok(Ok(info)) => {
-                            update_storage_info_cache(cache, &host, &info);
+                        Ok(Ok(mut info)) => {
+                            normalize_and_cache_peer_storage_info(cache, &host, &mut info);
                             Some(info)
                         }
                         Ok(Err(err)) => {
@@ -756,6 +769,14 @@ impl NotificationSys {
     }
 
     pub async fn load_bucket_metadata(&self, bucket: &str) -> Result<()> {
+        self.load_bucket_metadata_with_scanner_maintenance(bucket, false).await
+    }
+
+    pub async fn load_bucket_metadata_for_scanner_maintenance(&self, bucket: &str) -> Result<()> {
+        self.load_bucket_metadata_with_scanner_maintenance(bucket, true).await
+    }
+
+    async fn load_bucket_metadata_with_scanner_maintenance(&self, bucket: &str, scanner_maintenance_change: bool) -> Result<()> {
         let operation = format!("load_bucket_metadata({bucket})");
         let mut failures = Vec::new();
         let mut futures = Vec::with_capacity(self.peer_clients.len());
@@ -763,7 +784,12 @@ impl NotificationSys {
             if let Some(client) = client {
                 let host = client.host.to_string();
                 let b = bucket.to_string();
-                futures.push(async move { client.load_bucket_metadata(&b).await.map_err(|err| (host, err)) });
+                futures.push(async move {
+                    client
+                        .load_bucket_metadata(&b, scanner_maintenance_change)
+                        .await
+                        .map_err(|err| (host, err))
+                });
             } else {
                 failures.push(format!("peer[{idx}] {operation} failed: peer is not reachable"));
             }
@@ -976,6 +1002,36 @@ impl NotificationSys {
         join_all(futures).await
     }
 
+    pub async fn scanner_activity_snapshots(&self) -> Result<Vec<(String, ScannerPeerActivity)>> {
+        if self.peer_clients.is_empty() {
+            return Err(Error::other("scanner activity probe has no remote peers"));
+        }
+        if self.all_peer_clients.len() != self.peer_clients.len() + 1 {
+            return Err(Error::other(format!(
+                "scanner activity peer topology is incomplete: {} remote peers for {} cluster members",
+                self.peer_clients.len(),
+                self.all_peer_clients.len()
+            )));
+        }
+
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for (idx, client) in self.peer_clients.iter().cloned().enumerate() {
+            futures.push(async move {
+                let client = client.ok_or_else(|| Error::other(format!("scanner activity peer[{idx}] is unreachable")))?;
+                let host = client.grid_host.clone();
+                scanner_activity_with_timeout(SCANNER_ACTIVITY_PROBE_TIMEOUT, &host, client.scanner_activity())
+                    .await
+                    .map(|activity| (host, activity))
+            });
+        }
+
+        let mut generations = Vec::with_capacity(futures.len());
+        for result in join_all(futures).await {
+            generations.push(result?);
+        }
+        Ok(generations)
+    }
+
     pub async fn reload_site_replication_config(&self) -> Vec<NotificationPeerErr> {
         let mut futures = Vec::with_capacity(self.peer_clients.len());
         for client in self.peer_clients.iter() {
@@ -1027,6 +1083,59 @@ impl NotificationSys {
         }
         join_all(futures).await
     }
+
+    pub async fn prepare_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter().cloned() {
+            let payload = canonical_payload.clone();
+            futures.push(async move {
+                if let Some(client) = client {
+                    notification_peer_result(client.host.to_string(), client.prepare_tier_mutation(mutation_id, payload).await)
+                } else {
+                    unreachable_notification_peer_err()
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
+    pub async fn commit_tier_mutation(&self, mutation_id: Uuid, canonical_payload: Bytes) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter().cloned() {
+            let payload = canonical_payload.clone();
+            futures.push(async move {
+                if let Some(client) = client {
+                    notification_peer_result(client.host.to_string(), client.commit_tier_mutation(mutation_id, payload).await)
+                } else {
+                    unreachable_notification_peer_err()
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
+    pub async fn abort_tier_mutation(&self, mutation_id: Uuid) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter().cloned() {
+            futures.push(async move {
+                if let Some(client) = client {
+                    notification_peer_result(client.host.to_string(), client.abort_tier_mutation(mutation_id).await)
+                } else {
+                    unreachable_notification_peer_err()
+                }
+            });
+        }
+        join_all(futures).await
+    }
+}
+
+async fn scanner_activity_with_timeout<F>(timeout_duration: Duration, host: &str, activity: F) -> Result<ScannerPeerActivity>
+where
+    F: Future<Output = Result<ScannerPeerActivity>>,
+{
+    timeout(timeout_duration, activity)
+        .await
+        .map_err(|_| Error::other(format!("scanner activity peer {host} timed out after {timeout_duration:?}")))?
 }
 
 async fn call_peer_with_timeout<F, Fut>(
@@ -1104,7 +1213,13 @@ fn handle_peer_failure(
     None
 }
 
-fn update_storage_info_cache(cache: Option<&Mutex<PeerAdminCache>>, host: &str, info: &StorageInfo) {
+fn normalize_and_cache_peer_storage_info(cache: Option<&Mutex<PeerAdminCache>>, host: &str, info: &mut StorageInfo) {
+    // `Disk::local` is relative to this aggregator, not to the peer that
+    // produced the response.
+    for disk in &mut info.disks {
+        disk.local = false;
+    }
+
     let Some(cache) = cache else {
         return;
     };
@@ -1568,6 +1683,72 @@ mod tests {
         assert!(msg.contains("peer[0]"));
     }
 
+    #[tokio::test]
+    async fn scanner_activity_probe_reports_unreachable_peers() {
+        let sys = NotificationSys {
+            peer_clients: vec![None],
+            all_peer_clients: vec![None, None],
+            peer_admin_caches: vec![Mutex::new(PeerAdminCache::new())],
+        };
+
+        let err = sys
+            .scanner_activity_snapshots()
+            .await
+            .expect_err("unreachable peers must disable scanner idle backoff");
+
+        assert!(err.to_string().contains("scanner activity peer[0] is unreachable"));
+    }
+
+    #[tokio::test]
+    async fn scanner_activity_probe_rejects_an_empty_peer_set() {
+        let sys = NotificationSys {
+            peer_clients: Vec::new(),
+            all_peer_clients: Vec::new(),
+            peer_admin_caches: Vec::new(),
+        };
+
+        let err = sys
+            .scanner_activity_snapshots()
+            .await
+            .expect_err("a missing peer set must disable scanner idle backoff");
+
+        assert!(err.to_string().contains("no remote peers"));
+    }
+
+    #[tokio::test]
+    async fn scanner_activity_probe_rejects_an_incomplete_peer_topology() {
+        let client = PeerRestClient::new(
+            "127.0.0.1:9000".to_string().try_into().expect("peer host should parse"),
+            "http://127.0.0.1:9000".to_string(),
+        );
+        let sys = NotificationSys {
+            peer_clients: vec![Some(client)],
+            all_peer_clients: vec![None],
+            peer_admin_caches: vec![Mutex::new(PeerAdminCache::new())],
+        };
+
+        let err = sys
+            .scanner_activity_snapshots()
+            .await
+            .expect_err("an incomplete peer topology must disable scanner idle backoff");
+
+        assert!(err.to_string().contains("peer topology is incomplete"));
+    }
+
+    #[tokio::test]
+    async fn scanner_activity_probe_times_out() {
+        let err = scanner_activity_with_timeout(
+            Duration::from_millis(5),
+            "peer-1",
+            std::future::pending::<Result<ScannerPeerActivity>>(),
+        )
+        .await
+        .expect_err("a stalled peer must not block scanner scheduling");
+
+        assert!(err.to_string().contains("timed out"));
+        assert!(err.to_string().contains("peer-1"));
+    }
+
     #[tokio::test]
     async fn load_bucket_metadata_reports_unreachable_peers() {
         let sys = NotificationSys {
@@ -1602,6 +1783,36 @@ mod tests {
         assert!(results[0].err.as_ref().unwrap().to_string().contains("peer is not reachable"));
     }
 
+    #[tokio::test]
+    async fn tier_mutation_fanout_reports_unreachable_peers_fail_closed() {
+        let sys = NotificationSys {
+            peer_clients: vec![None],
+            all_peer_clients: Vec::new(),
+            peer_admin_caches: vec![Mutex::new(PeerAdminCache::new())],
+        };
+        let mutation_id = Uuid::from_u128(1);
+
+        let prepare = sys.prepare_tier_mutation(mutation_id, Bytes::from_static(b"prepare")).await;
+        assert_eq!(prepare.len(), 1);
+        assert!(prepare[0].host.is_empty());
+        assert!(
+            prepare[0]
+                .err
+                .as_ref()
+                .expect("unreachable prepare peer should carry an error")
+                .to_string()
+                .contains("peer is not reachable")
+        );
+
+        let commit = sys.commit_tier_mutation(mutation_id, Bytes::from_static(b"commit")).await;
+        assert_eq!(commit.len(), 1);
+        assert!(commit[0].err.is_some());
+
+        let abort = sys.abort_tier_mutation(mutation_id).await;
+        assert_eq!(abort.len(), 1);
+        assert!(abort[0].err.is_some());
+    }
+
     // --- Tests for handle_peer_failure / handle_server_info_failure caching ---
 
     #[test]
@@ -1642,6 +1853,53 @@ mod tests {
         assert_eq!(cache.lock().unwrap().storage_failures, 1);
     }
 
+    #[test]
+    fn normalize_and_cache_peer_storage_info_marks_disks_remote() {
+        let cache = Mutex::new(PeerAdminCache::new());
+        let mut info = StorageInfo {
+            disks: vec![
+                rustfs_madmin::Disk {
+                    endpoint: "http://node2:9000/media/rustfs-01".to_string(),
+                    drive_path: "/media/rustfs-01".to_string(),
+                    local: true,
+                    ..Default::default()
+                },
+                rustfs_madmin::Disk {
+                    endpoint: "http://node3:9000/media/rustfs-01".to_string(),
+                    drive_path: "/media/rustfs-01".to_string(),
+                    local: true,
+                    ..Default::default()
+                },
+                rustfs_madmin::Disk {
+                    endpoint: "http://node4:9000/media/rustfs-01".to_string(),
+                    drive_path: "/media/rustfs-01".to_string(),
+                    local: true,
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        normalize_and_cache_peer_storage_info(Some(&cache), "peer-1", &mut info);
+
+        assert!(info.disks.iter().all(|disk| !disk.local));
+        let cached = cache.lock().expect("peer cache must remain available");
+        assert!(
+            cached
+                .last_storage_info
+                .as_ref()
+                .expect("successful peer response must be cached")
+                .disks
+                .iter()
+                .all(|disk| !disk.local)
+        );
+        drop(cached);
+
+        let degraded = handle_peer_failure(Some(&cache), "peer-1", &EndpointServerPools::default())
+            .expect("first peer failure must return the cached snapshot");
+        assert!(degraded.disks.iter().all(|disk| !disk.local));
+    }
+
     #[test]
     fn handle_peer_failure_returns_offline_after_threshold_exceeded() {
         let cached_info = StorageInfo {
@@ -1941,10 +2199,10 @@ mod tests {
             panic!("poison server cache mutex");
         });
 
-        update_storage_info_cache(
+        normalize_and_cache_peer_storage_info(
             Some(&storage_cache),
             "peer-1",
-            &StorageInfo {
+            &mut StorageInfo {
                 disks: vec![rustfs_madmin::Disk {
                     endpoint: "disk-0".to_string(),
                     state: "ok".to_string(),

crates/ecstore/src/services/tier/mod.rs

@@ -19,6 +19,8 @@ pub mod tier_admin;
 pub mod tier_config;
 pub mod tier_gen;
 pub mod tier_handlers;
+pub(crate) mod tier_mutation_intent;
+pub mod tier_mutation_peer;
 pub mod warm_backend;
 pub mod warm_backend_aliyun;
 pub mod warm_backend_azure;
@@ -30,3 +32,4 @@ pub mod warm_backend_rustfs;
 pub mod warm_backend_s3;
 pub mod warm_backend_s3sdk;
 pub mod warm_backend_tencent;
+pub mod warm_backend_wasabi;

crates/ecstore/src/services/tier/test_util.rs

@@ -57,23 +57,42 @@
 use std::collections::HashMap;
 use std::io::Cursor;
 use std::path::Path;
-use std::sync::Arc;
+use std::sync::{
+    Arc,
+    atomic::{AtomicBool, AtomicUsize, Ordering},
+};
 use std::time::Duration;
 
 use async_trait::async_trait;
 use tokio::io::AsyncReadExt;
-use tokio::sync::{Mutex, RwLock};
+use tokio::sync::{Mutex, Notify, RwLock};
 use uuid::Uuid;
 
 use crate::client::transition_api::{ReadCloser, ReaderImpl};
 use crate::disk::endpoint::Endpoint;
-use crate::disk::{DiskAPI, DiskOption, STORAGE_FORMAT_FILE, new_disk};
+use crate::disk::format::FormatV3;
+use crate::disk::{DiskAPI, DiskOption, FORMAT_CONFIG_FILE, RUSTFS_META_BUCKET, STORAGE_FORMAT_FILE, new_disk};
 use crate::services::tier::tier::TierConfigMgr;
 use crate::services::tier::tier_config::{TierConfig, TierMinIO, TierType};
 use crate::services::tier::warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options};
 use rustfs_filemeta::FileMeta;
 use rustfs_utils::path::path_join_buf;
 
+/// One-shot barrier before rejected transition cleanup resolves its ECStore.
+pub struct TransitionCleanupStoreBarrier(crate::set_disk::SetDiskTransitionCleanupStoreBarrier);
+
+impl TransitionCleanupStoreBarrier {
+    /// Install the barrier for the next rejected transition cleanup.
+    pub fn install() -> Self {
+        Self(crate::set_disk::SetDiskTransitionCleanupStoreBarrier::install())
+    }
+
+    /// Wait until the rejected transition reaches cleanup-store resolution.
+    pub async fn wait_until_paused(&self) {
+        self.0.wait_until_paused().await;
+    }
+}
+
 /// Default polling cadence used by the `wait_for_*` helpers.
 const POLL_INTERVAL: Duration = Duration::from_millis(50);
 
@@ -139,7 +158,129 @@ pub struct MockStoredObject {
 struct MockWarmBackendInner {
     objects: Mutex<HashMap<String, MockStoredObject>>,
     faults: Mutex<FaultConfig>,
+    put_read_limit: Mutex<Option<usize>>,
+    put_remote_version: Mutex<Option<String>>,
+    reject_non_empty_remote_versions: AtomicBool,
+    fail_remove: AtomicBool,
+    exact_remove_count: AtomicUsize,
     op_log: Mutex<Vec<MockWarmOp>>,
+    put_versions: Mutex<Vec<(String, String)>>,
+    remove_versions: Mutex<Vec<(String, String)>>,
+    put_barrier: Mutex<Option<Arc<MockPutBarrierState>>>,
+    get_barrier: Mutex<Option<Arc<MockGetBarrierState>>>,
+    remove_barrier: Mutex<Option<Arc<MockRemoveBarrierState>>>,
+}
+
+#[derive(Default)]
+struct MockPutBarrierState {
+    arrived: Notify,
+    release: Notify,
+}
+
+#[derive(Default)]
+struct MockGetBarrierState {
+    arrived: Notify,
+    release: Notify,
+    fail_after_release: bool,
+}
+
+#[derive(Default)]
+struct MockRemoveBarrierState {
+    arrived: Notify,
+    release: Notify,
+    operation_dropped: Notify,
+}
+
+struct MockRemoveOperationGuard {
+    state: Arc<MockRemoveBarrierState>,
+}
+
+impl Drop for MockRemoveOperationGuard {
+    fn drop(&mut self) {
+        self.state.operation_dropped.notify_one();
+    }
+}
+
+/// One-shot barrier that pauses a mock tier PUT after storing its remote body.
+pub struct MockPutBarrier {
+    state: Arc<MockPutBarrierState>,
+}
+
+impl MockPutBarrier {
+    /// Wait until the remote body is stored and the PUT is paused before returning.
+    pub async fn wait_until_paused(&self) {
+        tokio::time::timeout(Duration::from_secs(30), self.state.arrived.notified())
+            .await
+            .expect("mock tier PUT should reach the deterministic barrier");
+    }
+
+    /// Release the paused PUT.
+    pub fn release(&self) {
+        self.state.release.notify_one();
+    }
+}
+
+impl Drop for MockPutBarrier {
+    fn drop(&mut self) {
+        self.state.release.notify_one();
+    }
+}
+
+/// One-shot barrier that pauses a mock tier GET before it reads remote bytes.
+pub struct MockGetBarrier {
+    state: Arc<MockGetBarrierState>,
+}
+
+impl MockGetBarrier {
+    /// Wait until the GET has reached the deterministic pause point.
+    pub async fn wait_until_paused(&self) {
+        tokio::time::timeout(Duration::from_secs(30), self.state.arrived.notified())
+            .await
+            .expect("mock tier GET should reach the deterministic barrier");
+    }
+
+    /// Release the paused GET.
+    pub fn release(&self) {
+        self.state.release.notify_one();
+    }
+}
+
+impl Drop for MockGetBarrier {
+    fn drop(&mut self) {
+        self.state.release.notify_one();
+    }
+}
+
+/// One-shot barrier that pauses and then fails a mock tier DELETE.
+pub struct MockRemoveBarrier {
+    state: Arc<MockRemoveBarrierState>,
+}
+
+impl MockRemoveBarrier {
+    /// Wait until DELETE reaches the deterministic failure point.
+    pub async fn wait_until_paused(&self) {
+        tokio::time::timeout(Duration::from_secs(30), self.state.arrived.notified())
+            .await
+            .expect("mock tier DELETE should reach the deterministic barrier");
+    }
+
+    /// Release the paused DELETE, which then returns an injected error.
+    pub fn release(&self) {
+        self.state.release.notify_one();
+    }
+
+    /// Wait until the paused DELETE future completes or is cancelled.
+    pub async fn wait_until_operation_dropped(&self) {
+        tokio::time::timeout(Duration::from_secs(30), self.state.operation_dropped.notified())
+            .await
+            .expect("mock tier DELETE operation should be dropped");
+    }
+}
+
+impl Drop for MockRemoveBarrier {
+    fn drop(&mut self) {
+        self.state.release.notify_one();
+    }
 }
 
 /// In-memory [`WarmBackend`] for lifecycle / tiering integration tests.
@@ -158,6 +299,41 @@ impl MockWarmBackend {
         Self::default()
     }
 
+    /// Arm a one-shot pause after the next tier PUT stores its remote body.
+    pub async fn arm_put_barrier(&self) -> MockPutBarrier {
+        let state = Arc::new(MockPutBarrierState::default());
+        *self.inner.put_barrier.lock().await = Some(Arc::clone(&state));
+        MockPutBarrier { state }
+    }
+
+    /// Pause and then fail the next DELETE after it reaches the backend.
+    pub async fn arm_failing_remove_barrier(&self) -> MockRemoveBarrier {
+        let state = Arc::new(MockRemoveBarrierState::default());
+        let mut barrier = self.inner.remove_barrier.lock().await;
+        assert!(barrier.is_none(), "mock tier DELETE barrier is already armed");
+        *barrier = Some(state.clone());
+        MockRemoveBarrier { state }
+    }
+
+    /// Arm a one-shot pause before the next tier GET, then return an error
+    /// after the test releases it.
+    pub async fn arm_failing_get_barrier(&self) -> MockGetBarrier {
+        let state = Arc::new(MockGetBarrierState {
+            fail_after_release: true,
+            ..Default::default()
+        });
+        *self.inner.get_barrier.lock().await = Some(Arc::clone(&state));
+        MockGetBarrier { state }
+    }
+
+    /// Arm a one-shot pause before the next tier GET, then continue normally
+    /// after the test releases it.
+    pub async fn arm_get_barrier(&self) -> MockGetBarrier {
+        let state = Arc::new(MockGetBarrierState::default());
+        *self.inner.get_barrier.lock().await = Some(Arc::clone(&state));
+        MockGetBarrier { state }
+    }
+
     // ---- fault injection -------------------------------------------------
 
     /// Replace the entire fault configuration.
@@ -190,6 +366,28 @@ impl MockWarmBackend {
         *self.inner.faults.lock().await = FaultConfig::default();
     }
 
+    /// Limit how many body bytes a successful mock PUT consumes. `None` drains
+    /// the complete body. This models a backend that incorrectly accepts a
+    /// truncated stream while still returning success.
+    pub async fn set_put_read_limit(&self, limit: Option<usize>) {
+        *self.inner.put_read_limit.lock().await = limit;
+    }
+
+    /// Override the remote version returned by subsequent successful PUTs.
+    pub async fn set_put_remote_version(&self, remote_version: Option<String>) {
+        *self.inner.put_remote_version.lock().await = remote_version;
+    }
+
+    /// Reject non-empty remote versions before transition metadata is committed.
+    pub fn set_reject_non_empty_remote_versions(&self, reject: bool) {
+        self.inner.reject_non_empty_remote_versions.store(reject, Ordering::Release);
+    }
+
+    /// Enable or disable a persistent remove failure for durability tests.
+    pub fn set_remove_failure(&self, fail: bool) {
+        self.inner.fail_remove.store(fail, Ordering::Release);
+    }
+
     async fn precondition(&self) -> Result<(), std::io::Error> {
         let (latency, error) = {
             let faults = self.inner.faults.lock().await;
@@ -231,6 +429,21 @@ impl MockWarmBackend {
             .count()
     }
 
+    /// Number of exact-version trait remove calls, including failed attempts.
+    pub fn exact_remove_count(&self) -> usize {
+        self.inner.exact_remove_count.load(Ordering::Acquire)
+    }
+
+    /// Return the exact object/version pairs produced by successful tier PUTs.
+    pub async fn put_versions(&self) -> Vec<(String, String)> {
+        self.inner.put_versions.lock().await.clone()
+    }
+
+    /// Return the exact object/version pairs passed to successful tier removes.
+    pub async fn remove_versions(&self) -> Vec<(String, String)> {
+        self.inner.remove_versions.lock().await.clone()
+    }
+
     /// Number of `get` calls recorded — useful to assert restore reads hit the
     /// local copy rather than the remote tier.
     pub async fn get_count(&self) -> usize {
@@ -327,7 +540,13 @@ impl MockWarmBackend {
     // ---- internal helpers -----------------------------------------------
 
     async fn put_bytes(&self, object: &str, bytes: Vec<u8>, metadata: HashMap<String, String>) -> String {
-        let remote_version_id = Uuid::new_v4().to_string();
+        let remote_version_id = self
+            .inner
+            .put_remote_version
+            .lock()
+            .await
+            .clone()
+            .unwrap_or_else(|| Uuid::new_v4().to_string());
         self.inner.objects.lock().await.insert(
             object.to_string(),
             MockStoredObject {
@@ -340,11 +559,18 @@ impl MockWarmBackend {
     }
 
     async fn read_bytes(&self, reader: ReaderImpl) -> Result<Vec<u8>, std::io::Error> {
+        let limit = *self.inner.put_read_limit.lock().await;
         match reader {
-            ReaderImpl::Body(bytes) => Ok(bytes.to_vec()),
+            ReaderImpl::Body(bytes) => Ok(bytes.slice(..limit.unwrap_or(bytes.len()).min(bytes.len())).to_vec()),
             ReaderImpl::ObjectBody(mut reader) => {
                 let mut buf = Vec::new();
-                reader.stream.read_to_end(&mut buf).await?;
+                if let Some(limit) = limit {
+                    let limit =
+                        u64::try_from(limit).map_err(|_| std::io::Error::other("mock PUT read limit exceeds u64::MAX"))?;
+                    reader.stream.take(limit).read_to_end(&mut buf).await?;
+                } else {
+                    reader.stream.read_to_end(&mut buf).await?;
+                }
                 Ok(buf)
             }
         }
@@ -353,10 +579,22 @@ impl MockWarmBackend {
 
 #[async_trait]
 impl WarmBackend for MockWarmBackend {
+    fn validate_remote_version_id(&self, remote_version_id: &str) -> Result<(), std::io::Error> {
+        if self.inner.reject_non_empty_remote_versions.load(Ordering::Acquire) && !remote_version_id.is_empty() {
+            return Err(std::io::Error::other("mock warm backend requires an unversioned remote object"));
+        }
+        Ok(())
+    }
+
     async fn put(&self, object: &str, r: ReaderImpl, _length: i64) -> Result<String, std::io::Error> {
         self.precondition().await?;
         let bytes = self.read_bytes(r).await?;
         let version = self.put_bytes(object, bytes, HashMap::new()).await;
+        self.inner
+            .put_versions
+            .lock()
+            .await
+            .push((object.to_string(), version.clone()));
         self.record(MockWarmOp::Put {
             object: object.to_string(),
         })
@@ -397,6 +635,16 @@ impl WarmBackend for MockWarmBackend {
             metadata.insert("x-amz-object-lock-legal-hold".to_string(), opts.legalhold.as_str().to_string());
         }
         let version = self.put_bytes(object, bytes, metadata).await;
+        self.inner
+            .put_versions
+            .lock()
+            .await
+            .push((object.to_string(), version.clone()));
+        let barrier = self.inner.put_barrier.lock().await.take();
+        if let Some(barrier) = barrier {
+            barrier.arrived.notify_one();
+            barrier.release.notified().await;
+        }
         self.record(MockWarmOp::Put {
             object: object.to_string(),
         })
@@ -406,6 +654,14 @@ impl WarmBackend for MockWarmBackend {
 
     async fn get(&self, object: &str, _rv: &str, opts: WarmBackendGetOpts) -> Result<ReadCloser, std::io::Error> {
         self.precondition().await?;
+        let barrier = self.inner.get_barrier.lock().await.take();
+        if let Some(barrier) = barrier {
+            barrier.arrived.notify_one();
+            barrier.release.notified().await;
+            if barrier.fail_after_release {
+                return Err(std::io::Error::other("mock warm backend GET failed after barrier"));
+            }
+        }
         self.record(MockWarmOp::Get {
             object: object.to_string(),
         })
@@ -426,9 +682,31 @@ impl WarmBackend for MockWarmBackend {
         Ok(tokio::io::BufReader::new(Cursor::new(bytes[start.min(bytes.len())..end].to_vec())))
     }
 
-    async fn remove(&self, object: &str, _rv: &str) -> Result<(), std::io::Error> {
+    async fn remove(&self, object: &str, rv: &str) -> Result<(), std::io::Error> {
         self.precondition().await?;
-        self.inner.objects.lock().await.remove(object);
+        if let Some(barrier) = self.inner.remove_barrier.lock().await.take() {
+            let _operation = MockRemoveOperationGuard { state: barrier.clone() };
+            barrier.arrived.notify_one();
+            barrier.release.notified().await;
+            return Err(std::io::Error::other("mock warm backend remove failure after barrier"));
+        }
+        if self.inner.fail_remove.load(Ordering::Acquire) {
+            return Err(std::io::Error::other("mock warm backend remove failure"));
+        }
+        let mut objects = self.inner.objects.lock().await;
+        if let Some(stored) = objects.get(object)
+            && !rv.is_empty()
+            && stored.remote_version_id != rv
+        {
+            return Err(std::io::Error::new(std::io::ErrorKind::NotFound, "NoSuchVersion"));
+        }
+        objects.remove(object);
+        drop(objects);
+        self.inner
+            .remove_versions
+            .lock()
+            .await
+            .push((object.to_string(), rv.to_string()));
         self.record(MockWarmOp::Remove {
             object: object.to_string(),
         })
@@ -436,10 +714,21 @@ impl WarmBackend for MockWarmBackend {
         Ok(())
     }
 
+    async fn remove_exact(&self, object: &str, rv: &str) -> Result<(), std::io::Error> {
+        if rv.is_empty() {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "an exact mock tier delete requires a remote version ID",
+            ));
+        }
+        self.inner.exact_remove_count.fetch_add(1, Ordering::AcqRel);
+        self.remove(object, rv).await
+    }
+
     async fn in_use(&self) -> Result<bool, std::io::Error> {
         self.precondition().await?;
         self.record(MockWarmOp::InUse).await;
-        Ok(false)
+        Ok(!self.inner.objects.lock().await.is_empty())
     }
 }
 
@@ -478,7 +767,9 @@ pub async fn register_mock_tier_backend(handle: &Arc<RwLock<TierConfigMgr>>, tie
             ..Default::default()
         },
     );
-    tier_config_mgr.driver_cache.insert(tier_name.to_string(), Box::new(backend));
+    tier_config_mgr
+        .install_test_driver(tier_name, Box::new(backend))
+        .expect("mock tier driver should install");
 }
 
 /// The transition-state tuple read from an on-disk `xl.meta`, plus the object's
@@ -498,10 +789,19 @@ pub struct TransitionMeta {
 }
 
 async fn open_disk(disk_path: &Path) -> Option<crate::disk::DiskStore> {
+    // `LocalDisk::new` rejects an endpoint whose (set_idx, disk_idx) disagrees
+    // with the position recorded in the disk's own format.json, so derive the
+    // real indices instead of assuming slot 0 (rustfs/backlog#1303).
+    let format_data = tokio::fs::read(disk_path.join(RUSTFS_META_BUCKET).join(FORMAT_CONFIG_FILE))
+        .await
+        .ok()?;
+    let format = FormatV3::try_from(format_data.as_slice()).ok()?;
+    let (set_idx, disk_idx) = format.find_disk_index_by_disk_id(format.erasure.this).ok()?;
+
     let mut endpoint = Endpoint::try_from(disk_path.to_str()?).ok()?;
     endpoint.set_pool_index(0);
-    endpoint.set_set_index(0);
-    endpoint.set_disk_index(0);
+    endpoint.set_set_index(set_idx);
+    endpoint.set_disk_index(disk_idx);
     new_disk(
         &endpoint,
         &DiskOption {