feat(table-catalog): add product API surface (#3320)

* feat(table-catalog): add product API surface

* fix(table-catalog): harden product API commits

* fix(table-catalog): close product API review gaps

* fix(table-catalog): validate rollback metadata before commit

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>

.agents/skills/code-change-verification/SKILL.md

@@ -41,6 +41,19 @@ Use this skill to review code changes consistently before merge, before release,
   - hidden coupling to shared helpers/constants/features
 - If a point is uncertain, mark it as an open question instead of guessing.
 
+#### Rust-specific checks (apply to all Rust changes)
+
+- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
+- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
+- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
+- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
+- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
+- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
+- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
+- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
+- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
+- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
+
 ### 4) Findings-first output
 - Order findings by severity:
   - P0: critical failure, security breach, or data loss risk

.agents/skills/rust-code-quality/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: rust-code-quality
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+---
+
+# Rust Code Quality Gate
+
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
+
+## Quick Start
+
+1. Identify changed `.rs` files.
+2. Run automated checks on changed files.
+3. Run manual review checklist on the diff.
+4. Report findings; block merge if P0/P1 issues exist.
+
+## Automated Checks
+
+Run these on every changed `.rs` file (excluding test modules):
+
+```bash
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
+
+# 2. Silent type truncation via `as` cast
+rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
+
+# 3. String as error type
+rg -n 'Result<.*String>' <changed-files> | grep -v test
+
+# 4. Box<dyn Error> in public APIs
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
+
+# 5. println/eprintln in production
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
+
+# 6. Ordering::Relaxed usage (verify each is intentional)
+rg -n 'Ordering::Relaxed' <changed-files>
+```
+
+## Manual Review Checklist
+
+For every Rust code change, verify:
+
+### Error Handling
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
+- [ ] No `Result<_, String>` in public API signatures
+- [ ] No `Box<dyn Error>` in public trait/struct methods
+- [ ] `Error::source()` is overridden when inner error is stored
+- [ ] Error messages are actionable (what failed, with what input)
+
+### Type Safety
+- [ ] No silent `as` truncation (negative→unsigned, large→small)
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
+
+### Concurrency
+- [ ] Lock acquisition order is documented when multiple locks are used
+- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
+- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
+
+### Memory and Performance
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
+
+### Recursion Safety
+- [ ] Recursive functions have a depth limit or use iterative traversal
+- [ ] Tree/cache traversals handle corrupted/cyclic input safely
+
+### Testing
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
+
+### Serde
+- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
+- [ ] `#[serde(default)]` not used on security-critical fields without validation
+
+### Code Hygiene
+- [ ] No `#![allow(dead_code)]` at crate root
+- [ ] No camelCase statics or Hungarian notation
+- [ ] New string literals don't duplicate existing constants
+
+## Severity Classification
+
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
+
+## Output Template
+
+```
+## Rust Code Quality Report
+
+### Automated Scan
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
+
+### Findings
+- [P1] `path:line` — description
+  - Fix: ...
+  - Validation: ...
+
+### Verdict
+PASS / BLOCKED (list blocking findings)
+```

.agents/skills/rust-code-quality/references/checklist.md

@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```

.agents/skills/security-advisory-lessons/SKILL.md

@@ -53,6 +53,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
 - Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
 - Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
 - Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
 
 ### IAM and service accounts
@@ -70,6 +71,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
 - Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
 - Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
 
 ### Secrets, default credentials, and crypto
 - Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
@@ -86,6 +88,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
 - Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
 - RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
 
 ### Browser, CORS, and console surfaces
 - Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
@@ -115,5 +118,6 @@ Use these prompts while reviewing a diff:
 - Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
 - Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -20,6 +20,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
 - `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
 - `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
+- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
 - `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
 
 ### IAM import, service accounts, and privilege boundaries
@@ -38,6 +39,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
 - `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
 
 ### Secrets, defaults, and cryptographic misuse
 
@@ -55,6 +57,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
 - `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
 
 ### Browser, CORS, and console isolation
 
@@ -69,6 +72,13 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
+### Serde deserialization and input validation
+
+- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
+- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
+- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
+- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
+
 ## Useful Search Seeds
 
 Use these targeted searches when a diff touches security-sensitive code:
@@ -76,10 +86,12 @@ Use these targeted searches when a diff touches security-sensitive code:
 ```bash
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
 ## Minimum Regression Test Expectations
@@ -87,7 +99,8 @@ rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-
 - Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/build-docker.mak

@@ -9,7 +9,7 @@ build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
 build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
 build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
 	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
-	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
+	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
 	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
 
 .PHONY: docker-inspect-multiarch
@@ -19,4 +19,4 @@ docker-inspect-multiarch: ## Check image architecture support
 		exit 1; \
 	fi
 	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
-	docker buildx imagetools inspect $(IMAGE)
+	docker buildx imagetools inspect $(IMAGE)

.config/make/lint-fmt.mak

@@ -21,6 +21,11 @@ unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
 	@echo "🔒 Checking unsafe_code allowances..."
 	./scripts/check_unsafe_code_allowances.sh
 
+.PHONY: architecture-migration-check
+architecture-migration-check: ## Check architecture migration guardrails
+	@echo "🏗️ Checking architecture migration guardrails..."
+	./scripts/check_architecture_migration_rules.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -7,5 +7,5 @@ setup-hooks: ## Set up git hooks
 	@echo "✅ Git hooks setup complete!"
 
 .PHONY: pre-commit
-pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
+pre-commit: fmt unsafe-code-check architecture-migration-check clippy-check compilation-check test ## Run pre-commit checks
 	@echo "✅ All pre-commit checks passed!"

.docker/compose/docker-compose.cluster.local-build.yml

@@ -23,13 +23,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -53,13 +55,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -83,13 +87,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -113,13 +119,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"

.docker/compose/docker-compose.cluster.yaml

@@ -21,8 +21,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9003:9000" # Map port 9004 of the host to port 9000 of the container

.docker/observability/grafana/dashboards/rustfs.json

@@ -4551,7 +4551,7 @@
                   "index": 0,
                   "text": "INACTIVE"
                 },
-                "to": 1e-09
+                "to": 1e-9
               },
               "type": "range"
             }
@@ -5281,7 +5281,10 @@
       "id": 102,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5368,7 +5371,10 @@
       "id": 103,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5455,7 +5461,10 @@
       "id": 104,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5533,7 +5542,9 @@
       "id": 105,
       "options": {
         "legend": {
-          "calcs": ["mean"],
+          "calcs": [
+            "mean"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5629,7 +5640,10 @@
       "id": 106,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5725,7 +5739,10 @@
       "id": 107,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5803,7 +5820,10 @@
       "id": 108,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -8071,6 +8091,374 @@
         "x": 0,
         "y": 195
       },
+      "id": 150,
+      "panels": [],
+      "title": "Object Lock Diagnostics",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 196
+      },
+      "id": 151,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_acquire_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Acquires",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 196
+      },
+      "id": 152,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_hold_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Holds",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 204
+      },
+      "id": 153,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_acquire_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Acquire Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 204
+      },
+      "id": 154,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_hold_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Hold Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [
+            {
+              "options": {
+                "0": {
+                  "text": "Disabled"
+                },
+                "1": {
+                  "text": "Enabled"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": 0
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 212
+      },
+      "id": 155,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "percentChangeColorMode": "standard",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max(rustfs_object_lock_diag_enabled{job=~\"$job\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Diagnostics Enabled",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 213
+      },
       "id": 137,
       "panels": [],
       "title": "Debug / Raw Explorer",
@@ -8138,7 +8526,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 196
+        "y": 214
       },
       "id": 138,
       "options": {
@@ -8235,7 +8623,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 196
+        "y": 214
       },
       "id": 139,
       "options": {
@@ -8332,7 +8720,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 204
+        "y": 222
       },
       "id": 140,
       "options": {
@@ -8429,7 +8817,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 204
+        "y": 222
       },
       "id": 141,
       "options": {
@@ -8526,7 +8914,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 212
+        "y": 230
       },
       "id": 142,
       "options": {
@@ -8623,7 +9011,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 212
+        "y": 230
       },
       "id": 143,
       "options": {
@@ -8720,7 +9108,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 220
+        "y": 238
       },
       "id": 144,
       "options": {
@@ -8817,7 +9205,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 220
+        "y": 238
       },
       "id": 145,
       "options": {
@@ -8914,7 +9302,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 228
+        "y": 246
       },
       "id": 146,
       "options": {
@@ -9011,7 +9399,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 228
+        "y": 246
       },
       "id": 147,
       "options": {
@@ -9108,7 +9496,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 236
+        "y": 254
       },
       "id": 148,
       "options": {
@@ -9205,7 +9593,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 236
+        "y": 254
       },
       "id": 149,
       "options": {
@@ -9239,6 +9627,416 @@
       ],
       "title": "Cluster Config / API Rejected (All)",
       "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 262
+      },
+      "id": 507,
+      "panels": [],
+      "title": "Internode Reliability",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ops"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 263
+      },
+      "id": 508,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, classification) (rate(rustfs_system_network_internode_operation_classified_errors_total{job=~\"$job\", operation=~\"put_file_stream|walk_dir|read_file_stream\"}[$__rate_interval]))",
+          "legendFormat": "{{operation}} | {{classification}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Internode Classified Errors",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ops"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 263
+      },
+      "id": 509,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, classification) (rate(rustfs_system_network_internode_operation_retries_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval]))",
+          "legendFormat": "retry | {{classification}}",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, classification) (rate(rustfs_system_network_internode_operation_retry_successes_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval]))",
+          "legendFormat": "retry success | {{classification}}",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Internode Open Write Retries",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "yellow",
+                "value": 0.25
+              },
+              {
+                "color": "red",
+                "value": 0.5
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 271
+      },
+      "id": 510,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (classification) (rate(rustfs_system_network_internode_operation_retry_successes_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval])) / clamp_min(sum by (classification) (rate(rustfs_system_network_internode_operation_retries_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval])), 1e-9)",
+          "legendFormat": "{{classification}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Internode Retry Success Ratio",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ops"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 271
+      },
+      "id": 511,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (stage, dominant_error) (rate(rustfs_system_storage_erasure_write_quorum_failures_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{stage}} | {{dominant_error}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Erasure Write Quorum Failures",
+      "type": "timeseries"
     }
   ],
   "preload": false,
@@ -9382,5 +10180,5 @@
   "timezone": "browser",
   "title": "RustFS",
   "uid": "rustfs-s3",
-  "version": 13
+  "version": 14
 }

.github/AGENTS.md

@@ -32,5 +32,9 @@ Current `test-and-lint` gate includes:
 - `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
 - `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
 - `cargo fmt --all --check`
-- `cargo clippy --all-targets --all-features -- -D warnings`
+- `cargo clippy --all-targets -- -D warnings`
+- `cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2`
+- `cargo test -p rustfs --doc --features rio-v2`
+- `cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings`
 - `./scripts/check_layer_dependencies.sh`
+- `./scripts/check_architecture_migration_rules.sh`

.github/actions/setup/action.yml

@@ -54,6 +54,8 @@ runs:
           build-essential \
           pkg-config \
           libssl-dev \
+          ripgrep \
+          unzip \
           protobuf-compiler
 
     - name: Install protoc
@@ -83,7 +85,7 @@ runs:
       uses: taiki-e/install-action@cargo-zigbuild
 
     - name: Install cargo-nextest
-      uses: taiki-e/install-action@cargo-nextest
+      uses: taiki-e/install-action@nextest
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@v2

.github/dependabot.yml

@@ -40,6 +40,8 @@ updates:
         versions: [ "1.x" ]
       - dependency-name: "ratelimit"
         versions: [ "2.x" ]
+      - dependency-name: "pyroscope"
+        versions: [ "2.x" ]
     groups:
       s3s:
         update-types:

.github/s3tests/s3tests.conf

@@ -11,8 +11,8 @@
 # host set for RustFS - will be substituted via envsubst
 host = ${S3_HOST}
 
-# port for RustFS
-port = 9000
+# port for RustFS - will be substituted via envsubst
+port = ${S3_PORT}
 
 ## say "False" to disable TLS
 is_secure = False

.github/workflows/architecture-migration-rules.yml

@@ -0,0 +1,43 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Architecture Migration Rules
+
+on:
+  pull_request:
+    branches: [ main ]
+    paths:
+      - "ARCHITECTURE.md"
+      - "docs/architecture/**"
+      - "scripts/check_architecture_migration_rules.sh"
+      - ".github/workflows/architecture-migration-rules.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  architecture-migration-rules:
+    name: Architecture Migration Rules
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install ripgrep
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y ripgrep
+
+      - name: Check architecture migration rules
+        run: ./scripts/check_architecture_migration_rules.sh

.github/workflows/audit.yml

@@ -40,7 +40,7 @@ env:
 jobs:
   security-audit:
     name: Security Audit
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 15
     steps:
       - name: Checkout repository
@@ -65,7 +65,7 @@ jobs:
 
   dependency-review:
     name: Dependency Review
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: github.event_name == 'pull_request'
     permissions:
       contents: read

.github/workflows/build.yml

@@ -73,7 +73,7 @@ jobs:
   # Build strategy check - determine build type based on trigger
   build-check:
     name: Build Strategy Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       build_type: ${{ steps.check.outputs.build_type }}
@@ -146,7 +146,7 @@ jobs:
   # Build RustFS binaries
   prepare-platform-matrix:
     name: Prepare Platform Matrix
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
       selected: ${{ steps.select.outputs.selected }}
@@ -169,7 +169,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-26-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -202,7 +202,7 @@ jobs:
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
-    timeout-minutes: 60
+    timeout-minutes: 90
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
       # Always enable Tokio unstable features (required by dial9-tokio-telemetry).
@@ -549,7 +549,7 @@ jobs:
     name: Build Summary
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Build completion summary
         shell: bash
@@ -601,7 +601,7 @@ jobs:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     outputs:
@@ -687,7 +687,7 @@ jobs:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
       actions: read
@@ -768,7 +768,7 @@ jobs:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/')
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Update latest.json
         env:
@@ -793,9 +793,10 @@ jobs:
 
           curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
           unzip "$OSSUTIL_ZIP"
-          mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+          OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
+          mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
           rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-          chmod +x /usr/local/bin/ossutil
+          chmod +x "$OSSUTIL_BIN"
 
           # Create latest.json
           cat > latest.json << EOF
@@ -809,7 +810,7 @@ jobs:
           EOF
 
           # Upload to OSS
-          ossutil cp latest.json oss://rustfs-version/latest.json --force
+          "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
 
           echo "✅ Updated latest.json for stable release $VERSION"
 
@@ -818,7 +819,7 @@ jobs:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     steps:

.github/workflows/ci.yml

@@ -54,7 +54,7 @@ on:
       - ".github/workflows/audit.yml"
       - ".github/workflows/performance.yml"
   merge_group:
-    types: [checks_requested]
+    types: [ checks_requested ]
   schedule:
     - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
   workflow_dispatch:
@@ -78,7 +78,7 @@ jobs:
     permissions:
       actions: write
       contents: read
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_skip: ${{ steps.skip_check.outputs.should_skip }}
     steps:
@@ -95,7 +95,7 @@ jobs:
     name: Typos
     needs: skip-check
     if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
       - uses: dtolnay/rust-toolchain@stable
@@ -134,11 +134,42 @@ jobs:
         run: ./scripts/check_unsafe_code_allowances.sh
 
       - name: Run clippy lints
-        run: cargo clippy --all-targets --all-features -- -D warnings
+        run: cargo clippy --all-targets -- -D warnings
 
       - name: Check layered dependencies
         run: ./scripts/check_layer_dependencies.sh
 
+      - name: Check architecture migration rules
+        run: ./scripts/check_architecture_migration_rules.sh
+
+  test-and-lint-rio-v2:
+    name: Test and Lint (rio-v2)
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-test-rio-v2-${{ hashFiles('**/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Run rio-v2 feature tests
+        run: |
+          cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2
+          cargo test -p rustfs --doc --features rio-v2
+
+      - name: Run rio-v2 clippy lints
+        run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
+
   build-rustfs-debug-binary:
     name: Build RustFS Debug Binary
     needs: skip-check
@@ -172,12 +203,92 @@ jobs:
           if-no-files-found: error
           retention-days: 1
 
+  build-rustfs-debug-binary-rio-v2:
+    name: Build RustFS Debug Binary (rio-v2)
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-rustfs-debug-binary-rio-v2-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build debug binary with rio-v2
+        run: |
+          touch rustfs/build.rs
+          cargo build -p rustfs --bins --features rio-v2 --jobs 2
+
+      - name: Upload debug binary
+        uses: actions/upload-artifact@v6
+        with:
+          name: rustfs-debug-binary-rio-v2
+          path: target/debug/rustfs
+          if-no-files-found: error
+          retention-days: 1
+
   e2e-tests:
     name: End-to-End Tests
     needs: [ skip-check, build-rustfs-debug-binary ]
     if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-2
     timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Setup Rust toolchain for s3s-e2e installation
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Install s3s-e2e test tool
+        uses: taiki-e/cache-cargo-install-action@v2
+        with:
+          tool: s3s-e2e
+          git: https://github.com/s3s-project/s3s.git
+          rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
+
+      - name: Run end-to-end tests
+        run: |
+          s3s-e2e --version
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          RUSTFS_TEST_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
+          RUSTFS_TEST_PORT="${RUSTFS_TEST_PORT}" \
+          RUSTFS_TEST_LOG="${RUN_ROOT}/rustfs.log" \
+          ./scripts/e2e-run.sh ./target/debug/rustfs "${RUN_ROOT}/data"
+
+      - name: Upload test logs
+        if: failure()
+        uses: actions/upload-artifact@v6
+        with:
+          name: e2e-test-logs-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
+          retention-days: 3
+
+  e2e-tests-rio-v2:
+    name: End-to-End Tests (rio-v2)
+    needs: [ skip-check, build-rustfs-debug-binary-rio-v2 ]
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 30
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -190,7 +301,7 @@ jobs:
       - name: Download debug binary
         uses: actions/download-artifact@v7
         with:
-          name: rustfs-debug-binary
+          name: rustfs-debug-binary-rio-v2
           path: target/debug
 
       - name: Make binary executable
@@ -215,13 +326,13 @@ jobs:
         if: failure()
         uses: actions/upload-artifact@v6
         with:
-          name: e2e-test-logs-${{ github.run_number }}
+          name: e2e-test-logs-rio-v2-${{ github.run_number }}
           path: /tmp/rustfs.log
           retention-days: 3
 
   s3-implemented-tests:
     name: S3 Implemented Tests
-    needs: [ skip-check, build-rustfs-debug-binary ]
+    needs: [ skip-check, build-rustfs-debug-binary, e2e-tests ]
     if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-4
     timeout-minutes: 60
@@ -240,10 +351,16 @@ jobs:
 
       - name: Run implemented s3-tests
         run: |
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
           DEPLOY_MODE=binary \
           RUSTFS_BINARY=./target/debug/rustfs \
           TEST_MODE=single \
           MAXFAIL=1 \
+          S3_PORT="${S3_PORT}" \
+          DATA_ROOT="${RUN_ROOT}" \
+          S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
           ./scripts/s3-tests/run.sh
 
       - name: Upload s3 test artifacts

.github/workflows/docker.yml

@@ -73,7 +73,7 @@ jobs:
   # Check if we should build Docker images
   build-check:
     name: Docker Build Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       should_push: ${{ steps.check.outputs.should_push }}
@@ -421,7 +421,7 @@ jobs:
     name: Docker Build Summary
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Docker build completion summary
         run: |

.github/workflows/e2e-s3tests.yml

@@ -40,8 +40,8 @@ on:
 
 env:
   # main user
-  S3_ACCESS_KEY: rustfs-ci-admin
-  S3_SECRET_KEY: rustfs-ci-secret
+  S3_ACCESS_KEY: rustfsadmin
+  S3_SECRET_KEY: rustfsadmin
   # alt user (must be different from main for many s3-tests)
   S3_ALT_ACCESS_KEY: rustfsalt
   S3_ALT_SECRET_KEY: rustfsalt

.github/workflows/helm-package.yml

@@ -31,7 +31,7 @@ permissions:
 
 jobs:
   build-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: |
       github.event_name == 'workflow_dispatch' ||
       (
@@ -99,7 +99,7 @@ jobs:
           retention-days: 1
 
   publish-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     needs: [ build-helm-package ]
     if: needs.build-helm-package.result == 'success'
 

.github/workflows/issue-translator.yml

@@ -25,7 +25,7 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubicloud-standard-4
+    runs-on: ubuntu-latest
     steps:
       - uses: usthe/issues-translate-action@v2.7
         with:

.github/workflows/nix.yml

@@ -38,41 +38,68 @@ concurrency:
   cancel-in-progress: true
 permissions:
   contents: read
+  id-token: write
 
 jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
     timeout-minutes: 60
+    permissions:
+      contents: read
+      id-token: write
     env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      NIX_CURL_FLAGS: -A cargo/stable
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
 
       - name: Install Nix
-        uses: cachix/install-nix-action@v31
+        uses: DeterminateSystems/determinate-nix-action@v3.21.0
         with:
-          github_access_token: ${{ secrets.GITHUB_TOKEN }}
-          extra_nix_config: |
-            experimental-features = nix-command flakes
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          extra-conf: |
+            experimental-features = nix-command flakes configurable-impure-env
+            impure-env = NIX_CURL_FLAGS
+            max-jobs = 1
 
-      - name: Setup Magic Nix Cache
-        uses: DeterminateSystems/magic-nix-cache-action@v13
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@v3.21.0
 
-      - name: Setup Flake Checker
+      - name: Check Nix Flake Inputs
         uses: DeterminateSystems/flake-checker-action@v12
+        with:
+          fail-mode: true
+          ignore-missing-flake-lock: false
 
       - name: Verify Flake
         run: |
           echo "Checking flake structure and evaluation..."
           nix flake show
-          nix flake check --print-build-logs
+          for attempt in 1 2 3; do
+            if nix flake check --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix flake check failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Build RustFS
         run: |
           echo "Building the default package..."
-          nix build .#default --print-build-logs
+          for attempt in 1 2 3; do
+            if nix build .#default --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix build failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Test Binary
         run: |

.gitignore

@@ -42,6 +42,11 @@ artifacts/
 PR_DESCRIPTION.md
 scripts/s3-tests/selected_tests.txt
 docs
+__pycache__/
+!docs/
+docs/*
+!docs/architecture/
+!docs/architecture/**
 
 # nix stuff
 result*
@@ -53,3 +58,4 @@ rustfs-webdav.code-workspace
 # Local test/benchmark artifacts
 benchmarks.logs
 tmp/
+crates/*/docs

.vscode/launch.json

@@ -224,6 +224,47 @@
                 "rust"
             ],
         },
+        {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug executable 'rustfs' with rio-v2 full matrix",
+            "cargo": {
+                "args": [
+                    "build",
+                    "--bin=rustfs",
+                    "--package=rustfs",
+                    "--features=full,rio-v2"
+                ],
+                "filter": {
+                    "name": "rustfs",
+                    "kind": "bin"
+                }
+            },
+            "env": {
+                "RUST_LOG": "rustfs=error,ecstore=error,s3s=error,iam=error",
+                "RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS": "true",
+                "RUST_BACKTRACE": "full",
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/rio-v2/test{1...4}",
+                "RUSTFS_SCANNER_ENABLED": "false",
+                "RUSTFS_HEAL_ENABLED": "false",
+                "RUSTFS_REGION": "us-east-1",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs/rio-v2",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                "RUSTFS_COMPRESSION_ENABLED": "true",
+                "RUSTFS_KMS_ENABLE": "true",
+                "RUSTFS_KMS_BACKEND": "local",
+                "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
+                "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}"
+        },
         {
             "name": "E2E test executable target/debug/rustfs",
             "type":  "lldb",
@@ -256,4 +297,4 @@
             ],
         },
     ]
-}
+}

AGENTS.md

@@ -103,6 +103,26 @@ Do not open a PR with code changes when the required checks fail.
 - Use environment variables or vault tooling for sensitive configuration.
 - For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
+## Tools
+
+### xl.meta decode tool Quick Use
+
+```
+cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
+```
+
+## Serde Safety
+
+- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs).
+- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
+- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
+
+## Naming Conventions
+
+- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
+- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
+- If existing code violates naming conventions, do not widen the violation in new code. Fix opportunistically when touching the surrounding area.
+
 ## Scoped Guidance in This Repository
 
 - `.github/AGENTS.md`

ARCHITECTURE.md

@@ -6,6 +6,8 @@
 > If you want to familiarize yourself with the code base, you are in the right place!
 >
 > See also [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow.
+> See also [docs/architecture](docs/architecture/overview.md) for active
+> architecture migration guardrails.
 
 ## Bird's Eye View
 

CLAUDE.md

@@ -0,0 +1,188 @@
+# RustFS — CLAUDE.md
+
+S3-compatible object store in Rust, derived from MinIO. Erasure-coded, multi-pool, supports ILM tiering/lifecycle.
+
+## Commands
+
+```bash
+cargo build --release --bin rustfs        # production binary
+cargo build                               # dev build
+cargo check -p <crate>                    # fast type-check one crate
+cargo test -p <crate>                     # test one crate
+cargo fmt --all                           # format (required before PR)
+make pre-commit                           # full pre-PR gate (fmt + clippy + test)
+make build-docker BUILD_OS=ubuntu22.04    # Docker cross-build
+```
+
+> **Docker build note**: `buildx build` without `--load` keeps the image in the buildx cache only — `docker run` will use a stale local image. The Makefile already includes `--load`; if you suspect a stale binary, add `--no-cache` to the `buildx build` invocation inside `.config/make/build-docker.mak`.
+
+> Agent/PR rules: see `.github/copilot-instructions.md`.
+> Crate membership: `Cargo.toml` `[workspace].members`.
+> CI gates: `.github/workflows/ci.yml`.
+
+## Workspace layout
+
+```
+rustfs/src/main.rs                          # binary entry point
+crates/ecstore/src/
+  set_disk.rs                               # ErasureSet: transition_object, restore_transitioned_object
+  store.rs / store_api/                     # ECStore trait + ObjectInfo / TransitionedObject types
+  bucket/lifecycle/
+    bucket_lifecycle_ops.rs                 # ILM actions: transition_object, expire_transitioned_object,
+                                            #   get_transitioned_object_reader, gen_transition_objname
+    tier_sweeper.rs                         # background sweep: delete_object_from_remote_tier
+  tier/
+    warm_backend.rs                         # WarmBackend trait (put/get/remove/in_use)
+    warm_backend_s3.rs                      # HTTP-client based (TransitionClient) — used for S3/MinIO
+    warm_backend_s3sdk.rs                   # aws-sdk-s3 based — alternative S3 backend
+    warm_backend_minio.rs / _rustfs.rs / …  # per-provider wrappers (all delegate to _s3 or _s3sdk)
+    tier.rs                                 # TierConfigMgr, new_warm_backend dispatch
+  client/transition_api.rs                  # TransitionClient HTTP plumbing; UploadInfo, to_object_info
+  client/api_put_object_streaming.rs        # put_object_do → UploadInfo (version_id from x-amz-version-id)
+crates/filemeta/src/
+  filemeta.rs                               # FileMeta (xl.meta top-level), is_skip_meta_key
+  filemeta/version.rs                       # FileMetaVersion, MetaObject, MetaDeleteMarker
+                                            #   → to_fileinfo() reads transition_version_id
+                                            #   → set_transition() writes raw UUID bytes
+                                            #   → From<FileInfo> for MetaObject writes all meta
+  fileinfo.rs                               # FileInfo struct (transition_version_id: Option<Uuid>)
+  examples/
+    dump_fileinfo.rs                        # CLI: parse xl.meta, print transition fields + metadata
+    dump_versions.rs                        # CLI: list all versions in xl.meta
+crates/utils/src/http/metadata_compat.rs    # SUFFIX_* constants, insert_bytes/get_bytes (dual RustFS+MinIO keys)
+```
+
+## Metadata key conventions
+
+Internal metadata is stored under **both** `x-rustfs-internal-<suffix>` and `x-minio-internal-<suffix>` for MinIO interoperability. `get_bytes` prefers the RustFS key with MinIO fallback.
+
+Key suffixes (from `metadata_compat.rs`):
+| Suffix | Meaning |
+|--------|---------|
+| `transition-status` | `"complete"` when tiered |
+| `transitioned-object` | tier key path (without prefix) |
+| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) |
+| `transition-tier` | tier name |
+| `tier-free-versionID` | delete-marker version for free-version sweep |
+
+## Tier / ILM transition architecture
+
+### Transition flow (hot → cold)
+1. `transition_object` (lifecycle_ops) → `ECStore::transition_object` → `set_disk.rs`
+2. `gen_transition_objname(bucket)` → `{sha256_hash[0..16]}/{uuid[0..2]}/{uuid[2..4]}/{uuid}` (unique per object version)
+3. `tgt_client.put_with_meta(dest_obj, …)` → returns `rv: String` (remote S3 version_id, or `""`)
+4. `fi.transition_version_id = if rv.is_empty() { None } else { Some(Uuid::parse_str(&rv)?) }`
+5. `fi.transitioned_objname = dest_obj` (without tier prefix)
+6. Written to xl.meta via `MetaObject::from(FileInfo)` → `insert_bytes(SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes())` (16 raw bytes)
+
+### Tier GET flow (restore/read)
+`get_transitioned_object_reader` (lifecycle_ops):
+- reads `oi.transitioned_object.name` (= `fi.transitioned_objname`)
+- reads `oi.transitioned_object.version_id` (= `fi.transition_version_id.to_string()` or `""`)
+- calls `warm_backend.get(name, version_id, opts)`
+- `warm_backend_s3.rs::get`: adds `?versionId=…` only when `rv != ""`
+
+### Tier prefix handling
+`WarmBackendS3::get_dest(object)` prepends `self.prefix` to the object name.
+`transitioned_objname` is stored **without** the prefix — `get_dest` adds it on every call.
+
+### xl.meta on disk
+Path: `{disk}/{bucket}/{object}/xl.meta` — one per erasure shard disk.
+All shards should be identical for a healthy object.
+
+## Known bugs & fixes
+
+### Bug 1: `NoSuchVersion` on tier GET — nil UUID sent as versionId
+**Root cause**: `transitioned-versionID` metadata key exists with empty string value (0 bytes). Old reading code:
+```rust
+// OLD — unwrap_or_default() converts 0-byte or wrong-length slice to Uuid::nil()
+get_bytes(…).map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default())
+// → Some(Uuid::nil()) → sends ?versionId=00000000-… → NoSuchVersion
+```
+**Fix** (version.rs, `MetaObject::to_fileinfo` + `MetaDeleteMarker::to_fileinfo`):
+```rust
+get_bytes(…)
+    .and_then(|v| Uuid::from_slice(v.as_slice()).ok())  // None for wrong-length bytes
+    .filter(|u| !u.is_nil())                            // None for nil UUID (old write-back)
+```
+**Regression tests** (`crates/filemeta/src/filemeta/version.rs` `mod tests`): 6 tests cover absent key, empty bytes, nil UUID, and valid UUID round-trip for both `MetaObject` and `MetaDeleteMarker` paths.
+
+### Bug 2: `warm_backend_s3sdk.rs` ignored rv and range opts
+**Fix**: added `req.version_id(rv)` and `req.range(…)` to GET; `req.version_id(rv)` to DELETE.
+
+### Bug 4: `set_disk::copy_object` returns 501 for tiered objects (storage class restore)
+**Root cause**: `set_disk::copy_object` immediately returns `StorageError::NotImplemented` when `src_info.metadata_only = false`. For tiered objects, `metadata_only` is never set to `true` (guarded by `transitioned_object.tier.is_empty()`). So `mc cp --storage-class STANDARD obj obj` on a tiered object always returns 501.
+**Fix** (`crates/ecstore/src/set_disk.rs`, `copy_object`):
+```rust
+if !src_info.metadata_only {
+    if path_join_buf(&[src_bucket, src_object]) == path_join_buf(&[dst_bucket, dst_object]) {
+        if let Some(mut put_reader) = src_info.put_object_reader.take() {
+            return self.put_object(dst_bucket, dst_object, &mut put_reader, dst_opts).await;
+        }
+    }
+    return Err(StorageError::NotImplemented);
+}
+```
+When a self-copy has a `put_object_reader` (data already fetched from tier in `execute_copy_object`), writes it back locally via `put_object`, effectively de-tiering the object.
+**How `mc cp --storage-class STANDARD` flows**:
+1. mc sends `PUT /bucket/key` with `x-amz-copy-source`, `x-amz-metadata-directive: REPLACE`, `x-amz-storage-class: STANDARD`
+2. `execute_copy_object` → `get_object_reader` fetches data from tier backend → stores in `src_info.put_object_reader`
+3. `store.copy_object(...)` → now calls `put_object` with tier data and STANDARD storage class in `dst_opts`
+4. New xl.meta written locally with STANDARD class, no tier metadata → object de-tiered
+
+### Bug 3 (open): race in `expire_transitioned_object`
+Order is: delete remote tier version → delete local object.
+A concurrent GET between those two steps fetches a valid stored version_id but the tier version is already gone → `NoSuchVersion`.
+The proper fix is to delete local metadata first (making the object unreachable) before deleting the remote tier version.
+
+## Debugging tier issues
+
+### Inspect xl.meta directly
+```bash
+cargo build -p rustfs-filemeta --example dump_fileinfo
+./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta
+# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id
+```
+`transition_ver_id: <none>` → no versionId will be sent to tier (correct for non-versioned tier bucket).
+`transition_ver_id: <uuid>` → that UUID will be sent as `?versionId=<uuid>`.
+
+### Check what versionId is being sent at runtime
+Enable debug logging:
+```bash
+RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs …
+```
+Log line: `fetching transitioned object from tier` (DEBUG before request).
+Log line: `tier GET failed` (ERROR on failure, includes `tier_version_id`).
+
+### Metadata key to watch
+```
+x-minio-internal-transitioned-versionID=   ← empty string = will cause NoSuchVersion with old code
+x-rustfs-internal-transitioned-versionID=  ← same
+```
+If both are empty string, the object was transitioned to a non-versioned tier bucket. The versionId should NOT be sent — fixed by Bug 1 above.
+
+## Common patterns
+
+### Writing internal metadata (binary values)
+```rust
+insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes().to_vec());
+// stores under both x-rustfs-internal-* and x-minio-internal-* keys
+```
+
+### Reading internal metadata (binary values)
+```rust
+get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)
+    .and_then(|v| Uuid::from_slice(v.as_slice()).ok())
+    .filter(|u| !u.is_nil())
+// Returns None for: absent, wrong-length bytes, nil UUID
+```
+
+### WarmBackend trait
+```rust
+put_with_meta(object, reader, length, meta) -> Result<String>  // returns S3 version_id or ""
+put(object, reader, length) -> Result<String>
+get(object, rv, opts) -> Result<ReadCloser>  // rv="" means no versionId
+remove(object, rv) -> Result<()>
+in_use() -> Result<bool>
+```
+`rv` = remote version, always pass as empty string when `transition_version_id` is None.

Cargo.toml

@@ -38,13 +38,16 @@ members = [
     "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
+    "crates/rio-v2", # Next-generation Rust I/O compatibility layer
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-types", # S3 event type definitions
     "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
     "crates/scanner", # Scanner for data integrity checks and health monitoring
+    "crates/security-governance", # Security governance contracts
     "crates/signer", # client signer
+    "crates/storage-api", # Storage API contracts
     "crates/targets", # Target-specific configurations and utilities
     "crates/trusted-proxies", # Trusted proxies management
     "crates/tls-runtime", # Project-wide TLS runtime foundation
@@ -59,8 +62,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.95.0"
-version = "1.0.0-beta.4"
+rust-version = "1.96.0"
+version = "1.0.0-beta.8"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -77,67 +80,70 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.4" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.4" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.4" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.4" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.4" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.4" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.4" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.4" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.4" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.4" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.4" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.4" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.4" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.4" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.4" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.4" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.4" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.4" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.4" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.4" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.4" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.4" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.4" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.4" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.4" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.4" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.4" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.4" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.4" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.4" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.4" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.4" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.4" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.4" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.4" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.4" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.4" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.8" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.8" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.8" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.8" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.8" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.8" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.8" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.8" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.8" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.8" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.8" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.8" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.8" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.8" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.8" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.8" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.8" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.8" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.8" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.8" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.8" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.8" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.8" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.8" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.8" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.8" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.8" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.8" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.8" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.8" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.8" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.8" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.8" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.8" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.8" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.8" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.8" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.8" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.8" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.8" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "2bad388283bc3ce48801fc2ffcd22445eb6f3d24" }
+mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-async-nats = "0.48.0"
+async-nats = "0.49.1"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
-pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
+pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime"] }
 lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
-hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
+hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
-http = "1.4.0"
+http = "1.4.2"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
-reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
-socket2 = { version = "0.6.3", features = ["all"] }
+socket2 = { version = "0.6.4", features = ["all"] }
 tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
@@ -155,7 +161,7 @@ bytesize = "2.3.1"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
-prost = "0.14.3"
+prost = "0.14.4"
 quick-xml = "0.40.1"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
@@ -164,7 +170,7 @@ serde_json = { version = "1.0.150", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
-aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
+aes-gcm = { version = "0.11.0-rc.4", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.8" }
 blake2 = "0.11.0-rc.6"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
@@ -183,9 +189,9 @@ subtle = "2.6"
 zeroize = { version = "1.8.2", features = ["derive"] }
 
 # Time and Date
-chrono = { version = "0.4.44", features = ["serde"] }
+chrono = { version = "0.4.45", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.24", features = ["serde"] }
+jiff = { version = "0.2.28", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Database
@@ -197,16 +203,16 @@ tokio-postgres-rustls = "0.14.0"
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
 astral-tokio-tar = "0.6.2"
-atoi = "2.0.0"
+atoi = "3.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.17" }
+aws-config = { version = "1.8.18" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.133.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.8" }
+aws-sdk-s3 = { version = "1.135.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.13", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.9" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
-brotli = "8.0.2"
+brotli = "8.0.3"
 clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
@@ -215,14 +221,14 @@ crossbeam-queue = "0.3.12"
 crossbeam-channel = "0.5.15"
 crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
-datafusion = "53.1.0"
+datafusion = "54.0.0"
 derive_builder = "0.20.2"
 enumset = "1.1.13"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.12.0"
-google-cloud-auth = "1.10.0"
+google-cloud-storage = "1.14.0"
+google-cloud-auth = "1.12.0"
 hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
@@ -254,23 +260,23 @@ ratelimit = "0.10.1"
 rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
-regex = { version = "1.12.3" }
+regex = { version = "1.12.4" }
 rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
-redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
+redis = { version = "1.2.2", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
 s3s = { git = "https://github.com/rustfs/s3s", rev = "507e1312b211c3ddc214b03875d6fabd15d22ed5", features = ["minio"] }
-serial_test = "3.4.0"
+serial_test = "3.5.0"
 shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
-snafu = "0.9.0"
+snafu = "0.9.1"
 snap = "1.1.1"
 starshard = { version = "2.2.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.39.2"
+sysinfo = "0.39.3"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -283,7 +289,7 @@ tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.23.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.23.3", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
@@ -298,17 +304,17 @@ dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.32.0" }
 opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
 opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.32.0", features = ["rt-tokio"] }
+opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
 opentelemetry-semantic-conventions = { version = "0.32.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.0.4", features = ["backend-pprof-rs"] }
+pyroscope = { version = "=2.0.5", features = ["backend-pprof-rs", "backend-jemalloc"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
-suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+suppaftp = { version = "8.0.4", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.8"
-russh = { version = "0.60.3", git = "https://github.com/Eugeny/russh", rev = "fc6e3ab4cd4338e94ae64e17aeed2acee9335e6b" }
+russh = { version = "0.61.2", features = ["serde"] }
 russh-sftp = "2.3.0"
 
 # WebDAV
@@ -324,7 +330,9 @@ tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profilin
 jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
 # pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
-# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
+# Use pprof-pyroscope-fork to match pyroscope 2.0.5's internal pprof dependency (v0.1500.4).
+# Cargo unifies them into a single instance, avoiding duplicate perf_signal_handler symbols.
+# NOTE: Do NOT upgrade pyroscope to >=2.0.6 — it vendors pprof-rs internally, causing symbol collision.
 pprof = { package = "pprof-pyroscope-fork", version = "0.1500.4", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]

Dockerfile.glibc

@@ -105,6 +105,8 @@ RUN groupadd -g 10001 rustfs && \
 
 ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_CONSOLE_ADDRESS=":9001" \
+    RUSTFS_ACCESS_KEY="rustfsadmin" \
+    RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \

README.md

@@ -7,6 +7,7 @@
   <a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="Build and Push Docker Images" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
   <img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
   <img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
+  <a href="https://discord.gg/NcKBCEJp6P"><img alt="Discord" src="https://img.shields.io/badge/Discord-Join%20Chat-5865F2?logo=discord&logoColor=white" /></a>
   <a href="https://hellogithub.com/repository/rustfs/rustfs" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=b95bcb72bdc340b68f16fdf6790b7d5b&claim_uid=MsbvjYeLDKAH457&theme=small" alt="Featured｜HelloGitHub" /></a>
 </p>
 
@@ -102,7 +103,7 @@ curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
 
 ### 2. Docker Quick Start (Option 2)
 
-The RustFS container runs as a non-root user `rustfs` (UID `10001`). If you run Docker with `-v` to mount a host directory, please ensure the host directory owner is set to `10001`, otherwise you will encounter permission denied errors.
+The RustFS container runs as a non-root user `rustfs` (UID/GID `10001:10001`). If you bind-mount host directories with Docker or Compose, every mounted path must be writable by that user, otherwise startup may fail with permission denied errors. This applies to data directories, log directories, and TLS certificate directories when `RUSTFS_TLS_PATH` is enabled.
 
 ```bash
 # Create data and logs directories
@@ -115,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -124,12 +125,26 @@ If you use [podman](https://github.com/containers/podman) instead of docker, you
 podman run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 ```
 
+If you enable TLS with a bind-mounted certificate directory, prepare that mount the same way:
+
+```bash
+mkdir -p certs
+chown -R 10001:10001 certs
+```
+
 You can also use Docker Compose. Using the `docker-compose.yml` file in the root directory:
 
 ```bash
 docker compose --profile observability up -d
 ```
 
+Before running Compose with host bind mounts:
+
+- Ensure every mounted host path is writable by `10001:10001`.
+- If you enable TLS, ensure the certificate mount for `/opt/tls` is also readable by `10001:10001`.
+- If matching host ownership is not practical, run the `rustfs` service with `user: "<host-uid>:<host-gid>"` instead.
+- `docker-compose-simple.yml` includes a `volume-permission-helper` service for named volumes. `docker-compose.yml` relies on you to prepare bind-mounted host paths in advance.
+
 Similarly, you can run the command with podman
 
 ```bash
@@ -295,6 +310,7 @@ If you have any questions or need assistance:
 - [Documentation](https://docs.rustfs.com) - The manual you should read
 - [Changelog](https://github.com/rustfs/rustfs/releases) - What we broke and fixed
 - [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Where the community lives
+- [Discord](https://discord.gg/NcKBCEJp6P) - Chat with the RustFS community
 
 ## Contact
 

README_ZH.md

@@ -100,20 +100,27 @@ curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
 
 ### 2\. Docker 快速启动 (选项 2)
 
-RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录，请务必确保宿主机目录的所有者已更改为 `10001`，否则会遇到权限拒绝错误。
+RustFS 容器以非 root 用户 `rustfs` (UID/GID `10001:10001`) 运行。如果您通过 Docker 或 Compose 绑定挂载宿主机目录，请务必确保所有挂载路径都对该用户可写，否则启动时可能出现权限拒绝错误。这不仅适用于数据目录和日志目录，也适用于启用 `RUSTFS_TLS_PATH` 时挂载的 TLS 证书目录。
 
 ```bash
- # 创建数据和日志目录
- mkdir -p data logs
+# 创建数据和日志目录
+mkdir -p data logs
 
- # 更改这两个目录的所有者
- chown -R 10001:10001 data logs
+# 更改这两个目录的所有者
+chown -R 10001:10001 data logs
 
- # 使用最新版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
+# 使用最新版本运行
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
- # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
+# 使用指定版本运行
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
+```
+
+如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
+
+```bash
+mkdir -p certs
+chown -R 10001:10001 certs
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：
@@ -122,6 +129,13 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
 docker compose --profile observability up -d
 ```
 
+在使用 Compose 且包含宿主机绑定挂载时，请先确认：
+
+- 所有挂载到容器内的宿主机路径都对 `10001:10001` 可写。
+- 如果启用了 TLS，挂载到 `/opt/tls` 的证书目录也需要对 `10001:10001` 可读。
+- 如果不方便调整宿主机目录归属，可以为 `rustfs` 服务显式设置 `user: "<host-uid>:<host-gid>"`。
+- `docker-compose-simple.yml` 为命名 volume 提供了 `volume-permission-helper` 服务；`docker-compose.yml` 使用宿主机绑定挂载，因此需要您在启动前自行准备目录权限。
+
 **注意**: 我们建议您在运行前查看 `docker-compose.yml` 文件。该文件定义了包括 Grafana、Prometheus 和 Jaeger 在内的多个服务，有助于 RustFS 的可观测性监控。如果您还想启动 Redis 或 Nginx 容器，可以指定相应的 profile。
 
 ### 3\. 源码编译 (选项 3) - 进阶用户

crates/AGENTS.md

@@ -14,6 +14,41 @@ Applies to all paths under `crates/`.
 - Keep integration tests under each crate's `tests/` directory.
 - Add regression tests for bug fixes and behavior changes.
 
+## Error Type Design
+
+- Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`.
+- Do not use `Box<dyn Error>` or `Box<dyn Error + Send + Sync>` in public trait methods or struct methods. Define a concrete error type with specific variants.
+- When implementing `std::error::Error`, always override `fn source()` if you store an inner error. Breaking the error chain makes debugging impossible.
+- Internal helpers that return `Result<_, String>` and are immediately wrapped via `.map_err(Error::other)` should return the actual error type directly.
+
+## Concurrency
+
+- Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths.
+- Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics).
+- When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff.
+- `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`.
+
+## Recursion Safety
+
+- Recursive tree/graph traversals must have a depth limit (e.g., `max_depth` counter) or use an iterative approach with an explicit `Vec` stack.
+- This applies to cache trees, directory walks, and any user-influenced hierarchy.
+- A corrupted or malicious input must not be able to overflow the thread stack.
+
+## Type Casting
+
+- Never use `as` for numeric conversions that may truncate or overflow. Use `try_into()` with explicit error handling, or clamp with `value.max(0) as usize` when the domain is bounded.
+- `f64 as usize` saturates but is fragile; clamp to `[0, usize::MAX as f64]` first.
+- Treat every `as` cast in a PR review as a potential bug; require justification.
+
+## Testing
+
+- Keep unit tests close to the module they test.
+- Keep integration tests under each crate's `tests/` directory.
+- Add regression tests for bug fixes and behavior changes.
+- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test.
+- In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input.
+
 ## Async and Performance
 
 - Keep async paths non-blocking.

crates/common/src/heal_channel.rs

@@ -110,6 +110,25 @@ pub enum HealScanMode {
     Deep = 2,
 }
 
+impl HealScanMode {
+    pub const fn as_str(self) -> &'static str {
+        match self {
+            Self::Unknown => "unknown",
+            Self::Normal => "normal",
+            Self::Deep => "deep",
+        }
+    }
+
+    pub const fn from_u8(value: u8) -> Option<Self> {
+        match value {
+            0 => Some(Self::Unknown),
+            1 => Some(Self::Normal),
+            2 => Some(Self::Deep),
+            _ => None,
+        }
+    }
+}
+
 impl Serialize for HealScanMode {
     fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
     where
@@ -137,12 +156,7 @@ impl<'de> Deserialize<'de> for HealScanMode {
             where
                 E: serde::de::Error,
             {
-                match value {
-                    0 => Ok(HealScanMode::Unknown),
-                    1 => Ok(HealScanMode::Normal),
-                    2 => Ok(HealScanMode::Deep),
-                    _ => Err(E::custom(format!("invalid HealScanMode value: {value}"))),
-                }
+                HealScanMode::from_u8(value).ok_or_else(|| E::custom(format!("invalid HealScanMode value: {value}")))
             }
 
             fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>

crates/config/README.md

@@ -65,11 +65,34 @@ Current guidance:
 ## Scanner environment aliases
 
 - `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)
+- `RUSTFS_SCANNER_DELAY` (canonical)
+- `RUSTFS_SCANNER_MAX_WAIT_SECS` (canonical)
 - `RUSTFS_SCANNER_CYCLE` (canonical, also accepts `MINIO_SCANNER_CYCLE`)
 - `RUSTFS_SCANNER_START_DELAY_SECS` (canonical)
 - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` (deprecated alias for compatibility)
 - `RUSTFS_SCANNER_IDLE_MODE` (canonical)
 - `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical)
+
+## Health compatibility switches
+
+- `RUSTFS_HEALTH_ENDPOINT_ENABLE`
+  - controls canonical `/health`, `/health/live`, and `/health/ready` endpoint exposure.
+- `RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE`
+  - enables minimal payload mode for GET health responses (`status`, `ready` only).
+- `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
+  - TTL for readiness cache evaluation.
+- `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
+  - enables busy protection behavior for health probes.
+  - default is `false`.
+- `RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS`
+  - max active HTTP requests; health probes return `429` when active requests reach or exceed this value.
+  - `0` disables thresholding even if busy protection is enabled.
+- `RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE`
+  - enables KMS readiness enforcement for `/health/ready`.
+  - default is `false`.
 
 ## Drive timeout environment variables
 

crates/config/src/audit/kafka.rs

@@ -21,10 +21,14 @@ pub const ENV_AUDIT_KAFKA_TLS_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_TLS_ENABLE";
 pub const ENV_AUDIT_KAFKA_TLS_CA: &str = "RUSTFS_AUDIT_KAFKA_TLS_CA";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_KAFKA_SASL_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_SASL_ENABLE";
+pub const ENV_AUDIT_KAFKA_SASL_MECHANISM: &str = "RUSTFS_AUDIT_KAFKA_SASL_MECHANISM";
+pub const ENV_AUDIT_KAFKA_SASL_USERNAME: &str = "RUSTFS_AUDIT_KAFKA_SASL_USERNAME";
+pub const ENV_AUDIT_KAFKA_SASL_PASSWORD: &str = "RUSTFS_AUDIT_KAFKA_SASL_PASSWORD";
 pub const ENV_AUDIT_KAFKA_QUEUE_DIR: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_DIR";
 pub const ENV_AUDIT_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 14] = &[
     ENV_AUDIT_KAFKA_ENABLE,
     ENV_AUDIT_KAFKA_BROKERS,
     ENV_AUDIT_KAFKA_TOPIC,
@@ -33,6 +37,10 @@ pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
     ENV_AUDIT_KAFKA_TLS_CA,
     ENV_AUDIT_KAFKA_TLS_CLIENT_CERT,
     ENV_AUDIT_KAFKA_TLS_CLIENT_KEY,
+    ENV_AUDIT_KAFKA_SASL_ENABLE,
+    ENV_AUDIT_KAFKA_SASL_MECHANISM,
+    ENV_AUDIT_KAFKA_SASL_USERNAME,
+    ENV_AUDIT_KAFKA_SASL_PASSWORD,
     ENV_AUDIT_KAFKA_QUEUE_DIR,
     ENV_AUDIT_KAFKA_QUEUE_LIMIT,
 ];
@@ -47,6 +55,10 @@ pub const AUDIT_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,

crates/config/src/constants/app.rs

@@ -155,12 +155,6 @@ pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
-/// Environment variable to explicitly allow public default root credentials.
-///
-/// This is intended for local development only. Production startup paths should
-/// provide non-default `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` values.
-pub const ENV_RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS: &str = "RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS";
-
 /// Environment variable controlling startup behavior when unsupported filesystem types are detected.
 ///
 /// Accepted values:

crates/config/src/constants/heal.rs

@@ -12,6 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+/// Heal admin config subsystem name.
+pub const HEAL_SUB_SYS: &str = "heal";
+
+/// Heal config key setting the scanner-driven periodic deep bitrot scan cycle in seconds.
+pub const HEAL_BITROT_CYCLE: &str = "bitrot_cycle";
+
+/// Heal config keys supported by the admin config subsystem.
+pub const HEAL_KEYS: &[&str] = &[HEAL_BITROT_CYCLE];
+
+/// Default scanner-driven bitrot scan cycle used by heal/scanner runtime config.
+pub const DEFAULT_HEAL_BITROT_CYCLE_SECS: u64 = 30 * 24 * 60 * 60;
+
 /// Environment variable name that enables or disables auto-heal functionality.
 /// - Purpose: Control whether the system automatically performs heal operations.
 /// - Valid values: "true" or "false" (case insensitive).

crates/config/src/constants/health.rs

@@ -26,3 +26,20 @@ pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
 /// When enabled, only `status` and `ready` fields are returned.
 pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
 pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;
+
+/// Enable busy protection for health probes.
+/// When enabled with a positive request threshold, alias health probes may
+/// return 429 when active HTTP requests exceed the threshold.
+pub const ENV_HEALTH_COMPAT_BUSY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_CHECK_ENABLE: bool = false;
+
+/// Max active HTTP requests; alias health probes report busy (429) when active requests reach or exceed this value.
+/// Set to 0 to disable thresholding even when busy protection is enabled.
+pub const ENV_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: &str = "RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: usize = 0;
+
+/// Enable KMS readiness check for alias readiness probes.
+/// When enabled, `/health/ready` additionally requires KMS service to be
+/// in running state if a global KMS manager exists.
+pub const ENV_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: bool = false;

crates/config/src/constants/object.rs

@@ -281,6 +281,40 @@ pub const ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT: &str = "RUSTFS_OBJECT_LOCK_ACQUIRE_TI
 /// Default lock acquisition timeout: 5 seconds.
 pub const DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT: u64 = 5;
 
+/// Environment variable to enable object namespace lock diagnostics.
+///
+/// When enabled, RustFS emits slow lock acquisition and long lock hold
+/// warnings for object-level namespace locks. This is intended for
+/// production debugging of contention on hot object keys.
+///
+/// Default: false (disabled, can be overridden by `RUSTFS_OBJECT_LOCK_DIAG_ENABLE`).
+pub const ENV_OBJECT_LOCK_DIAG_ENABLE: &str = "RUSTFS_OBJECT_LOCK_DIAG_ENABLE";
+
+/// Default: object lock diagnostics are disabled.
+pub const DEFAULT_OBJECT_LOCK_DIAG_ENABLE: bool = false;
+
+/// Environment variable for the slow object lock acquisition threshold in milliseconds.
+///
+/// When a read or write namespace lock takes at least this long to acquire,
+/// RustFS emits a warning with the operation name and object key.
+///
+/// Default: 500 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS";
+
+/// Default slow object lock acquisition threshold: 500 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: u64 = 500;
+
+/// Environment variable for the long object lock hold threshold in milliseconds.
+///
+/// When a namespace lock guard is held for at least this long, RustFS emits
+/// a warning when the guard is dropped.
+///
+/// Default: 1000 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_HOLD_MS";
+
+/// Default long object lock hold threshold: 1000 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: u64 = 1000;
+
 // ============================================================================
 // I/O priority scheduling configuration
 // ============================================================================

crates/config/src/constants/oidc.rs

@@ -28,6 +28,7 @@ pub const OIDC_GROUPS_CLAIM: &str = "groups_claim";
 pub const OIDC_ROLES_CLAIM: &str = "roles_claim";
 pub const OIDC_EMAIL_CLAIM: &str = "email_claim";
 pub const OIDC_USERNAME_CLAIM: &str = "username_claim";
+pub const OIDC_HIDE_FROM_UI: &str = "hide_from_ui";
 
 // Environment variable names for OIDC
 pub const ENV_IDENTITY_OPENID_ENABLE: &str = "RUSTFS_IDENTITY_OPENID_ENABLE";
@@ -46,9 +47,10 @@ pub const ENV_IDENTITY_OPENID_GROUPS_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_GROUP
 pub const ENV_IDENTITY_OPENID_ROLES_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_ROLES_CLAIM";
 pub const ENV_IDENTITY_OPENID_EMAIL_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM";
 pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM";
+pub const ENV_IDENTITY_OPENID_HIDE_FROM_UI: &str = "RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI";
 
 /// List of all environment variable keys for an OIDC provider.
-pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
+pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
     ENV_IDENTITY_OPENID_ENABLE,
     ENV_IDENTITY_OPENID_CONFIG_URL,
     ENV_IDENTITY_OPENID_CLIENT_ID,
@@ -65,6 +67,7 @@ pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
     ENV_IDENTITY_OPENID_ROLES_CLAIM,
     ENV_IDENTITY_OPENID_EMAIL_CLAIM,
     ENV_IDENTITY_OPENID_USERNAME_CLAIM,
+    ENV_IDENTITY_OPENID_HIDE_FROM_UI,
 ];
 
 /// A list of all valid configuration keys for an OIDC provider.
@@ -85,6 +88,7 @@ pub const IDENTITY_OPENID_KEYS: &[&str] = &[
     OIDC_ROLES_CLAIM,
     OIDC_EMAIL_CLAIM,
     OIDC_USERNAME_CLAIM,
+    OIDC_HIDE_FROM_UI,
     crate::COMMENT_KEY,
 ];
 

crates/config/src/constants/runtime.rs

@@ -69,6 +69,10 @@ pub const DEFAULT_TRANSITION_QUEUE_CAPACITY: usize = 1000;
 pub const DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS: usize = 100;
 /// Test-only fault injection env var that forces the immediate transition enqueue timeout path.
 pub const ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT: &str = "RUSTFS_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT";
+/// Test-only fault injection env var that forces a number of IAM bootstrap failures.
+pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS";
+/// Test-only env var that overrides the deferred IAM retry interval in debug builds.
+pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
 /// Runtime env var controlling the transition worker count.
 pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
 /// Runtime env var controlling the absolute maximum transition workers.

crates/config/src/constants/scanner.rs

@@ -14,6 +14,84 @@
 
 use std::time::Duration;
 
+/// Scanner admin config subsystem name.
+pub const SCANNER_SUB_SYS: &str = "scanner";
+
+/// Scanner config key selecting the speed preset.
+pub const SCANNER_SPEED: &str = "speed";
+
+/// Scanner config key overriding the scanner sleep multiplier.
+pub const SCANNER_DELAY: &str = "delay";
+
+/// Scanner config key overriding the maximum scanner sleep in seconds.
+pub const SCANNER_MAX_WAIT: &str = "max_wait";
+
+/// Scanner config key overriding the cycle interval in seconds.
+pub const SCANNER_CYCLE: &str = "cycle";
+
+/// Scanner config key setting the startup delay in seconds.
+///
+/// For compatibility, this also acts as the scanner cycle interval when
+/// `cycle` is unset.
+pub const SCANNER_START_DELAY: &str = "start_delay";
+
+/// Scanner config key capping one cycle's runtime in seconds.
+pub const SCANNER_CYCLE_MAX_DURATION: &str = "cycle_max_duration";
+
+/// Scanner config key capping objects processed by one cycle.
+pub const SCANNER_CYCLE_MAX_OBJECTS: &str = "cycle_max_objects";
+
+/// Scanner config key capping directories entered by one cycle.
+pub const SCANNER_CYCLE_MAX_DIRECTORIES: &str = "cycle_max_directories";
+
+/// Scanner config key setting the periodic bitrot scan cycle in seconds.
+pub const SCANNER_BITROT_CYCLE: &str = "bitrot_cycle";
+
+/// Scanner config key controlling whether scanner throttling is enabled.
+pub const SCANNER_IDLE_MODE: &str = "idle_mode";
+
+/// Scanner config key controlling scanner cache save timeout in seconds.
+pub const SCANNER_CACHE_SAVE_TIMEOUT: &str = "cache_save_timeout";
+
+/// Scanner config key capping concurrent scanner set tasks.
+pub const SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "max_concurrent_set_scans";
+
+/// Scanner config key capping concurrent scanner disk bucket walks per set.
+pub const SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "max_concurrent_disk_scans";
+
+/// Scanner config key controlling how often object loops yield.
+pub const SCANNER_YIELD_EVERY_N_OBJECTS: &str = "yield_every_n_objects";
+
+/// Scanner config key controlling object version count alerts.
+pub const SCANNER_ALERT_EXCESS_VERSIONS: &str = "alert_excess_versions";
+
+/// Scanner config key controlling retained version size alerts.
+pub const SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "alert_excess_version_size";
+
+/// Scanner config key controlling direct subfolder count alerts.
+pub const SCANNER_ALERT_EXCESS_FOLDERS: &str = "alert_excess_folders";
+
+/// Scanner config keys supported by the admin config subsystem.
+pub const SCANNER_KEYS: &[&str] = &[
+    SCANNER_SPEED,
+    SCANNER_DELAY,
+    SCANNER_MAX_WAIT,
+    SCANNER_CYCLE,
+    SCANNER_START_DELAY,
+    SCANNER_CYCLE_MAX_DURATION,
+    SCANNER_CYCLE_MAX_OBJECTS,
+    SCANNER_CYCLE_MAX_DIRECTORIES,
+    SCANNER_BITROT_CYCLE,
+    SCANNER_IDLE_MODE,
+    SCANNER_CACHE_SAVE_TIMEOUT,
+    SCANNER_MAX_CONCURRENT_SET_SCANS,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
+    SCANNER_ALERT_EXCESS_VERSIONS,
+    SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_FOLDERS,
+];
+
 /// Canonical environment variable name that specifies the scanner start delay in seconds.
 /// If set, this overrides the cycle interval derived from `RUSTFS_SCANNER_SPEED`.
 /// - Unit: seconds (u64).
@@ -31,15 +109,52 @@ pub const ENV_DATA_SCANNER_START_DELAY_SECS: &str = "RUSTFS_DATA_SCANNER_START_D
 /// - Example: `export RUSTFS_SCANNER_CYCLE=3600` (1 hour)
 pub const ENV_SCANNER_CYCLE: &str = "RUSTFS_SCANNER_CYCLE";
 
+/// Environment variable that caps one scanner cycle's runtime in seconds.
+/// A value of `0` disables the cycle runtime budget.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS=1800`
+pub const ENV_SCANNER_CYCLE_MAX_DURATION_SECS: &str = "RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS";
+
+/// Environment variable that caps objects processed by one scanner cycle.
+/// A value of `0` disables the per-cycle object budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_OBJECTS=1000000`
+pub const ENV_SCANNER_CYCLE_MAX_OBJECTS: &str = "RUSTFS_SCANNER_CYCLE_MAX_OBJECTS";
+
+/// Environment variable that caps directories entered by one scanner cycle.
+/// A value of `0` disables the per-cycle directory budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000`
+pub const ENV_SCANNER_CYCLE_MAX_DIRECTORIES: &str = "RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES";
+
 /// Environment variable that selects the scanner speed preset.
 /// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
 /// Controls the sleep factor, maximum sleep duration, and cycle interval.
 /// - Example: `export RUSTFS_SCANNER_SPEED=slow`
 pub const ENV_SCANNER_SPEED: &str = "RUSTFS_SCANNER_SPEED";
 
+/// Environment variable that overrides the scanner sleep multiplier.
+/// - Example: `export RUSTFS_SCANNER_DELAY=30.0`
+pub const ENV_SCANNER_DELAY: &str = "RUSTFS_SCANNER_DELAY";
+
+/// Environment variable that overrides the maximum scanner sleep in seconds.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_MAX_WAIT_SECS=15`
+pub const ENV_SCANNER_MAX_WAIT_SECS: &str = "RUSTFS_SCANNER_MAX_WAIT_SECS";
+
 /// Default scanner speed preset.
 pub const DEFAULT_SCANNER_SPEED: &str = "default";
 
+/// Default scanner cycle runtime budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS: u64 = 0;
+
+/// Default scanner per-cycle object budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_OBJECTS: u64 = 0;
+
+/// Default scanner per-cycle directory budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES: u64 = 0;
+
 /// Environment variable that specifies the periodic bitrot scan cycle in seconds.
 /// When set to `0`, `true`, `on`, or `yes`, every scanner cycle runs in deep mode.
 /// When set to `false`, `off`, `no`, or `disabled`, periodic deep scans are disabled.
@@ -65,11 +180,12 @@ pub const ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "RUSTFS_SCANNER_ALERT_EX
 pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE: u64 = 1024 * 1024 * 1024 * 1024;
 
 /// Environment variable that controls how many subfolders trigger scanner alerts.
-/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS=50000`
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS=65538`
 pub const ENV_SCANNER_ALERT_EXCESS_FOLDERS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS";
 
 /// Default subfolder count that triggers scanner alerts.
-pub const DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS: u64 = 50_000;
+/// Allows Proxmox Backup Server's chunk namespace layout.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS: u64 = 65_538;
 
 /// Environment variable that controls whether the scanner sleeps between operations.
 /// When `true` (default), the scanner throttles itself. When `false`, it runs at full speed.
@@ -82,9 +198,38 @@ pub const ENV_SCANNER_IDLE_MODE: &str = "RUSTFS_SCANNER_IDLE_MODE";
 /// - Example: `export RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS=30`
 pub const ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS: &str = "RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS";
 
+/// Default scanner cache save timeout in seconds.
+pub const DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS: u64 = 30;
+
+/// Environment variable that caps concurrent scanner set tasks.
+/// A value of `0` keeps the existing topology-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS=2`
+pub const ENV_SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS";
+
+/// Environment variable that caps concurrent scanner disk bucket walks per set.
+/// A value of `0` keeps the existing disk-count-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS=1`
+pub const ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS";
+
+/// Environment variable that controls how often scanner object loops yield to the async runtime.
+/// A value of `0` disables this extra object-count yield.
+/// - Example: `export RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS=32`
+pub const ENV_SCANNER_YIELD_EVERY_N_OBJECTS: &str = "RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS";
+
 /// Default scanner idle mode.
 pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
 
+/// Default set scan concurrency budget.
+/// `0` means no additional limit beyond deployment topology.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 0;
+
+/// Default disk scan concurrency budget.
+/// `0` means no additional limit beyond available disks in the set.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 0;
+
+/// Default object interval for cooperative scanner yields.
+pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
+
 /// Compatibility flag kept for Patch 3 rollback windows.
 ///
 /// Inline scanner heal execution has been removed in favor of heal-candidate enqueue.
@@ -109,7 +254,9 @@ pub const DEFAULT_SCANNER_INLINE_HEAL_ENABLE: bool = false;
 /// | `slow`    | 10x    | 15 seconds| 1 minute       |
 /// | `slowest` | 100x   | 15 seconds| 30 minutes     |
 ///
-/// The cycle interval can be overridden by `RUSTFS_SCANNER_CYCLE`.
+/// The sleep factor, max sleep, and cycle interval can be overridden by
+/// `RUSTFS_SCANNER_DELAY`, `RUSTFS_SCANNER_MAX_WAIT_SECS`, and
+/// `RUSTFS_SCANNER_CYCLE`.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
 pub enum ScannerSpeed {
     Fastest,
@@ -148,15 +295,20 @@ impl ScannerSpeed {
         }
     }
 
-    pub fn from_env_str(s: &str) -> Self {
+    pub fn parse_str(s: &str) -> Option<Self> {
         match s.trim().to_ascii_lowercase().as_str() {
-            "fastest" => Self::Fastest,
-            "fast" => Self::Fast,
-            "slow" => Self::Slow,
-            "slowest" => Self::Slowest,
-            _ => Self::Default,
+            "fastest" => Some(Self::Fastest),
+            "fast" => Some(Self::Fast),
+            "default" => Some(Self::Default),
+            "slow" => Some(Self::Slow),
+            "slowest" => Some(Self::Slowest),
+            _ => None,
         }
     }
+
+    pub fn from_env_str(s: &str) -> Self {
+        Self::parse_str(s).unwrap_or_default()
+    }
 }
 
 impl std::fmt::Display for ScannerSpeed {

crates/config/src/constants/targets.rs

@@ -49,6 +49,10 @@ pub const KAFKA_TLS_ENABLE: &str = "tls_enable";
 pub const KAFKA_TLS_CA: &str = "tls_ca";
 pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const KAFKA_SASL_ENABLE: &str = "sasl_enable";
+pub const KAFKA_SASL_MECHANISM: &str = "sasl_mechanism";
+pub const KAFKA_SASL_USERNAME: &str = "sasl_username";
+pub const KAFKA_SASL_PASSWORD: &str = "sasl_password";
 
 pub const AMQP_URL: &str = "url";
 pub const AMQP_EXCHANGE: &str = "exchange";

crates/config/src/notify/kafka.rs

@@ -22,6 +22,10 @@ pub const NOTIFY_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,
@@ -36,10 +40,14 @@ pub const ENV_NOTIFY_KAFKA_TLS_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_TLS_ENABLE";
 pub const ENV_NOTIFY_KAFKA_TLS_CA: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CA";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_KAFKA_SASL_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_SASL_ENABLE";
+pub const ENV_NOTIFY_KAFKA_SASL_MECHANISM: &str = "RUSTFS_NOTIFY_KAFKA_SASL_MECHANISM";
+pub const ENV_NOTIFY_KAFKA_SASL_USERNAME: &str = "RUSTFS_NOTIFY_KAFKA_SASL_USERNAME";
+pub const ENV_NOTIFY_KAFKA_SASL_PASSWORD: &str = "RUSTFS_NOTIFY_KAFKA_SASL_PASSWORD";
 pub const ENV_NOTIFY_KAFKA_QUEUE_DIR: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_DIR";
 pub const ENV_NOTIFY_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 14] = &[
     ENV_NOTIFY_KAFKA_ENABLE,
     ENV_NOTIFY_KAFKA_BROKERS,
     ENV_NOTIFY_KAFKA_TOPIC,
@@ -48,6 +56,10 @@ pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
     ENV_NOTIFY_KAFKA_TLS_CA,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY,
+    ENV_NOTIFY_KAFKA_SASL_ENABLE,
+    ENV_NOTIFY_KAFKA_SASL_MECHANISM,
+    ENV_NOTIFY_KAFKA_SASL_USERNAME,
+    ENV_NOTIFY_KAFKA_SASL_PASSWORD,
     ENV_NOTIFY_KAFKA_QUEUE_DIR,
     ENV_NOTIFY_KAFKA_QUEUE_LIMIT,
 ];

crates/data-usage/src/data_usage.rs

@@ -688,15 +688,14 @@ impl DataUsageCache {
             return;
         }
 
-        let mut leaves = Vec::new();
+        let mut candidates = Vec::new();
         let mut remove = total - limit;
-        add(self, path, &mut leaves);
-        leaves.sort_by_key(|a| a.objects);
+        add(self, path, &mut candidates);
+        candidates.sort_by_key(|a| a.objects);
 
-        while remove > 0 && !leaves.is_empty() {
-            let Some(e) = leaves.first() else {
-                break;
-            };
+        let mut candidate_index = 0;
+        while remove > 0 && candidate_index < candidates.len() {
+            let e = &candidates[candidate_index];
             let candidate = e.path.clone();
             if candidate == *path && !compact_self {
                 break;
@@ -705,7 +704,7 @@ impl DataUsageCache {
             let mut flat = match self.size_recursive(&candidate.key()) {
                 Some(flat) => flat,
                 None => {
-                    leaves.remove(0);
+                    candidate_index += 1;
                     continue;
                 }
             };
@@ -714,8 +713,8 @@ impl DataUsageCache {
             self.delete_recursive(&candidate);
             self.replace_hashed(&candidate, &None, &flat);
 
-            remove -= removing;
-            leaves.remove(0);
+            remove = remove.saturating_sub(removing);
+            candidate_index += 1;
         }
     }
 
@@ -869,23 +868,24 @@ struct Inner {
     path: DataUsageHash,
 }
 
-fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, leaves: &mut Vec<Inner>) {
+fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, candidates: &mut Vec<Inner>) -> usize {
     let e = match data_usage_cache.cache.get(&path.key()) {
         Some(e) => e,
-        None => return,
+        None => return 0,
     };
-    if !e.children.is_empty() {
-        return;
-    }
-
-    let sz = data_usage_cache.size_recursive(&path.key()).unwrap_or_default();
-    leaves.push(Inner {
-        objects: sz.objects,
-        path: path.clone(),
-    });
+    let mut objects = e.objects;
     for ch in e.children.iter() {
-        add(data_usage_cache, &DataUsageHash(ch.clone()), leaves);
+        objects += add(data_usage_cache, &DataUsageHash(ch.clone()), candidates);
     }
+    // Collect internal nodes (with children) as compaction candidates.
+    // Leaf nodes have no children to remove, so compacting them is a no-op.
+    if !e.children.is_empty() {
+        candidates.push(Inner {
+            objects,
+            path: path.clone(),
+        });
+    }
+    objects
 }
 
 fn mark(duc: &DataUsageCache, entry: &DataUsageEntry, found: &mut HashSet<String>) {
@@ -1361,4 +1361,161 @@ mod tests {
         assert_eq!(child_entry.size, 30);
         assert_eq!(child_entry.objects, 3);
     }
+
+    // --- Tests for `add` and `reduce_children_of` (bug fixes) ---
+
+    /// Build a small tree: root -> child1 (leaf), child2 -> grandchild (leaf).
+    fn build_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let c1 = hash_path("bucket/a");
+        let c2 = hash_path("bucket/b");
+        let gc = hash_path("bucket/b/c");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(&root, &None, &DataUsageEntry::default());
+        cache.replace_hashed(
+            &c1,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                size: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &c2,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 2,
+                size: 20,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &gc,
+            &Some(c2.clone()),
+            &DataUsageEntry {
+                objects: 3,
+                size: 30,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    fn build_underflow_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let small = hash_path("bucket/small");
+        let small_a = hash_path("bucket/small/a");
+        let small_b = hash_path("bucket/small/b");
+        let large = hash_path("bucket/large");
+        let large_a = hash_path("bucket/large/a");
+        let large_b = hash_path("bucket/large/b");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(
+            &root,
+            &None,
+            &DataUsageEntry {
+                objects: 100,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&small, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &small_a,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &small_b,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&large, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &large_a,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &large_b,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    #[test]
+    fn test_add_collects_internal_nodes_as_compaction_candidates() {
+        let (cache, root) = build_test_tree();
+        let mut candidates = Vec::new();
+        add(&cache, &root, &mut candidates);
+
+        let mut paths: Vec<String> = candidates.iter().map(|l| l.path.key()).collect();
+        paths.sort();
+        assert_eq!(paths.len(), 2, "add() should find internal nodes with children");
+        assert!(paths.contains(&hash_path("bucket").key()));
+        assert!(paths.contains(&hash_path("bucket/b").key()));
+    }
+
+    #[test]
+    fn test_add_skips_leaf_node() {
+        let mut cache = DataUsageCache::default();
+        let h = hash_path("single-leaf");
+        cache.replace_hashed(
+            &h,
+            &None,
+            &DataUsageEntry {
+                objects: 5,
+                size: 50,
+                ..Default::default()
+            },
+        );
+
+        let mut candidates = Vec::new();
+        add(&cache, &h, &mut candidates);
+        assert!(candidates.is_empty(), "leaf node should not be a compaction candidate");
+    }
+
+    #[test]
+    fn test_reduce_children_of_compacts_internal_node() {
+        let (mut cache, root) = build_test_tree();
+        cache.reduce_children_of(&root, 2, false);
+
+        let entry_c2 = cache.find("bucket/b").unwrap();
+        assert!(entry_c2.compacted, "internal node 'bucket/b' should be compacted");
+        let entry_c1 = cache.find("bucket/a").unwrap();
+        assert!(!entry_c1.compacted, "leaf 'bucket/a' should not be compacted");
+        assert!(cache.find("bucket/b/c").is_none(), "grandchild should be removed");
+    }
+
+    #[test]
+    fn test_reduce_children_of_usize_underflow_saturates() {
+        let (mut cache, root) = build_underflow_test_tree();
+
+        // total children=6, limit=5, remove=1. The smallest candidate removes
+        // two descendants, so plain subtraction would underflow and compact the
+        // next candidate too.
+        cache.reduce_children_of(&root, 5, false);
+
+        assert!(cache.find("bucket/small").is_some_and(|entry| entry.compacted));
+        assert!(cache.find("bucket/small/a").is_none());
+        assert!(cache.find("bucket/small/b").is_none());
+        assert!(cache.find("bucket/large").is_some_and(|entry| !entry.compacted));
+        assert!(cache.find("bucket/large/a").is_some());
+        assert!(cache.find("bucket/large/b").is_some());
+    }
 }

crates/e2e_test/src/admin_timeout_regression_test.rs

@@ -0,0 +1,208 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestClusterEnvironment, init_logging, local_http_client};
+use aws_sdk_s3::primitives::ByteStream;
+use http::header::HOST;
+use reqwest::StatusCode;
+use rustfs_madmin::{ITEM_OFFLINE, InfoMessage, StorageInfo};
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serial_test::serial;
+use std::error::Error;
+use std::process::Command;
+use tokio::time::{Duration, sleep, timeout};
+use uuid::Uuid;
+
+const BUCKET: &str = "issue-2525-admin-timeout";
+
+async fn signed_admin_get(
+    url: &str,
+    access_key: &str,
+    secret_key: &str,
+) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+    let request = http::Request::builder()
+        .method(http::Method::GET)
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+        .body(Body::empty())?;
+    let signed = sign_v4(request, 0, access_key, secret_key, "", "us-east-1");
+
+    let mut builder = local_http_client().get(url);
+    for (name, value) in signed.headers() {
+        builder = builder.header(name, value);
+    }
+
+    Ok(builder.send().await?)
+}
+
+async fn fetch_info(cluster: &RustFSTestClusterEnvironment) -> Result<InfoMessage, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}/rustfs/admin/v3/info", cluster.nodes[0].url);
+    let response = signed_admin_get(&url, &cluster.access_key, &cluster.secret_key).await?;
+    parse_json_response(response, "cluster info").await
+}
+
+async fn fetch_storage_info(cluster: &RustFSTestClusterEnvironment) -> Result<StorageInfo, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}/rustfs/admin/v3/storageinfo", cluster.nodes[0].url);
+    let response = signed_admin_get(&url, &cluster.access_key, &cluster.secret_key).await?;
+    parse_json_response(response, "storage info").await
+}
+
+async fn parse_json_response<T: serde::de::DeserializeOwned>(
+    response: reqwest::Response,
+    label: &str,
+) -> Result<T, Box<dyn Error + Send + Sync>> {
+    let status = response.status();
+    let body = response.bytes().await?;
+    if status != StatusCode::OK {
+        return Err(format!("{label} request failed: {status} {}", String::from_utf8_lossy(body.as_ref())).into());
+    }
+
+    Ok(serde_json::from_slice(&body)?)
+}
+
+fn signal_process(pid: u32, signal: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let output = Command::new("kill").arg(format!("-{signal}")).arg(pid.to_string()).output()?;
+    if output.status.success() {
+        return Ok(());
+    }
+
+    Err(format!("kill -{signal} {pid} failed: {}", String::from_utf8_lossy(&output.stderr)).into())
+}
+
+fn offline_server_count(info: &InfoMessage) -> usize {
+    info.servers
+        .as_ref()
+        .map(|servers| servers.iter().filter(|server| server.state == ITEM_OFFLINE).count())
+        .unwrap_or_default()
+}
+
+#[tokio::test(flavor = "multi_thread")]
+#[serial]
+async fn test_single_admin_timeout_does_not_immediately_mark_peer_offline() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let warm_info = fetch_info(&cluster).await?;
+    assert_eq!(
+        warm_info
+            .backend
+            .as_ref()
+            .map(|backend| backend.offline_disks)
+            .unwrap_or_default(),
+        0,
+        "cluster should start with zero offline disks"
+    );
+
+    let warm_storage = fetch_storage_info(&cluster).await?;
+    assert_eq!(
+        warm_storage.backend.offline_disks.sum(),
+        0,
+        "storage info should start with zero offline disks"
+    );
+
+    let suspended_pid = cluster.nodes[1]
+        .process
+        .as_ref()
+        .ok_or("cluster node 1 process missing")?
+        .id();
+    signal_process(suspended_pid, "STOP")?;
+
+    let resume = tokio::spawn(async move {
+        sleep(Duration::from_secs(6)).await;
+        signal_process(suspended_pid, "CONT")
+    });
+
+    let client = cluster.create_s3_client(0)?;
+    let key = format!("issue-2525/{}", Uuid::new_v4().simple());
+    let payload = b"admin timeout regression payload".to_vec();
+
+    let during = timeout(Duration::from_secs(20), async {
+        tokio::join!(fetch_info(&cluster), fetch_storage_info(&cluster), async {
+            client
+                .put_object()
+                .bucket(BUCKET)
+                .key(&key)
+                .body(ByteStream::from(payload.clone()))
+                .send()
+                .await?;
+            Ok::<(), Box<dyn Error + Send + Sync>>(())
+        })
+    })
+    .await;
+
+    resume.await??;
+
+    let (info_during, storage_during, put_result) =
+        during.map_err(|_| "timed out while waiting for admin queries during suspended peer")?;
+    let info_during = info_during?;
+    let storage_during = storage_during?;
+    put_result?;
+
+    assert_eq!(
+        info_during
+            .backend
+            .as_ref()
+            .map(|backend| backend.offline_disks)
+            .unwrap_or_default(),
+        0,
+        "single admin timeout must not synthesize offline disks in /info"
+    );
+    assert_eq!(
+        offline_server_count(&info_during),
+        0,
+        "single admin timeout must not mark any server offline in /info"
+    );
+    assert_eq!(
+        storage_during.backend.offline_disks.sum(),
+        0,
+        "single admin timeout must not synthesize offline disks in /storageinfo"
+    );
+    assert!(
+        storage_during.disks.iter().all(|disk| disk.state != ITEM_OFFLINE),
+        "single admin timeout must not synthesize offline disk entries in /storageinfo"
+    );
+
+    let recovered_info = fetch_info(&cluster).await?;
+    assert_eq!(
+        recovered_info
+            .backend
+            .as_ref()
+            .map(|backend| backend.offline_disks)
+            .unwrap_or_default(),
+        0,
+        "offline disk count should stay at zero after the peer resumes"
+    );
+
+    let body = client
+        .get_object()
+        .bucket(BUCKET)
+        .key(&key)
+        .send()
+        .await?
+        .body
+        .collect()
+        .await?
+        .into_bytes();
+    assert_eq!(body.as_ref(), b"admin timeout regression payload");
+
+    Ok(())
+}

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -60,6 +60,125 @@ mod tests {
         assert_eq!(body.as_ref(), expected, "object body changed for {key}");
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn test_auto_heal_rebuilds_runtime_wiped_disk_without_restart() {
+        init_logging();
+        info!("Issue #1533: auto heal should rebuild a runtime-wiped disk in a 4-disk single-node erasure set without restart");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        let root = PathBuf::from(env.temp_dir.clone());
+        let disk0 = root.join("disk0");
+        let disk1 = root.join("disk1");
+        let disk2 = root.join("disk2");
+        let disk3 = root.join("disk3");
+        for disk in [&disk0, &disk1, &disk2, &disk3] {
+            std::fs::create_dir_all(disk).expect("disk directory should be created");
+        }
+
+        env.temp_dir = disk3.to_string_lossy().to_string();
+        let disk0_arg = disk0.to_string_lossy().to_string();
+        let disk1_arg = disk1.to_string_lossy().to_string();
+        let disk2_arg = disk2.to_string_lossy().to_string();
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[
+                ("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true"),
+                ("RUSTFS_HEAL_INTERVAL_SECS", "10"),
+            ],
+        )
+        .await
+        .expect("Failed to start 4-disk RustFS");
+        env.temp_dir = root.to_string_lossy().to_string();
+
+        let client = env.create_s3_client();
+        let bucket = "heal-runtime-wiped-disk";
+        let heal_timeout_secs = std::env::var("RUSTFS_AUTO_HEAL_RUNTIME_WIPE_TIMEOUT_SECS")
+            .ok()
+            .and_then(|value| value.parse::<u64>().ok())
+            .unwrap_or(45);
+
+        let objects: Vec<(String, Vec<u8>, &'static str)> = vec![
+            (
+                "runtime/alpha.txt".to_string(),
+                b"alpha payload for runtime wipe heal".to_vec(),
+                "text/plain; charset=utf-8",
+            ),
+            ("runtime/beta.bin".to_string(), (0..=127).collect::<Vec<u8>>(), "application/octet-stream"),
+            (
+                "runtime/dir/emoji-free-name.json".to_string(),
+                br#"{"status":"runtime-heal"}"#.to_vec(),
+                "application/json",
+            ),
+            (
+                "runtime/dir/gamma.txt".to_string(),
+                b"gamma payload for runtime wipe heal".to_vec(),
+                "text/plain; charset=utf-8",
+            ),
+        ];
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("bucket create should succeed");
+        for (key, body, content_type) in &objects {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .content_type(*content_type)
+                .body(ByteStream::from(body.clone()))
+                .send()
+                .await
+                .expect("PUT should succeed");
+        }
+
+        for (key, body, _) in &objects {
+            assert_object_body(&env, bucket, key, body).await;
+            assert!(
+                object_metadata_exists_on_disk(&disk0, bucket, key),
+                "disk0 should contain xl.meta before runtime wipe for {key}"
+            );
+        }
+
+        std::fs::remove_dir_all(&disk0).expect("disk0 wipe should succeed while server is running");
+        std::fs::create_dir_all(&disk0).expect("disk0 should be recreated empty while server is running");
+        assert!(!has_file_under(&disk0), "disk0 must be empty immediately after runtime wipe");
+
+        let mut remaining_rebuild_keys: HashSet<String> = objects.iter().map(|(key, _, _)| key.clone()).collect();
+        for _ in 0..heal_timeout_secs {
+            for (key, body, _) in &objects {
+                assert_object_body(&env, bucket, key, body).await;
+            }
+
+            if !remaining_rebuild_keys.is_empty() {
+                let mut rebuilt = Vec::new();
+                for key in &remaining_rebuild_keys {
+                    if object_metadata_exists_on_disk(&disk0, bucket, key) {
+                        rebuilt.push(key.clone());
+                    }
+                }
+                for key in rebuilt {
+                    let _ = remaining_rebuild_keys.remove(&key);
+                }
+            }
+
+            if remaining_rebuild_keys.is_empty() {
+                assert!(
+                    disk0.join(".rustfs.sys").join("format.json").is_file(),
+                    "runtime-wiped disk should have format.json restored by auto heal"
+                );
+                return;
+            }
+
+            sleep(Duration::from_secs(1)).await;
+        }
+
+        panic!("auto heal did not rebuild all files on the runtime-wiped disk within timeout");
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_admin_deep_heal_rebuilds_cleared_disk_in_single_node_erasure_set() {

crates/e2e_test/src/kms/README.md

@@ -191,7 +191,7 @@ RUST_LOG=debug cargo test test_local_kms_end_to_end -- --nocapture
 4. **Monitor ports**
    ```bash
    netstat -an | grep 9050
-   curl http://127.0.0.1:9050/minio/health/ready
+   curl http://127.0.0.1:9050/health/ready
    ```
 
 ## 📊 Coverage

crates/e2e_test/src/kms/bucket_default_encryption_test.rs

@@ -520,3 +520,75 @@ async fn test_explicit_encryption_overrides_bucket_default() -> Result<(), Box<d
 
     Ok(())
 }
+
+/// Test 5: Setting SSE-KMS without a specific key ID should auto-populate the
+/// default KMS key ID so that GetBucketEncryption returns it (issue #3039).
+#[tokio::test]
+#[serial]
+async fn test_sse_kms_without_key_id_populates_default() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Testing SSE-KMS without explicit key ID populates default key");
+
+    let mut kms_env = LocalKMSTestEnvironment::new().await?;
+    let default_key_id = kms_env.start_rustfs_for_local_kms().await?;
+    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
+
+    let s3_client = kms_env.base_env.create_s3_client();
+    kms_env.base_env.create_test_bucket(TEST_BUCKET).await?;
+
+    // Set bucket encryption to SSE-KMS WITHOUT specifying a key ID
+    info!("Setting bucket default encryption to SSE-KMS without key ID");
+    let encryption_config = ServerSideEncryptionConfiguration::builder()
+        .rules(
+            ServerSideEncryptionRule::builder()
+                .apply_server_side_encryption_by_default(
+                    ServerSideEncryptionByDefault::builder()
+                        .sse_algorithm(ServerSideEncryption::AwsKms)
+                        .kms_master_key_id("")
+                        .build()
+                        .unwrap(),
+                )
+                .build(),
+        )
+        .build()
+        .unwrap();
+
+    s3_client
+        .put_bucket_encryption()
+        .bucket(TEST_BUCKET)
+        .server_side_encryption_configuration(encryption_config)
+        .send()
+        .await
+        .expect("Failed to set bucket SSE-KMS encryption without key ID");
+
+    // GetBucketEncryption should return the default KMS key ID
+    info!("Verifying GetBucketEncryption returns default KMS key ID");
+    let get_response = s3_client
+        .get_bucket_encryption()
+        .bucket(TEST_BUCKET)
+        .send()
+        .await
+        .expect("Failed to get bucket encryption");
+
+    let config = get_response.server_side_encryption_configuration();
+    let rule = config
+        .and_then(|c| c.rules().first())
+        .expect("Should have at least one encryption rule");
+    let by_default = rule
+        .apply_server_side_encryption_by_default()
+        .expect("Should have ApplyServerSideEncryptionByDefault");
+
+    assert_eq!(by_default.sse_algorithm(), &ServerSideEncryption::AwsKms, "Algorithm should be aws:kms");
+    assert!(
+        by_default.kms_master_key_id().is_some(),
+        "KMS key ID should be populated with the default key (was None)"
+    );
+    assert_eq!(
+        by_default.kms_master_key_id().unwrap(),
+        &default_key_id,
+        "KMS key ID should match the configured default key"
+    );
+
+    info!("Test passed: SSE-KMS without key ID correctly populates default key '{}'", default_key_id);
+    Ok(())
+}

crates/e2e_test/src/lib.rs

@@ -63,6 +63,10 @@ mod archive_download_integrity_test;
 #[cfg(test)]
 mod list_objects_v2_pagination_test;
 
+// Regression test for Issue #3107: mc mirror small-bucket listing must not time out.
+#[cfg(test)]
+mod mc_mirror_small_bucket_test;
+
 // Policy variables tests
 #[cfg(test)]
 mod policy;
@@ -142,4 +146,10 @@ mod replication_extension_test;
 #[cfg(test)]
 mod snowball_auto_extract_test;
 
+#[cfg(test)]
+mod namespace_lock_quorum_test;
+
+#[cfg(test)]
+mod admin_timeout_regression_test;
+
 pub mod tls_gen;

crates/e2e_test/src/mc_mirror_small_bucket_test.rs

@@ -0,0 +1,124 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, RustFSTestEnvironment};
+use serial_test::serial;
+use std::path::Path;
+use std::process::Command;
+use std::time::Duration;
+use tokio::fs;
+use tokio::time::timeout;
+use tracing::info;
+
+const BUCKET: &str = "ddd";
+const OBJECT_COUNT: usize = 484;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn create_issue_3107_fixture(root: &Path) -> TestResult {
+    for idx in 0..OBJECT_COUNT {
+        let object_path = root.join("ddd").join("requirements").join(format!("file-{idx:04}.txt"));
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, format!("issue-3107 payload {idx}\n").repeat(128)).await?;
+    }
+
+    for dir in ["aaa", "ccc", "ccc-package"] {
+        let object_path = root.join(dir).join("marker.txt");
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, dir.as_bytes()).await?;
+    }
+
+    Ok(())
+}
+
+fn mc_available() -> bool {
+    Command::new("mc")
+        .arg("--version")
+        .output()
+        .is_ok_and(|output| output.status.success())
+}
+
+fn run_mc(args: &[&str]) -> TestResult {
+    let output = Command::new("mc").args(args).output()?;
+    if !output.status.success() {
+        return Err(format!(
+            "mc {:?} failed: status={:?}, stdout={}, stderr={}",
+            args,
+            output.status.code(),
+            String::from_utf8_lossy(&output.stdout),
+            String::from_utf8_lossy(&output.stderr)
+        )
+        .into());
+    }
+    Ok(())
+}
+
+fn count_files(root: &Path) -> usize {
+    walkdir::WalkDir::new(root)
+        .into_iter()
+        .filter_map(Result::ok)
+        .filter(|entry| entry.file_type().is_file())
+        .count()
+}
+
+#[tokio::test]
+#[serial]
+async fn test_mc_mirror_small_bucket_completes_without_list_timeout() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting issue #3107 mc mirror regression test");
+    if !mc_available() {
+        info!("Skipping issue #3107 mc mirror regression test because mc is not installed");
+        return Ok(());
+    }
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    env.create_test_bucket(BUCKET).await?;
+
+    let alias = format!("rustfs-issue-3107-{}", uuid::Uuid::new_v4());
+    run_mc(&["alias", "set", &alias, &env.url, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY])?;
+
+    let fixture = Path::new(&env.temp_dir).join("issue-3107-fixture");
+    let backup = Path::new(&env.temp_dir).join("issue-3107-backup");
+    fs::create_dir_all(&fixture).await?;
+    fs::create_dir_all(&backup).await?;
+    create_issue_3107_fixture(&fixture).await?;
+
+    let bucket_target = format!("{alias}/{BUCKET}");
+    let source_path = fixture.to_string_lossy().to_string();
+    run_mc(&["mirror", "--overwrite", &source_path, &bucket_target])?;
+
+    let backup_path = backup.join("ddd-backup");
+    let backup_path_string = backup_path.to_string_lossy().to_string();
+    timeout(
+        Duration::from_secs(20),
+        tokio::task::spawn_blocking({
+            let bucket_target = bucket_target.clone();
+            let backup_path_string = backup_path_string.clone();
+            move || run_mc(&["mirror", "--overwrite", &bucket_target, &backup_path_string])
+        }),
+    )
+    .await
+    .map_err(|_| "mc mirror timed out while listing/comparing a small bucket")???;
+
+    let mirrored_count = count_files(&backup_path);
+    assert_eq!(
+        mirrored_count,
+        OBJECT_COUNT + 3,
+        "mc mirror should copy every object from the issue #3107 small bucket fixture"
+    );
+
+    run_mc(&["alias", "remove", &alias])?;
+    Ok(())
+}

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -0,0 +1,234 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::RustFSTestClusterEnvironment;
+use aws_sdk_s3::Client;
+use aws_sdk_s3::error::SdkError;
+use bytes::Bytes;
+use serial_test::serial;
+use std::sync::Arc;
+use tokio::sync::Barrier;
+use tracing::{info, warn};
+
+const BUCKET: &str = "namespace-lock-quorum-bucket";
+const KEY: &str = "thumb/79/concurrent-overwrite.jpg";
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .body(Bytes::from(payload).into())
+        .send()
+        .await
+        .map(|_| ())
+        .map_err(|err| format_s3_error(err, writer_id))
+}
+
+fn format_s3_error(err: SdkError<aws_sdk_s3::operation::put_object::PutObjectError>, writer_id: usize) -> String {
+    match err {
+        SdkError::ServiceError(service_err) => {
+            let err = service_err.err();
+            let code = err.meta().code().unwrap_or("<unknown>");
+            let message = err.meta().message().unwrap_or("<empty>");
+            format!("writer {writer_id} returned service error {code}: {message}")
+        }
+        other => format!("writer {writer_id} returned SDK error: {other:?}"),
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting namespace lock quorum regression test with auto cluster");
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    // Keep the regression focused on false quorum-loss errors, not ordinary lock
+    // wait exhaustion under a heavily contended same-key overwrite workload.
+    cluster.set_env("RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT", "20");
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let clients = cluster.create_all_clients()?;
+    let first_payload = b"initial object contents".to_vec();
+    put_object(clients[0].clone(), first_payload, 0).await?;
+
+    let writer_count = clients.len() * 2;
+    let barrier = Arc::new(Barrier::new(writer_count));
+    let mut handles = Vec::with_capacity(writer_count);
+
+    for writer_id in 0..writer_count {
+        let client = clients[writer_id % clients.len()].clone();
+        let barrier = barrier.clone();
+        let payload = format!("replacement payload from writer {writer_id:02}").into_bytes();
+        handles.push(tokio::spawn(async move {
+            barrier.wait().await;
+            put_object(client, payload, writer_id).await
+        }));
+    }
+
+    let mut failures = Vec::new();
+    for handle in handles {
+        match handle.await {
+            Ok(Ok(())) => {}
+            Ok(Err(err)) => failures.push(err),
+            Err(err) => failures.push(format!("writer task join failed: {err}")),
+        }
+    }
+
+    if !failures.is_empty() {
+        for failure in &failures {
+            warn!("concurrent overwrite failure: {}", failure);
+        }
+    }
+
+    assert!(
+        failures.is_empty(),
+        "concurrent overwrites must not surface namespace lock quorum failures: {failures:#?}"
+    );
+
+    let body = clients[0]
+        .get_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .send()
+        .await?
+        .body
+        .collect()
+        .await?
+        .into_bytes();
+    let body = std::str::from_utf8(&body)?;
+    assert!(
+        body.starts_with("replacement payload from writer "),
+        "final object body should be one of the successful overwrite payloads, got {body:?}"
+    );
+
+    clients[0].delete_object().bucket(BUCKET).key(KEY).send().await?;
+    Ok(())
+}
+
+/// Regression test: concurrent PUTs to the same key must return 503 (ServiceUnavailable)
+/// on lock contention, never 500 (InternalError).
+///
+/// Before the fix, `map_namespace_lock_error` wrapped lock timeout/conflict errors as
+/// `StorageError::other(...)` → `StorageError::Io(...)`, which fell through to
+/// `S3ErrorCode::InternalError` (500) in the error mapping.
+#[tokio::test]
+#[serial]
+async fn test_concurrent_put_same_key_never_returns_500() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting concurrent PUT 500 regression test");
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    // Short lock timeout to trigger contention errors quickly
+    cluster.set_env("RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT", "3");
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let clients = cluster.create_all_clients()?;
+    let writer_count = clients.len() * 4; // 16 writers for heavy contention
+    let barrier = Arc::new(Barrier::new(writer_count));
+
+    // Seed initial object
+    let first_payload = b"initial object for 500 regression".to_vec();
+    put_object(clients[0].clone(), first_payload, 0).await?;
+
+    let err_500_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+    let err_503_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+    let unexpected_err_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+    let ok_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+
+    let mut handles = Vec::with_capacity(writer_count);
+    for writer_id in 0..writer_count {
+        let client = clients[writer_id % clients.len()].clone();
+        let barrier = barrier.clone();
+        let payload = format!("payload from writer {writer_id:02}").into_bytes();
+        let err_500 = err_500_count.clone();
+        let err_503 = err_503_count.clone();
+        let unexpected_err = unexpected_err_count.clone();
+        let ok = ok_count.clone();
+
+        handles.push(tokio::spawn(async move {
+            barrier.wait().await;
+            match client
+                .put_object()
+                .bucket(BUCKET)
+                .key(KEY)
+                .body(Bytes::from(payload).into())
+                .send()
+                .await
+            {
+                Ok(_) => {
+                    ok.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                }
+                Err(err) => match &err {
+                    SdkError::ServiceError(service_err) => {
+                        let code = service_err.err().meta().code().unwrap_or("<unknown>");
+                        if code == "500" || code == "InternalError" {
+                            err_500.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                            warn!("writer {writer_id} returned 500: {code}");
+                        } else if code == "503" || code == "ServiceUnavailable" {
+                            err_503.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                        } else {
+                            unexpected_err.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                            warn!("writer {writer_id} returned unexpected error: {code}");
+                        }
+                    }
+                    other => {
+                        unexpected_err.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                        warn!("writer {writer_id} returned SDK error: {other:?}");
+                    }
+                },
+            }
+        }));
+    }
+
+    for handle in handles {
+        if let Err(err) = handle.await {
+            unexpected_err_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+            warn!("writer task join failed: {err}");
+        }
+    }
+
+    let ok = ok_count.load(std::sync::atomic::Ordering::Relaxed);
+    let err_503 = err_503_count.load(std::sync::atomic::Ordering::Relaxed);
+    let err_500 = err_500_count.load(std::sync::atomic::Ordering::Relaxed);
+    let unexpected_err = unexpected_err_count.load(std::sync::atomic::Ordering::Relaxed);
+    let total = ok + err_503 + err_500 + unexpected_err;
+
+    info!("Concurrent PUT 500 regression: total={total}, ok={ok}, 503={err_503}, 500={err_500}, unexpected={unexpected_err}");
+
+    assert_eq!(
+        total, writer_count as u64,
+        "every concurrent PUT writer must be classified as success, 503, 500, or unexpected error"
+    );
+
+    assert_eq!(
+        unexpected_err, 0,
+        "Concurrent PUTs to the same key must only succeed or return 503. \
+         Got {unexpected_err} unexpected errors out of {total} requests. 503 count: {err_503}, ok: {ok}"
+    );
+
+    assert_eq!(
+        err_500, 0,
+        "Concurrent PUTs to the same key must NEVER return 500 InternalError. \
+         Got {err_500} out of {total} requests. 503 count: {err_503}, ok: {ok}"
+    );
+
+    clients[0].delete_object().bucket(BUCKET).key(KEY).send().await?;
+    Ok(())
+}

crates/e2e_test/src/snowball_auto_extract_test.rs

@@ -70,6 +70,36 @@ mod tests {
         Ok(builder.into_inner().await?.into_inner())
     }
 
+    fn build_archive_with_parent_dir_entry(victim_bucket: &str) -> Vec<u8> {
+        let path = format!("../{victim_bucket}/evil-injected.txt");
+        let data = b"injected-body";
+        let mut header = [0u8; 512];
+
+        header[..path.len()].copy_from_slice(path.as_bytes());
+        header[100..108].copy_from_slice(b"0000644\0");
+        header[108..116].copy_from_slice(b"0000000\0");
+        header[116..124].copy_from_slice(b"0000000\0");
+        let size = format!("{:011o}\0", data.len());
+        header[124..136].copy_from_slice(size.as_bytes());
+        header[136..148].copy_from_slice(b"00000000000\0");
+        header[148..156].fill(b' ');
+        header[156] = b'0';
+        header[257..263].copy_from_slice(b"ustar\0");
+        header[263..265].copy_from_slice(b"00");
+
+        let checksum: u32 = header.iter().map(|byte| *byte as u32).sum();
+        let checksum = format!("{:06o}\0 ", checksum);
+        header[148..156].copy_from_slice(checksum.as_bytes());
+
+        let mut archive = Vec::new();
+        archive.extend_from_slice(&header);
+        archive.extend_from_slice(data);
+        let padding = (512 - (data.len() % 512)) % 512;
+        archive.extend(std::iter::repeat_n(0, padding));
+        archive.extend_from_slice(&[0u8; 1024]);
+        archive
+    }
+
     #[tokio::test]
     #[serial]
     async fn snowball_auto_extract_supports_minio_prefix_and_directory_markers() -> Result<(), Box<dyn Error + Send + Sync>> {
@@ -273,6 +303,49 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn snowball_auto_extract_rejects_parent_dir_entry_without_cross_bucket_write()
+    -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(vec![]).await?;
+
+        let client = env.create_s3_client();
+        let attacker_bucket = "snowball-traversal-source";
+        let victim_bucket = "snowball-traversal-victim";
+        let archive = build_archive_with_parent_dir_entry(victim_bucket);
+
+        client.create_bucket().bucket(attacker_bucket).send().await?;
+        client.create_bucket().bucket(victim_bucket).send().await?;
+
+        let err = client
+            .put_object()
+            .bucket(attacker_bucket)
+            .key("fixture.tar")
+            .metadata("Snowball-Auto-Extract", "true")
+            .body(ByteStream::from(archive))
+            .send()
+            .await
+            .expect_err("parent directory archive entry should be rejected");
+        let service_err = err.into_service_error();
+        assert_eq!(service_err.code(), Some("InvalidArgument"));
+
+        let victim_err = client
+            .head_object()
+            .bucket(victim_bucket)
+            .key("evil-injected.txt")
+            .send()
+            .await
+            .expect_err("rejected archive entry must not write into the victim bucket");
+        let victim_service_err = victim_err.into_service_error();
+        assert_eq!(victim_service_err.code(), Some("NotFound"));
+
+        env.stop_server();
+        Ok(())
+    }
+
     #[tokio::test]
     #[serial]
     async fn snowball_auto_extract_prefers_exact_minio_prefix_over_suffix_fallback() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/ecstore/AGENTS.md

@@ -14,6 +14,30 @@ Applies to `crates/ecstore/`.
 - Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
 - Benchmark-sensitive changes should include measurable rationale.
 
+## Allocation Discipline in Hot Paths
+
+- Do not implement `Clone` on structs with >5 heap-allocated fields without considering `Arc` for heavy fields.
+- Before cloning a struct in a loop or per-request path, check if a reference or `Cow` would suffice.
+- When a struct contains a large buffer (e.g., erasure coding block), wrap it in `Arc` and pass by reference rather than cloning.
+- Use `&str` and `Cow<str>` instead of `String` for temporary computations (header parsing, signature building, path manipulation).
+- Use `HashMap::with_capacity()` / `Vec::with_capacity()` when the size is known or estimable.
+
+## Lock Ordering
+
+- When a function acquires multiple locks, document the acquisition order in a comment.
+- Never acquire the same set of locks in different orders across code paths — this is a deadlock.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters.
+
+## Recursion Safety
+
+- Recursive tree traversals (cache trees, directory walks) must have a depth limit or use iterative traversal with an explicit `Vec` stack.
+- `flatten()`, `delete_recursive()`, `copy_with_children()`, `total_children_rec()`, `mark()` — all must handle deep or corrupted trees safely.
+
+## Dead Code
+
+- Do not add `#![allow(dead_code)]` at the crate root. If code is unused, remove it or gate it behind a feature flag.
+- Each `#[allow(dead_code)]` annotation must have a comment explaining why the code is kept.
+
 ## Cross-Module Coordination
 
 - Validate behavior impacts on:

crates/ecstore/Cargo.toml

@@ -32,12 +32,15 @@ workspace = true
 
 [features]
 default = []
+rio-v2 = ["dep:rustfs-rio-v2"]
 
 [dependencies]
 rustfs-filemeta.workspace = true
 rustfs-utils = { workspace = true, features = ["full"] }
 rustfs-rio.workspace = true
+rustfs-rio-v2 = { workspace = true, optional = true }
 rustfs-signer.workspace = true
+rustfs-storage-api.workspace = true
 rustfs-tls-runtime.workspace = true
 rustfs-checksums.workspace = true
 rustfs-config = { workspace = true, features = ["constants", "notify", "audit"] }
@@ -131,7 +134,7 @@ aws-smithy-http-client.workspace = true
 metrics = { workspace = true }
 
 [dev-dependencies]
-tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
+tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true, features = ["async_closure"] }
 tracing-subscriber = { workspace = true }
@@ -153,5 +156,9 @@ harness = false
 name = "rename_data_meta_benchmark"
 harness = false
 
+[[bench]]
+name = "single_block_non_inline_benchmark"
+harness = false
+
 [lib]
 doctest = false

crates/ecstore/benches/single_block_non_inline_benchmark.rs

@@ -0,0 +1,126 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
+use rustfs_ecstore::erasure_coding::{BitrotWriterWrapper, CustomWriter, Erasure};
+use rustfs_utils::HashAlgorithm;
+use std::io::Cursor;
+use std::sync::Arc;
+use std::time::Duration;
+
+#[derive(Clone, Debug)]
+struct BenchConfig {
+    name: String,
+    payload_size: usize,
+    data_shards: usize,
+    parity_shards: usize,
+    block_size: usize,
+}
+
+impl BenchConfig {
+    fn new(payload_size: usize, data_shards: usize, parity_shards: usize, block_size: usize) -> Self {
+        Self {
+            name: format!(
+                "payload={}KB_{}+{}_block={}KB",
+                payload_size / 1024,
+                data_shards,
+                parity_shards,
+                block_size / 1024
+            ),
+            payload_size,
+            data_shards,
+            parity_shards,
+            block_size,
+        }
+    }
+}
+
+fn generate_payload(size: usize) -> Vec<u8> {
+    (0..size).map(|i| (i % 251) as u8).collect()
+}
+
+fn build_non_inline_writers(config: &BenchConfig) -> Vec<Option<BitrotWriterWrapper>> {
+    let erasure = Erasure::new(config.data_shards, config.parity_shards, config.block_size);
+    let total_shards = config.data_shards + config.parity_shards;
+    let shard_size = erasure.shard_size();
+    (0..total_shards)
+        .map(|_| {
+            Some(BitrotWriterWrapper::new(
+                CustomWriter::new_tokio_writer(tokio::io::sink()),
+                shard_size,
+                HashAlgorithm::HighwayHash256S,
+            ))
+        })
+        .collect()
+}
+
+fn bench_single_block_non_inline_fast_path(c: &mut Criterion) {
+    let configs = vec![
+        BenchConfig::new(4 * 1024, 4, 2, 128 * 1024),
+        BenchConfig::new(64 * 1024, 4, 2, 128 * 1024),
+        BenchConfig::new(128 * 1024, 4, 2, 128 * 1024),
+    ];
+
+    let rt = tokio::runtime::Builder::new_current_thread()
+        .enable_all()
+        .build()
+        .expect("benchmark runtime");
+
+    let mut group = c.benchmark_group("single_block_non_inline_write_path");
+    group.sample_size(20);
+    group.measurement_time(Duration::from_secs(6));
+
+    for config in configs {
+        let payload = generate_payload(config.payload_size);
+        let erasure = Arc::new(Erasure::new(config.data_shards, config.parity_shards, config.block_size));
+        group.throughput(Throughput::Bytes(config.payload_size as u64));
+
+        group.bench_with_input(BenchmarkId::new("encode_pipeline", &config.name), &config, |b, config| {
+            let payload = payload.clone();
+            let erasure = Arc::clone(&erasure);
+            b.iter(|| {
+                let mut writers = build_non_inline_writers(config);
+                let reader = tokio::io::BufReader::new(Cursor::new(payload.clone()));
+                rt.block_on(async {
+                    erasure
+                        .clone()
+                        .encode(reader, &mut writers, config.data_shards)
+                        .await
+                        .expect("encode pipeline benchmark");
+                });
+            });
+        });
+
+        group.bench_with_input(BenchmarkId::new("single_block_candidate", &config.name), &config, |b, config| {
+            let payload = payload.clone();
+            let erasure = Arc::clone(&erasure);
+            b.iter(|| {
+                let mut writers = build_non_inline_writers(config);
+                let reader = tokio::io::BufReader::new(Cursor::new(payload.clone()));
+                rt.block_on(async {
+                    erasure
+                        .clone()
+                        .encode_single_block_non_inline(reader, &mut writers, config.data_shards)
+                        .await
+                        .expect("single block candidate benchmark");
+                });
+            });
+        });
+    }
+
+    group.finish();
+}
+
+criterion_group!(benches, bench_single_block_non_inline_fast_path);
+criterion_main!(benches);

crates/ecstore/docs/s3-table/s3-table-concept-model.md

@@ -0,0 +1,130 @@
+# RustFS S3 Table Core Concepts
+
+This document keeps only the core concepts for adding table catalog capability
+to RustFS. The direction is to build an Iceberg REST Catalog-compatible layer
+on top of existing RustFS object storage.
+
+## Concept Model
+
+```text
+TableBucket
+  Warehouse
+    Namespace
+      TableIdentifier
+        TableMetadataLocation
+        TableMetadata
+          Schema
+          Snapshot
+          Manifest
+          DataFile
+```
+
+## TableBucket
+
+`TableBucket` is a normal RustFS bucket enabled for table catalog use. It is not
+a new physical bucket type.
+
+RustFS can mark it with internal bucket metadata such as `table-bucket.json`.
+That marker is an implementation detail, not a public compatibility API.
+
+## Warehouse
+
+`Warehouse` is the catalog storage root exposed to Iceberg clients. In the first
+RustFS model, a warehouse maps to a table-enabled bucket.
+
+Clients should interact with the warehouse through catalog APIs, not by knowing
+RustFS internal metadata paths.
+
+## Namespace
+
+`Namespace` groups tables inside a warehouse. It is a first-class catalog
+resource, not a raw object prefix.
+
+Namespace validation must be conservative: no path traversal, encoded
+separators, empty segments, control characters, or ambiguous normalization.
+
+## TableIdentifier
+
+`TableIdentifier` names one table inside a warehouse and namespace.
+
+```text
+warehouse + namespace + table name
+```
+
+It is resolved by the catalog layer to internal metadata objects. It should not
+be treated as an S3 object key.
+
+## TableMetadataLocation
+
+`TableMetadataLocation` points to the current Iceberg table metadata file.
+
+It is the main mutable table pointer. Only catalog commit code may update it.
+Updates must use conflict detection so stale writers cannot overwrite newer
+table state.
+
+## TableMetadata
+
+`TableMetadata` is the Iceberg metadata JSON referenced by the current metadata
+location.
+
+RustFS should preserve Iceberg metadata exactly where possible. Full parsing can
+be added incrementally for fields RustFS must enforce, such as schemas,
+snapshots, manifest lists, table location, and properties.
+
+## Schema
+
+`Schema` describes table columns.
+
+RustFS does not need to interpret every schema detail at the beginning, but it
+must preserve schema metadata without corrupting unknown Iceberg fields.
+
+## Snapshot
+
+`Snapshot` represents an immutable table state at a point in time.
+
+Catalog commits advance the current metadata to a new snapshot. Maintenance
+features such as rollback and snapshot expiration can be added later.
+
+## Manifest
+
+`Manifest` and manifest lists describe table data-file additions, deletions, and
+references.
+
+RustFS should store and protect these as catalog-owned metadata. Full manifest
+parsing can be deferred until validation or maintenance requires it.
+
+## DataFile
+
+`DataFile` is a table data object, commonly Parquet, Avro, or ORC.
+
+Data files live in object storage. The catalog metadata decides which data files
+belong to a table snapshot.
+
+## Reserved Metadata Boundary
+
+Catalog-owned metadata should live under a reserved internal prefix, for
+example:
+
+```text
+.rustfs-table/
+  warehouses/
+    <warehouse-id>/
+      namespaces/
+        <namespace-id>/
+          tables/
+            <table-id>/
+              current.json
+              metadata/
+              manifests/
+```
+
+Ordinary S3 mutating APIs must not create, overwrite, copy into, delete, or
+mutate objects under this reserved prefix for table-enabled buckets.
+
+## Catalog Boundary
+
+S3 object APIs remain object APIs.
+
+Namespace, table, metadata pointer, and commit operations belong to a separate
+catalog layer. Public compatibility should target Iceberg REST Catalog semantics
+rather than exposing RustFS internal metadata objects directly.

crates/ecstore/src/batch_processor.rs

@@ -274,10 +274,10 @@ mod tests {
         assert!(successes.len() >= 2);
     }
 
-    #[tokio::test]
+    #[tokio::test(start_paused = true)]
     async fn test_batch_processor_quorum_returns_before_slow_tail() {
         let processor = AsyncBatchProcessor::new(4);
-        let started = std::time::Instant::now();
+        let started = tokio::time::Instant::now();
 
         let tasks: Vec<_> = [(10_u64, Ok(1_i32)), (15, Ok(2)), (250, Ok(3))]
             .into_iter()
@@ -295,10 +295,10 @@ mod tests {
         assert!(started.elapsed() < Duration::from_millis(100));
     }
 
-    #[tokio::test]
+    #[tokio::test(start_paused = true)]
     async fn test_batch_processor_quorum_fails_once_quorum_becomes_impossible() {
         let processor = AsyncBatchProcessor::new(4);
-        let started = std::time::Instant::now();
+        let started = tokio::time::Instant::now();
 
         let tasks: Vec<_> = vec![
             (10_u64, Ok(1_i32)),

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -45,11 +45,11 @@ use futures::Future;
 use http::HeaderMap;
 use lazy_static::lazy_static;
 use rustfs_common::heal_channel::rep_has_active_rules;
-use rustfs_common::metrics::{IlmAction, Metrics};
+use rustfs_common::metrics::{IlmAction, Metrics, global_metrics};
 use rustfs_config::{
     DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
-    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, ENV_TRANSITION_QUEUE_CAPACITY,
-    ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
+    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TRANSITION_QUEUE_CAPACITY, ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS,
+    ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
 };
 use rustfs_data_usage::TierStats;
 use rustfs_filemeta::{
@@ -75,6 +75,8 @@ use time::OffsetDateTime;
 use tokio::select;
 use tokio::sync::mpsc::{Receiver, Sender};
 use tokio::sync::{RwLock, mpsc};
+use tokio::task::JoinHandle;
+use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, info, warn};
 use uuid::Uuid;
 use xxhash_rust::xxh64;
@@ -143,9 +145,15 @@ fn is_immediate_transition_source(src: &LcEventSrc) -> bool {
     )
 }
 
+fn record_scanner_lifecycle_enqueue_result(src: &LcEventSrc, count: u64, queued: bool) {
+    if matches!(src, LcEventSrc::Scanner) {
+        global_metrics().record_scanner_ilm_enqueue_result(count, queued);
+    }
+}
+
 #[cfg(any(test, debug_assertions))]
 fn should_force_immediate_transition_enqueue_timeout() -> bool {
-    env::var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
+    env::var(rustfs_config::ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
         .ok()
         .is_some_and(|value| value == "1")
 }
@@ -355,7 +363,7 @@ impl ExpiryState {
         }
     }
 
-    pub async fn enqueue_by_days(&mut self, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) {
+    pub async fn enqueue_by_days(&mut self, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) -> bool {
         let task = ExpiryTask {
             obj_info: oi.clone(),
             event: event.clone(),
@@ -364,22 +372,29 @@ impl ExpiryState {
         let wrkr = self.get_worker_ch(task.op_hash());
         if wrkr.is_none() {
             *self.stats.as_mut().expect("stats lock").missed_expiry_tasks.get_mut() += 1;
-            return;
+            record_scanner_lifecycle_enqueue_result(src, 1, false);
+            return false;
         }
         let wrkr = wrkr.expect("worker channel should exist after None check");
-        select! {
-            //_ -> GlobalContext.Done() => {}
-            _ = wrkr.send(Some(Box::new(task))) => (),
-            else => {
-                *self.stats.as_mut().expect("stats lock").missed_expiry_tasks.get_mut() += 1;
-            }
+        let queued = wrkr.send(Some(Box::new(task))).await.is_ok();
+        if !queued {
+            *self.stats.as_mut().expect("stats lock").missed_expiry_tasks.get_mut() += 1;
         }
+        record_scanner_lifecycle_enqueue_result(src, 1, queued);
+        queued
     }
 
-    pub async fn enqueue_by_newer_noncurrent(&mut self, bucket: &str, versions: Vec<ObjectToDelete>, lc_event: lifecycle::Event) {
+    pub async fn enqueue_by_newer_noncurrent(
+        &mut self,
+        bucket: &str,
+        versions: Vec<ObjectToDelete>,
+        lc_event: lifecycle::Event,
+        src: &LcEventSrc,
+    ) -> bool {
         if versions.is_empty() {
-            return;
+            return true;
         }
+        let version_count = u64::try_from(versions.len()).unwrap_or(u64::MAX);
 
         let task = NewerNoncurrentTask {
             bucket: String::from(bucket),
@@ -389,16 +404,16 @@ impl ExpiryState {
         let wrkr = self.get_worker_ch(task.op_hash());
         if wrkr.is_none() {
             *self.stats.as_mut().expect("stats lock").missed_expiry_tasks.get_mut() += 1;
-            return;
+            record_scanner_lifecycle_enqueue_result(src, version_count, false);
+            return false;
         }
         let wrkr = wrkr.expect("worker channel should exist after None check");
-        select! {
-            //_ -> GlobalContext.Done() => {}
-            _ = wrkr.send(Some(Box::new(task))) => (),
-            else => {
-                *self.stats.as_mut().expect("stats lock").missed_expiry_tasks.get_mut() += 1;
-            }
+        let queued = wrkr.send(Some(Box::new(task))).await.is_ok();
+        if !queued {
+            *self.stats.as_mut().expect("stats lock").missed_expiry_tasks.get_mut() += 1;
         }
+        record_scanner_lifecycle_enqueue_result(src, version_count, queued);
+        queued
     }
 
     pub fn get_worker_ch(&self, h: u64) -> Option<Sender<Option<ExpiryOpType>>> {
@@ -587,12 +602,16 @@ impl ExpiryOp for TransitionTask {
     }
 }
 
+struct TransitionWorker {
+    cancel: CancellationToken,
+    handle: JoinHandle<()>,
+}
+
 pub struct TransitionState {
     transition_tx: A_Sender<Option<TransitionTask>>,
     transition_rx: A_Receiver<Option<TransitionTask>>,
     pub num_workers: AtomicI64,
-    kill_tx: A_Sender<()>,
-    kill_rx: A_Receiver<()>,
+    workers: Mutex<Vec<TransitionWorker>>,
     transition_queue_capacity: usize,
     transition_queue_send_timeout: StdDuration,
     active_tasks: AtomicI64,
@@ -621,13 +640,11 @@ impl TransitionState {
         let capacity = capacity.max(1);
         let queue_send_timeout = resolve_transition_queue_send_timeout();
         let (tx1, rx1) = bounded(capacity);
-        let (tx2, rx2) = bounded(1);
         Arc::new(Self {
             transition_tx: tx1,
             transition_rx: rx1,
             num_workers: AtomicI64::new(0),
-            kill_tx: tx2,
-            kill_rx: rx2,
+            workers: Mutex::new(Vec::new()),
             transition_queue_capacity: capacity,
             transition_queue_send_timeout: queue_send_timeout,
             active_tasks: AtomicI64::new(0),
@@ -735,10 +752,11 @@ impl TransitionState {
         }
     }
 
-    pub async fn queue_transition_task(self: &Arc<Self>, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) {
+    pub async fn queue_transition_task(self: &Arc<Self>, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) -> bool {
         if is_immediate_transition_source(src) && should_force_immediate_transition_enqueue_timeout() {
             self.handle_immediate_enqueue_failure(oi, src, ImmediateEnqueueFailure::ForcedTimeout);
-            return;
+            record_scanner_lifecycle_enqueue_result(src, 1, false);
+            return false;
         }
 
         let task = TransitionTask {
@@ -746,14 +764,15 @@ impl TransitionState {
             src: src.clone(),
             event: event.clone(),
         };
+        let mut queued = false;
         if is_immediate_transition_source(src) {
             match self.transition_tx.try_send(Some(task)) {
-                Ok(()) => {}
+                Ok(()) => queued = true,
                 Err(async_channel::TrySendError::Full(task)) => {
                     Self::inc_counter(&self.queue_full_tasks);
                     let send_timeout = self.transition_queue_send_timeout;
                     match tokio::time::timeout(send_timeout, self.transition_tx.send(task)).await {
-                        Ok(Ok(())) => {}
+                        Ok(Ok(())) => queued = true,
                         Ok(Err(_)) => {
                             self.handle_immediate_enqueue_failure(
                                 oi,
@@ -778,29 +797,31 @@ impl TransitionState {
                     self.handle_immediate_enqueue_failure(oi, src, ImmediateEnqueueFailure::QueueClosed { timeout_ms: None });
                 }
             }
-            return;
+            record_scanner_lifecycle_enqueue_result(src, 1, queued);
+            return queued;
         }
 
-        if let Err(err) = self.transition_tx.try_send(Some(task)) {
-            match err {
-                async_channel::TrySendError::Full(_) => {
-                    debug!(
-                        bucket = %oi.bucket,
-                        object = %oi.name,
-                        source = ?src,
-                        "transition queue is full; deferring to scanner/backfill"
-                    );
-                }
-                async_channel::TrySendError::Closed(_) => {
-                    warn!(
-                        bucket = %oi.bucket,
-                        object = %oi.name,
-                        source = ?src,
-                        "transition enqueue failed because the queue is closed"
-                    );
-                }
+        match self.transition_tx.try_send(Some(task)) {
+            Ok(()) => queued = true,
+            Err(async_channel::TrySendError::Full(_)) => {
+                debug!(
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    source = ?src,
+                    "transition queue is full; deferring to scanner/backfill"
+                );
+            }
+            Err(async_channel::TrySendError::Closed(_)) => {
+                warn!(
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    source = ?src,
+                    "transition enqueue failed because the queue is closed"
+                );
             }
         }
+        record_scanner_lifecycle_enqueue_result(src, 1, queued);
+        queued
     }
 
     pub async fn init(api: Arc<ECStore>) {
@@ -849,10 +870,12 @@ impl TransitionState {
         Self::counter_value(&self.compensation_running_tasks)
     }
 
-    pub async fn worker(api: Arc<ECStore>) {
+    async fn worker_with_cancel(api: Arc<ECStore>, cancel_token: CancellationToken) {
         loop {
             select! {
-                _ = GLOBAL_TransitionState.kill_rx.recv() => {
+                biased;
+
+                _ = cancel_token.cancelled() => {
                     return;
                 }
                 task = GLOBAL_TransitionState.transition_rx.recv() => {
@@ -952,33 +975,45 @@ impl TransitionState {
         }
         // Allow environment override of maximum workers
         let absolute_max = resolve_transition_workers_absolute_max();
-        n = std::cmp::min(n, absolute_max);
+        n = n.clamp(0, absolute_max);
 
-        let previous_num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        let mut num_workers = previous_num_workers;
-        while num_workers < n {
+        Self::resize_workers_to(api, n, requested, absolute_max);
+    }
+
+    fn resize_workers_to(api: Arc<ECStore>, n: i64, requested: i64, absolute_max: i64) {
+        let target = n as usize;
+        let mut workers = GLOBAL_TransitionState.workers.lock().unwrap();
+        let tracked_workers = workers.len();
+        workers.retain(|worker| !worker.handle.is_finished());
+        let pruned_finished_workers = tracked_workers.saturating_sub(workers.len());
+        let previous_num_workers = workers.len() as i64;
+
+        while workers.len() < target {
             let clone_api = api.clone();
-            tokio::spawn(async move {
-                TransitionState::worker(clone_api).await;
+            let cancel = CancellationToken::new();
+            let worker_cancel = cancel.clone();
+            let handle = tokio::spawn(async move {
+                TransitionState::worker_with_cancel(clone_api, worker_cancel).await;
             });
-            num_workers += 1;
-            GLOBAL_TransitionState.num_workers.fetch_add(1, Ordering::SeqCst);
+            workers.push(TransitionWorker { cancel, handle });
         }
 
-        let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        while num_workers > n {
-            let worker = GLOBAL_TransitionState.kill_tx.clone();
-            let _ = worker.send(()).await;
-            num_workers -= 1;
-            GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
+        while workers.len() > target {
+            if let Some(worker) = workers.pop() {
+                worker.cancel.cancel();
+            }
         }
 
+        let current_workers = workers.len() as i64;
+        GLOBAL_TransitionState.num_workers.store(current_workers, Ordering::SeqCst);
+
         info!(
             requested_transition_workers = requested,
             effective_transition_workers = n,
             absolute_max_workers = absolute_max,
             previous_transition_workers = previous_num_workers,
-            current_transition_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst),
+            current_transition_workers = current_workers,
+            pruned_finished_transition_workers = pruned_finished_workers,
             "transition workers updated"
         );
     }
@@ -1464,7 +1499,7 @@ pub async fn enqueue_immediate_expiry(oi: &ObjectInfo, src: LcEventSrc) {
         GLOBAL_ExpiryState
             .write()
             .await
-            .enqueue_by_newer_noncurrent(&oi.bucket, to_delete_objs, event)
+            .enqueue_by_newer_noncurrent(&oi.bucket, to_delete_objs, event, &src)
             .await;
     }
 }
@@ -1752,11 +1787,31 @@ pub async fn get_transitioned_object_reader(
         gopts.length = length;
     }
 
-    //return Ok(HttpFileReader::new(rs, &oi, opts, &h));
-    //timeTierAction := auditTierActions(oi.transitioned_object.Tier, length)
+    debug!(
+        bucket = %bucket,
+        object = %object,
+        tier = %oi.transitioned_object.tier,
+        tier_object = %oi.transitioned_object.name,
+        tier_version_id = %oi.transitioned_object.version_id,
+        start_offset = gopts.start_offset,
+        length = gopts.length,
+        "fetching transitioned object from tier"
+    );
     let reader = tgt_client
         .get(&oi.transitioned_object.name, &oi.transitioned_object.version_id, gopts)
-        .await?;
+        .await
+        .map_err(|e| {
+            tracing::error!(
+                bucket = %bucket,
+                object = %object,
+                tier = %oi.transitioned_object.tier,
+                tier_object = %oi.transitioned_object.name,
+                tier_version_id = %oi.transitioned_object.version_id,
+                error = %e,
+                "tier GET failed"
+            );
+            e
+        })?;
     Ok(get_fn(reader, h.clone()))
 }
 
@@ -1850,11 +1905,11 @@ pub async fn put_restore_opts(
             ..Default::default()
         });
     }
-    for (k, v) in &oi.user_defined {
+    for (k, v) in oi.user_defined.iter() {
         meta.insert(k.to_string(), v.clone());
     }
     if !oi.user_tags.is_empty() {
-        meta.insert(AMZ_OBJECT_TAGGING.to_string(), oi.user_tags.clone());
+        meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
     }
     let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
     meta.insert(
@@ -1885,7 +1940,7 @@ impl LifecycleOps for ObjectInfo {
     fn to_lifecycle_opts(&self) -> lifecycle::ObjectOpts {
         lifecycle::ObjectOpts {
             name: self.name.clone(),
-            user_tags: self.user_tags.clone(),
+            user_tags: (*self.user_tags).clone(),
             version_id: self.version_id,
             mod_time: self.mod_time,
             size: self.size as usize,
@@ -2007,8 +2062,7 @@ pub async fn apply_transition_rule(event: &lifecycle::Event, src: &LcEventSrc, o
     if oi.delete_marker || oi.is_dir {
         return false;
     }
-    GLOBAL_TransitionState.queue_transition_task(oi, event, src).await;
-    true
+    GLOBAL_TransitionState.queue_transition_task(oi, event, src).await
 }
 
 pub async fn apply_expiry_on_transitioned_object(
@@ -2097,8 +2151,7 @@ pub async fn apply_expiry_on_non_transitioned_objects(
 
 pub async fn apply_expiry_rule(event: &lifecycle::Event, src: &LcEventSrc, oi: &ObjectInfo) -> bool {
     let mut expiry_state = GLOBAL_ExpiryState.write().await;
-    expiry_state.enqueue_by_days(oi, event, src).await;
-    true
+    expiry_state.enqueue_by_days(oi, event, src).await
 }
 
 fn lifecycle_deleted_object(oi: &ObjectInfo, dobj: &ObjectInfo) -> crate::store_api::DeletedObject {
@@ -2259,13 +2312,15 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc,
 mod tests {
     use super::{
         DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
-        DEFAULT_TRANSITION_WORKERS_CAP, GLOBAL_TransitionState, StaleMultipartUploadCandidate, TransitionState,
+        DEFAULT_TRANSITION_WORKERS_CAP, ExpiryState, GLOBAL_TransitionState, StaleMultipartUploadCandidate, TransitionState,
         cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at, lifecycle_deleted_object,
         lifecycle_rule_has_date_expiration, lifecycle_version_purge_state_from_completed_targets,
         mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate, replication_state_for_delete,
         resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
-        should_defer_date_expiry_for_recent_config_update, should_reuse_lifecycle_delete_replication_state,
+        resolve_transition_workers_absolute_max, should_defer_date_expiry_for_recent_config_update,
+        should_reuse_lifecycle_delete_replication_state,
     };
+    use crate::bucket::lifecycle::bucket_lifecycle_audit::LcEventSrc;
     use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
     use crate::bucket::metadata_sys;
     use crate::disk::RUSTFS_META_MULTIPART_BUCKET;
@@ -2278,6 +2333,7 @@ mod tests {
         BucketOperations, BucketOptions, MakeBucketOptions, MultipartOperations, ObjectInfo, ObjectOptions, PutObjReader,
     };
     use futures::FutureExt;
+    use rustfs_common::metrics::IlmAction;
     use rustfs_config::ENV_TRANSITION_WORKERS_ABSOLUTE_MAX;
     use rustfs_filemeta::{ReplicateDecision, VersionPurgeStatusType};
     use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, Timestamp};
@@ -2294,6 +2350,48 @@ mod tests {
     use tokio_util::sync::CancellationToken;
     use uuid::Uuid;
 
+    #[tokio::test]
+    async fn expiry_enqueue_reports_missed_without_worker_channel() {
+        let state = ExpiryState::new();
+        let mut state = state.write().await;
+        let object = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object".to_string(),
+            ..Default::default()
+        };
+        let event = crate::bucket::lifecycle::lifecycle::Event {
+            action: IlmAction::DeleteAction,
+            ..Default::default()
+        };
+
+        let queued = state.enqueue_by_days(&object, &event, &LcEventSrc::Scanner).await;
+
+        assert!(!queued);
+        let stats = state.stats.as_ref().expect("expiry stats should exist");
+        assert_eq!(stats.missed_tasks(), 1);
+    }
+
+    #[tokio::test]
+    async fn scanner_transition_enqueue_reports_full_queue() {
+        let state = TransitionState::new_with_capacity(1);
+        let object = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object".to_string(),
+            ..Default::default()
+        };
+        let event = crate::bucket::lifecycle::lifecycle::Event {
+            action: IlmAction::TransitionAction,
+            ..Default::default()
+        };
+
+        let first = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
+        let second = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
+
+        assert!(first);
+        assert!(!second);
+        assert_eq!(state.transition_rx.len(), 1);
+    }
+
     #[test]
     fn mark_delete_opts_skip_decommissioned_on_remote_success_sets_flag_on_success() {
         let mut opts = ObjectOptions::default();
@@ -2655,15 +2753,45 @@ mod tests {
         })
         .await;
 
-        let current_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        if original_workers > 0 {
-            TransitionState::update_workers(ecstore, original_workers).await;
-        } else {
-            for _ in 0..current_workers {
-                let _ = GLOBAL_TransitionState.kill_tx.send(()).await;
-                GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
-            }
-        }
+        let absolute_max = resolve_transition_workers_absolute_max();
+        TransitionState::resize_workers_to(ecstore, original_workers, original_workers, absolute_max);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn transition_worker_resize_cancels_removed_workers_directly() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let original_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        let absolute_max = resolve_transition_workers_absolute_max();
+
+        TransitionState::resize_workers_to(ecstore.clone(), 0, 0, absolute_max);
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 0);
+
+        TransitionState::resize_workers_to(ecstore.clone(), 2, 2, absolute_max);
+        let worker_tokens = {
+            let workers = GLOBAL_TransitionState.workers.lock().unwrap();
+            assert_eq!(workers.len(), 2);
+            workers.iter().map(|worker| worker.cancel.clone()).collect::<Vec<_>>()
+        };
+
+        TransitionState::resize_workers_to(ecstore.clone(), 1, 1, absolute_max);
+
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 1);
+        assert_eq!(worker_tokens.iter().filter(|token| token.is_cancelled()).count(), 1);
+
+        let remaining_token = {
+            let workers = GLOBAL_TransitionState.workers.lock().unwrap();
+            assert_eq!(workers.len(), 1);
+            let token = workers[0].cancel.clone();
+            assert!(!token.is_cancelled());
+            token
+        };
+
+        TransitionState::resize_workers_to(ecstore.clone(), 0, 0, absolute_max);
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 0);
+        assert!(remaining_token.is_cancelled());
+
+        TransitionState::resize_workers_to(ecstore, original_workers, original_workers, absolute_max);
     }
 
     #[test]
@@ -2994,6 +3122,52 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[ignore = "requires isolated global object layer state"]
+    #[serial]
+    async fn ecstore_new_succeeds_on_fresh_local_volumes() {
+        let test_base_dir = format!("/tmp/rustfs_ecstore_empty_boot_{}", Uuid::new_v4());
+        let temp_dir = PathBuf::from(&test_base_dir);
+        if temp_dir.exists() {
+            fs::remove_dir_all(&temp_dir).await.ok();
+        }
+        fs::create_dir_all(&temp_dir).await.unwrap();
+
+        let disk_paths = vec![
+            temp_dir.join("disk1"),
+            temp_dir.join("disk2"),
+            temp_dir.join("disk3"),
+            temp_dir.join("disk4"),
+        ];
+
+        for disk_path in &disk_paths {
+            fs::create_dir_all(disk_path).await.unwrap();
+        }
+
+        let mut endpoints = Vec::new();
+        for (i, disk_path) in disk_paths.iter().enumerate() {
+            let mut endpoint = Endpoint::try_from(disk_path.to_str().unwrap()).unwrap();
+            endpoint.set_pool_index(0);
+            endpoint.set_set_index(0);
+            endpoint.set_disk_index(i);
+            endpoints.push(endpoint);
+        }
+
+        let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 4,
+            endpoints: Endpoints::from(endpoints),
+            cmd_line: "fresh-boot-test".to_string(),
+            platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
+        }]);
+
+        crate::store::init_local_disks(endpoint_pools.clone()).await.unwrap();
+
+        let ecstore = ECStore::new("127.0.0.1:0".parse().unwrap(), endpoint_pools, CancellationToken::new()).await;
+        assert!(ecstore.is_ok(), "fresh local ECStore boot should succeed, got {ecstore:?}");
+    }
+
     #[tokio::test]
     #[serial]
     async fn stale_multipart_cleanup_uses_default_expiry_without_lifecycle() {
@@ -3138,6 +3312,76 @@ mod tests {
         assert!(!parts.user_defined.contains_key(RUSTFS_MULTIPART_OBJECT_KEY));
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn repeated_upload_part_overwrites_previous_part_state() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let bucket = format!("multipart-overwrite-{}", Uuid::new_v4().simple());
+        let object = "overwrite/object.txt";
+        create_test_bucket(&ecstore, &bucket).await;
+
+        let upload = ecstore
+            .new_multipart_upload(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect("multipart upload should be created");
+
+        let mut first = PutObjReader::from_vec(vec![1, 2, 3]);
+        let first_part = ecstore
+            .put_object_part(&bucket, object, &upload.upload_id, 1, &mut first, &ObjectOptions::default())
+            .await
+            .expect("first multipart part should be uploaded");
+
+        let mut second = PutObjReader::from_vec(vec![4, 5, 6, 7]);
+        let second_part = ecstore
+            .put_object_part(&bucket, object, &upload.upload_id, 1, &mut second, &ObjectOptions::default())
+            .await
+            .expect("second multipart part should overwrite the previous part");
+
+        assert_ne!(
+            first_part.etag, second_part.etag,
+            "the overwrite path should persist the latest part metadata rather than reusing stale state"
+        );
+
+        let parts = ecstore
+            .list_object_parts(
+                &bucket,
+                object,
+                &upload.upload_id,
+                None,
+                crate::set_disk::MAX_PARTS_COUNT,
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("multipart parts should be readable after overwrite");
+
+        assert_eq!(parts.parts.len(), 1, "only the latest version of part 1 should remain visible");
+        assert_eq!(parts.parts[0].part_num, 1);
+        assert_eq!(parts.parts[0].etag, second_part.etag);
+        assert_eq!(parts.parts[0].size, second_part.size);
+        assert_eq!(parts.parts[0].actual_size, second_part.actual_size);
+
+        let completed = ecstore
+            .complete_multipart_upload(
+                &bucket,
+                object,
+                &upload.upload_id,
+                vec![crate::store_api::CompletePart {
+                    part_num: 1,
+                    etag: second_part.etag.clone(),
+                    checksum_crc32: None,
+                    checksum_crc32c: None,
+                    checksum_sha1: None,
+                    checksum_sha256: None,
+                    checksum_crc64nvme: None,
+                }],
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("complete multipart upload should succeed with the latest overwritten part");
+
+        assert_eq!(completed.size, second_part.size as i64);
+    }
+
     #[tokio::test]
     #[serial]
     async fn cleanup_removes_empty_multipart_sha_dirs() {

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -881,7 +881,7 @@ impl ObjectOpts {
     pub fn from_object_info(oi: &ObjectInfo) -> Self {
         Self {
             name: oi.name.clone(),
-            user_tags: oi.user_tags.clone(),
+            user_tags: (*oi.user_tags).clone(),
             mod_time: oi.mod_time,
             size: oi.size as usize,
             version_id: oi.version_id,
@@ -894,7 +894,7 @@ impl ObjectOpts {
             restore_expires: oi.restore_expires,
             versioned: false,
             version_suspended: false,
-            user_defined: oi.user_defined.clone(),
+            user_defined: (*oi.user_defined).clone(),
             version_purge_status: oi.version_purge_status.clone(),
             replication_status: oi.replication_status.clone(),
         }

crates/ecstore/src/bucket/metadata.rs

@@ -30,6 +30,7 @@ use s3s::dto::{
     ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
 };
 use serde::Serializer;
+use sha2::{Digest, Sha256};
 use std::collections::HashMap;
 use std::io::{Read, Write};
 use std::sync::Arc;
@@ -244,6 +245,30 @@ pub const BUCKET_ACCELERATE_CONFIG: &str = "accelerate.xml";
 pub const BUCKET_REQUEST_PAYMENT_CONFIG: &str = "request-payment.xml";
 pub const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = "public-access-block.xml";
 pub const BUCKET_ACL_CONFIG: &str = "bucket-acl.json";
+pub const BUCKET_TABLE_CONFIG: &str = "table-bucket.json";
+pub const BUCKET_TABLE_RESERVED_PREFIX: &str = ".rustfs-table";
+pub const BUCKET_TABLE_CATALOG_META_PREFIX: &str = "s3tables/catalog";
+pub const BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX: &str = "table-buckets";
+
+pub fn table_catalog_path_hash(value: &str) -> String {
+    let digest = Sha256::digest(value.as_bytes());
+    let mut output = String::with_capacity(digest.len() * 2);
+    const HEX: &[u8; 16] = b"0123456789abcdef";
+    for byte in digest {
+        output.push(char::from(HEX[usize::from(byte >> 4)]));
+        output.push(char::from(HEX[usize::from(byte & 0x0f)]));
+    }
+    output
+}
+
+pub fn table_bucket_catalog_metadata_prefix(bucket: &str) -> String {
+    format!(
+        "{}/{}/{}",
+        BUCKET_TABLE_CATALOG_META_PREFIX,
+        BUCKET_TABLE_CATALOG_TABLE_BUCKETS_PREFIX,
+        table_catalog_path_hash(bucket)
+    )
+}
 
 #[derive(Debug, Clone)]
 pub struct BucketMetadata {
@@ -268,6 +293,7 @@ pub struct BucketMetadata {
     pub request_payment_config_xml: Vec<u8>,
     pub public_access_block_config_xml: Vec<u8>,
     pub bucket_acl_config_json: Vec<u8>,
+    pub table_bucket_config_json: Vec<u8>,
 
     pub policy_config_updated_at: OffsetDateTime,
     pub object_lock_config_updated_at: OffsetDateTime,
@@ -287,6 +313,7 @@ pub struct BucketMetadata {
     pub request_payment_config_updated_at: OffsetDateTime,
     pub public_access_block_config_updated_at: OffsetDateTime,
     pub bucket_acl_config_updated_at: OffsetDateTime,
+    pub table_bucket_config_updated_at: OffsetDateTime,
 
     pub new_field_updated_at: OffsetDateTime,
 
@@ -334,6 +361,7 @@ impl Default for BucketMetadata {
             request_payment_config_xml: Default::default(),
             public_access_block_config_xml: Default::default(),
             bucket_acl_config_json: Default::default(),
+            table_bucket_config_json: Default::default(),
             policy_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             object_lock_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             encryption_config_updated_at: OffsetDateTime::UNIX_EPOCH,
@@ -352,6 +380,7 @@ impl Default for BucketMetadata {
             request_payment_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             public_access_block_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             bucket_acl_config_updated_at: OffsetDateTime::UNIX_EPOCH,
+            table_bucket_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             new_field_updated_at: OffsetDateTime::UNIX_EPOCH,
             policy_config: Default::default(),
             notification_config: Default::default(),
@@ -397,6 +426,10 @@ impl BucketMetadata {
         self.lock_enabled || (self.versioning_config.as_ref().is_some_and(|v| v.enabled()))
     }
 
+    pub fn table_bucket_enabled(&self) -> bool {
+        !self.table_bucket_config_json.is_empty()
+    }
+
     /// Decode from msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn decode_from<R: Read>(&mut self, rd: &mut R) -> Result<()> {
         let mut fields = rmp::decode::read_map_len(rd)?;
@@ -447,6 +480,7 @@ impl BucketMetadata {
                     self.public_access_block_config_xml = read_msgp_bin(rd)?
                 }
                 "BucketAclConfigJSON" | "BucketAclConfigJson" => self.bucket_acl_config_json = read_msgp_bin(rd)?,
+                "TableBucketConfigJSON" | "TableBucketConfigJson" => self.table_bucket_config_json = read_msgp_bin(rd)?,
                 "CorsConfigUpdatedAt" => self.cors_config_updated_at = read_msgp_time_value(rd)?,
                 "LoggingConfigUpdatedAt" => self.logging_config_updated_at = read_msgp_time_value(rd)?,
                 "WebsiteConfigUpdatedAt" => self.website_config_updated_at = read_msgp_time_value(rd)?,
@@ -454,6 +488,7 @@ impl BucketMetadata {
                 "RequestPaymentConfigUpdatedAt" => self.request_payment_config_updated_at = read_msgp_time_value(rd)?,
                 "PublicAccessBlockConfigUpdatedAt" => self.public_access_block_config_updated_at = read_msgp_time_value(rd)?,
                 "BucketAclConfigUpdatedAt" => self.bucket_acl_config_updated_at = read_msgp_time_value(rd)?,
+                "TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
                 other => {
                     tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
                     skip_msgp_value(rd)?;
@@ -466,8 +501,8 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Map size: MinIO fields (25) + RustFS extensions (14)
-        let map_len: u32 = 39;
+        // Map size: MinIO fields (25) + RustFS extensions (16)
+        let map_len: u32 = 41;
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -523,6 +558,7 @@ impl BucketMetadata {
         write_bin_field(wr, "RequestPaymentConfigXML", &self.request_payment_config_xml)?;
         write_bin_field(wr, "PublicAccessBlockConfigXML", &self.public_access_block_config_xml)?;
         write_bin_field(wr, "BucketAclConfigJSON", &self.bucket_acl_config_json)?;
+        write_bin_field(wr, "TableBucketConfigJSON", &self.table_bucket_config_json)?;
         rmp::encode::write_str(wr, "CorsConfigUpdatedAt")?;
         write_msgp_time(wr, self.cors_config_updated_at)?;
         rmp::encode::write_str(wr, "LoggingConfigUpdatedAt")?;
@@ -537,6 +573,8 @@ impl BucketMetadata {
         write_msgp_time(wr, self.public_access_block_config_updated_at)?;
         rmp::encode::write_str(wr, "BucketAclConfigUpdatedAt")?;
         write_msgp_time(wr, self.bucket_acl_config_updated_at)?;
+        rmp::encode::write_str(wr, "TableBucketConfigUpdatedAt")?;
+        write_msgp_time(wr, self.table_bucket_config_updated_at)?;
 
         Ok(())
     }
@@ -632,6 +670,9 @@ impl BucketMetadata {
         if self.bucket_acl_config_updated_at == OffsetDateTime::UNIX_EPOCH {
             self.bucket_acl_config_updated_at = self.created
         }
+        if self.table_bucket_config_updated_at == OffsetDateTime::UNIX_EPOCH {
+            self.table_bucket_config_updated_at = self.created
+        }
     }
 
     pub fn update_config(&mut self, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
@@ -709,6 +750,10 @@ impl BucketMetadata {
                 self.bucket_acl_config_json = data;
                 self.bucket_acl_config_updated_at = updated;
             }
+            BUCKET_TABLE_CONFIG => {
+                self.table_bucket_config_json = data;
+                self.table_bucket_config_updated_at = updated;
+            }
             _ => return Err(Error::other(format!("config file not found : {config_file}"))),
         }
 
@@ -1028,6 +1073,10 @@ mod test {
         bm.bucket_acl_config_json = bucket_acl.as_bytes().to_vec();
         bm.bucket_acl_config_updated_at = OffsetDateTime::now_utc();
 
+        let table_bucket_marker = r#"{"enabled":true}"#;
+        bm.table_bucket_config_json = table_bucket_marker.as_bytes().to_vec();
+        bm.table_bucket_config_updated_at = OffsetDateTime::now_utc();
+
         // Test serialization
         let buf = bm.marshal_msg().unwrap();
         assert!(!buf.is_empty(), "Serialized buffer should not be empty");
@@ -1049,6 +1098,7 @@ mod test {
         assert_eq!(bm.quota_config_json, deserialized_bm.quota_config_json);
         assert_eq!(bm.public_access_block_config_xml, deserialized_bm.public_access_block_config_xml);
         assert_eq!(bm.bucket_acl_config_json, deserialized_bm.bucket_acl_config_json);
+        assert_eq!(bm.table_bucket_config_json, deserialized_bm.table_bucket_config_json);
         assert_eq!(bm.object_lock_config_xml, deserialized_bm.object_lock_config_xml);
         assert_eq!(bm.notification_config_xml, deserialized_bm.notification_config_xml);
         assert_eq!(bm.replication_config_xml, deserialized_bm.replication_config_xml);
@@ -1100,6 +1150,11 @@ mod test {
             bm.bucket_targets_config_meta_updated_at.unix_timestamp(),
             deserialized_bm.bucket_targets_config_meta_updated_at.unix_timestamp()
         );
+        assert_eq!(
+            bm.table_bucket_config_updated_at.unix_timestamp(),
+            deserialized_bm.table_bucket_config_updated_at.unix_timestamp()
+        );
+        assert!(deserialized_bm.table_bucket_enabled());
 
         // Test that the serialized data contains expected content
         let buf_str = String::from_utf8_lossy(&buf);
@@ -1116,6 +1171,20 @@ mod test {
         println!("   - Serialized buffer size: {} bytes", buf.len());
     }
 
+    #[test]
+    fn table_bucket_marker_tracks_config_presence() {
+        let mut bm = BucketMetadata::new("table-bucket");
+        assert!(!bm.table_bucket_enabled());
+
+        bm.update_config(BUCKET_TABLE_CONFIG, br#"{"enabled":true}"#.to_vec())
+            .unwrap();
+        assert!(bm.table_bucket_enabled());
+        assert!(!bm.table_bucket_config_json.is_empty());
+
+        bm.update_config(BUCKET_TABLE_CONFIG, Vec::new()).unwrap();
+        assert!(!bm.table_bucket_enabled());
+    }
+
     /// After policy deletion (policy_config_json cleared), parse_policy_config sets policy_config to None.
     #[test]
     fn test_parse_policy_config_clears_cache_when_json_empty() {

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -68,6 +68,30 @@ pub const MRF_WORKER_AUTO_DEFAULT: usize = 4;
 pub const LARGE_WORKER_COUNT: usize = 10;
 pub const MIN_LARGE_OBJ_SIZE: i64 = 128 * 1024 * 1024; // 128MiB
 
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub enum ReplicationQueueAdmission {
+    #[default]
+    Skipped,
+    Queued,
+    Missed,
+}
+
+impl ReplicationQueueAdmission {
+    fn merge(&mut self, other: Self) {
+        *self = match (*self, other) {
+            (Self::Missed, _) | (_, Self::Missed) => Self::Missed,
+            (Self::Queued, _) | (_, Self::Queued) => Self::Queued,
+            (Self::Skipped, Self::Skipped) => Self::Skipped,
+        };
+    }
+}
+
+#[derive(Debug, Clone, Default)]
+pub struct ReplicationHealQueueResult {
+    pub object_info: ReplicateObjectInfo,
+    pub admission: ReplicationQueueAdmission,
+}
+
 /// Priority levels for replication
 #[derive(Debug, Clone, PartialEq)]
 pub enum ReplicationPriority {
@@ -513,7 +537,7 @@ impl<S: StorageAPI> ReplicationPool<S> {
     }
 
     /// Queues a replica task
-    pub async fn queue_replica_task(&self, ri: ReplicateObjectInfo) {
+    pub async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission {
         // If object is large, queue it to a static set of large workers
         if ri.size >= MIN_LARGE_OBJ_SIZE {
             use std::collections::hash_map::DefaultHasher;
@@ -532,7 +556,11 @@ impl<S: StorageAPI> ReplicationPool<S> {
                     && worker.try_send(ReplicationOperation::Object(Box::new(ri.clone()))).is_err()
                 {
                     // Queue to MRF if worker is busy
-                    let _ = self.mrf_save_tx.try_send(ri.to_mrf_entry());
+                    let admission = if self.mrf_save_tx.try_send(ri.to_mrf_entry()).is_ok() {
+                        ReplicationQueueAdmission::Queued
+                    } else {
+                        ReplicationQueueAdmission::Missed
+                    };
 
                     // Try to add more workers if possible
                     let max_l_workers = *self.max_l_workers.read().await;
@@ -543,9 +571,12 @@ impl<S: StorageAPI> ReplicationPool<S> {
                         drop(lrg_workers);
                         self.resize_lrg_workers(workers, existing).await;
                     }
+                    return admission;
                 }
+
+                return ReplicationQueueAdmission::Queued;
             }
-            return;
+            return ReplicationQueueAdmission::Missed;
         }
 
         // Handle regular sized objects
@@ -555,85 +586,105 @@ impl<S: StorageAPI> ReplicationPool<S> {
             _ => self.get_worker_ch(&ri.bucket, &ri.name, ri.size).await,
         };
 
-        if let Some(channel) = ch
-            && channel.try_send(ReplicationOperation::Object(Box::new(ri.clone()))).is_err()
-        {
-            // Queue to MRF if all workers are busy
-            let _ = self.mrf_save_tx.try_send(ri.to_mrf_entry());
+        let Some(channel) = ch else {
+            return ReplicationQueueAdmission::Missed;
+        };
 
-            // Try to scale up workers based on priority
-            let priority = self.priority.read().await.clone();
-            let max_workers = *self.max_workers.read().await;
+        if channel.try_send(ReplicationOperation::Object(Box::new(ri.clone()))).is_ok() {
+            return ReplicationQueueAdmission::Queued;
+        }
 
-            match priority {
-                ReplicationPriority::Fast => {
-                    // Log warning about unable to keep up
-                    info!("Warning: Unable to keep up with incoming traffic");
+        // Queue to MRF if all workers are busy
+        let admission = if self.mrf_save_tx.try_send(ri.to_mrf_entry()).is_ok() {
+            ReplicationQueueAdmission::Queued
+        } else {
+            ReplicationQueueAdmission::Missed
+        };
+
+        // Try to scale up workers based on priority
+        let priority = self.priority.read().await.clone();
+        let max_workers = *self.max_workers.read().await;
+
+        match priority {
+            ReplicationPriority::Fast => {
+                // Log warning about unable to keep up
+                info!("Warning: Unable to keep up with incoming traffic");
+            }
+            ReplicationPriority::Slow => {
+                info!("Warning: Unable to keep up with incoming traffic - recommend increasing replication priority to auto");
+            }
+            ReplicationPriority::Auto => {
+                let max_w = std::cmp::min(max_workers, WORKER_MAX_LIMIT);
+                let active_workers = self.active_workers();
+
+                if active_workers < max_w as i32 {
+                    let workers = self.workers.read().await;
+                    let new_count = std::cmp::min(workers.len() + 1, max_w);
+                    let existing = workers.len();
+
+                    drop(workers);
+                    self.resize_workers(new_count, existing).await;
                 }
-                ReplicationPriority::Slow => {
-                    info!("Warning: Unable to keep up with incoming traffic - recommend increasing replication priority to auto");
-                }
-                ReplicationPriority::Auto => {
-                    let max_w = std::cmp::min(max_workers, WORKER_MAX_LIMIT);
-                    let active_workers = self.active_workers();
 
-                    if active_workers < max_w as i32 {
-                        let workers = self.workers.read().await;
-                        let new_count = std::cmp::min(workers.len() + 1, max_w);
-                        let existing = workers.len();
+                let max_mrf_workers = std::cmp::min(max_workers, MRF_WORKER_MAX_LIMIT);
+                let active_mrf = self.active_mrf_workers();
 
-                        drop(workers);
-                        self.resize_workers(new_count, existing).await;
-                    }
+                if active_mrf < max_mrf_workers as i32 {
+                    let current_mrf = self.mrf_worker_size.load(Ordering::SeqCst);
+                    let new_mrf = std::cmp::min(current_mrf + 1, max_mrf_workers as i32);
 
-                    let max_mrf_workers = std::cmp::min(max_workers, MRF_WORKER_MAX_LIMIT);
-                    let active_mrf = self.active_mrf_workers();
-
-                    if active_mrf < max_mrf_workers as i32 {
-                        let current_mrf = self.mrf_worker_size.load(Ordering::SeqCst);
-                        let new_mrf = std::cmp::min(current_mrf + 1, max_mrf_workers as i32);
-
-                        self.resize_failed_workers(new_mrf).await;
-                    }
+                    self.resize_failed_workers(new_mrf).await;
                 }
             }
         }
+
+        admission
     }
 
     /// Queues a replica delete task
-    pub async fn queue_replica_delete_task(&self, doi: DeletedObjectReplicationInfo) {
+    pub async fn queue_replica_delete_task(&self, doi: DeletedObjectReplicationInfo) -> ReplicationQueueAdmission {
         let ch = match doi.op_type {
             ReplicationType::Heal | ReplicationType::ExistingObject => Some(self.mrf_replica_tx.clone()),
             _ => self.get_worker_ch(&doi.bucket, &doi.delete_object.object_name, 0).await,
         };
 
-        if let Some(channel) = ch
-            && channel.try_send(ReplicationOperation::Delete(Box::new(doi.clone()))).is_err()
-        {
-            let _ = self.mrf_save_tx.try_send(doi.to_mrf_entry());
+        let Some(channel) = ch else {
+            return ReplicationQueueAdmission::Missed;
+        };
 
-            let priority = self.priority.read().await.clone();
-            let max_workers = *self.max_workers.read().await;
+        if channel.try_send(ReplicationOperation::Delete(Box::new(doi.clone()))).is_ok() {
+            return ReplicationQueueAdmission::Queued;
+        }
 
-            match priority {
-                ReplicationPriority::Fast => {
-                    info!("Warning: Unable to keep up with incoming deletes");
-                }
-                ReplicationPriority::Slow => {
-                    info!("Warning: Unable to keep up with incoming deletes - recommend increasing replication priority to auto");
-                }
-                ReplicationPriority::Auto => {
-                    let max_w = std::cmp::min(max_workers, WORKER_MAX_LIMIT);
-                    if self.active_workers() < max_w as i32 {
-                        let workers = self.workers.read().await;
-                        let new_count = std::cmp::min(workers.len() + 1, max_w);
-                        let existing = workers.len();
-                        drop(workers);
-                        self.resize_workers(new_count, existing).await;
-                    }
+        let admission = if self.mrf_save_tx.try_send(doi.to_mrf_entry()).is_ok() {
+            ReplicationQueueAdmission::Queued
+        } else {
+            ReplicationQueueAdmission::Missed
+        };
+
+        let priority = self.priority.read().await.clone();
+        let max_workers = *self.max_workers.read().await;
+
+        match priority {
+            ReplicationPriority::Fast => {
+                info!("Warning: Unable to keep up with incoming deletes");
+            }
+            ReplicationPriority::Slow => {
+                info!("Warning: Unable to keep up with incoming deletes - recommend increasing replication priority to auto");
+            }
+            ReplicationPriority::Auto => {
+                let max_w = std::cmp::min(max_workers, WORKER_MAX_LIMIT);
+                if self.active_workers() < max_w as i32 {
+                    let workers = self.workers.read().await;
+                    let new_count = std::cmp::min(workers.len() + 1, max_w);
+                    let existing = workers.len();
+                    drop(workers);
+                    self.resize_workers(new_count, existing).await;
                 }
             }
         }
+
+        admission
     }
 
     /// Queues an MRF save operation
@@ -959,8 +1010,8 @@ pub trait ReplicationPoolTrait: std::fmt::Debug {
     fn active_workers(&self) -> i32;
     fn active_mrf_workers(&self) -> i32;
     fn active_lrg_workers(&self) -> i32;
-    async fn queue_replica_task(&self, ri: ReplicateObjectInfo);
-    async fn queue_replica_delete_task(&self, ri: DeletedObjectReplicationInfo);
+    async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission;
+    async fn queue_replica_delete_task(&self, ri: DeletedObjectReplicationInfo) -> ReplicationQueueAdmission;
     async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize);
     async fn get_bucket_resync_status(&self, bucket: &str) -> Result<BucketReplicationResyncStatus, EcstoreError>;
     async fn cancel_bucket_resync(&self, opts: ResyncOpts) -> Result<(), EcstoreError>;
@@ -987,12 +1038,12 @@ impl<S: StorageAPI> ReplicationPoolTrait for ReplicationPool<S> {
         ReplicationPool::<S>::active_lrg_workers(self)
     }
 
-    async fn queue_replica_task(&self, ri: ReplicateObjectInfo) {
-        self.queue_replica_task(ri).await;
+    async fn queue_replica_task(&self, ri: ReplicateObjectInfo) -> ReplicationQueueAdmission {
+        self.queue_replica_task(ri).await
     }
 
-    async fn queue_replica_delete_task(&self, ri: DeletedObjectReplicationInfo) {
-        self.queue_replica_delete_task(ri).await;
+    async fn queue_replica_delete_task(&self, ri: DeletedObjectReplicationInfo) -> ReplicationQueueAdmission {
+        self.queue_replica_delete_task(ri).await
     }
 
     async fn resize(&self, priority: ReplicationPriority, max_workers: usize, max_l_workers: usize) {
@@ -1079,7 +1130,7 @@ pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc:
         target_statuses: tgt_statuses,
         target_purge_statuses: purge_statuses,
         replication_timestamp: tm,
-        user_tags: oi.user_tags,
+        user_tags: (*oi.user_tags).clone(),
         checksum: None,
         retry_count: 0,
         event_type: "".to_string(),
@@ -1093,13 +1144,13 @@ pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc:
     if dsc.is_synchronous() {
         replicate_object(ri, o).await
     } else if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
-        pool.queue_replica_task(ri).await;
+        let _ = pool.queue_replica_task(ri).await;
     }
 }
 
 pub async fn schedule_replication_delete(dv: DeletedObjectReplicationInfo) {
     if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
-        pool.queue_replica_delete_task(dv.clone()).await;
+        let _ = pool.queue_replica_delete_task(dv.clone()).await;
     }
 
     if let (Some(rs), Some(stats)) = (dv.delete_object.replication_state, GLOBAL_REPLICATION_STATS.get()) {
@@ -1153,23 +1204,32 @@ pub async fn queue_replication_heal_internal(
     oi: ObjectInfo,
     rcfg: ReplicationConfig,
     retry_count: u32,
-) -> ReplicateObjectInfo {
+) -> ReplicationHealQueueResult {
     let mut roi = ReplicateObjectInfo::default();
 
     // ignore modtime zero objects
     if oi.mod_time.is_none() || oi.mod_time == Some(OffsetDateTime::UNIX_EPOCH) {
-        return roi;
+        return ReplicationHealQueueResult {
+            object_info: roi,
+            admission: ReplicationQueueAdmission::Skipped,
+        };
     }
 
     if rcfg.config.is_none() || rcfg.remotes.is_none() {
-        return roi;
+        return ReplicationHealQueueResult {
+            object_info: roi,
+            admission: ReplicationQueueAdmission::Skipped,
+        };
     }
 
     roi = get_heal_replicate_object_info(&oi, &rcfg).await;
     roi.retry_count = retry_count;
 
     if !roi.dsc.replicate_any() {
-        return roi;
+        return ReplicationHealQueueResult {
+            object_info: roi,
+            admission: ReplicationQueueAdmission::Skipped,
+        };
     }
 
     // early return if replication already done, otherwise we need to determine if this
@@ -1178,7 +1238,10 @@ pub async fn queue_replication_heal_internal(
         && roi.version_purge_status.is_empty()
         && !roi.existing_obj_resync.must_resync()
     {
-        return roi;
+        return ReplicationHealQueueResult {
+            object_info: roi,
+            admission: ReplicationQueueAdmission::Skipped,
+        };
     }
 
     if roi.delete_marker || !roi.version_purge_status.is_empty() {
@@ -1210,10 +1273,15 @@ pub async fn queue_replication_heal_internal(
             || roi.version_purge_status == VersionPurgeStatusType::Failed
             || roi.version_purge_status == VersionPurgeStatusType::Pending
         {
-            if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
-                pool.queue_replica_delete_task(dv).await;
-            }
-            return roi;
+            let admission = if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
+                pool.queue_replica_delete_task(dv).await
+            } else {
+                ReplicationQueueAdmission::Missed
+            };
+            return ReplicationHealQueueResult {
+                object_info: roi,
+                admission,
+            };
         }
 
         // if replication status is Complete on DeleteMarker and existing object resync required
@@ -1221,11 +1289,17 @@ pub async fn queue_replication_heal_internal(
         if existing_obj_resync.must_resync()
             && (roi.replication_status == ReplicationStatusType::Completed || roi.replication_status.is_empty())
         {
-            queue_replicate_deletes_wrapper(dv, existing_obj_resync).await;
-            return roi;
+            let admission = queue_replicate_deletes_wrapper(dv, existing_obj_resync).await;
+            return ReplicationHealQueueResult {
+                object_info: roi,
+                admission,
+            };
         }
 
-        return roi;
+        return ReplicationHealQueueResult {
+            object_info: roi,
+            admission: ReplicationQueueAdmission::Skipped,
+        };
     }
 
     if roi.existing_obj_resync.must_resync() {
@@ -1235,34 +1309,72 @@ pub async fn queue_replication_heal_internal(
     match roi.replication_status {
         ReplicationStatusType::Pending | ReplicationStatusType::Failed => {
             roi.event_type = REPLICATE_HEAL.to_string();
-            if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
-                pool.queue_replica_task(roi.clone()).await;
-            }
-            return roi;
+            let admission = if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
+                pool.queue_replica_task(roi.clone()).await
+            } else {
+                ReplicationQueueAdmission::Missed
+            };
+            return ReplicationHealQueueResult {
+                object_info: roi,
+                admission,
+            };
         }
         _ => {}
     }
 
     if roi.existing_obj_resync.must_resync() {
         roi.event_type = REPLICATE_EXISTING.to_string();
-        if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
-            pool.queue_replica_task(roi.clone()).await;
-        }
+        let admission = if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
+            pool.queue_replica_task(roi.clone()).await
+        } else {
+            ReplicationQueueAdmission::Missed
+        };
+        return ReplicationHealQueueResult {
+            object_info: roi,
+            admission,
+        };
     }
 
-    roi
+    ReplicationHealQueueResult {
+        object_info: roi,
+        admission: ReplicationQueueAdmission::Skipped,
+    }
 }
 
 /// Wrapper function for queueing replicate deletes with resync decision
-async fn queue_replicate_deletes_wrapper(doi: DeletedObjectReplicationInfo, existing_obj_resync: ResyncDecision) {
+async fn queue_replicate_deletes_wrapper(
+    doi: DeletedObjectReplicationInfo,
+    existing_obj_resync: ResyncDecision,
+) -> ReplicationQueueAdmission {
+    let mut admission = ReplicationQueueAdmission::Skipped;
     for (k, v) in existing_obj_resync.targets.iter() {
         if v.replicate {
             let mut dv = doi.clone();
             dv.reset_id = v.reset_id.clone();
             dv.target_arn = k.clone();
-            if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
-                pool.queue_replica_delete_task(dv).await;
-            }
+            let target_admission = if let Some(pool) = GLOBAL_REPLICATION_POOL.get() {
+                pool.queue_replica_delete_task(dv).await
+            } else {
+                ReplicationQueueAdmission::Missed
+            };
+            admission.merge(target_admission);
         }
     }
+    admission
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn replication_queue_admission_combines_target_results() {
+        let mut admission = ReplicationQueueAdmission::Skipped;
+
+        admission.merge(ReplicationQueueAdmission::Queued);
+        assert_eq!(admission, ReplicationQueueAdmission::Queued);
+
+        admission.merge(ReplicationQueueAdmission::Missed);
+        assert_eq!(admission, ReplicationQueueAdmission::Missed);
+    }
 }

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -934,7 +934,7 @@ fn heal_should_use_check_replicate_delete(oi: &ObjectInfo) -> bool {
 
 pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationConfig) -> ReplicateObjectInfo {
     let mut oi = oi.clone();
-    let mut user_defined = oi.user_defined.clone();
+    let mut user_defined = (*oi.user_defined).clone();
 
     if let Some(rc) = rcfg.config.as_ref()
         && !rc.role.is_empty()
@@ -982,7 +982,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
             &oi.name,
             MustReplicateOptions::new(
                 &user_defined,
-                oi.user_tags.clone(),
+                (*oi.user_tags).clone(),
                 ReplicationStatusType::Empty,
                 ReplicationType::Heal,
                 ObjectOptions::default(),
@@ -1020,7 +1020,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
         target_purge_statuses,
         replication_timestamp: None,
         ssec: false, // TODO: add ssec support
-        user_tags: oi.user_tags.clone(),
+        user_tags: (*oi.user_tags).clone(),
         checksum: oi.checksum.clone(),
         retry_count: 0,
     }
@@ -1158,7 +1158,7 @@ impl ReplicationConfig {
             return self.resync_internal(oi, dsc, status);
         }
 
-        let mut user_defined = oi.user_defined.clone();
+        let mut user_defined = (*oi.user_defined).clone();
         user_defined.remove(AMZ_BUCKET_REPLICATION_STATUS);
 
         let dsc = must_replicate(
@@ -1166,7 +1166,7 @@ impl ReplicationConfig {
             &oi.name,
             MustReplicateOptions::new(
                 &user_defined,
-                oi.user_tags.clone(),
+                (*oi.user_tags).clone(),
                 ReplicationStatusType::Empty,
                 ReplicationType::ExistingObject,
                 ObjectOptions::default(),
@@ -1296,7 +1296,7 @@ impl MustReplicateOptions {
     }
 
     pub fn from_object_info(oi: &ObjectInfo, op_type: ReplicationType, opts: ObjectOptions) -> Self {
-        Self::new(&oi.user_defined, oi.user_tags.clone(), oi.replication_status.clone(), op_type, opts)
+        Self::new(&oi.user_defined, (*oi.user_tags).clone(), oi.replication_status.clone(), op_type, opts)
     }
 
     pub fn replication_status(&self) -> ReplicationStatusType {
@@ -1411,7 +1411,7 @@ fn delete_replication_object_opts(dobj: &ObjectToDelete, oi: &ObjectInfo) -> Obj
     ObjectOpts {
         name: dobj.object_name.clone(),
         ssec: is_ssec_encrypted(&oi.user_defined),
-        user_tags: oi.user_tags.clone(),
+        user_tags: (*oi.user_tags).clone(),
         delete_marker: oi.delete_marker,
         version_id: dobj.version_id,
         op_type: ReplicationType::Delete,
@@ -2995,7 +2995,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             mod_time: self.mod_time,
             version_id: self.version_id,
             size: self.size,
-            user_tags: self.user_tags.clone(),
+            user_tags: Arc::new(self.user_tags.clone()),
             actual_size: self.actual_size,
             replication_status_internal: self.replication_status_internal.clone(),
             replication_status: self.replication_status.clone(),
@@ -3221,7 +3221,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
     }
 
     // Use case-insensitive lookup for headers
-    let lk_map = object_info.user_defined.clone();
+    let lk_map = &*object_info.user_defined;
 
     if let Some(lang) = lk_map.lookup(CONTENT_LANGUAGE) {
         put_op.content_language = lang.to_string();
@@ -3500,7 +3500,7 @@ fn get_replication_action(oi1: &ObjectInfo, oi2: &HeadObjectOutput, op_type: Rep
 
     // compare metadata on both maps to see if meta is identical
     let mut compare_meta1 = HashMap::new();
-    for (k, v) in &oi1.user_defined {
+    for (k, v) in oi1.user_defined.iter() {
         let mut found = false;
         for prefix in &compare_keys {
             if strings_has_prefix_fold(k, prefix) {

crates/ecstore/src/cache_value/metacache_set.rs

@@ -56,6 +56,7 @@ async fn peek_with_timeout<R: AsyncRead + Unpin>(reader: &mut MetacacheReader<R>
 pub(crate) enum TestReaderBehavior {
     Eof,
     Stall,
+    IgnoreCancel,
     ProducerError(DiskError),
     PartialThenTimeout(Vec<MetaCacheEntry>),
 }
@@ -72,6 +73,7 @@ pub struct ListPathRawOptions {
     pub min_disks: usize,
     pub report_not_found: bool,
     pub per_disk_limit: i32,
+    pub skip_walkdir_total_timeout: bool,
     pub agreed: Option<AgreedFn>,
     pub partial: Option<PartialFn>,
     pub finished: Option<FinishedFn>,
@@ -97,6 +99,7 @@ impl Clone for ListPathRawOptions {
             min_disks: self.min_disks,
             report_not_found: self.report_not_found,
             per_disk_limit: self.per_disk_limit,
+            skip_walkdir_total_timeout: self.skip_walkdir_total_timeout,
             #[cfg(test)]
             test_reader_behaviors: self.test_reader_behaviors.clone(),
             #[cfg(test)]
@@ -137,6 +140,11 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                         cancel_rx_clone.cancelled().await;
                         return Ok(());
                     }
+                    TestReaderBehavior::IgnoreCancel => {
+                        let _held_writer = wr;
+                        std::future::pending::<()>().await;
+                        return Ok(());
+                    }
                     TestReaderBehavior::ProducerError(err) => {
                         record_producer_error(&producer_errs_clone, disk_idx, &err);
                         return Err(err);
@@ -162,6 +170,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 filter_prefix: opts_clone.filter_prefix.clone(),
                 forward_to: opts_clone.forward_to.clone(),
                 limit: opts_clone.per_disk_limit,
+                skip_total_timeout: opts_clone.skip_walkdir_total_timeout,
                 ..Default::default()
             };
 
@@ -213,6 +222,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                             filter_prefix: opts_clone.filter_prefix.clone(),
                             forward_to: opts_clone.forward_to.clone(),
                             limit: opts_clone.per_disk_limit,
+                            skip_total_timeout: opts_clone.skip_walkdir_total_timeout,
                             ..Default::default()
                         },
                         &mut wr,
@@ -442,10 +452,13 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 {
                     finished_fn(&errs).await;
                 }
-                if errs.iter().flatten().any(|err| *err == DiskError::Timeout) {
-                    return Err(DiskError::Timeout);
-                }
-
+                // All remaining readers are either at EOF or failed. Earlier logic
+                // returned Timeout here for even a single stalled drive, despite
+                // `has_err` being within the tolerated drive-failure budget. That
+                // makes small distributed listings fail once healthy quorum readers
+                // reach EOF but one remote walk stream is slow/stalled. Only the
+                // `has_err > opts.disks.len() - opts.min_disks` branch above should
+                // turn tolerated reader failures into request failures.
                 // error!("list_path_raw: at_eof + has_err == readers.len() break {:?}", &errs);
                 break;
             }
@@ -482,20 +495,42 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
     if let Err(err) = revjob.await.map_err(std::io::Error::other)? {
         error!("list_path_raw: revjob err {:?}", err);
         cancel_rx.cancel();
+        for job in jobs {
+            job.abort();
+        }
 
         return Err(err);
     }
 
+    // The merge consumer can finish successfully before every producer finishes
+    // (for example after reaching EOF quorum while a tolerated drive is stalled,
+    // or after the requested listing limit is satisfied). Cancel remaining walk
+    // jobs before aborting them so list calls do not wait for slow remote streams.
+    cancel_rx.cancel();
+
+    for job in jobs.iter() {
+        if !job.is_finished() {
+            job.abort();
+        }
+    }
+
     let results = join_all(jobs).await;
     let mut job_errs = Vec::new();
     for result in results {
         match result {
             Ok(Ok(())) => {}
             Ok(Err(err)) => {
-                error!("list_path_raw producer err {:?}", err);
+                if matches!(err, DiskError::FileNotFound | DiskError::VolumeNotFound) {
+                    warn!("list_path_raw producer missing path {:?}", err);
+                } else {
+                    error!("list_path_raw producer err {:?}", err);
+                }
                 job_errs.push(err);
             }
             Err(err) => {
+                if err.is_cancelled() {
+                    continue;
+                }
                 error!("list_path_raw join err {:?}", err);
                 job_errs.push(err.into());
             }
@@ -541,18 +576,59 @@ mod tests {
             CancellationToken::new(),
             ListPathRawOptions {
                 disks: vec![None, None],
-                min_disks: 1,
+                min_disks: 2,
                 test_reader_behaviors: vec![TestReaderBehavior::Stall, TestReaderBehavior::Eof],
                 peek_timeout: Some(Duration::from_millis(20)),
                 ..Default::default()
             },
         )
         .await
-        .expect_err("stalled reader should make listing fail explicitly");
+        .expect_err("stalled reader should fail when read quorum cannot be met");
 
         assert_eq!(err, DiskError::Timeout);
     }
 
+    #[tokio::test]
+    async fn list_path_raw_tolerates_stalled_reader_after_quorum_eof() {
+        list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None, None, None],
+                min_disks: 2,
+                test_reader_behaviors: vec![TestReaderBehavior::Eof, TestReaderBehavior::Eof, TestReaderBehavior::Stall],
+                peek_timeout: Some(Duration::from_millis(20)),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("listing should complete when healthy quorum reached EOF and only a tolerated drive stalled");
+    }
+
+    #[tokio::test]
+    async fn list_path_raw_aborts_unresponsive_producer_after_quorum_eof() {
+        let result = timeout(
+            Duration::from_millis(200),
+            list_path_raw(
+                CancellationToken::new(),
+                ListPathRawOptions {
+                    disks: vec![None, None, None],
+                    min_disks: 2,
+                    test_reader_behaviors: vec![
+                        TestReaderBehavior::Eof,
+                        TestReaderBehavior::Eof,
+                        TestReaderBehavior::IgnoreCancel,
+                    ],
+                    peek_timeout: Some(Duration::from_millis(20)),
+                    ..Default::default()
+                },
+            ),
+        )
+        .await;
+
+        let listing = result.expect("list_path_raw should abort unresponsive producer instead of hanging");
+        assert!(listing.is_ok());
+    }
+
     #[tokio::test]
     async fn list_path_raw_returns_timeout_when_producer_fails_after_partial_entry() {
         let seen = Arc::new(Mutex::new(Vec::new()));

crates/ecstore/src/config/audit.rs

@@ -17,23 +17,23 @@ use rustfs_config::audit::AUDIT_REDIS_DEFAULT_CHANNEL;
 use rustfs_config::{
     AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
     AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
-    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA,
-    KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL,
-    MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT,
-    MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST,
-    MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA,
-    MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR,
-    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
-    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
-    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
-    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
-    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
-    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
-    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
-    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
-    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY,
-    WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
-    WEBHOOK_SKIP_TLS_VERIFY,
+    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
+    KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
+    KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
+    MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
+    MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
+    MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
+    MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
+    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING,
+    POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT,
+    POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR,
+    PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC,
+    PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS,
+    REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT,
+    REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT,
+    REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA,
+    WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR,
+    WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;
 
@@ -636,6 +636,26 @@ pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
             value: "".to_owned(),
             hidden_if_empty: true,
         },
+        KV {
+            key: KAFKA_SASL_ENABLE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: KAFKA_SASL_MECHANISM.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: KAFKA_SASL_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: KAFKA_SASL_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
         KV {
             key: KAFKA_QUEUE_DIR.to_owned(),
             value: EVENT_DEFAULT_DIR.to_owned(),

crates/ecstore/src/config/com.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::config::{Config, GLOBAL_STORAGE_CLASS, KVS, audit, notify, oidc, storageclass};
+use crate::config::{Config, KVS, audit, notify, oidc, set_global_storage_class, storageclass};
 use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result};
 use crate::global::is_first_cluster_node_local;
@@ -1164,11 +1164,8 @@ async fn apply_dynamic_config_for_sub_sys<S: StorageAPI>(cfg: &mut Config, api:
         for (i, count) in set_drive_counts.iter().enumerate() {
             match storageclass::lookup_config(&kvs, *count) {
                 Ok(res) => {
-                    if i == 0
-                        && GLOBAL_STORAGE_CLASS.get().is_none()
-                        && let Err(r) = GLOBAL_STORAGE_CLASS.set(res)
-                    {
-                        error!("GLOBAL_STORAGE_CLASS.set failed {:?}", r);
+                    if i == 0 {
+                        set_global_storage_class(res);
                     }
                 }
                 Err(err) => {
@@ -1358,7 +1355,7 @@ mod tests {
                 size: self.data.len() as i64,
                 actual_size: self.data.len() as i64,
                 is_dir: false,
-                user_defined: HashMap::new(),
+                user_defined: Arc::new(HashMap::new()),
                 parity_blocks: 0,
                 data_blocks: 0,
                 version_id: None,
@@ -1366,8 +1363,8 @@ mod tests {
                 transitioned_object: Default::default(),
                 restore_ongoing: false,
                 restore_expires: None,
-                user_tags: String::new(),
-                parts: Vec::new(),
+                user_tags: Arc::new(String::new()),
+                parts: Arc::new(Vec::new()),
                 is_latest: true,
                 content_type: Some("application/json".to_string()),
                 content_encoding: None,
@@ -2638,4 +2635,87 @@ mod tests {
         assert_eq!(object_info.bucket, crate::disk::RUSTFS_META_BUCKET);
         assert_eq!(object_info.name, "config/test.json");
     }
+
+    #[test]
+    fn test_encode_decode_roundtrip_preserves_storage_class() {
+        use crate::config::STORAGE_CLASS_SUB_SYS;
+
+        // Create a config with custom storage class values
+        let mut cfg = Config::new();
+        let kvs = storage_class_kvs_mut(&mut cfg);
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        kvs.insert("rrs".to_string(), "EC:2".to_string());
+        kvs.insert("optimize".to_string(), "availability".to_string());
+
+        // Encode to external format (what save_server_config produces)
+        let encoded = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+
+        // Verify the encoded data uses "storageclass" (no underscore)
+        let v: serde_json::Value = serde_json::from_slice(&encoded).expect("should be valid json");
+        assert!(v.get("storageclass").is_some(), "encoded should have 'storageclass' key");
+        assert!(v.get("storage_class").is_none(), "encoded should NOT have 'storage_class' key");
+
+        // Decode back (what load_server_config_from_store produces)
+        let decoded = decode_server_config_blob(&encoded).expect("decode should succeed");
+
+        // Verify the decoded config has "storage_class" (with underscore) subsystem
+        let kvs = decoded
+            .get_value(STORAGE_CLASS_SUB_SYS, crate::config::DEFAULT_DELIMITER)
+            .expect("decoded config should have storage_class subsystem");
+        assert_eq!(kvs.get("standard"), "EC:4", "standard should be EC:4");
+        assert_eq!(kvs.get("rrs"), "EC:2", "rrs should be EC:2");
+        assert_eq!(kvs.get("optimize"), "availability", "optimize should be availability");
+    }
+
+    #[test]
+    fn test_set_then_load_preserves_storage_class_values() {
+        use crate::config::STORAGE_CLASS_SUB_SYS;
+
+        // Step 1: Start with a fresh config (simulates initial server state)
+        let mut cfg = Config::new();
+
+        // Step 2: Apply set directives (simulates "mc admin config set storage_class standard=EC:4 rrs=EC:2")
+        let kvs = cfg
+            .0
+            .entry(STORAGE_CLASS_SUB_SYS.to_string())
+            .or_default()
+            .entry(DEFAULT_DELIMITER.to_string())
+            .or_default();
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        kvs.insert("rrs".to_string(), "EC:2".to_string());
+        cfg.set_defaults();
+
+        // Step 3: Save (encode to external format)
+        let encoded = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+
+        // Step 4: Load (decode from external format)
+        let loaded = decode_server_config_blob(&encoded).expect("decode should succeed");
+
+        // Step 5: Verify the values are preserved
+        let loaded_kvs = loaded
+            .get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("should have storage_class");
+        assert_eq!(loaded_kvs.get("standard"), "EC:4");
+        assert_eq!(loaded_kvs.get("rrs"), "EC:2");
+
+        // Step 6: Verify the subsystem is accessible by name (what render_selected_config does)
+        let targets = loaded
+            .0
+            .get(STORAGE_CLASS_SUB_SYS)
+            .expect("storage_class subsystem should exist");
+        let target_kvs = targets.get(DEFAULT_DELIMITER).expect("default target should exist");
+        assert_eq!(target_kvs.get("standard"), "EC:4");
+        assert_eq!(target_kvs.get("rrs"), "EC:2");
+    }
+
+    #[test]
+    fn test_config_unmarshal_fails_on_external_format() {
+        // This verifies that the external "storageclass" format cannot be
+        // mistakenly parsed by Config::unmarshal (which expects internal format).
+        // If unmarshal succeeds, the config would have "storageclass" instead of
+        // "storage_class", causing render_selected_config to fail.
+        let external_json = r#"{"version":"33","storageclass":{"standard":"EC:4","rrs":"EC:2"}}"#;
+        let result = Config::unmarshal(external_json.as_bytes());
+        assert!(result.is_err(), "Config::unmarshal should fail on external format, but got: {:?}", result);
+    }
 }

crates/ecstore/src/config/heal.rs

@@ -12,10 +12,21 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use crate::config::{KV, KVS};
 use crate::error::{Error, Result};
+use rustfs_config::{DEFAULT_HEAL_BITROT_CYCLE_SECS, HEAL_BITROT_CYCLE};
 use rustfs_utils::string::parse_bool;
+use std::sync::LazyLock;
 use std::time::Duration;
 
+pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![KV {
+        key: HEAL_BITROT_CYCLE.to_owned(),
+        value: DEFAULT_HEAL_BITROT_CYCLE_SECS.to_string(),
+        hidden_if_empty: false,
+    }])
+});
+
 #[derive(Debug, Default)]
 pub struct Config {
     pub bitrot: String,

crates/ecstore/src/config/mod.rs

@@ -18,6 +18,7 @@ pub mod com;
 pub mod heal;
 mod notify;
 mod oidc;
+mod scanner;
 pub mod storageclass;
 
 use crate::error::Result;
@@ -25,6 +26,7 @@ use crate::store::ECStore;
 use com::{STORAGE_CLASS_SUB_SYS, lookup_configs, read_config_without_migrate};
 use rustfs_config::COMMENT_KEY;
 use rustfs_config::DEFAULT_DELIMITER;
+use rustfs_config::HEAL_SUB_SYS;
 use rustfs_config::audit::{
     AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_MYSQL_SUB_SYS, AUDIT_NATS_SUB_SYS, AUDIT_POSTGRES_SUB_SYS,
     AUDIT_PULSAR_SUB_SYS, AUDIT_REDIS_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS,
@@ -37,11 +39,12 @@ use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::sync::LazyLock;
-use std::sync::{Arc, OnceLock};
+use std::sync::{Arc, OnceLock, RwLock};
 
-pub static GLOBAL_STORAGE_CLASS: LazyLock<OnceLock<storageclass::Config>> = LazyLock::new(OnceLock::new);
+pub static GLOBAL_STORAGE_CLASS: LazyLock<RwLock<storageclass::Config>> =
+    LazyLock::new(|| RwLock::new(storageclass::Config::default()));
 pub static DEFAULT_KVS: LazyLock<OnceLock<HashMap<String, KVS>>> = LazyLock::new(OnceLock::new);
-pub static GLOBAL_SERVER_CONFIG: LazyLock<OnceLock<Config>> = LazyLock::new(OnceLock::new);
+pub static GLOBAL_SERVER_CONFIG: LazyLock<RwLock<Option<Config>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_CONFIG_SYS: LazyLock<ConfigSys> = LazyLock::new(ConfigSys::new);
 
 pub static RUSTFS_CONFIG_PREFIX: &str = "config";
@@ -63,14 +66,30 @@ impl ConfigSys {
 
         lookup_configs(&mut cfg, api).await;
 
-        let _ = GLOBAL_SERVER_CONFIG.set(cfg);
+        set_global_server_config(cfg);
 
         Ok(())
     }
 }
 
 pub fn get_global_server_config() -> Option<Config> {
-    GLOBAL_SERVER_CONFIG.get().cloned()
+    GLOBAL_SERVER_CONFIG.read().ok().and_then(|guard| (*guard).clone())
+}
+
+pub fn set_global_server_config(cfg: Config) {
+    if let Ok(mut guard) = GLOBAL_SERVER_CONFIG.write() {
+        *guard = Some(cfg);
+    }
+}
+
+pub fn get_global_storage_class() -> Option<storageclass::Config> {
+    GLOBAL_STORAGE_CLASS.read().ok().map(|guard| (*guard).clone())
+}
+
+pub fn set_global_storage_class(cfg: storageclass::Config) {
+    if let Ok(mut guard) = GLOBAL_STORAGE_CLASS.write() {
+        *guard = cfg;
+    }
 }
 
 pub async fn init_global_config_sys(api: Arc<ECStore>) -> Result<()> {
@@ -237,6 +256,8 @@ pub fn init() {
     let mut kvs = HashMap::new();
     // Load storageclass default configuration
     kvs.insert(STORAGE_CLASS_SUB_SYS.to_owned(), storageclass::DEFAULT_KVS.clone());
+    kvs.insert(rustfs_config::SCANNER_SUB_SYS.to_owned(), scanner::DEFAULT_KVS.clone());
+    kvs.insert(HEAL_SUB_SYS.to_owned(), heal::DEFAULT_KVS.clone());
     // New: Loading default configurations for notify_webhook and notify_mqtt
     // Referring subsystem names through constants to improve the readability and maintainability of the code
     kvs.insert(NOTIFY_WEBHOOK_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_WEBHOOK_KVS.clone());
@@ -262,3 +283,49 @@ pub fn init() {
     // Register all default configurations
     register_default_kvs(kvs)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use rustfs_config::{
+        DEFAULT_DELIMITER, DEFAULT_HEAL_BITROT_CYCLE_SECS, DEFAULT_SCANNER_SPEED, HEAL_BITROT_CYCLE, SCANNER_CYCLE_MAX_OBJECTS,
+        SCANNER_DELAY, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_SUB_SYS,
+    };
+
+    #[test]
+    fn global_server_config_set_and_get_roundtrip() {
+        init();
+        let mut cfg = Config::new();
+        let mut kvs = KVS::new();
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        cfg.0
+            .insert(STORAGE_CLASS_SUB_SYS.to_string(), HashMap::from([("_".to_string(), kvs)]));
+
+        set_global_server_config(cfg.clone());
+        let loaded = get_global_server_config().expect("global config should be set");
+        let sc_kvs = loaded
+            .get_value(STORAGE_CLASS_SUB_SYS, "_")
+            .expect("storage_class should exist");
+        assert_eq!(sc_kvs.get("standard"), "EC:4");
+    }
+
+    #[test]
+    fn scanner_defaults_are_registered_for_admin_config() {
+        init();
+        let cfg = Config::new();
+        let scanner_kvs = cfg
+            .get_value(SCANNER_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("scanner defaults should exist");
+
+        assert_eq!(scanner_kvs.get(SCANNER_SPEED), DEFAULT_SCANNER_SPEED);
+        assert_eq!(scanner_kvs.get(SCANNER_DELAY), "");
+        assert_eq!(scanner_kvs.get(SCANNER_MAX_WAIT), "");
+        assert_eq!(scanner_kvs.get(SCANNER_CYCLE_MAX_OBJECTS), "0");
+
+        let heal_kvs = cfg
+            .get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("heal defaults should exist");
+
+        assert_eq!(heal_kvs.get(HEAL_BITROT_CYCLE), DEFAULT_HEAL_BITROT_CYCLE_SECS.to_string());
+    }
+}

crates/ecstore/src/config/notify.rs

@@ -17,22 +17,22 @@ use rustfs_config::notify::NOTIFY_REDIS_DEFAULT_CHANNEL;
 use rustfs_config::{
     AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
     AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
-    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_TLS_CA,
-    KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY, KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL,
-    MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT,
-    MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST,
-    MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA,
-    MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR,
-    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
-    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
-    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
-    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
-    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
-    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
-    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
-    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
-    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
-    WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
+    EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
+    KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
+    KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
+    MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
+    MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
+    MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
+    MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
+    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING,
+    POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT,
+    POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR,
+    PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC,
+    PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS,
+    REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT,
+    REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT,
+    REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA,
+    WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;
 
@@ -613,6 +613,26 @@ pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
             value: "".to_owned(),
             hidden_if_empty: true,
         },
+        KV {
+            key: KAFKA_SASL_ENABLE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: KAFKA_SASL_MECHANISM.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: KAFKA_SASL_USERNAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: KAFKA_SASL_PASSWORD.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: true,
+        },
         KV {
             key: KAFKA_QUEUE_DIR.to_owned(),
             value: EVENT_DEFAULT_DIR.to_owned(),

crates/ecstore/src/config/scanner.rs

@@ -0,0 +1,116 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::config::{KV, KVS};
+use rustfs_config::{
+    DEFAULT_SCANNER_BITROT_CYCLE_SECS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES,
+    DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_IDLE_MODE,
+    DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_SPEED,
+    DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE,
+    SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_SPEED, SCANNER_START_DELAY,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
+};
+use std::sync::LazyLock;
+
+pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: SCANNER_SPEED.to_owned(),
+            value: DEFAULT_SCANNER_SPEED.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_DELAY.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_MAX_WAIT.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_CYCLE.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_START_DELAY.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_DURATION.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_OBJECTS.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_OBJECTS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_DIRECTORIES.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_BITROT_CYCLE.to_owned(),
+            value: DEFAULT_SCANNER_BITROT_CYCLE_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_IDLE_MODE.to_owned(),
+            value: DEFAULT_SCANNER_IDLE_MODE.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CACHE_SAVE_TIMEOUT.to_owned(),
+            value: DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_MAX_CONCURRENT_SET_SCANS.to_owned(),
+            value: DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_MAX_CONCURRENT_DISK_SCANS.to_owned(),
+            value: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_YIELD_EVERY_N_OBJECTS.to_owned(),
+            value: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_VERSIONS.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_VERSION_SIZE.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_FOLDERS.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS.to_string(),
+            hidden_if_empty: false,
+        },
+    ])
+});

crates/ecstore/src/config/storageclass.rs

@@ -101,13 +101,13 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
 });
 
 // StorageClass - holds storage class information
-#[derive(Serialize, Deserialize, Debug, Default)]
+#[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct StorageClass {
     parity: usize,
 }
 
 // Config storage class configuration
-#[derive(Serialize, Deserialize, Debug, Default)]
+#[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Config {
     standard: StorageClass,
     rrs: StorageClass,
@@ -205,7 +205,7 @@ pub fn lookup_config(kvs: &KVS, set_drive_count: usize) -> Result<Config> {
             if let Ok(ssc_str) = env::var(RRS_ENV) {
                 ssc_str
             } else {
-                kvs.get(RRS)
+                kvs.get(CLASS_RRS)
             }
         };
 
@@ -336,3 +336,36 @@ pub fn validate_parity_inner(ss_parity: usize, rrs_parity: usize, set_drive_coun
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn lookup_config_reads_rrs_from_class_rrs_key() {
+        // Regression: kvs.get(RRS) used RRS="REDUCED_REDUNDANCY" instead of
+        // CLASS_RRS="rrs", so admin-written RRS values were never read back.
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
+        kvs.insert(CLASS_RRS.to_string(), "EC:2".to_string());
+
+        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        assert_eq!(cfg.standard.parity, 4, "standard parity should be 4");
+        assert_eq!(cfg.rrs.parity, 2, "rrs parity should be 2");
+    }
+
+    #[test]
+    fn lookup_config_ignores_redundancy_key_name() {
+        // Ensure the old key name "REDUCED_REDUNDANCY" is NOT read.
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
+        kvs.insert(RRS.to_string(), "EC:2".to_string());
+
+        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        assert_eq!(cfg.standard.parity, 4);
+        assert_eq!(
+            cfg.rrs.parity, DEFAULT_RRS_PARITY,
+            "rrs should fall back to default when CLASS_RRS key is absent"
+        );
+    }
+}

crates/ecstore/src/data_movement.rs

@@ -12,11 +12,14 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::error::{Error, Result};
+use crate::error::{Error, Result, is_err_data_movement_overwrite, is_err_object_not_found, is_err_version_not_found};
 use crate::store::ECStore;
-use crate::store_api::{CompletePart, GetObjectReader, MultipartOperations, ObjectIO, ObjectInfo, ObjectOptions, PutObjReader};
+use crate::store_api::{
+    CompletePart, GetObjectReader, MultipartOperations, ObjectIO, ObjectInfo, ObjectOperations, ObjectOptions, PutObjReader,
+};
 use bytes::Bytes;
 use rustfs_rio::{EtagResolvable, HashReader, HashReaderDetector, Index, TryGetIndex};
+use rustfs_utils::path::encode_dir_object;
 use std::io::Cursor;
 use std::pin::Pin;
 use std::sync::{
@@ -94,7 +97,7 @@ fn data_movement_new_multipart_opts(object_info: &ObjectInfo, src_pool_idx: usiz
     ObjectOptions {
         versioned: object_info.version_id.is_some(),
         version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
-        user_defined: object_info.user_defined.clone(),
+        user_defined: (*object_info.user_defined).clone(),
         preserve_etag: object_info.etag.clone(),
         src_pool_idx,
         data_movement: true,
@@ -102,13 +105,14 @@ fn data_movement_new_multipart_opts(object_info: &ObjectInfo, src_pool_idx: usiz
     }
 }
 
-fn data_movement_complete_multipart_opts(object_info: &ObjectInfo) -> ObjectOptions {
+fn data_movement_complete_multipart_opts(object_info: &ObjectInfo, src_pool_idx: usize) -> ObjectOptions {
     ObjectOptions {
         versioned: object_info.version_id.is_some(),
         version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
         data_movement: true,
         mod_time: object_info.mod_time,
         preserve_etag: object_info.etag.clone(),
+        src_pool_idx,
         ..Default::default()
     }
 }
@@ -120,7 +124,7 @@ fn data_movement_put_object_opts(object_info: &ObjectInfo, src_pool_idx: usize)
         data_movement: true,
         version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
         mod_time: object_info.mod_time,
-        user_defined: object_info.user_defined.clone(),
+        user_defined: (*object_info.user_defined).clone(),
         preserve_etag: object_info.etag.clone(),
         ..Default::default()
     }
@@ -139,6 +143,111 @@ fn resolve_data_movement_abort_result(
     ))
 }
 
+fn data_movement_stage_error(op_label: &str, stage: &str, bucket: &str, object: &str, err: impl std::fmt::Display) -> Error {
+    Error::other(format!("{op_label}: {stage} failed for {bucket}/{object}: {err}"))
+}
+
+fn should_check_data_movement_overwrite_resume(err: &Error) -> bool {
+    is_err_data_movement_overwrite(err)
+}
+
+fn effective_actual_size(info: &ObjectInfo) -> Option<i64> {
+    info.get_actual_size().ok()
+}
+
+fn is_equivalent_data_movement_object(source: &ObjectInfo, target: &ObjectInfo) -> bool {
+    source.version_id == target.version_id
+        && source.delete_marker == target.delete_marker
+        && source.size == target.size
+        && effective_actual_size(source) == effective_actual_size(target)
+        && source.etag == target.etag
+        && source.checksum == target.checksum
+        && source.mod_time == target.mod_time
+}
+
+fn should_check_data_movement_resume_target(src_pool_idx: usize, target_pool_idx: usize) -> bool {
+    target_pool_idx != src_pool_idx
+}
+
+async fn find_data_movement_target_info(
+    store: &ECStore,
+    target_pool_idx: usize,
+    bucket: &str,
+    object_info: &ObjectInfo,
+) -> Result<Option<ObjectInfo>> {
+    let opts = ObjectOptions {
+        versioned: object_info.version_id.is_some(),
+        version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
+        no_lock: true,
+        ..Default::default()
+    };
+    let object = encode_dir_object(object_info.name.as_str());
+
+    let Some(pool) = store.pools.get(target_pool_idx) else {
+        return Err(Error::other(format!(
+            "data movement resume target pool {target_pool_idx} is out of range for {bucket}/{object}"
+        )));
+    };
+
+    match pool.get_object_info(bucket, object.as_str(), &opts).await {
+        Ok(target_info) => Ok(Some(target_info)),
+        Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => Ok(None),
+        Err(err) => Err(err),
+    }
+}
+
+fn resolve_data_movement_overwrite_resume_result(
+    err: &Error,
+    target_result: Result<Option<ObjectInfo>>,
+    source: &ObjectInfo,
+    src_pool_idx: usize,
+    target_pool_idx: usize,
+) -> Result<bool> {
+    if !should_check_data_movement_overwrite_resume(err)
+        || !should_check_data_movement_resume_target(src_pool_idx, target_pool_idx)
+    {
+        return Ok(false);
+    }
+
+    let Some(target) = target_result? else {
+        return Ok(false);
+    };
+
+    Ok(is_equivalent_data_movement_object(source, &target))
+}
+
+async fn should_treat_data_movement_overwrite_as_complete(
+    store: &ECStore,
+    src_pool_idx: usize,
+    target_pool_idx: usize,
+    bucket: &str,
+    object_info: &ObjectInfo,
+    err: &Error,
+) -> Result<bool> {
+    if !should_check_data_movement_overwrite_resume(err) {
+        return Ok(false);
+    }
+
+    resolve_data_movement_overwrite_resume_result(
+        err,
+        find_data_movement_target_info(store, target_pool_idx, bucket, object_info).await,
+        object_info,
+        src_pool_idx,
+        target_pool_idx,
+    )
+}
+
+fn data_movement_part_stage_error(
+    op_label: &str,
+    stage: &str,
+    bucket: &str,
+    object: &str,
+    part_number: usize,
+    err: impl std::fmt::Display,
+) -> Error {
+    Error::other(format!("{op_label}: {stage} failed for {bucket}/{object} part {part_number}: {err}"))
+}
+
 pub(crate) async fn migrate_object(
     store: Arc<ECStore>,
     pool_idx: usize,
@@ -149,14 +258,24 @@ pub(crate) async fn migrate_object(
     let object_info = rd.object_info.clone();
 
     if object_info.is_multipart() {
-        let res = match store
-            .new_multipart_upload(&bucket, &object_info.name, &data_movement_new_multipart_opts(&object_info, pool_idx))
+        let (res, target_pool_idx) = match store
+            .handle_new_multipart_upload_with_pool_idx(
+                &bucket,
+                &object_info.name,
+                &data_movement_new_multipart_opts(&object_info, pool_idx),
+            )
             .await
         {
             Ok(res) => res,
             Err(err) => {
                 error!("{op_label}: new_multipart_upload err {:?}", &err);
-                return Err(err);
+                return Err(data_movement_stage_error(
+                    op_label,
+                    "new_multipart_upload",
+                    bucket.as_str(),
+                    object_info.name.as_str(),
+                    err,
+                ));
             }
         };
 
@@ -167,12 +286,39 @@ pub(crate) async fn migrate_object(
 
             for (i, part) in object_info.parts.iter().enumerate() {
                 let mut chunk = vec![0u8; part.size];
-                reader.read_exact(&mut chunk).await?;
+                reader.read_exact(&mut chunk).await.map_err(|err| {
+                    data_movement_part_stage_error(
+                        op_label,
+                        "read_part",
+                        bucket.as_str(),
+                        object_info.name.as_str(),
+                        part.number,
+                        Error::other(err.to_string()),
+                    )
+                })?;
 
-                let part_size = i64::try_from(part.size).map_err(|_| Error::other("part size overflow"))?;
+                let part_size = i64::try_from(part.size).map_err(|_| {
+                    data_movement_part_stage_error(
+                        op_label,
+                        "prepare_part",
+                        bucket.as_str(),
+                        object_info.name.as_str(),
+                        part.number,
+                        Error::other("part size overflow"),
+                    )
+                })?;
                 let part_actual_size = if part.actual_size > 0 { part.actual_size } else { part_size };
                 let index = decode_part_index(part.index.as_ref());
-                let mut data = put_obj_reader_from_chunk(chunk, part_size, part_actual_size, index)?;
+                let mut data = put_obj_reader_from_chunk(chunk, part_size, part_actual_size, index).map_err(|err| {
+                    data_movement_part_stage_error(
+                        op_label,
+                        "prepare_part",
+                        bucket.as_str(),
+                        object_info.name.as_str(),
+                        part.number,
+                        err,
+                    )
+                })?;
 
                 let pi = match store
                     .put_object_part(
@@ -191,7 +337,14 @@ pub(crate) async fn migrate_object(
                     Ok(pi) => pi,
                     Err(err) => {
                         error!("{op_label}: put_object_part {i} err {:?}", &err);
-                        return Err(err);
+                        return Err(data_movement_part_stage_error(
+                            op_label,
+                            "put_object_part",
+                            bucket.as_str(),
+                            object_info.name.as_str(),
+                            part.number,
+                            err,
+                        ));
                     }
                 };
 
@@ -209,12 +362,32 @@ pub(crate) async fn migrate_object(
                     &object_info.name,
                     &res.upload_id,
                     parts,
-                    &data_movement_complete_multipart_opts(&object_info),
+                    &data_movement_complete_multipart_opts(&object_info, pool_idx),
                 )
                 .await
             {
+                if should_treat_data_movement_overwrite_as_complete(
+                    store.as_ref(),
+                    pool_idx,
+                    target_pool_idx,
+                    bucket.as_str(),
+                    &object_info,
+                    &err,
+                )
+                .await?
+                {
+                    mark_multipart_upload_completed(&abort_multipart_flag);
+                    return Ok(());
+                }
+
                 error!("{op_label}: complete_multipart_upload err {:?}", &err);
-                return Err(err);
+                return Err(data_movement_stage_error(
+                    op_label,
+                    "complete_multipart_upload",
+                    bucket.as_str(),
+                    object_info.name.as_str(),
+                    err,
+                ));
             }
 
             mark_multipart_upload_completed(&abort_multipart_flag);
@@ -248,13 +421,18 @@ pub(crate) async fn migrate_object(
         return Ok(());
     }
 
-    let actual_size = object_info.get_actual_size()?;
+    let actual_size = object_info.get_actual_size().map_err(|err| {
+        data_movement_stage_error(op_label, "prepare_put_object", bucket.as_str(), object_info.name.as_str(), err)
+    })?;
     let index = object_info
         .parts
         .first()
         .and_then(|part| decode_part_index(part.index.as_ref()));
     let reader = IndexedDataMovementReader::new(BufReader::new(rd.stream), index);
-    let hrd = HashReader::from_stream(reader, object_info.size, actual_size, object_info.etag.clone(), None, false)?;
+    let hrd =
+        HashReader::from_stream(reader, object_info.size, actual_size, object_info.etag.clone(), None, false).map_err(|err| {
+            data_movement_stage_error(op_label, "prepare_put_object", bucket.as_str(), object_info.name.as_str(), err)
+        })?;
     let mut data = PutObjReader::new(hrd);
 
     if let Err(err) = store
@@ -267,7 +445,13 @@ pub(crate) async fn migrate_object(
         .await
     {
         error!("{op_label}: put_object err {:?}", &err);
-        return Err(err);
+        return Err(data_movement_stage_error(
+            op_label,
+            "put_object",
+            bucket.as_str(),
+            object_info.name.as_str(),
+            err,
+        ));
     }
 
     Ok(())
@@ -309,6 +493,33 @@ mod tests {
         assert!(message.contains(Error::SlowDown.to_string().as_str()));
     }
 
+    #[test]
+    fn test_data_movement_stage_error_includes_stage_and_object() {
+        let err = data_movement_stage_error("rebalance_object", "put_object", "bucket-a", "object-a", Error::SlowDown);
+        let message = err.to_string();
+        assert!(message.contains("rebalance_object: put_object failed for bucket-a/object-a"));
+        assert!(message.contains(Error::SlowDown.to_string().as_str()));
+    }
+
+    #[test]
+    fn test_data_movement_part_stage_error_includes_stage_object_and_part() {
+        let err =
+            data_movement_part_stage_error("rebalance_object", "put_object_part", "bucket-a", "object-a", 7, Error::SlowDown);
+        let message = err.to_string();
+        assert!(message.contains("rebalance_object: put_object_part failed for bucket-a/object-a part 7"));
+        assert!(message.contains(Error::SlowDown.to_string().as_str()));
+    }
+
+    #[test]
+    fn test_should_check_data_movement_overwrite_resume_only_for_overwrite_error() {
+        assert!(should_check_data_movement_overwrite_resume(&Error::DataMovementOverwriteErr(
+            "bucket-a".to_string(),
+            "object-a".to_string(),
+            "version-a".to_string(),
+        )));
+        assert!(!should_check_data_movement_overwrite_resume(&Error::SlowDown));
+    }
+
     #[test]
     fn test_decode_part_index_returns_none_when_absent() {
         assert!(decode_part_index(None).is_none());
@@ -341,7 +552,7 @@ mod tests {
         let object_info = ObjectInfo {
             version_id: Some(version_id),
             etag: Some("etag-value".to_string()),
-            user_defined: std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())]),
+            user_defined: Arc::new(std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())])),
             ..Default::default()
         };
 
@@ -366,13 +577,14 @@ mod tests {
             ..Default::default()
         };
 
-        let opts = data_movement_complete_multipart_opts(&object_info);
+        let opts = data_movement_complete_multipart_opts(&object_info, 7);
 
         assert!(opts.versioned);
         assert!(opts.data_movement);
         assert_eq!(opts.mod_time, Some(mod_time));
         assert_eq!(opts.version_id.as_deref(), Some(version_id.to_string().as_str()));
         assert_eq!(opts.preserve_etag.as_deref(), Some("etag-value"));
+        assert_eq!(opts.src_pool_idx, 7);
     }
 
     #[test]
@@ -382,7 +594,7 @@ mod tests {
             version_id: Some(version_id),
             mod_time: Some(OffsetDateTime::UNIX_EPOCH),
             etag: Some("etag-value".to_string()),
-            user_defined: std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())]),
+            user_defined: Arc::new(std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())])),
             ..Default::default()
         };
 
@@ -396,4 +608,130 @@ mod tests {
         assert!(opts.data_movement);
         assert_eq!(opts.mod_time, object_info.mod_time);
     }
+
+    #[test]
+    fn test_is_equivalent_data_movement_object_accepts_matching_metadata() {
+        let version_id = Uuid::nil();
+        let info = ObjectInfo {
+            version_id: Some(version_id),
+            size: 128,
+            actual_size: 96,
+            etag: Some("etag-value".to_string()),
+            checksum: Some(Bytes::from_static(b"checksum")),
+            mod_time: Some(OffsetDateTime::UNIX_EPOCH),
+            ..Default::default()
+        };
+
+        assert!(is_equivalent_data_movement_object(&info, &info.clone()));
+    }
+
+    #[test]
+    fn test_is_equivalent_data_movement_object_rejects_content_mismatch() {
+        let source = ObjectInfo {
+            version_id: Some(Uuid::nil()),
+            size: 128,
+            actual_size: 96,
+            etag: Some("etag-source".to_string()),
+            checksum: Some(Bytes::from_static(b"checksum-source")),
+            mod_time: Some(OffsetDateTime::UNIX_EPOCH),
+            ..Default::default()
+        };
+        let target = ObjectInfo {
+            etag: Some("etag-target".to_string()),
+            checksum: Some(Bytes::from_static(b"checksum-target")),
+            ..source.clone()
+        };
+
+        assert!(!is_equivalent_data_movement_object(&source, &target));
+    }
+
+    #[test]
+    fn test_is_equivalent_data_movement_object_uses_effective_actual_size() {
+        let source = ObjectInfo {
+            size: 128,
+            actual_size: 0,
+            etag: Some("etag-value".to_string()),
+            ..Default::default()
+        };
+        let target = ObjectInfo {
+            size: 128,
+            actual_size: 128,
+            etag: Some("etag-value".to_string()),
+            ..Default::default()
+        };
+
+        assert!(is_equivalent_data_movement_object(&source, &target));
+    }
+
+    #[test]
+    fn test_resolve_data_movement_overwrite_resume_result_accepts_equivalent_target() {
+        let source = ObjectInfo {
+            version_id: Some(Uuid::nil()),
+            size: 128,
+            etag: Some("etag-value".to_string()),
+            mod_time: Some(OffsetDateTime::UNIX_EPOCH),
+            ..Default::default()
+        };
+        let err = Error::DataMovementOverwriteErr("bucket".to_string(), "object".to_string(), "version".to_string());
+
+        let should_resume = resolve_data_movement_overwrite_resume_result(&err, Ok(Some(source.clone())), &source, 0, 1)
+            .expect("equivalent overwrite target should be evaluated");
+
+        assert!(should_resume);
+    }
+
+    #[test]
+    fn test_resolve_data_movement_overwrite_resume_result_rejects_source_pool_target() {
+        let source = ObjectInfo {
+            version_id: Some(Uuid::nil()),
+            size: 128,
+            etag: Some("etag-value".to_string()),
+            mod_time: Some(OffsetDateTime::UNIX_EPOCH),
+            ..Default::default()
+        };
+        let err = Error::DataMovementOverwriteErr("bucket".to_string(), "object".to_string(), "version".to_string());
+
+        let should_resume = resolve_data_movement_overwrite_resume_result(&err, Ok(Some(source.clone())), &source, 0, 0)
+            .expect("source-pool target should be rejected before target lookup");
+
+        assert!(!should_resume);
+    }
+
+    #[test]
+    fn test_resolve_data_movement_overwrite_resume_result_rejects_non_equivalent_target() {
+        let source = ObjectInfo {
+            version_id: Some(Uuid::nil()),
+            size: 128,
+            etag: Some("etag-source".to_string()),
+            ..Default::default()
+        };
+        let target = ObjectInfo {
+            etag: Some("etag-target".to_string()),
+            ..source.clone()
+        };
+        let err = Error::DataMovementOverwriteErr("bucket".to_string(), "object".to_string(), "version".to_string());
+
+        let should_resume = resolve_data_movement_overwrite_resume_result(&err, Ok(Some(target)), &source, 0, 1)
+            .expect("non-equivalent overwrite target should be evaluated");
+
+        assert!(!should_resume);
+    }
+
+    #[test]
+    fn test_resolve_data_movement_overwrite_resume_result_propagates_target_lookup_error() {
+        let source = ObjectInfo::default();
+        let err = Error::DataMovementOverwriteErr("bucket".to_string(), "object".to_string(), "version".to_string());
+        let result = resolve_data_movement_overwrite_resume_result(&err, Err(Error::SlowDown), &source, 0, 1);
+
+        assert!(matches!(result, Err(Error::SlowDown)));
+    }
+
+    #[test]
+    fn test_resolve_data_movement_overwrite_resume_result_ignores_non_overwrite_error() {
+        let source = ObjectInfo::default();
+        let result = resolve_data_movement_overwrite_resume_result(&Error::SlowDown, Err(Error::FileAccessDenied), &source, 0, 1)
+            .expect("non-overwrite errors should not query target equivalence");
+
+        assert!(!result);
+    }
 }

crates/ecstore/src/disk/disk_store.rs

@@ -1136,11 +1136,18 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn walk_dir<W: tokio::io::AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
+        let timeout_duration = if opts.skip_total_timeout {
+            Duration::ZERO
+        } else {
+            get_drive_walkdir_timeout()
+        };
+
         self.track_disk_health_with_op_and_timeout_action(
             "walk_dir",
             || async { self.disk.walk_dir(opts, wr).await },
-            get_drive_walkdir_timeout(),
-            self.scanner_timeout_health_action(),
+            timeout_duration,
+            // Listing/scanner backpressure should fail only the current walk, not poison drive health.
+            TimeoutHealthAction::IgnoreFailure,
         )
         .await
     }
@@ -1241,7 +1248,8 @@ impl DiskAPI for LocalDiskWrapper {
         dst_volume: &str,
         dst_path: &str,
     ) -> Result<RenameDataResp> {
-        self.track_disk_health(
+        self.track_disk_health_with_op(
+            "rename_data",
             || async { self.disk.rename_data(src_volume, src_path, fi, dst_volume, dst_path).await },
             get_max_timeout_duration(),
         )
@@ -1603,7 +1611,7 @@ mod tests {
     }
 
     #[tokio::test]
-    async fn walk_dir_writer_backpressure_timeout_marks_drive_failure_by_default() {
+    async fn walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure_by_default() {
         temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
             let dir = tempfile::tempdir().expect("temp dir should be created");
             let endpoint =
@@ -1639,7 +1647,106 @@ mod tests {
                 .await;
 
             assert_eq!(result.expect_err("walk_dir should time out"), DiskError::Timeout);
-            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Suspect);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    async fn walk_dir_skip_total_timeout_keeps_stream_pending() {
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+            let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+            let wrapper = LocalDiskWrapper::new(disk, false);
+            let bucket = "test-bucket";
+            let object = "test-object";
+
+            wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+            let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+            file_info.volume = bucket.to_string();
+            file_info.name = object.to_string();
+            file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+            file_info.erasure.index = 1;
+
+            wrapper
+                .write_metadata("", bucket, object, file_info)
+                .await
+                .expect("object metadata should be written");
+
+            let mut writer = PendingWriter;
+            let result = tokio::time::timeout(
+                Duration::from_millis(20),
+                wrapper.walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        skip_total_timeout: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                ),
+            )
+            .await;
+
+            assert!(result.is_err(), "skip_total_timeout should leave backpressured walk pending");
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    async fn walk_dir_timeout_does_not_break_followup_stat_volume() {
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+            let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+            let wrapper = LocalDiskWrapper::new(disk, false);
+            let bucket = "test-bucket";
+            let object = "test-object";
+
+            wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+            let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+            file_info.volume = bucket.to_string();
+            file_info.name = object.to_string();
+            file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+            file_info.erasure.index = 1;
+
+            wrapper
+                .write_metadata("", bucket, object, file_info)
+                .await
+                .expect("object metadata should be written");
+
+            let mut writer = PendingWriter;
+            let walk_err = wrapper
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await
+                .expect_err("walk_dir should time out");
+
+            assert_eq!(walk_err, DiskError::Timeout);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+
+            let info = wrapper
+                .stat_volume(bucket)
+                .await
+                .expect("follow-up bucket stat should still succeed after walk timeout");
+            assert_eq!(info.name, bucket);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
         })
         .await;
     }

crates/ecstore/src/disk/endpoint.rs

@@ -19,6 +19,23 @@ use std::{fmt::Display, path::Path};
 use tracing::debug;
 use url::{ParseError, Url};
 
+#[cfg(windows)]
+pub(crate) fn windows_fallback_local_path(
+    path: &str,
+    canonicalize_error: &std::io::Error,
+    context: &'static str,
+) -> std::io::Result<std::path::PathBuf> {
+    let absolute = Path::new(path).absolutize()?.to_path_buf();
+    tracing::warn!(
+        path = %path,
+        canonicalize_error = %canonicalize_error,
+        resolved = ?absolute,
+        context = context,
+        "using windows fallback path resolution for local endpoint"
+    );
+    Ok(absolute)
+}
+
 /// enum for endpoint type.
 #[derive(PartialEq, Eq, Debug)]
 pub enum EndpointType {

crates/ecstore/src/disk/error.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use rustfs_rio::{InternodeHttpError, InternodeHttpErrorKind};
 use std::hash::{Hash, Hasher};
 use std::io::{self};
 use std::path::PathBuf;
@@ -181,6 +182,26 @@ impl DiskError {
         matches!(err, &DiskError::FileVersionNotFound)
     }
 
+    pub fn is_retryable_internode_write_failure(&self) -> bool {
+        match self {
+            DiskError::Io(io_error) => io_error
+                .get_ref()
+                .and_then(|source| source.downcast_ref::<InternodeHttpError>())
+                .is_some_and(|err| err.kind().is_retryable()),
+            _ => false,
+        }
+    }
+
+    pub fn internode_http_error_kind(&self) -> Option<InternodeHttpErrorKind> {
+        match self {
+            DiskError::Io(io_error) => io_error
+                .get_ref()
+                .and_then(|source| source.downcast_ref::<InternodeHttpError>())
+                .map(InternodeHttpError::kind),
+            _ => None,
+        }
+    }
+
     // /// If all errors are of the same fatal disk error type, returns the corresponding error.
     // /// Otherwise, returns Ok.
     // pub fn check_disk_fatal_errs(errs: &[Option<Error>]) -> Result<()> {
@@ -241,7 +262,10 @@ impl From<rustfs_filemeta::Error> for DiskError {
 
 impl From<std::io::Error> for DiskError {
     fn from(e: std::io::Error) -> Self {
-        e.downcast::<DiskError>().unwrap_or_else(DiskError::Io)
+        match e.downcast::<DiskError>() {
+            Ok(disk_error) => disk_error,
+            Err(io_error) => DiskError::Io(io_error),
+        }
     }
 }
 

crates/ecstore/src/disk/error_reduce.rs

@@ -14,6 +14,19 @@
 
 use crate::disk::error::Error;
 
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct WriteQuorumFailureSummary {
+    pub required: usize,
+    pub achieved: usize,
+    pub failed: usize,
+    pub total: usize,
+    pub offline_disks: usize,
+    pub ignored_failures: usize,
+    pub retryable_failures: usize,
+    pub dominant_error: Option<Error>,
+    pub dominant_error_label: &'static str,
+}
+
 pub static OBJECT_OP_IGNORED_ERRS: &[Error] = &[
     Error::DiskNotFound,
     Error::FaultyDisk,
@@ -77,6 +90,58 @@ pub fn reduce_errs(errors: &[Option<Error>], ignored_errs: &[Error]) -> (usize,
     }
 }
 
+pub fn build_write_quorum_failure_summary(
+    errors: &[Option<Error>],
+    ignored_errs: &[Error],
+    quorum: usize,
+) -> WriteQuorumFailureSummary {
+    let total = errors.len();
+    let achieved = errors.iter().filter(|err| err.is_none()).count();
+    let failed = total.saturating_sub(achieved);
+    let offline_disks = count_errs(errors, &Error::DiskNotFound);
+    let ignored_failures = errors
+        .iter()
+        .filter_map(|err| err.as_ref())
+        .filter(|err| is_ignored_err(ignored_errs, err))
+        .count();
+    let retryable_failures = count_retryable_failures(errors);
+    let (_, dominant_error) = reduce_errs(errors, ignored_errs);
+    let dominant_error_label = dominant_error_label(errors, ignored_errs, dominant_error.as_ref());
+
+    WriteQuorumFailureSummary {
+        required: quorum,
+        achieved,
+        failed,
+        total,
+        offline_disks,
+        ignored_failures,
+        retryable_failures,
+        dominant_error,
+        dominant_error_label,
+    }
+}
+
+fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
+    let Some(dominant_error) = dominant_error else {
+        return "nil_dominated";
+    };
+
+    if dominant_error == &Error::DiskNotFound {
+        return "disk_not_found";
+    }
+    if dominant_error == &Error::ShortWrite {
+        return "short_write";
+    }
+
+    errors
+        .iter()
+        .filter_map(|err| err.as_ref())
+        .find(|err| !is_ignored_err(ignored_errs, err) && *err == dominant_error)
+        .and_then(Error::internode_http_error_kind)
+        .map(|kind| kind.metric_label())
+        .unwrap_or("other_error")
+}
+
 pub fn is_ignored_err(ignored_errs: &[Error], err: &Error) -> bool {
     ignored_errs.iter().any(|e| e == err)
 }
@@ -85,6 +150,14 @@ pub fn count_errs(errors: &[Option<Error>], err: &Error) -> usize {
     errors.iter().filter(|&e| e.as_ref() == Some(err)).count()
 }
 
+pub fn count_retryable_failures(errors: &[Option<Error>]) -> usize {
+    errors
+        .iter()
+        .filter_map(|err| err.as_ref())
+        .filter(|err| err.is_retryable_internode_write_failure())
+        .count()
+}
+
 pub fn is_all_buckets_not_found(errs: &[Option<Error>]) -> bool {
     for err in errs.iter() {
         if let Some(err) = err {
@@ -163,6 +236,51 @@ mod tests {
         assert!(!is_ignored_err(&ignored, &e2));
     }
 
+    #[test]
+    fn test_build_write_quorum_failure_summary() {
+        let retryable = Error::from(rustfs_rio::new_test_internode_http_io_error(
+            rustfs_rio::InternodeHttpErrorKind::ConnectionReset,
+        ));
+        let non_retryable = err_io("other");
+        let errors = vec![
+            None,
+            None,
+            None,
+            None,
+            None,
+            Some(retryable),
+            Some(non_retryable),
+            Some(Error::DiskNotFound),
+        ];
+
+        let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 6);
+        assert_eq!(summary.required, 6);
+        assert_eq!(summary.achieved, 5);
+        assert_eq!(summary.failed, 3);
+        assert_eq!(summary.total, 8);
+        assert_eq!(summary.offline_disks, 1);
+        assert_eq!(summary.ignored_failures, 1);
+        assert_eq!(summary.retryable_failures, 1);
+        assert_eq!(summary.dominant_error, None);
+        assert_eq!(summary.dominant_error_label, "nil_dominated");
+    }
+
+    #[test]
+    fn test_build_write_quorum_failure_summary_preserves_internode_label() {
+        let retryable_a = Error::from(rustfs_rio::new_test_internode_http_io_error(
+            rustfs_rio::InternodeHttpErrorKind::ConnectionReset,
+        ));
+        let retryable_b = Error::from(rustfs_rio::new_test_internode_http_io_error(
+            rustfs_rio::InternodeHttpErrorKind::ConnectionReset,
+        ));
+        let errors = vec![Some(retryable_a), Some(retryable_b)];
+
+        let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
+
+        assert_eq!(summary.retryable_failures, 2);
+        assert_eq!(summary.dominant_error_label, "connection_reset");
+    }
+
     #[test]
     fn test_reduce_errs_nil_tiebreak() {
         // Error::Nil and another error have the same count, should prefer Nil

crates/ecstore/src/disk/fs.rs

@@ -161,33 +161,39 @@ pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
     fs::create_dir_all(path.as_ref()).await
 }
 
+fn is_dir_error(e: &io::Error) -> bool {
+    e.raw_os_error() == Some(libc::EISDIR)
+        || e.kind() == io::ErrorKind::IsADirectory
+        // macOS: remove_file on a directory returns EPERM
+        || (cfg!(target_os = "macos") && e.raw_os_error() == Some(libc::EPERM))
+}
+
 #[tracing::instrument(level = "debug", skip_all)]
 pub async fn remove(path: impl AsRef<Path>) -> io::Result<()> {
-    let meta = fs::metadata(path.as_ref()).await?;
-    if meta.is_dir() {
-        fs::remove_dir(path.as_ref()).await
-    } else {
-        fs::remove_file(path.as_ref()).await
+    // Try remove_file first; fall back to remove_dir if it's a directory
+    match fs::remove_file(path.as_ref()).await {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => fs::remove_dir(path.as_ref()).await,
+        Err(e) => Err(e),
     }
 }
 
 pub async fn remove_all(path: impl AsRef<Path>) -> io::Result<()> {
-    let meta = fs::metadata(path.as_ref()).await?;
-    if meta.is_dir() {
-        fs::remove_dir_all(path.as_ref()).await
-    } else {
-        fs::remove_file(path.as_ref()).await
+    // Try remove_file first; fall back to remove_dir_all if it's a directory
+    match fs::remove_file(path.as_ref()).await {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => fs::remove_dir_all(path.as_ref()).await,
+        Err(e) => Err(e),
     }
 }
 
 #[tracing::instrument(level = "debug", skip_all)]
 pub fn remove_std(path: impl AsRef<Path>) -> io::Result<()> {
-    let path = path.as_ref();
-    let meta = std::fs::metadata(path)?;
-    if meta.is_dir() {
-        std::fs::remove_dir(path)
-    } else {
-        std::fs::remove_file(path)
+    // Try remove_file first; fall back to remove_dir if it's a directory
+    match std::fs::remove_file(path.as_ref()) {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => std::fs::remove_dir(path.as_ref()),
+        Err(e) => Err(e),
     }
 }
 

crates/ecstore/src/disk/local.rs

@@ -26,7 +26,7 @@ use crate::disk::{
     format::FormatV3,
     fs::{O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, lstat, lstat_std, remove, remove_all_std, remove_std, rename},
     os,
-    os::{check_path_length, is_empty_dir, is_root_disk, rename_all},
+    os::{check_path_length, is_empty_dir, is_root_disk, rename_all, rename_all_ignore_missing_source},
 };
 use crate::erasure_coding::bitrot_verify;
 use crate::global::{GLOBAL_IsErasureSD, GLOBAL_RootDiskThreshold};
@@ -57,14 +57,15 @@ use std::{
 use time::OffsetDateTime;
 use tokio::fs::{self, File};
 use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWrite, AsyncWriteExt, ErrorKind};
-use tokio::sync::RwLock;
-use tokio::time::interval;
+use tokio::sync::{Notify, RwLock};
+use tokio::time::{Instant, interval_at, timeout};
 use tracing::{debug, error, info, warn};
 use uuid::Uuid;
 
 const DELETED_OBJECTS_CLEANUP_INTERVAL: Duration = Duration::from_secs(60 * 5);
 const STALE_TMP_OBJECT_EXPIRY: Duration = Duration::from_secs(24 * 60 * 60);
 const RUSTFS_META_TMP_OLD_BUCKET: &str = ".rustfs.sys/tmp-old";
+const STARTUP_CLEANUP_WAIT_TIMEOUT: Duration = Duration::from_secs(2);
 
 #[derive(Debug, Clone)]
 pub struct FormatInfo {
@@ -331,6 +332,8 @@ pub struct LocalDisk {
     // pub format_data: Mutex<Vec<u8>>,
     // pub format_file_info: Mutex<Option<Metadata>>,
     // pub format_last_check: Mutex<Option<OffsetDateTime>>,
+    startup_cleanup_ready: Arc<AtomicU32>,
+    startup_cleanup_notify: Arc<Notify>,
     exit_signal: Option<tokio::sync::broadcast::Sender<()>>,
 }
 
@@ -353,24 +356,72 @@ impl Debug for LocalDisk {
     }
 }
 
+/// Resolve the local disk root path from an endpoint path.
+///
+/// Tries `canonicalize` first (fast path). On Windows, if canonicalization reports
+/// `NotFound` for paths that may still be valid mount roots, falls back to
+/// `absolutize` + metadata check to accept valid local directory roots that
+/// don't support full canonicalization.
+fn resolve_local_disk_root(ep_path: &str) -> Result<PathBuf> {
+    match rustfs_utils::canonicalize(ep_path) {
+        Ok(path) => Ok(path),
+        Err(err) => {
+            if err.kind() != ErrorKind::NotFound {
+                return Err(to_file_error(err).into());
+            }
+
+            #[cfg(windows)]
+            {
+                // On Windows, canonicalize can fail for ZFS volumes, junction points,
+                // subst drives, and other non-standard filesystem mounts. Try a fallback
+                // path resolution using absolutize + metadata check.
+                let absolute = match crate::disk::endpoint::windows_fallback_local_path(ep_path, &err, "local disk root") {
+                    Ok(path) => path,
+                    Err(_) => {
+                        return Err(DiskError::VolumeNotFound);
+                    }
+                };
+
+                match std::fs::metadata(&absolute) {
+                    Ok(metadata) => {
+                        if !metadata.is_dir() {
+                            return Err(DiskError::DiskNotDir);
+                        }
+                        return Ok(absolute);
+                    }
+                    Err(meta_err) => {
+                        if meta_err.kind() == ErrorKind::NotFound {
+                            return Err(DiskError::VolumeNotFound);
+                        }
+                        return Err(to_file_error(meta_err).into());
+                    }
+                }
+            }
+
+            #[cfg(not(windows))]
+            {
+                Err(DiskError::VolumeNotFound)
+            }
+        }
+    }
+}
+
 impl LocalDisk {
     pub async fn new(ep: &Endpoint, cleanup: bool) -> Result<Self> {
         debug!("Creating local disk");
-        // Use optimized path resolution instead of absolutize() for better performance
-        // Use dunce::canonicalize instead of std::fs::canonicalize to avoid UNC paths on Windows
-        let root = match rustfs_utils::canonicalize(ep.get_file_path()) {
-            Ok(path) => path,
-            Err(e) => {
-                if e.kind() == ErrorKind::NotFound {
-                    return Err(DiskError::VolumeNotFound);
-                }
-                return Err(to_file_error(e).into());
-            }
-        };
+        let root = resolve_local_disk_root(&ep.get_file_path())?;
 
         ensure_data_usage_layout(&root).await.map_err(DiskError::from)?;
 
-        if cleanup && let Err(err) = Self::cleanup_tmp_on_startup(&root).await {
+        let startup_cleanup_ready = Arc::new(AtomicU32::new(u32::from(!cleanup)));
+        let startup_cleanup_notify = Arc::new(Notify::new());
+
+        if cleanup
+            && let Err(err) =
+                Self::cleanup_tmp_on_startup(&root, startup_cleanup_ready.clone(), startup_cleanup_notify.clone()).await
+        {
+            startup_cleanup_ready.store(1, Ordering::Release);
+            startup_cleanup_notify.notify_waiters();
             warn!(root = ?root, error = ?err, "failed to cleanup temporary data during disk startup");
         }
 
@@ -466,6 +517,8 @@ impl LocalDisk {
             // format_last_check: Mutex::new(format_last_check),
             path_cache: Arc::new(ParkingLotRwLock::new(HashMap::with_capacity(2048))),
             current_dir: Arc::new(OnceLock::new()),
+            startup_cleanup_ready,
+            startup_cleanup_notify,
             exit_signal: None,
         };
         let (info, _root) = get_disk_info(root).await?;
@@ -497,7 +550,8 @@ impl LocalDisk {
     }
 
     async fn cleanup_deleted_objects_loop(root: PathBuf, mut exit_rx: tokio::sync::broadcast::Receiver<()>) {
-        let mut interval = interval(DELETED_OBJECTS_CLEANUP_INTERVAL);
+        let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);
         loop {
             tokio::select! {
                 _ = interval.tick() => {
@@ -525,12 +579,18 @@ impl LocalDisk {
         root.join(meta_path)
     }
 
-    async fn cleanup_tmp_on_startup(root: &Path) -> Result<()> {
+    async fn cleanup_tmp_on_startup(
+        root: &Path,
+        startup_cleanup_ready: Arc<AtomicU32>,
+        startup_cleanup_notify: Arc<Notify>,
+    ) -> Result<()> {
         let tmp_path = Self::meta_path(root, RUSTFS_META_TMP_BUCKET);
         let tmp_old_path = Self::meta_path(root, RUSTFS_META_TMP_OLD_BUCKET).join(Uuid::new_v4().to_string());
 
         rename_all(&tmp_path, &tmp_old_path, root).await?;
 
+        tokio::fs::create_dir_all(Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET)).await?;
+
         let tmp_old_root = Self::meta_path(root, RUSTFS_META_TMP_OLD_BUCKET);
         tokio::spawn(async move {
             if let Err(err) = tokio::fs::remove_dir_all(&tmp_old_root).await
@@ -538,12 +598,35 @@ impl LocalDisk {
             {
                 warn!(path = ?tmp_old_root, error = ?err, "failed to remove old temporary data");
             }
+            startup_cleanup_ready.store(1, Ordering::Release);
+            startup_cleanup_notify.notify_waiters();
         });
 
-        tokio::fs::create_dir_all(Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET)).await?;
         Ok(())
     }
 
+    async fn wait_for_startup_cleanup(&self) {
+        if self.startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+            return;
+        }
+
+        if wait_for_startup_cleanup_signal(
+            self.startup_cleanup_ready.as_ref(),
+            self.startup_cleanup_notify.as_ref(),
+            STARTUP_CLEANUP_WAIT_TIMEOUT,
+        )
+        .await
+        {
+            debug!(disk = %self.endpoint, "startup cleanup barrier released before walk_dir");
+        } else {
+            warn!(
+                disk = %self.endpoint,
+                timeout_ms = STARTUP_CLEANUP_WAIT_TIMEOUT.as_millis(),
+                "startup cleanup barrier timed out; continuing walk_dir"
+            );
+        }
+    }
+
     async fn cleanup_stale_tmp_objects(root: PathBuf) -> Result<()> {
         Self::cleanup_stale_tmp_objects_with_expiry(root, STALE_TMP_OBJECT_EXPIRY).await
     }
@@ -846,19 +929,20 @@ impl LocalDisk {
         // }
 
         let err = if recursive {
-            rename_all(delete_path, trash_path, self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?)
+            rename_all_ignore_missing_source(delete_path, trash_path, self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?)
                 .await
                 .err()
         } else {
-            rename(&delete_path, &trash_path)
-                .await
-                .map_err(|e| to_file_error(e).into())
-                .err()
+            match rename(&delete_path, &trash_path).await {
+                Ok(()) => None,
+                Err(err) if err.kind() == ErrorKind::NotFound => None,
+                Err(err) => Some(to_file_error(err).into()),
+            }
         };
 
         if immediate_purge || delete_path.to_string_lossy().ends_with(SLASH_SEPARATOR) {
             let trash_path2 = self.get_object_path(RUSTFS_META_TMP_DELETED_BUCKET, Uuid::new_v4().to_string().as_str())?;
-            let _ = rename_all(
+            let _ = rename_all_ignore_missing_source(
                 encode_dir_object(delete_path.to_string_lossy().as_ref()),
                 trash_path2,
                 self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?,
@@ -1367,12 +1451,9 @@ impl LocalDisk {
                 let name = entry.trim_end_matches(SLASH_SEPARATOR);
                 let name = decode_dir_object(format!("{}/{}", &current, &name).as_str());
 
-                // if opts.limit > 0
-                //     && let Ok(meta) = FileMeta::load(&metadata)
-                //     && !meta.all_hidden(true)
-                // {
-                *objs_returned += 1;
-                // }
+                if opts.limit <= 0 || metadata_counts_toward_limit(&metadata) {
+                    *objs_returned += 1;
+                }
 
                 out.write_obj(&MetaCacheEntry {
                     name: name.clone(),
@@ -1475,15 +1556,19 @@ impl LocalDisk {
 
                     out.write_obj(&meta).await?;
 
-                    // if let Ok(meta) = FileMeta::load(&meta.metadata)
-                    //     && !meta.all_hidden(true)
-                    // {
-                    *objs_returned += 1;
-                    // }
+                    let file_meta = if opts.limit > 0 || opts.recursive {
+                        FileMeta::load(&res).ok()
+                    } else {
+                        None
+                    };
+
+                    if opts.limit <= 0 || file_meta.as_ref().is_none_or(file_meta_counts_toward_limit) {
+                        *objs_returned += 1;
+                    }
 
                     if opts.recursive {
                         let mut dir_to_skip = HashSet::new();
-                        if let Ok(file_meta) = FileMeta::load(&res)
+                        if let Some(file_meta) = file_meta.as_ref()
                             && let Ok(data_dirs) = file_meta.get_data_dirs()
                         {
                             for data_dir in data_dirs.iter().flatten() {
@@ -1552,6 +1637,15 @@ fn is_root_path(path: impl AsRef<Path>) -> bool {
     path.as_ref().components().count() == 1 && path.as_ref().has_root()
 }
 
+fn metadata_counts_toward_limit(metadata: &[u8]) -> bool {
+    FileMeta::load(metadata).map_or(true, |meta| file_meta_counts_toward_limit(&meta))
+}
+
+fn file_meta_counts_toward_limit(meta: &FileMeta) -> bool {
+    meta.into_fileinfo("", "", "", false, true, false)
+        .map_or_else(|_| !meta.all_hidden(true), |latest| !latest.deleted && !latest.tier_free_version())
+}
+
 // Filter std::io::ErrorKind::NotFound
 async fn read_file_exists(path: impl AsRef<Path>) -> Result<(Bytes, Option<Metadata>)> {
     let p = path.as_ref();
@@ -1699,20 +1793,28 @@ impl DiskAPI for LocalDisk {
 
         let id = format_info.id;
 
-        // if format_info.last_check_valid() {
-        //     return Ok(id);
-        // }
-
         if format_info.file_info.is_some() && id.is_some() {
-            // check last check time
+            // Reuse the cached disk id only when the cached format check is fresh.
             if let Some(last_check) = format_info.last_check
-                && last_check.unix_timestamp() + 1 < OffsetDateTime::now_utc().unix_timestamp()
+                && last_check.unix_timestamp() + 1 >= OffsetDateTime::now_utc().unix_timestamp()
             {
                 return Ok(id);
             }
         }
 
-        let file_meta = self.check_format_json().await?;
+        let file_meta = match self.check_format_json().await {
+            Ok(meta) => meta,
+            Err(err) => {
+                if matches!(err, DiskError::UnformattedDisk | DiskError::DiskNotFound) {
+                    let mut format_info = self.format_info.write().await;
+                    format_info.id = None;
+                    format_info.data = Bytes::new();
+                    format_info.file_info = None;
+                    format_info.last_check = None;
+                }
+                return Err(err);
+            }
+        };
 
         if let Some(file_info) = &format_info.file_info
             && super::fs::same_file(&file_meta, file_info)
@@ -2375,6 +2477,8 @@ impl DiskAPI for LocalDisk {
     // FIXME: TODO: io.writer TODO cancel
     #[tracing::instrument(level = "debug", skip(self, wr))]
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
+        self.wait_for_startup_cleanup().await;
+
         let volume_dir = self.get_bucket_path(&opts.bucket)?;
 
         if !skip_access_checks(&opts.bucket)
@@ -2512,75 +2616,57 @@ impl DiskAPI for LocalDisk {
         check_path_length(src_file_path.to_string_lossy().to_string().as_str())?;
         check_path_length(dst_file_path.to_string_lossy().to_string().as_str())?;
 
-        // Read the previous xl.meta
-
-        let has_dst_buf = match super::fs::read_file(&dst_file_path).await {
-            Ok(res) => Some(res),
-            Err(e) => {
-                let e: DiskError = to_file_error(e).into();
-
-                if e != DiskError::FileNotFound {
-                    return Err(e);
-                }
-
-                None
-            }
-        };
-
-        let mut xlmeta = FileMeta::new();
-
-        if let Some(dst_buf) = has_dst_buf.as_ref()
-            && FileMeta::is_xl2_v1_format(dst_buf)
-            && let Ok(nmeta) = FileMeta::load(dst_buf)
-        {
-            xlmeta = nmeta
-        }
-
-        let mut skip_parent = dst_volume_dir.clone();
-        if has_dst_buf.as_ref().is_some()
-            && let Some(parent) = dst_file_path.parent()
-        {
-            skip_parent = parent.to_path_buf();
-        }
-
-        // TODO: Healing
-
-        let version_id = fi.version_id.unwrap_or_default();
-        let search_version_id = Some(version_id);
         let no_inline = fi.data.is_none() && fi.size > 0;
 
-        // Check if there's an existing version with the same version_id that has a data_dir to clean up
-        // Reuse one metadata scan to find the version data_dir and determine whether it is shared.
-        let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(search_version_id);
-        if let Some(old_data_dir) = has_old_data_dir.as_ref() {
-            let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
-        }
-
-        xlmeta.add_version(fi)?;
-
-        if xlmeta.versions.len() <= 10 {
-            // TODO: Sign
-        }
-
-        if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
-            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
-            let meta_skip_parent = if no_inline {
-                src_file_parent
-            } else {
-                src_volume_dir.as_path()
+        if no_inline {
+            // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
+            let has_dst_buf = match super::fs::read_file(&dst_file_path).await {
+                Ok(res) => Some(res),
+                Err(e) => {
+                    let e: DiskError = to_file_error(e).into();
+                    if e != DiskError::FileNotFound {
+                        return Err(e);
+                    }
+                    None
+                }
             };
+
+            let mut xlmeta = FileMeta::new();
+            if let Some(dst_buf) = has_dst_buf.as_ref()
+                && FileMeta::is_xl2_v1_format(dst_buf)
+                && let Ok(nmeta) = FileMeta::load(dst_buf)
+            {
+                xlmeta = nmeta
+            }
+
+            let mut skip_parent = dst_volume_dir.clone();
+            if has_dst_buf.as_ref().is_some()
+                && let Some(parent) = dst_file_path.parent()
+            {
+                skip_parent = parent.to_path_buf();
+            }
+
+            let version_id = fi.version_id.unwrap_or_default();
+            let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
+            if let Some(old_data_dir) = has_old_data_dir.as_ref() {
+                let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
+            }
+            xlmeta.add_version(fi)?;
             let new_dst_buf = xlmeta.marshal_msg()?;
 
+            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
             self.write_all_private(
                 src_volume,
-                format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(),
+                &format!("{}/{}", &src_path, STORAGE_FORMAT_FILE),
                 new_dst_buf.into(),
                 true,
-                meta_skip_parent,
+                src_file_parent,
             )
             .await?;
 
-            if no_inline && let Err(err) = rename_all(&src_data_path, &dst_data_path, &skip_parent).await {
+            if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref()
+                && let Err(err) = rename_all(src_data_path, dst_data_path, &skip_parent).await
+            {
                 let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
                 info!(
                     "rename all failed src_data_path: {:?}, dst_data_path: {:?}, err: {:?}",
@@ -2588,19 +2674,21 @@ impl DiskAPI for LocalDisk {
                 );
                 return Err(err);
             }
-        } else {
-            let new_dst_buf = xlmeta.marshal_msg()?;
-            self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
-                .await?;
-        }
 
-        if let Some(old_data_dir) = has_old_data_dir {
-            // preserve current xl.meta inside the oldDataDir.
-            if let Some(dst_buf) = has_dst_buf
+            if let Err(err) = rename_all(&src_file_path, &dst_file_path, &skip_parent).await {
+                if let Some((_, dst_data_path)) = has_data_dir_path.as_ref() {
+                    let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
+                }
+                info!("rename all failed err: {:?}", err);
+                return Err(err);
+            }
+
+            if let Some(old_data_dir) = has_old_data_dir
+                && let Some(dst_buf) = has_dst_buf
                 && let Err(err) = self
                     .write_all_private(
                         dst_volume,
-                        format!("{}/{}/{}", &dst_path, &old_data_dir.to_string(), STORAGE_FORMAT_FILE).as_str(),
+                        &format!("{}/{}/{}", &dst_path, &old_data_dir.to_string(), STORAGE_FORMAT_FILE),
                         dst_buf.into(),
                         true,
                         &skip_parent,
@@ -2610,30 +2698,106 @@ impl DiskAPI for LocalDisk {
                 info!("write_all_private failed err: {:?}", err);
                 return Err(err);
             }
-        }
 
-        if let Err(err) = rename_all(&src_file_path, &dst_file_path, &skip_parent).await {
-            if let Some((_, dst_data_path)) = has_data_dir_path.as_ref() {
-                let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
+            if let Some(src_file_path_parent) = src_file_path.parent() {
+                if src_volume != super::RUSTFS_META_MULTIPART_BUCKET {
+                    let _ = remove_std(src_file_path_parent);
+                } else {
+                    let _ = self
+                        .delete_file(&dst_volume_dir, &src_file_path_parent.to_path_buf(), true, false)
+                        .await;
+                }
             }
-            info!("rename all failed err: {:?}", err);
-            return Err(err);
-        }
 
-        if let Some(src_file_path_parent) = src_file_path.parent() {
-            if src_volume != super::RUSTFS_META_MULTIPART_BUCKET {
-                let _ = remove_std(src_file_path_parent);
+            Ok(RenameDataResp {
+                old_data_dir: has_old_data_dir,
+                sign: None,
+            })
+        } else {
+            // Inline: merge read + parse + write + rename into single spawn_blocking
+            let src = src_file_path.clone();
+            let dst = dst_file_path.clone();
+            let cleanup_path = if src_volume == super::RUSTFS_META_MULTIPART_BUCKET {
+                src_file_path.parent().map(|p| p.to_path_buf())
             } else {
-                let _ = self
-                    .delete_file(&dst_volume_dir, &src_file_path_parent.to_path_buf(), true, false)
-                    .await;
-            }
-        }
+                None
+            };
 
-        Ok(RenameDataResp {
-            old_data_dir: has_old_data_dir,
-            sign: None, // TODO:
-        })
+            let (old_data_dir, _dst_buf) = tokio::task::spawn_blocking(move || {
+                // Read existing xl.meta
+                let has_dst_buf = match std::fs::read(&dst) {
+                    Ok(buf) => Some(Bytes::from(buf)),
+                    Err(e) if e.kind() == std::io::ErrorKind::NotFound => None,
+                    Err(e) => return Err(to_file_error(e)),
+                };
+
+                let mut xlmeta = FileMeta::new();
+                if let Some(ref buf) = has_dst_buf
+                    && FileMeta::is_xl2_v1_format(buf)
+                    && let Ok(nmeta) = FileMeta::load(buf)
+                {
+                    xlmeta = nmeta
+                }
+
+                let version_id = fi.version_id.unwrap_or_default();
+                let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
+                if let Some(d) = old_data_dir.as_ref() {
+                    let _ = xlmeta.data.remove_two(version_id, *d);
+                }
+                xlmeta.add_version(fi)?;
+                let new_buf = xlmeta.marshal_msg()?;
+
+                // Write new xl.meta + rename
+                if let Some(parent) = src.parent() {
+                    std::fs::create_dir_all(parent)?;
+                }
+                let mut f = std::fs::OpenOptions::new()
+                    .create(true)
+                    .write(true)
+                    .truncate(true)
+                    .open(&src)?;
+                std::io::Write::write_all(&mut f, &new_buf)?;
+                match std::fs::rename(&src, &dst) {
+                    Ok(()) => Ok(()),
+                    Err(err) if err.kind() == std::io::ErrorKind::NotFound && !src.exists() => Ok(()),
+                    Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+                        if let Some(parent) = dst.parent() {
+                            std::fs::create_dir_all(parent)?;
+                        }
+                        std::fs::rename(&src, &dst).map_err(to_file_error)?;
+                        Ok(())
+                    }
+                    Err(err) => Err(to_file_error(err)),
+                }?;
+
+                if let Some(old_dir) = old_data_dir.as_ref()
+                    && let Some(ref buf) = has_dst_buf
+                    && let Some(dst_parent) = dst.parent()
+                {
+                    let old_path = dst_parent.join(old_dir.to_string()).join(STORAGE_FORMAT_FILE);
+                    if let Some(old_parent) = old_path.parent() {
+                        std::fs::create_dir_all(old_parent)?;
+                    }
+                    std::fs::write(&old_path, buf).map_err(to_file_error)?;
+                }
+
+                Ok::<(Option<uuid::Uuid>, Option<Bytes>), std::io::Error>((old_data_dir, has_dst_buf))
+            })
+            .await
+            .map_err(DiskError::from)??;
+
+            // Cleanup
+            if let Some(ref cleanup) = cleanup_path {
+                let _ = self.delete_file(&dst_volume_dir, cleanup, true, false).await;
+            } else if let Some(parent) = src_file_path.parent() {
+                let _ = remove_std(parent);
+            }
+
+            Ok(RenameDataResp {
+                old_data_dir,
+                sign: None,
+            })
+        }
     }
 
     #[tracing::instrument(skip(self))]
@@ -3105,6 +3269,31 @@ impl DiskAPI for LocalDisk {
     }
 }
 
+async fn wait_for_startup_cleanup_signal(
+    startup_cleanup_ready: &AtomicU32,
+    startup_cleanup_notify: &Notify,
+    wait_timeout: Duration,
+) -> bool {
+    if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+        return true;
+    }
+
+    timeout(wait_timeout, async {
+        loop {
+            if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+                return;
+            }
+            let notified = startup_cleanup_notify.notified();
+            if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+                return;
+            }
+            notified.await;
+        }
+    })
+    .await
+    .is_ok()
+}
+
 #[tracing::instrument]
 async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInfo, bool)> {
     let drive_path = drive_path.to_string_lossy().to_string();
@@ -3147,6 +3336,52 @@ mod test {
         }
     }
 
+    #[tokio::test]
+    async fn test_get_disk_id_invalidates_cache_after_format_removal() {
+        use crate::disk::FORMAT_CONFIG_FILE;
+        use crate::disk::format::FormatV3;
+        use tempfile::tempdir;
+
+        let dir = tempdir().unwrap();
+        let mut endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+        let meta_dir = dir.path().join(RUSTFS_META_BUCKET);
+        fs::create_dir_all(&meta_dir).await.expect("meta dir should be creatable");
+        let mut format = FormatV3::new(1, 1);
+        format.erasure.this = format.erasure.sets[0][0];
+        let format_json = format.to_json().expect("format should serialize");
+        fs::write(meta_dir.join(FORMAT_CONFIG_FILE), format_json)
+            .await
+            .expect("format.json should be writable");
+
+        let disk = LocalDisk::new(&endpoint, false)
+            .await
+            .expect("local disk should open after seeding format");
+
+        let initial_id = disk.get_disk_id().await.expect("disk id lookup should succeed");
+        assert!(initial_id.is_some(), "new disk should expose a disk id");
+
+        fs::remove_file(&disk.format_path)
+            .await
+            .expect("format.json should be removable");
+
+        tokio::time::sleep(Duration::from_secs(2)).await;
+
+        let err = disk
+            .get_disk_id()
+            .await
+            .expect_err("removed format.json should invalidate the cached disk id");
+        assert!(matches!(err, DiskError::UnformattedDisk));
+
+        let format_info = disk.format_info.read().await.clone();
+        assert!(format_info.id.is_none(), "cached disk id should be cleared");
+        assert!(format_info.data.is_empty(), "cached format bytes should be cleared");
+        assert!(format_info.file_info.is_none(), "cached file metadata should be cleared");
+        assert!(format_info.last_check.is_none(), "cached format timestamp should be cleared");
+    }
+
     #[tokio::test]
     async fn cleanup_tmp_on_startup_moves_existing_tmp_and_recreates_trash() {
         use tempfile::tempdir;
@@ -3157,7 +3392,9 @@ mod test {
         fs::create_dir_all(leftover.parent().unwrap()).await.unwrap();
         fs::write(&leftover, b"temporary").await.unwrap();
 
-        LocalDisk::cleanup_tmp_on_startup(dir.path()).await.unwrap();
+        LocalDisk::cleanup_tmp_on_startup(dir.path(), Arc::new(AtomicU32::new(0)), Arc::new(Notify::new()))
+            .await
+            .unwrap();
 
         assert!(!tmp.join("leftover").exists());
         assert!(LocalDisk::meta_path(dir.path(), RUSTFS_META_TMP_DELETED_BUCKET).exists());
@@ -3213,6 +3450,54 @@ mod test {
         assert!(entries.next_entry().await.unwrap().is_none());
     }
 
+    #[tokio::test(start_paused = true)]
+    async fn cleanup_loop_interval_does_not_tick_immediately() {
+        let start_at = tokio::time::Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);
+
+        assert!(tokio::time::timeout(Duration::from_secs(1), interval.tick()).await.is_err());
+
+        tokio::time::advance(DELETED_OBJECTS_CLEANUP_INTERVAL).await;
+        interval.tick().await;
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn startup_cleanup_barrier_waits_for_notification() {
+        let ready = Arc::new(AtomicU32::new(0));
+        let notify = Arc::new(Notify::new());
+
+        let wait = tokio::spawn({
+            let ready = ready.clone();
+            let notify = notify.clone();
+            async move { wait_for_startup_cleanup_signal(ready.as_ref(), notify.as_ref(), Duration::from_secs(2)).await }
+        });
+
+        tokio::task::yield_now().await;
+        assert!(!wait.is_finished());
+
+        ready.store(1, Ordering::Release);
+        notify.notify_waiters();
+
+        assert!(wait.await.unwrap());
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn startup_cleanup_barrier_times_out() {
+        let ready = Arc::new(AtomicU32::new(0));
+        let notify = Arc::new(Notify::new());
+
+        let wait = tokio::spawn({
+            let ready = ready.clone();
+            let notify = notify.clone();
+            async move { wait_for_startup_cleanup_signal(ready.as_ref(), notify.as_ref(), Duration::from_secs(2)).await }
+        });
+
+        tokio::task::yield_now().await;
+        tokio::time::advance(Duration::from_secs(2)).await;
+
+        assert!(!wait.await.unwrap());
+    }
+
     #[tokio::test]
     async fn test_scan_dir_includes_nested_object_dirs() {
         use rustfs_filemeta::MetacacheReader;
@@ -3420,6 +3705,121 @@ mod test {
         assert_eq!(double_count as usize, double_names.len());
     }
 
+    #[tokio::test]
+    async fn test_scan_dir_hidden_delete_markers_do_not_exhaust_limit() {
+        use rustfs_filemeta::MetacacheReader;
+        use tempfile::tempdir;
+
+        fn delete_marker_metadata(version_id: &str) -> Vec<u8> {
+            let mut fm = FileMeta::default();
+            fm.add_version(FileInfo {
+                deleted: true,
+                version_id: Some(Uuid::parse_str(version_id).expect("test version id should parse")),
+                mod_time: Some(OffsetDateTime::now_utc()),
+                ..Default::default()
+            })
+            .expect("delete marker metadata should be valid");
+            fm.marshal_msg().expect("delete marker metadata should encode")
+        }
+
+        fn delete_marker_with_old_object_metadata(delete_version_id: &str, object_version_id: &str) -> Vec<u8> {
+            let mut fm = FileMeta::default();
+            fm.add_version({
+                let mut fi = FileInfo::new("hidden", 1, 1);
+                fi.version_id = Some(Uuid::parse_str(object_version_id).expect("test version id should parse"));
+                fi.mod_time = Some(OffsetDateTime::now_utc() - time::Duration::seconds(1));
+                fi
+            })
+            .expect("object metadata should be valid");
+            fm.add_version(FileInfo {
+                deleted: true,
+                version_id: Some(Uuid::parse_str(delete_version_id).expect("test version id should parse")),
+                mod_time: Some(OffsetDateTime::now_utc()),
+                ..Default::default()
+            })
+            .expect("delete marker metadata should be valid");
+            fm.marshal_msg().expect("delete marker metadata should encode")
+        }
+
+        fn object_metadata(version_id: &str) -> Vec<u8> {
+            let mut fm = FileMeta::default();
+            let mut fi = FileInfo::new("visible", 1, 1);
+            fi.version_id = Some(Uuid::parse_str(version_id).expect("test version id should parse"));
+            fi.mod_time = Some(OffsetDateTime::now_utc());
+            fm.add_version(fi).expect("object metadata should be valid");
+            fm.marshal_msg().expect("object metadata should encode")
+        }
+
+        let dir = tempdir().unwrap();
+        let bucket = "test-bucket";
+        let bucket_dir = dir.path().join(bucket);
+
+        for (name, version_id) in [
+            ("shard/aaa-trash-0000", "11111111-1111-1111-1111-111111111111"),
+            ("shard/aaa-trash-0001", "22222222-2222-2222-2222-222222222222"),
+            ("shard/aaa-trash-0002", "33333333-3333-3333-3333-333333333333"),
+        ] {
+            let object_dir = bucket_dir.join(name);
+            fs::create_dir_all(&object_dir).await.unwrap();
+            fs::write(object_dir.join(STORAGE_FORMAT_FILE), delete_marker_metadata(version_id))
+                .await
+                .unwrap();
+        }
+
+        let hidden_versioned_dir = bucket_dir.join("shard/aaa-trash-0003");
+        fs::create_dir_all(&hidden_versioned_dir).await.unwrap();
+        fs::write(
+            hidden_versioned_dir.join(STORAGE_FORMAT_FILE),
+            delete_marker_with_old_object_metadata(
+                "44444444-4444-4444-4444-444444444444",
+                "55555555-5555-5555-5555-555555555555",
+            ),
+        )
+        .await
+        .unwrap();
+
+        let visible_dir = bucket_dir.join("shard/bbb-visible-0000");
+        fs::create_dir_all(&visible_dir).await.unwrap();
+        fs::write(
+            visible_dir.join(STORAGE_FORMAT_FILE),
+            object_metadata("66666666-6666-6666-6666-666666666666"),
+        )
+        .await
+        .unwrap();
+
+        let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
+        let disk = LocalDisk::new(&endpoint, false).await.unwrap();
+
+        let (reader, mut writer) = tokio::io::duplex(4096);
+        let mut out = MetacacheWriter::new(&mut writer);
+        let opts = WalkDirOptions {
+            bucket: bucket.to_string(),
+            base_dir: "".to_string(),
+            recursive: true,
+            limit: 1,
+            ..Default::default()
+        };
+        let mut objs_returned = 0;
+
+        disk.scan_dir("".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false, None)
+            .await
+            .unwrap();
+        out.close().await.unwrap();
+        drop(out);
+        drop(writer);
+
+        let mut reader = MetacacheReader::new(reader);
+        let has_visible_object = reader
+            .read_all()
+            .await
+            .unwrap()
+            .into_iter()
+            .any(|entry| !entry.metadata.is_empty() && entry.name == "shard/bbb-visible-0000");
+
+        assert!(has_visible_object);
+        assert_eq!(objs_returned, 1);
+    }
+
     #[tokio::test]
     async fn test_walk_dir_ignore_multipart_dirs() {
         use rustfs_filemeta::MetacacheReader;

crates/ecstore/src/disk/mod.rs

@@ -698,6 +698,10 @@ pub struct WalkDirOptions {
     // DiskID contains the disk ID of the disk.
     // Leave empty to not check disk ID.
     pub disk_id: String,
+
+    // Skip the wrapper-level total timeout for long streaming walks.
+    #[serde(default)]
+    pub skip_total_timeout: bool,
 }
 
 #[derive(Clone, Debug, Default)]
@@ -894,6 +898,7 @@ mod tests {
             forward_to: Some("object/path".to_string()),
             limit: 100,
             disk_id: "disk-123".to_string(),
+            skip_total_timeout: false,
         };
 
         assert_eq!(opts.bucket, "test-bucket");
@@ -904,6 +909,7 @@ mod tests {
         assert_eq!(opts.forward_to, Some("object/path".to_string()));
         assert_eq!(opts.limit, 100);
         assert_eq!(opts.disk_id, "disk-123");
+        assert!(!opts.skip_total_timeout);
     }
 
     /// Test DeleteOptions structure

crates/ecstore/src/disk/os.rs

@@ -140,36 +140,55 @@ pub async fn rename_all(
     Ok(())
 }
 
+#[tracing::instrument(level = "debug", skip_all)]
+pub async fn rename_all_ignore_missing_source(
+    src_file_path: impl AsRef<Path>,
+    dst_file_path: impl AsRef<Path>,
+    base_dir: impl AsRef<Path>,
+) -> Result<()> {
+    match reliable_rename_inner(src_file_path, dst_file_path.as_ref(), base_dir, false).await {
+        Ok(()) => Ok(()),
+        Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(()),
+        Err(err) => Err(to_file_error(err).into()),
+    }
+}
+
 async fn reliable_rename(
     src_file_path: impl AsRef<Path>,
     dst_file_path: impl AsRef<Path>,
     base_dir: impl AsRef<Path>,
+) -> io::Result<()> {
+    reliable_rename_inner(src_file_path, dst_file_path, base_dir, true).await
+}
+
+async fn reliable_rename_inner(
+    src_file_path: impl AsRef<Path>,
+    dst_file_path: impl AsRef<Path>,
+    base_dir: impl AsRef<Path>,
+    warn_on_failure: bool,
 ) -> io::Result<()> {
     if let Some(parent) = dst_file_path.as_ref().parent()
         && !file_exists(parent)
     {
-        // info!("reliable_rename reliable_mkdir_all parent: {:?}", parent);
         reliable_mkdir_all(parent, base_dir.as_ref()).await?;
     }
 
     let mut i = 0;
     loop {
         if let Err(e) = super::fs::rename_std(src_file_path.as_ref(), dst_file_path.as_ref()) {
-            if e.kind() == io::ErrorKind::NotFound {
-                break;
-            }
-
             if i == 0 {
                 i += 1;
                 continue;
             }
-            warn!(
-                "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
-                src_file_path.as_ref(),
-                dst_file_path.as_ref(),
-                base_dir.as_ref(),
-                e
-            );
+            if warn_on_failure {
+                warn!(
+                    "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
+                    src_file_path.as_ref(),
+                    dst_file_path.as_ref(),
+                    base_dir.as_ref(),
+                    e
+                );
+            }
             return Err(e);
         }
 
@@ -249,3 +268,36 @@ pub async fn os_mkdir_all(dir_path: impl AsRef<Path>, base_dir: impl AsRef<Path>
 pub fn file_exists(path: impl AsRef<Path>) -> bool {
     std::fs::metadata(path.as_ref()).map(|_| true).unwrap_or(false)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tempfile::tempdir;
+
+    #[tokio::test]
+    async fn rename_all_missing_source_returns_file_not_found() {
+        let temp_dir = tempdir().expect("create temp dir");
+        let src = temp_dir.path().join("missing");
+        let dst = temp_dir.path().join("dst");
+
+        let err = rename_all(&src, &dst, temp_dir.path())
+            .await
+            .expect_err("missing source must fail");
+
+        assert!(matches!(err, DiskError::FileNotFound));
+        assert!(!dst.exists());
+    }
+
+    #[tokio::test]
+    async fn rename_all_ignore_missing_source_returns_ok() {
+        let temp_dir = tempdir().expect("create temp dir");
+        let src = temp_dir.path().join("missing");
+        let dst = temp_dir.path().join("dst");
+
+        rename_all_ignore_missing_source(&src, &dst, temp_dir.path())
+            .await
+            .expect("missing cleanup source must be ignored");
+
+        assert!(!dst.exists());
+    }
+}

crates/ecstore/src/endpoints.rs

@@ -58,6 +58,40 @@ pub struct Node {
 #[derive(Debug, Default, Clone)]
 pub struct Endpoints(Vec<Endpoint>);
 
+#[derive(Debug, Clone)]
+struct LocalDiskValidationDiagnostic {
+    original_path: String,
+    canonical_path: Option<String>,
+    device_numbers: Option<String>,
+    device_ids: Option<Vec<String>>,
+}
+
+impl LocalDiskValidationDiagnostic {
+    fn new(original_path: &str) -> Self {
+        Self {
+            original_path: original_path.to_string(),
+            canonical_path: None,
+            device_numbers: None,
+            device_ids: None,
+        }
+    }
+
+    fn summary(&self) -> String {
+        let canonical_path = self.canonical_path.as_deref().unwrap_or("(unresolved)");
+        let device_numbers = self.device_numbers.as_deref().unwrap_or("(unavailable)");
+        let device_ids = self
+            .device_ids
+            .as_ref()
+            .map(|ids| ids.join(","))
+            .unwrap_or_else(|| "(unavailable)".to_string());
+
+        format!(
+            "path='{}', canonical='{}', st_dev='{}', device_ids=[{}]",
+            self.original_path, canonical_path, device_numbers, device_ids
+        )
+    }
+}
+
 impl AsRef<Vec<Endpoint>> for Endpoints {
     fn as_ref(&self) -> &Vec<Endpoint> {
         &self.0
@@ -683,14 +717,55 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
     }
 
     let mut device_paths = BTreeMap::<String, BTreeSet<String>>::new();
+    let mut diagnostics = Vec::with_capacity(local_paths.len());
+    #[cfg(not(windows))]
     let mut missing_paths = Vec::new();
 
     for path in &local_paths {
+        let mut diagnostic = LocalDiskValidationDiagnostic::new(path);
         let canonical = match rustfs_utils::canonicalize(path) {
             Ok(path) => path,
             Err(err) if err.kind() == ErrorKind::NotFound => {
-                missing_paths.push(path.clone());
-                continue;
+                // On Windows, canonicalize can fail for ZFS volumes, junction points,
+                // subst drives, and other non-standard mounts. Try absolutize as fallback.
+                #[cfg(windows)]
+                {
+                    match crate::disk::endpoint::windows_fallback_local_path(path, &err, "disk independence validation") {
+                        Ok(absolute) => {
+                            let abs_path = absolute.to_string_lossy().into_owned();
+                            diagnostic.canonical_path = Some(abs_path.clone());
+                            if let Ok(serial) = rustfs_utils::os::get_volume_serial_number(&abs_path) {
+                                diagnostic.device_numbers = Some(format!("serial:{serial:#010x}"));
+                            }
+                            match rustfs_utils::os::get_physical_device_ids(&abs_path) {
+                                Ok(ids) => {
+                                    diagnostic.device_ids = Some(ids.clone());
+                                    for device_id in ids {
+                                        device_paths.entry(device_id).or_default().insert(abs_path.clone());
+                                    }
+                                }
+                                Err(device_err) => {
+                                    return Err(Error::other(format!(
+                                        "failed to inspect physical disk for local endpoint '{abs_path}' after fallback path resolution: {device_err}"
+                                    )));
+                                }
+                            }
+                            diagnostics.push(diagnostic);
+                            continue;
+                        }
+                        Err(fallback_err) => {
+                            return Err(Error::other(format!(
+                                "failed to resolve local endpoint path '{path}' for disk validation: {err}; fallback resolution failed: {fallback_err}"
+                            )));
+                        }
+                    }
+                }
+                #[cfg(not(windows))]
+                {
+                    missing_paths.push(path.clone());
+                    diagnostics.push(diagnostic);
+                    continue;
+                }
             }
             Err(err) => {
                 return Err(Error::other(format!(
@@ -699,15 +774,27 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
             }
         };
         let canonical_path = canonical.to_string_lossy().into_owned();
+        diagnostic.canonical_path = Some(canonical_path.clone());
+        #[cfg(not(windows))]
+        if let Ok(stat) = rustix::fs::stat(canonical.as_path()) {
+            diagnostic.device_numbers = Some(format!("{}:{}", rustix::fs::major(stat.st_dev), rustix::fs::minor(stat.st_dev)));
+        }
+        #[cfg(windows)]
+        if let Ok(serial) = rustfs_utils::os::get_volume_serial_number(&canonical_path) {
+            diagnostic.device_numbers = Some(format!("serial:{serial:#010x}"));
+        }
         let device_ids = rustfs_utils::os::get_physical_device_ids(&canonical_path).map_err(|err| {
             Error::other(format!("failed to inspect physical disk for local endpoint '{canonical_path}': {err}"))
         })?;
+        diagnostic.device_ids = Some(device_ids.clone());
+        diagnostics.push(diagnostic);
 
         for device_id in device_ids {
             device_paths.entry(device_id).or_default().insert(canonical_path.clone());
         }
     }
 
+    #[cfg(not(windows))]
     if !missing_paths.is_empty() {
         warn!(
             missing_paths = ?missing_paths,
@@ -715,6 +802,15 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
         );
     }
 
+    warn!(
+        diagnostics = %diagnostics
+            .iter()
+            .map(LocalDiskValidationDiagnostic::summary)
+            .collect::<Vec<_>>()
+            .join("; "),
+        "Collected local endpoint disk-topology diagnostics before physical disk independence validation",
+    );
+
     let shared_devices = device_paths
         .into_iter()
         .filter_map(|(device_id, paths)| {
@@ -735,9 +831,15 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
         .map(|(device_id, paths)| format!("{device_id} => {}", paths.join(", ")))
         .collect::<Vec<_>>()
         .join("; ");
+    let diagnostics_summary = diagnostics
+        .iter()
+        .map(LocalDiskValidationDiagnostic::summary)
+        .collect::<Vec<_>>()
+        .join("; ");
 
     Err(Error::other(format!(
         "local erasure endpoints must use distinct physical disks; detected shared devices [{details}]. \
+validation diagnostics: [{diagnostics_summary}]. \
 Set {ENV_UNSAFE_BYPASS_DISK_CHECK}=true only for local testing or CI to bypass this safety check"
     )))
 }
@@ -1545,6 +1647,9 @@ mod test {
             let err_text = err.to_string();
             assert!(err_text.contains("distinct physical disks"), "unexpected error: {err_text}");
             assert!(err_text.contains(ENV_UNSAFE_BYPASS_DISK_CHECK), "unexpected error: {err_text}");
+            assert!(err_text.contains("validation diagnostics:"), "unexpected error: {err_text}");
+            assert!(err_text.contains("st_dev='"), "unexpected error: {err_text}");
+            assert!(err_text.contains("device_ids=["), "unexpected error: {err_text}");
         })
         .await;
     }

crates/ecstore/src/erasure_coding/decode.rs

@@ -433,6 +433,63 @@ mod tests {
         }
     }
 
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn test_erasure_decode_preserves_compressed_stream_near_block_boundary() {
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const BLOCK_SIZE: usize = 1024 * 1024;
+
+        use crate::rio::CompressReader;
+        use rustfs_utils::CompressionAlgorithm;
+        use tokio::io::AsyncReadExt;
+
+        let plaintext_size = 8 * BLOCK_SIZE + 123;
+        let plaintext = (0..plaintext_size)
+            .scan(0x9e37_79b9_7f4a_7c15u64, |state, _| {
+                *state ^= *state << 7;
+                *state ^= *state >> 9;
+                *state = state.wrapping_mul(0xbf58_476d_1ce4_e5b9);
+                Some((*state >> 32) as u8)
+            })
+            .collect::<Vec<_>>();
+
+        let mut compressor = CompressReader::new(Cursor::new(plaintext), CompressionAlgorithm::default());
+        let mut compressed = Vec::new();
+        compressor.read_to_end(&mut compressed).await.unwrap();
+
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        let total_shards = DATA_SHARDS + PARITY_SHARDS;
+        let shard_size = erasure.shard_size();
+        let hash_algo = HashAlgorithm::HighwayHash256;
+
+        let mut shard_writers: Vec<BitrotWriter<Cursor<Vec<u8>>>> = (0..total_shards)
+            .map(|_| BitrotWriter::new(Cursor::new(Vec::new()), shard_size, hash_algo.clone()))
+            .collect();
+
+        for block in compressed.chunks(BLOCK_SIZE) {
+            let shards = erasure.encode_data(block).unwrap();
+            for (i, shard) in shards.iter().enumerate() {
+                shard_writers[i].write(shard).await.unwrap();
+            }
+        }
+
+        let shard_bufs: Vec<Vec<u8>> = shard_writers.into_iter().map(|w| w.into_inner().into_inner()).collect();
+        let readers = shard_bufs
+            .iter()
+            .map(|buf| Some(BitrotReader::new(Cursor::new(buf.clone()), shard_size, hash_algo.clone(), false)))
+            .collect();
+
+        let mut decoded = Vec::new();
+        let (written, err) = erasure
+            .decode(&mut decoded, readers, 0, compressed.len(), compressed.len())
+            .await;
+
+        assert!(err.is_none(), "unexpected decode error: {err:?}");
+        assert_eq!(written, compressed.len());
+        assert_eq!(decoded, compressed);
+    }
+
     #[tokio::test]
     async fn test_parallel_reader_normal() {
         const BLOCK_SIZE: usize = 64;

crates/ecstore/src/erasure_coding/encode.rs

@@ -13,8 +13,9 @@
 // limitations under the License.
 
 use crate::disk::error::Error;
-use crate::disk::error_reduce::count_errs;
-use crate::disk::error_reduce::{OBJECT_OP_IGNORED_ERRS, reduce_write_quorum_errs};
+use crate::disk::error_reduce::{
+    OBJECT_OP_IGNORED_ERRS, WriteQuorumFailureSummary, build_write_quorum_failure_summary, reduce_write_quorum_errs,
+};
 use crate::erasure_coding::BitrotWriterWrapper;
 use crate::erasure_coding::Erasure;
 use bytes::Bytes;
@@ -27,8 +28,8 @@ use tokio::sync::mpsc;
 use tracing::error;
 
 const ENV_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: &str = "RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES";
-const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: usize = 8 * 1024 * 1024;
-const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS: usize = 8;
+const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES: usize = 32 * 1024 * 1024;
+const DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS: usize = 32;
 
 fn encode_channel_capacity(expanded_block_bytes: usize, max_inflight_bytes: usize) -> usize {
     if expanded_block_bytes == 0 {
@@ -50,6 +51,28 @@ async fn drain_queued_inflight_bytes(rx: &mut mpsc::Receiver<Vec<Bytes>>) {
     }
 }
 
+fn dominant_error_summary_label(summary: &WriteQuorumFailureSummary) -> &'static str {
+    summary.dominant_error_label
+}
+
+fn format_write_quorum_failure(summary: &WriteQuorumFailureSummary) -> String {
+    format!(
+        "erasure write quorum (required={}, achieved={}, failed={}, total={}, offline-disks={}/{}, retryable-failures={}, dominant-error={})",
+        summary.required,
+        summary.achieved,
+        summary.failed,
+        summary.total,
+        summary.offline_disks,
+        summary.total,
+        summary.retryable_failures,
+        dominant_error_summary_label(summary)
+    )
+}
+
+fn quorum_dominant_error_metric_label(summary: &WriteQuorumFailureSummary) -> &'static str {
+    dominant_error_summary_label(summary)
+}
+
 pub(crate) struct MultiWriter<'a> {
     writers: &'a mut [Option<BitrotWriterWrapper>],
     write_quorum: usize,
@@ -109,25 +132,18 @@ impl<'a> MultiWriter<'a> {
         }
 
         if let Some(write_err) = reduce_write_quorum_errs(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum) {
-            error!(
-                "reduce_write_quorum_errs: {:?}, offline-disks={}/{}, errs={:?}",
-                write_err,
-                count_errs(&self.errs, &Error::DiskNotFound),
-                self.writers.len(),
-                self.errs
-            );
-            return Err(std::io::Error::other(format!(
-                "Failed to write data: {} (offline-disks={}/{})",
-                write_err,
-                count_errs(&self.errs, &Error::DiskNotFound),
-                self.writers.len()
-            )));
+            let summary = build_write_quorum_failure_summary(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum);
+            let summary_text = format_write_quorum_failure(&summary);
+            rustfs_io_metrics::internode_metrics::global_internode_metrics()
+                .record_erasure_write_quorum_failure("write", quorum_dominant_error_metric_label(&summary));
+            error!("reduce_write_quorum_errs: {:?}, {}, errs={:?}", write_err, summary_text, self.errs);
+            return Err(std::io::Error::other(format!("Failed to write data: {summary_text}")));
         }
 
+        let summary = build_write_quorum_failure_summary(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum);
         Err(std::io::Error::other(format!(
-            "Failed to write data:  (offline-disks={}/{}): {}",
-            count_errs(&self.errs, &Error::DiskNotFound),
-            self.writers.len(),
+            "Failed to write data: {}: {}",
+            format_write_quorum_failure(&summary),
             self.errs
                 .iter()
                 .map(|e| e.as_ref().map_or_else(|| "<nil>".to_string(), |e| e.to_string()))
@@ -171,25 +187,21 @@ impl<'a> MultiWriter<'a> {
         }
 
         if let Some(write_err) = reduce_write_quorum_errs(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum) {
+            let summary = build_write_quorum_failure_summary(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum);
+            let summary_text = format_write_quorum_failure(&summary);
+            rustfs_io_metrics::internode_metrics::global_internode_metrics()
+                .record_erasure_write_quorum_failure("shutdown", quorum_dominant_error_metric_label(&summary));
             error!(
-                "reduce_write_quorum_errs during shutdown: {:?}, offline-disks={}/{}, errs={:?}",
-                write_err,
-                count_errs(&self.errs, &Error::DiskNotFound),
-                self.writers.len(),
-                self.errs
+                "reduce_write_quorum_errs during shutdown: {:?}, {}, errs={:?}",
+                write_err, summary_text, self.errs
             );
-            return Err(std::io::Error::other(format!(
-                "Failed to shutdown writers: {} (offline-disks={}/{})",
-                write_err,
-                count_errs(&self.errs, &Error::DiskNotFound),
-                self.writers.len()
-            )));
+            return Err(std::io::Error::other(format!("Failed to shutdown writers: {summary_text}")));
         }
 
+        let summary = build_write_quorum_failure_summary(&self.errs, OBJECT_OP_IGNORED_ERRS, self.write_quorum);
         Err(std::io::Error::other(format!(
-            "Failed to shutdown writers: (offline-disks={}/{}): {}",
-            count_errs(&self.errs, &Error::DiskNotFound),
-            self.writers.len(),
+            "Failed to shutdown writers: {}: {}",
+            format_write_quorum_failure(&summary),
             self.errs
                 .iter()
                 .map(|e| e.as_ref().map_or_else(|| "<nil>".to_string(), |e| e.to_string()))
@@ -200,6 +212,49 @@ impl<'a> MultiWriter<'a> {
 }
 
 impl Erasure {
+    async fn encode_small_direct<R>(
+        self: Arc<Self>,
+        mut reader: R,
+        writers: &mut [Option<BitrotWriterWrapper>],
+        quorum: usize,
+        require_single_block: bool,
+    ) -> std::io::Result<(R, usize)>
+    where
+        R: AsyncRead + Send + Sync + Unpin,
+    {
+        use tokio::io::AsyncReadExt;
+
+        let mut buf = Vec::with_capacity(self.block_size);
+        let total = if require_single_block {
+            let read_limit = self
+                .block_size
+                .checked_add(1)
+                .ok_or_else(|| std::io::Error::new(std::io::ErrorKind::InvalidInput, "erasure block_size is too large"))?;
+            let read_limit = u64::try_from(read_limit)
+                .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidInput, "erasure block_size exceeds u64"))?;
+            (&mut reader).take(read_limit).read_to_end(&mut buf).await?
+        } else {
+            reader.read_to_end(&mut buf).await?
+        };
+
+        if total == 0 {
+            return Ok((reader, 0));
+        }
+
+        if require_single_block && total > self.block_size {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "single-block non-inline fast path expects total <= block_size",
+            ));
+        }
+
+        let shards = self.encode_data_owned(buf)?;
+        let mut mw = MultiWriter::new(writers, quorum);
+        mw.write(shards).await?;
+        mw.shutdown().await?;
+        Ok((reader, total))
+    }
+
     pub async fn encode<R>(
         self: Arc<Self>,
         mut reader: R,
@@ -209,6 +264,13 @@ impl Erasure {
     where
         R: AsyncRead + Send + Sync + Unpin + 'static,
     {
+        if self.block_size == 0 {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "erasure block_size must be non-zero",
+            ));
+        }
+
         // Bound queued encoded blocks by memory budget to avoid per-request spikes.
         let expanded_block_bytes = self.shard_size().saturating_mul(self.total_shard_count());
         let max_inflight_bytes = rustfs_utils::get_env_usize(
@@ -223,10 +285,20 @@ impl Erasure {
             let mut total = 0;
             let mut buf = vec![0u8; block_size];
             loop {
-                match rustfs_utils::read_full(&mut reader, &mut buf).await {
-                    Ok(n) if n > 0 => {
+                match rustfs_utils::read_full_or_eof(&mut reader, &mut buf).await {
+                    Ok(Some(n)) => {
+                        debug_assert!(n > 0, "non-zero block_size prevents zero-length reads");
                         total += n;
-                        let res = self.encode_data(&buf[..n])?;
+                        let erasure = self.clone();
+                        let encode_buf = std::mem::take(&mut buf);
+                        let (res, returned_buf) = tokio::task::spawn_blocking(move || {
+                            let res = erasure.encode_data(&encode_buf[..n]);
+                            (res, encode_buf)
+                        })
+                        .await
+                        .map_err(|err| std::io::Error::other(format!("EC encode task failed: {err}")))?;
+                        buf = returned_buf;
+                        let res = res?;
                         let queued_bytes = queued_block_bytes(&res);
                         rustfs_io_metrics::add_ec_encode_inflight_bytes(queued_bytes);
                         if let Err(err) = tx.send(res).await {
@@ -234,7 +306,7 @@ impl Erasure {
                             return Err(std::io::Error::other(format!("Failed to send encoded data : {err}")));
                         }
                     }
-                    Ok(_) => {
+                    Ok(None) => {
                         break;
                     }
                     Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
@@ -244,7 +316,7 @@ impl Erasure {
                         {
                             return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()));
                         }
-                        break;
+                        return Err(e);
                     }
                     Err(e) => {
                         return Err(e);
@@ -285,13 +357,43 @@ impl Erasure {
         writers.shutdown().await?;
         Ok((reader, total))
     }
+
+    /// Fast path for small inline objects: skip tokio::spawn + mpsc channel.
+    /// Reads all data, encodes directly, writes shards sequentially.
+    pub async fn encode_inline_small<R>(
+        self: Arc<Self>,
+        reader: R,
+        writers: &mut [Option<BitrotWriterWrapper>],
+        quorum: usize,
+    ) -> std::io::Result<(R, usize)>
+    where
+        R: AsyncRead + Send + Sync + Unpin,
+    {
+        self.encode_small_direct(reader, writers, quorum, false).await
+    }
+
+    /// Fast path for single-block non-inline objects: avoids the producer/consumer
+    /// pipeline in `encode()` while keeping the same writer/quorum/shutdown semantics.
+    pub async fn encode_single_block_non_inline<R>(
+        self: Arc<Self>,
+        reader: R,
+        writers: &mut [Option<BitrotWriterWrapper>],
+        quorum: usize,
+    ) -> std::io::Result<(R, usize)>
+    where
+        R: AsyncRead + Send + Sync + Unpin,
+    {
+        self.encode_small_direct(reader, writers, quorum, true).await
+    }
 }
 
 #[cfg(test)]
 mod tests {
     use super::*;
     use crate::erasure_coding::{BitrotWriterWrapper, CustomWriter};
+    use rustfs_rio::HardLimitReader;
     use rustfs_utils::HashAlgorithm;
+    use std::io::Cursor;
     use std::pin::Pin;
     use std::sync::{Arc, Mutex};
     use std::task::{Context, Poll};
@@ -348,6 +450,172 @@ mod tests {
         assert!(!committed.lock().unwrap().is_empty());
     }
 
+    #[tokio::test]
+    async fn encode_returns_unexpected_eof_for_truncated_limited_reader() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed);
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 16));
+        let truncated = HardLimitReader::new(Cursor::new(b"short".to_vec()), 10);
+
+        let err = match erasure.encode(truncated, &mut writers, 1).await {
+            Ok(_) => panic!("truncated input must fail"),
+            Err(err) => err,
+        };
+
+        assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
+    }
+
+    #[tokio::test]
+    async fn encode_rejects_zero_block_size() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed);
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"payload".to_vec()));
+        let err = erasure
+            .encode(reader, &mut writers, 1)
+            .await
+            .expect_err("zero block size must be rejected");
+
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
+        assert!(err.to_string().contains("block_size"));
+    }
+
+    /// encode_inline_small: empty reader returns (reader, 0) without writing to any shard.
+    #[tokio::test]
+    async fn encode_inline_small_empty_stream_returns_zero() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed.clone());
+        // 1 data shard, 0 parity shards, block_size = 16
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 16));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(Vec::<u8>::new()));
+        let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, 1).await.unwrap();
+
+        assert_eq!(total, 0);
+        // No shutdown was called, so nothing should be committed
+        assert!(committed.lock().unwrap().is_empty());
+    }
+
+    /// encode_inline_small: small payload is encoded into the correct number of shards
+    /// and each writer receives data after shutdown.
+    #[tokio::test]
+    async fn encode_inline_small_payload_writes_all_shards() {
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const TOTAL_SHARDS: usize = DATA_SHARDS + PARITY_SHARDS;
+        const BLOCK_SIZE: usize = 64;
+
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..TOTAL_SHARDS).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+
+        let mut writers: Vec<Option<BitrotWriterWrapper>> = committed
+            .iter()
+            .map(|c| {
+                Some(BitrotWriterWrapper::new(
+                    CustomWriter::new_tokio_writer(DeferredCommitWriter::new(c.clone())),
+                    BLOCK_SIZE / DATA_SHARDS,
+                    HashAlgorithm::HighwayHash256S,
+                ))
+            })
+            .collect();
+
+        let payload = b"hello inline small";
+        let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
+        let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, DATA_SHARDS).await.unwrap();
+
+        assert_eq!(total, payload.len());
+        // All shards must have received data (shutdown flushed the bitrot header + shard bytes)
+        for (i, c) in committed.iter().enumerate() {
+            assert!(!c.lock().unwrap().is_empty(), "shard {i} should have received data");
+        }
+    }
+
+    #[tokio::test]
+    async fn encode_single_block_non_inline_payload_writes_all_shards() {
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const TOTAL_SHARDS: usize = DATA_SHARDS + PARITY_SHARDS;
+        const BLOCK_SIZE: usize = 64;
+
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..TOTAL_SHARDS).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+
+        let mut writers: Vec<Option<BitrotWriterWrapper>> = committed
+            .iter()
+            .map(|c| {
+                Some(BitrotWriterWrapper::new(
+                    CustomWriter::new_tokio_writer(DeferredCommitWriter::new(c.clone())),
+                    BLOCK_SIZE / DATA_SHARDS,
+                    HashAlgorithm::HighwayHash256S,
+                ))
+            })
+            .collect();
+
+        let payload = b"hello single block";
+        let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
+        let (_reader, total) = erasure
+            .encode_single_block_non_inline(reader, &mut writers, DATA_SHARDS)
+            .await
+            .unwrap();
+
+        assert_eq!(total, payload.len());
+        for (i, c) in committed.iter().enumerate() {
+            assert!(!c.lock().unwrap().is_empty(), "shard {i} should have received data");
+        }
+    }
+
+    #[tokio::test]
+    async fn encode_single_block_non_inline_rejects_multi_block_payload() {
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const TOTAL_SHARDS: usize = DATA_SHARDS + PARITY_SHARDS;
+        const BLOCK_SIZE: usize = 64;
+
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..TOTAL_SHARDS).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+
+        let mut writers: Vec<Option<BitrotWriterWrapper>> = committed
+            .iter()
+            .map(|c| {
+                Some(BitrotWriterWrapper::new(
+                    CustomWriter::new_tokio_writer(DeferredCommitWriter::new(c.clone())),
+                    BLOCK_SIZE / DATA_SHARDS,
+                    HashAlgorithm::HighwayHash256S,
+                ))
+            })
+            .collect();
+
+        let payload = vec![1u8; BLOCK_SIZE + 1];
+        let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload));
+        let err = erasure
+            .encode_single_block_non_inline(reader, &mut writers, DATA_SHARDS)
+            .await
+            .expect_err("single-block fast path must reject oversized readers");
+
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
+        assert!(err.to_string().contains("single-block non-inline fast path"));
+        for c in committed {
+            assert!(c.lock().unwrap().is_empty());
+        }
+    }
+
     #[test]
     fn encode_channel_capacity_never_returns_zero() {
         assert_eq!(encode_channel_capacity(0, 1024), 1);
@@ -355,9 +623,33 @@ mod tests {
         assert_eq!(encode_channel_capacity(4096, 1024), 1);
     }
 
+    #[test]
+    fn write_quorum_failure_summary_uses_stable_dominant_error_label() {
+        let err = Error::from(rustfs_rio::new_test_internode_http_io_error(
+            rustfs_rio::InternodeHttpErrorKind::ConnectionReset,
+        ));
+        let summary = WriteQuorumFailureSummary {
+            required: 2,
+            achieved: 0,
+            failed: 2,
+            total: 2,
+            offline_disks: 0,
+            ignored_failures: 0,
+            retryable_failures: 2,
+            dominant_error: Some(err),
+            dominant_error_label: "connection_reset",
+        };
+        let text = format_write_quorum_failure(&summary);
+
+        assert!(text.contains("dominant-error=connection_reset"));
+        assert!(!text.contains("/rustfs/rpc/put_file_stream"));
+        assert!(!text.contains("PUT "));
+    }
+
     #[test]
     fn encode_channel_capacity_respects_budget_and_hard_cap() {
         assert_eq!(encode_channel_capacity(4 * 1024 * 1024, 32 * 1024 * 1024), 8);
+        assert_eq!(encode_channel_capacity(1536 * 1024, 32 * 1024 * 1024), 21);
         assert_eq!(encode_channel_capacity(16 * 1024 * 1024, 32 * 1024 * 1024), 2);
         assert_eq!(encode_channel_capacity(1, usize::MAX), DEFAULT_RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BLOCKS);
     }

crates/ecstore/src/erasure_coding/erasure.rs

@@ -308,7 +308,6 @@ where
 /// - `encoder`: Optional ReedSolomon encoder instance.
 /// - `block_size`: Block size for each shard.
 /// - `_id`: Unique identifier for the erasure instance.
-/// - `_buf`: Internal buffer for block operations.
 ///
 /// # Example
 /// ```ignore
@@ -326,7 +325,6 @@ pub struct Erasure {
     pub block_size: usize,
     uses_legacy: bool,
     _id: Uuid,
-    _buf: Vec<u8>,
 }
 
 impl Default for Erasure {
@@ -339,7 +337,6 @@ impl Default for Erasure {
             block_size: 0,
             uses_legacy: false,
             _id: Uuid::nil(),
-            _buf: vec![],
         }
     }
 }
@@ -353,8 +350,7 @@ impl Clone for Erasure {
             legacy_encoder: self.legacy_encoder.clone(),
             block_size: self.block_size,
             uses_legacy: self.uses_legacy,
-            _id: Uuid::new_v4(), // Generate new ID for clone
-            _buf: vec![0u8; self.block_size],
+            _id: self._id, // Shared by clones; this field is unused in hot paths.
         }
     }
 }
@@ -399,7 +395,6 @@ impl Erasure {
             legacy_encoder,
             uses_legacy,
             _id: Uuid::new_v4(),
-            _buf: vec![0u8; block_size],
         }
     }
 
@@ -418,6 +413,9 @@ impl Erasure {
             calc_shard_size
         };
         let per_shard_size = shard_size_fn(data.len(), self.data_shards);
+        if per_shard_size == 0 {
+            return Ok(vec![Bytes::new(); self.total_shard_count()]);
+        }
         let need_total_size = per_shard_size * self.total_shard_count();
 
         let mut data_buffer = BytesMut::with_capacity(need_total_size);
@@ -453,6 +451,63 @@ impl Erasure {
         Ok(shards)
     }
 
+    /// Encode owned data, avoiding a copy when the caller already has a heap buffer.
+    /// Falls back to copying into a new buffer if zero-copy conversion fails.
+    pub fn encode_data_owned(&self, data: Vec<u8>) -> io::Result<Vec<Bytes>> {
+        let shard_size_fn = if self.uses_legacy {
+            calc_shard_size_legacy
+        } else {
+            calc_shard_size
+        };
+        let per_shard_size = shard_size_fn(data.len(), self.data_shards);
+        if per_shard_size == 0 {
+            return Ok(vec![Bytes::new(); self.total_shard_count()]);
+        }
+        let need_total_size = per_shard_size * self.total_shard_count();
+
+        // Try zero-copy: Vec<u8> -> Bytes -> BytesMut (succeeds when refcount == 1)
+        let mut data_buffer = match Bytes::from(data).try_into_mut() {
+            Ok(mut bm) => {
+                bm.resize(need_total_size, 0u8);
+                bm
+            }
+            Err(b) => {
+                // Rare path: refcount != 1, fall back to copy
+                let mut bm = BytesMut::with_capacity(need_total_size);
+                bm.extend_from_slice(&b);
+                bm.resize(need_total_size, 0u8);
+                bm
+            }
+        };
+
+        {
+            let data_slices: SmallVec<[&mut [u8]; 16]> = data_buffer.chunks_exact_mut(per_shard_size).collect();
+
+            if self.parity_shards > 0 {
+                if self.uses_legacy {
+                    if let Some(encoder) = self.legacy_encoder.as_ref() {
+                        encoder.encode(data_slices)?;
+                    } else {
+                        warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
+                    }
+                } else if let Some(encoder) = self.encoder.as_ref() {
+                    encoder.encode(data_slices)?;
+                } else {
+                    warn!("parity_shards > 0, but encoder is None");
+                }
+            }
+        }
+
+        let mut data_buffer = data_buffer.freeze();
+        let mut shards = Vec::with_capacity(self.total_shard_count());
+        for _ in 0..self.total_shard_count() {
+            let shard = data_buffer.split_to(per_shard_size);
+            shards.push(shard);
+        }
+
+        Ok(shards)
+    }
+
     /// Decode and reconstruct missing data shards in-place.
     ///
     /// # Arguments
@@ -579,18 +634,42 @@ impl Erasure {
         F: FnMut(std::io::Result<Vec<Bytes>>) -> Fut + Send,
         Fut: std::future::Future<Output = Result<(), E>> + Send,
     {
+        if self.block_size == 0 {
+            on_block(Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "erasure block_size must be non-zero",
+            )))
+            .await?;
+            return Ok(0);
+        }
+
         let block_size = self.block_size;
         let mut total = 0;
         let mut buf = vec![0u8; block_size];
         loop {
-            match rustfs_utils::read_full(&mut *reader, &mut buf).await {
-                Ok(n) if n > 0 => {
+            match rustfs_utils::read_full_or_eof(&mut *reader, &mut buf).await {
+                Ok(Some(n)) => {
+                    debug_assert!(n > 0, "non-zero block_size prevents zero-length reads");
                     warn!("encode_stream_callback_async read n={}", n);
                     total += n;
-                    let res = self.encode_data(&buf[..n]);
+                    let erasure = self.clone();
+                    let encode_buf = std::mem::take(&mut buf);
+                    let (res, returned_buf) = match tokio::task::spawn_blocking(move || {
+                        let res = erasure.encode_data(&encode_buf[..n]);
+                        (res, encode_buf)
+                    })
+                    .await
+                    {
+                        Ok(result) => result,
+                        Err(err) => {
+                            on_block(Err(std::io::Error::other(format!("EC encode task failed: {err}")))).await?;
+                            break;
+                        }
+                    };
+                    buf = returned_buf;
                     on_block(res).await?
                 }
-                Ok(_) => {
+                Ok(None) => {
                     warn!("encode_stream_callback_async read unexpected ok");
                     break;
                 }
@@ -618,6 +697,24 @@ mod tests {
         shards.iter().map(|shard| Some(shard.to_vec())).collect()
     }
 
+    fn assert_owned_encode_matches_borrowed(erasure: &Erasure, data: Vec<u8>) {
+        let borrowed = erasure.encode_data(&data).expect("borrowed encode should succeed");
+        let owned = erasure.encode_data_owned(data).expect("owned encode should succeed");
+
+        assert_eq!(owned, borrowed);
+    }
+
+    #[test]
+    fn encode_data_owned_matches_borrowed_path() {
+        for uses_legacy in [false, true] {
+            let erasure = Erasure::new_with_options(4, 2, 64, uses_legacy);
+
+            assert_owned_encode_matches_borrowed(&erasure, Vec::new());
+            assert_owned_encode_matches_borrowed(&erasure, b"small payload".to_vec());
+            assert_owned_encode_matches_borrowed(&erasure, (0_u8..37).collect());
+        }
+    }
+
     #[test]
     fn decode_data_keeps_missing_parity_shard_unreconstructed() {
         let erasure = Erasure::new(2, 2, 64);
@@ -1018,6 +1115,35 @@ mod tests {
         assert_eq!(&recovered, &data_clone);
     }
 
+    #[tokio::test]
+    async fn test_encode_stream_callback_async_reports_zero_block_size() {
+        use std::io::Cursor;
+        use std::sync::{Arc, Mutex};
+
+        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let mut reader = Cursor::new(b"payload".to_vec());
+        let observed = Arc::new(Mutex::new(None));
+        let observed_clone = observed.clone();
+
+        let total = erasure
+            .encode_stream_callback_async::<_, _, (), _>(&mut reader, move |res| {
+                let observed = observed_clone.clone();
+                async move {
+                    let err = res.expect_err("zero block size should report an error");
+                    *observed.lock().unwrap() = Some((err.kind(), err.to_string()));
+                    Ok(())
+                }
+            })
+            .await
+            .expect("callback should handle the zero block size error");
+
+        assert_eq!(total, 0);
+        let observed = observed.lock().unwrap();
+        let (kind, message) = observed.as_ref().expect("callback should be invoked once");
+        assert_eq!(*kind, std::io::ErrorKind::InvalidInput);
+        assert!(message.contains("block_size"));
+    }
+
     // SIMD mode specific tests
     mod simd_tests {
         use super::*;

crates/ecstore/src/error.rs

@@ -14,6 +14,7 @@
 
 use crate::bucket::error::BucketMetadataError;
 use crate::disk::error::DiskError;
+use rustfs_storage_api::StorageErrorCode;
 use rustfs_utils::path::decode_dir_object;
 use s3s::{S3Error, S3ErrorCode};
 
@@ -160,6 +161,14 @@ pub enum StorageError {
     NotModified,
     #[error("Invalid range specified: {0}")]
     InvalidRangeSpec(String),
+    #[error("Namespace lock quorum unavailable for {mode} lock on {bucket}/{object}: required {required}, achieved {achieved}")]
+    NamespaceLockQuorumUnavailable {
+        mode: &'static str,
+        bucket: String,
+        object: String,
+        required: usize,
+        achieved: usize,
+    },
 
     // ── Generic ──────────────────────────────────────────────────────
     #[error("Unexpected error")]
@@ -204,6 +213,7 @@ impl StorageError {
                 | StorageError::ErasureWriteQuorum
                 | StorageError::InsufficientReadQuorum(_, _)
                 | StorageError::InsufficientWriteQuorum(_, _)
+                | StorageError::NamespaceLockQuorumUnavailable { .. }
         )
     }
 }
@@ -433,153 +443,199 @@ impl Clone for StorageError {
             StorageError::NotModified => StorageError::NotModified,
             StorageError::InvalidPartNumber(a) => StorageError::InvalidPartNumber(*a),
             StorageError::InvalidRangeSpec(a) => StorageError::InvalidRangeSpec(a.clone()),
+            StorageError::NamespaceLockQuorumUnavailable {
+                mode,
+                bucket,
+                object,
+                required,
+                achieved,
+            } => StorageError::NamespaceLockQuorumUnavailable {
+                mode,
+                bucket: bucket.clone(),
+                object: object.clone(),
+                required: *required,
+                achieved: *achieved,
+            },
         }
     }
 }
 
 impl StorageError {
-    pub fn to_u32(&self) -> u32 {
+    fn code(&self) -> StorageErrorCode {
         match self {
-            StorageError::Io(_) => 0x01,
-            StorageError::FaultyDisk => 0x02,
-            StorageError::DiskFull => 0x03,
-            StorageError::VolumeNotFound => 0x04,
-            StorageError::VolumeExists => 0x05,
-            StorageError::FileNotFound => 0x06,
-            StorageError::FileVersionNotFound => 0x07,
-            StorageError::FileNameTooLong => 0x08,
-            StorageError::FileAccessDenied => 0x09,
-            StorageError::FileCorrupt => 0x0A,
-            StorageError::IsNotRegular => 0x0B,
-            StorageError::VolumeNotEmpty => 0x0C,
-            StorageError::VolumeAccessDenied => 0x0D,
-            StorageError::CorruptedFormat => 0x0E,
-            StorageError::CorruptedBackend => 0x0F,
-            StorageError::UnformattedDisk => 0x10,
-            StorageError::DiskNotFound => 0x11,
-            StorageError::DriveIsRoot => 0x12,
-            StorageError::FaultyRemoteDisk => 0x13,
-            StorageError::DiskAccessDenied => 0x14,
-            StorageError::Unexpected => 0x15,
-            StorageError::NotImplemented => 0x16,
-            StorageError::InvalidArgument(_, _, _) => 0x17,
-            StorageError::MethodNotAllowed => 0x18,
-            StorageError::BucketNotFound(_) => 0x19,
-            StorageError::BucketNotEmpty(_) => 0x1A,
-            StorageError::BucketNameInvalid(_) => 0x1B,
-            StorageError::ObjectNameInvalid(_, _) => 0x1C,
-            StorageError::BucketExists(_) => 0x1D,
-            StorageError::StorageFull => 0x1E,
-            StorageError::SlowDown => 0x1F,
-            StorageError::PrefixAccessDenied(_, _) => 0x20,
-            StorageError::InvalidUploadIDKeyCombination(_, _) => 0x21,
-            StorageError::MalformedUploadID(_) => 0x22,
-            StorageError::ObjectNameTooLong(_, _) => 0x23,
-            StorageError::ObjectNamePrefixAsSlash(_, _) => 0x24,
-            StorageError::ObjectNotFound(_, _) => 0x25,
-            StorageError::VersionNotFound(_, _, _) => 0x26,
-            StorageError::InvalidUploadID(_, _, _) => 0x27,
-            StorageError::InvalidVersionID(_, _, _) => 0x28,
-            StorageError::DataMovementOverwriteErr(_, _, _) => 0x29,
-            StorageError::ObjectExistsAsDirectory(_, _) => 0x2A,
-            // StorageError::InsufficientReadQuorum => 0x2B,
-            // StorageError::InsufficientWriteQuorum => 0x2C,
-            StorageError::DecommissionNotStarted => 0x2D,
-            StorageError::InvalidPart(_, _, _) => 0x2E,
-            StorageError::DoneForNow => 0x2F,
-            StorageError::DecommissionAlreadyRunning => 0x30,
-            StorageError::RebalanceAlreadyRunning => 0x40,
-            StorageError::OperationCanceled => 0x41,
-            StorageError::ErasureReadQuorum => 0x31,
-            StorageError::ErasureWriteQuorum => 0x32,
-            StorageError::NotFirstDisk => 0x33,
-            StorageError::FirstDiskWait => 0x34,
-            StorageError::ConfigNotFound => 0x35,
-            StorageError::TooManyOpenFiles => 0x36,
-            StorageError::NoHealRequired => 0x37,
-            StorageError::Lock(_) => 0x38,
-            StorageError::InsufficientReadQuorum(_, _) => 0x39,
-            StorageError::InsufficientWriteQuorum(_, _) => 0x3A,
-            StorageError::PreconditionFailed => 0x3B,
-            StorageError::EntityTooSmall(_, _, _) => 0x3C,
-            StorageError::InvalidRangeSpec(_) => 0x3D,
-            StorageError::NotModified => 0x3E,
-            StorageError::InvalidPartNumber(_) => 0x3F,
+            StorageError::Io(_) => StorageErrorCode::Io,
+            StorageError::FaultyDisk => StorageErrorCode::FaultyDisk,
+            StorageError::DiskFull => StorageErrorCode::DiskFull,
+            StorageError::VolumeNotFound => StorageErrorCode::VolumeNotFound,
+            StorageError::VolumeExists => StorageErrorCode::VolumeExists,
+            StorageError::FileNotFound => StorageErrorCode::FileNotFound,
+            StorageError::FileVersionNotFound => StorageErrorCode::FileVersionNotFound,
+            StorageError::FileNameTooLong => StorageErrorCode::FileNameTooLong,
+            StorageError::FileAccessDenied => StorageErrorCode::FileAccessDenied,
+            StorageError::FileCorrupt => StorageErrorCode::FileCorrupt,
+            StorageError::IsNotRegular => StorageErrorCode::IsNotRegular,
+            StorageError::VolumeNotEmpty => StorageErrorCode::VolumeNotEmpty,
+            StorageError::VolumeAccessDenied => StorageErrorCode::VolumeAccessDenied,
+            StorageError::CorruptedFormat => StorageErrorCode::CorruptedFormat,
+            StorageError::CorruptedBackend => StorageErrorCode::CorruptedBackend,
+            StorageError::UnformattedDisk => StorageErrorCode::UnformattedDisk,
+            StorageError::DiskNotFound => StorageErrorCode::DiskNotFound,
+            StorageError::DriveIsRoot => StorageErrorCode::DriveIsRoot,
+            StorageError::FaultyRemoteDisk => StorageErrorCode::FaultyRemoteDisk,
+            StorageError::DiskAccessDenied => StorageErrorCode::DiskAccessDenied,
+            StorageError::Unexpected => StorageErrorCode::Unexpected,
+            StorageError::NotImplemented => StorageErrorCode::NotImplemented,
+            StorageError::InvalidArgument(_, _, _) => StorageErrorCode::InvalidArgument,
+            StorageError::MethodNotAllowed => StorageErrorCode::MethodNotAllowed,
+            StorageError::BucketNotFound(_) => StorageErrorCode::BucketNotFound,
+            StorageError::BucketNotEmpty(_) => StorageErrorCode::BucketNotEmpty,
+            StorageError::BucketNameInvalid(_) => StorageErrorCode::BucketNameInvalid,
+            StorageError::ObjectNameInvalid(_, _) => StorageErrorCode::ObjectNameInvalid,
+            StorageError::BucketExists(_) => StorageErrorCode::BucketExists,
+            StorageError::StorageFull => StorageErrorCode::StorageFull,
+            StorageError::SlowDown => StorageErrorCode::SlowDown,
+            StorageError::PrefixAccessDenied(_, _) => StorageErrorCode::PrefixAccessDenied,
+            StorageError::InvalidUploadIDKeyCombination(_, _) => StorageErrorCode::InvalidUploadIDKeyCombination,
+            StorageError::MalformedUploadID(_) => StorageErrorCode::MalformedUploadID,
+            StorageError::ObjectNameTooLong(_, _) => StorageErrorCode::ObjectNameTooLong,
+            StorageError::ObjectNamePrefixAsSlash(_, _) => StorageErrorCode::ObjectNamePrefixAsSlash,
+            StorageError::ObjectNotFound(_, _) => StorageErrorCode::ObjectNotFound,
+            StorageError::VersionNotFound(_, _, _) => StorageErrorCode::VersionNotFound,
+            StorageError::InvalidUploadID(_, _, _) => StorageErrorCode::InvalidUploadID,
+            StorageError::InvalidVersionID(_, _, _) => StorageErrorCode::InvalidVersionID,
+            StorageError::DataMovementOverwriteErr(_, _, _) => StorageErrorCode::DataMovementOverwriteErr,
+            StorageError::ObjectExistsAsDirectory(_, _) => StorageErrorCode::ObjectExistsAsDirectory,
+            StorageError::DecommissionNotStarted => StorageErrorCode::DecommissionNotStarted,
+            StorageError::InvalidPart(_, _, _) => StorageErrorCode::InvalidPart,
+            StorageError::DoneForNow => StorageErrorCode::DoneForNow,
+            StorageError::DecommissionAlreadyRunning => StorageErrorCode::DecommissionAlreadyRunning,
+            StorageError::RebalanceAlreadyRunning => StorageErrorCode::RebalanceAlreadyRunning,
+            StorageError::OperationCanceled => StorageErrorCode::OperationCanceled,
+            StorageError::ErasureReadQuorum => StorageErrorCode::ErasureReadQuorum,
+            StorageError::ErasureWriteQuorum => StorageErrorCode::ErasureWriteQuorum,
+            StorageError::NotFirstDisk => StorageErrorCode::NotFirstDisk,
+            StorageError::FirstDiskWait => StorageErrorCode::FirstDiskWait,
+            StorageError::ConfigNotFound => StorageErrorCode::ConfigNotFound,
+            StorageError::TooManyOpenFiles => StorageErrorCode::TooManyOpenFiles,
+            StorageError::NoHealRequired => StorageErrorCode::NoHealRequired,
+            StorageError::Lock(_) => StorageErrorCode::Lock,
+            StorageError::InsufficientReadQuorum(_, _) => StorageErrorCode::InsufficientReadQuorum,
+            StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum,
+            StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed,
+            StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall,
+            StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec,
+            StorageError::NotModified => StorageErrorCode::NotModified,
+            StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
+            StorageError::NamespaceLockQuorumUnavailable { .. } => StorageErrorCode::NamespaceLockQuorumUnavailable,
         }
     }
 
+    pub fn to_u32(&self) -> u32 {
+        self.code().as_u32()
+    }
+
     pub fn from_u32(error: u32) -> Option<Self> {
-        match error {
-            0x01 => Some(StorageError::Io(std::io::Error::other("Io error"))),
-            0x02 => Some(StorageError::FaultyDisk),
-            0x03 => Some(StorageError::DiskFull),
-            0x04 => Some(StorageError::VolumeNotFound),
-            0x05 => Some(StorageError::VolumeExists),
-            0x06 => Some(StorageError::FileNotFound),
-            0x07 => Some(StorageError::FileVersionNotFound),
-            0x08 => Some(StorageError::FileNameTooLong),
-            0x09 => Some(StorageError::FileAccessDenied),
-            0x0A => Some(StorageError::FileCorrupt),
-            0x0B => Some(StorageError::IsNotRegular),
-            0x0C => Some(StorageError::VolumeNotEmpty),
-            0x0D => Some(StorageError::VolumeAccessDenied),
-            0x0E => Some(StorageError::CorruptedFormat),
-            0x0F => Some(StorageError::CorruptedBackend),
-            0x10 => Some(StorageError::UnformattedDisk),
-            0x11 => Some(StorageError::DiskNotFound),
-            0x12 => Some(StorageError::DriveIsRoot),
-            0x13 => Some(StorageError::FaultyRemoteDisk),
-            0x14 => Some(StorageError::DiskAccessDenied),
-            0x15 => Some(StorageError::Unexpected),
-            0x16 => Some(StorageError::NotImplemented),
-            0x17 => Some(StorageError::InvalidArgument(Default::default(), Default::default(), Default::default())),
-            0x18 => Some(StorageError::MethodNotAllowed),
-            0x19 => Some(StorageError::BucketNotFound(Default::default())),
-            0x1A => Some(StorageError::BucketNotEmpty(Default::default())),
-            0x1B => Some(StorageError::BucketNameInvalid(Default::default())),
-            0x1C => Some(StorageError::ObjectNameInvalid(Default::default(), Default::default())),
-            0x1D => Some(StorageError::BucketExists(Default::default())),
-            0x1E => Some(StorageError::StorageFull),
-            0x1F => Some(StorageError::SlowDown),
-            0x20 => Some(StorageError::PrefixAccessDenied(Default::default(), Default::default())),
-            0x21 => Some(StorageError::InvalidUploadIDKeyCombination(Default::default(), Default::default())),
-            0x22 => Some(StorageError::MalformedUploadID(Default::default())),
-            0x23 => Some(StorageError::ObjectNameTooLong(Default::default(), Default::default())),
-            0x24 => Some(StorageError::ObjectNamePrefixAsSlash(Default::default(), Default::default())),
-            0x25 => Some(StorageError::ObjectNotFound(Default::default(), Default::default())),
-            0x26 => Some(StorageError::VersionNotFound(Default::default(), Default::default(), Default::default())),
-            0x27 => Some(StorageError::InvalidUploadID(Default::default(), Default::default(), Default::default())),
-            0x28 => Some(StorageError::InvalidVersionID(Default::default(), Default::default(), Default::default())),
-            0x29 => Some(StorageError::DataMovementOverwriteErr(
+        match StorageErrorCode::from_u32(error)? {
+            StorageErrorCode::Io => Some(StorageError::Io(std::io::Error::other("Io error"))),
+            StorageErrorCode::FaultyDisk => Some(StorageError::FaultyDisk),
+            StorageErrorCode::DiskFull => Some(StorageError::DiskFull),
+            StorageErrorCode::VolumeNotFound => Some(StorageError::VolumeNotFound),
+            StorageErrorCode::VolumeExists => Some(StorageError::VolumeExists),
+            StorageErrorCode::FileNotFound => Some(StorageError::FileNotFound),
+            StorageErrorCode::FileVersionNotFound => Some(StorageError::FileVersionNotFound),
+            StorageErrorCode::FileNameTooLong => Some(StorageError::FileNameTooLong),
+            StorageErrorCode::FileAccessDenied => Some(StorageError::FileAccessDenied),
+            StorageErrorCode::FileCorrupt => Some(StorageError::FileCorrupt),
+            StorageErrorCode::IsNotRegular => Some(StorageError::IsNotRegular),
+            StorageErrorCode::VolumeNotEmpty => Some(StorageError::VolumeNotEmpty),
+            StorageErrorCode::VolumeAccessDenied => Some(StorageError::VolumeAccessDenied),
+            StorageErrorCode::CorruptedFormat => Some(StorageError::CorruptedFormat),
+            StorageErrorCode::CorruptedBackend => Some(StorageError::CorruptedBackend),
+            StorageErrorCode::UnformattedDisk => Some(StorageError::UnformattedDisk),
+            StorageErrorCode::DiskNotFound => Some(StorageError::DiskNotFound),
+            StorageErrorCode::DriveIsRoot => Some(StorageError::DriveIsRoot),
+            StorageErrorCode::FaultyRemoteDisk => Some(StorageError::FaultyRemoteDisk),
+            StorageErrorCode::DiskAccessDenied => Some(StorageError::DiskAccessDenied),
+            StorageErrorCode::Unexpected => Some(StorageError::Unexpected),
+            StorageErrorCode::NotImplemented => Some(StorageError::NotImplemented),
+            StorageErrorCode::InvalidArgument => {
+                Some(StorageError::InvalidArgument(Default::default(), Default::default(), Default::default()))
+            }
+            StorageErrorCode::MethodNotAllowed => Some(StorageError::MethodNotAllowed),
+            StorageErrorCode::BucketNotFound => Some(StorageError::BucketNotFound(Default::default())),
+            StorageErrorCode::BucketNotEmpty => Some(StorageError::BucketNotEmpty(Default::default())),
+            StorageErrorCode::BucketNameInvalid => Some(StorageError::BucketNameInvalid(Default::default())),
+            StorageErrorCode::ObjectNameInvalid => Some(StorageError::ObjectNameInvalid(Default::default(), Default::default())),
+            StorageErrorCode::BucketExists => Some(StorageError::BucketExists(Default::default())),
+            StorageErrorCode::StorageFull => Some(StorageError::StorageFull),
+            StorageErrorCode::SlowDown => Some(StorageError::SlowDown),
+            StorageErrorCode::PrefixAccessDenied => {
+                Some(StorageError::PrefixAccessDenied(Default::default(), Default::default()))
+            }
+            StorageErrorCode::InvalidUploadIDKeyCombination => {
+                Some(StorageError::InvalidUploadIDKeyCombination(Default::default(), Default::default()))
+            }
+            StorageErrorCode::MalformedUploadID => Some(StorageError::MalformedUploadID(Default::default())),
+            StorageErrorCode::ObjectNameTooLong => Some(StorageError::ObjectNameTooLong(Default::default(), Default::default())),
+            StorageErrorCode::ObjectNamePrefixAsSlash => {
+                Some(StorageError::ObjectNamePrefixAsSlash(Default::default(), Default::default()))
+            }
+            StorageErrorCode::ObjectNotFound => Some(StorageError::ObjectNotFound(Default::default(), Default::default())),
+            StorageErrorCode::VersionNotFound => {
+                Some(StorageError::VersionNotFound(Default::default(), Default::default(), Default::default()))
+            }
+            StorageErrorCode::InvalidUploadID => {
+                Some(StorageError::InvalidUploadID(Default::default(), Default::default(), Default::default()))
+            }
+            StorageErrorCode::InvalidVersionID => {
+                Some(StorageError::InvalidVersionID(Default::default(), Default::default(), Default::default()))
+            }
+            StorageErrorCode::DataMovementOverwriteErr => Some(StorageError::DataMovementOverwriteErr(
                 Default::default(),
                 Default::default(),
                 Default::default(),
             )),
-            0x2A => Some(StorageError::ObjectExistsAsDirectory(Default::default(), Default::default())),
-            // 0x2B => Some(StorageError::InsufficientReadQuorum),
-            // 0x2C => Some(StorageError::InsufficientWriteQuorum),
-            0x2D => Some(StorageError::DecommissionNotStarted),
-            0x2E => Some(StorageError::InvalidPart(Default::default(), Default::default(), Default::default())),
-            0x2F => Some(StorageError::DoneForNow),
-            0x30 => Some(StorageError::DecommissionAlreadyRunning),
-            0x40 => Some(StorageError::RebalanceAlreadyRunning),
-            0x41 => Some(StorageError::OperationCanceled),
-            0x31 => Some(StorageError::ErasureReadQuorum),
-            0x32 => Some(StorageError::ErasureWriteQuorum),
-            0x33 => Some(StorageError::NotFirstDisk),
-            0x34 => Some(StorageError::FirstDiskWait),
-            0x35 => Some(StorageError::ConfigNotFound),
-            0x36 => Some(StorageError::TooManyOpenFiles),
-            0x37 => Some(StorageError::NoHealRequired),
-            0x38 => Some(StorageError::Lock(rustfs_lock::LockError::internal("Generic lock error".to_string()))),
-            0x39 => Some(StorageError::InsufficientReadQuorum(Default::default(), Default::default())),
-            0x3A => Some(StorageError::InsufficientWriteQuorum(Default::default(), Default::default())),
-            0x3B => Some(StorageError::PreconditionFailed),
-            0x3C => Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default())),
-            0x3D => Some(StorageError::InvalidRangeSpec(Default::default())),
-            0x3E => Some(StorageError::NotModified),
-            0x3F => Some(StorageError::InvalidPartNumber(Default::default())),
-            _ => None,
+            StorageErrorCode::ObjectExistsAsDirectory => {
+                Some(StorageError::ObjectExistsAsDirectory(Default::default(), Default::default()))
+            }
+            StorageErrorCode::DecommissionNotStarted => Some(StorageError::DecommissionNotStarted),
+            StorageErrorCode::InvalidPart => {
+                Some(StorageError::InvalidPart(Default::default(), Default::default(), Default::default()))
+            }
+            StorageErrorCode::DoneForNow => Some(StorageError::DoneForNow),
+            StorageErrorCode::DecommissionAlreadyRunning => Some(StorageError::DecommissionAlreadyRunning),
+            StorageErrorCode::RebalanceAlreadyRunning => Some(StorageError::RebalanceAlreadyRunning),
+            StorageErrorCode::OperationCanceled => Some(StorageError::OperationCanceled),
+            StorageErrorCode::ErasureReadQuorum => Some(StorageError::ErasureReadQuorum),
+            StorageErrorCode::ErasureWriteQuorum => Some(StorageError::ErasureWriteQuorum),
+            StorageErrorCode::NotFirstDisk => Some(StorageError::NotFirstDisk),
+            StorageErrorCode::FirstDiskWait => Some(StorageError::FirstDiskWait),
+            StorageErrorCode::ConfigNotFound => Some(StorageError::ConfigNotFound),
+            StorageErrorCode::TooManyOpenFiles => Some(StorageError::TooManyOpenFiles),
+            StorageErrorCode::NoHealRequired => Some(StorageError::NoHealRequired),
+            StorageErrorCode::Lock => {
+                Some(StorageError::Lock(rustfs_lock::LockError::internal("Generic lock error".to_string())))
+            }
+            StorageErrorCode::InsufficientReadQuorum => {
+                Some(StorageError::InsufficientReadQuorum(Default::default(), Default::default()))
+            }
+            StorageErrorCode::InsufficientWriteQuorum => {
+                Some(StorageError::InsufficientWriteQuorum(Default::default(), Default::default()))
+            }
+            StorageErrorCode::PreconditionFailed => Some(StorageError::PreconditionFailed),
+            StorageErrorCode::EntityTooSmall => {
+                Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default()))
+            }
+            StorageErrorCode::InvalidRangeSpec => Some(StorageError::InvalidRangeSpec(Default::default())),
+            StorageErrorCode::NotModified => Some(StorageError::NotModified),
+            StorageErrorCode::InvalidPartNumber => Some(StorageError::InvalidPartNumber(Default::default())),
+            StorageErrorCode::NamespaceLockQuorumUnavailable => Some(StorageError::NamespaceLockQuorumUnavailable {
+                mode: "write",
+                bucket: Default::default(),
+                object: Default::default(),
+                required: Default::default(),
+                achieved: Default::default(),
+            }),
         }
     }
 }
@@ -984,6 +1040,17 @@ mod tests {
         assert_eq!(StorageError::DecommissionAlreadyRunning.to_u32(), 0x30);
         assert_eq!(StorageError::RebalanceAlreadyRunning.to_u32(), 0x40);
         assert_eq!(StorageError::OperationCanceled.to_u32(), 0x41);
+        assert_eq!(
+            StorageError::NamespaceLockQuorumUnavailable {
+                mode: "write",
+                bucket: "bucket".into(),
+                object: "object".into(),
+                required: 3,
+                achieved: 2,
+            }
+            .to_u32(),
+            0x42
+        );
     }
 
     #[test]
@@ -998,11 +1065,34 @@ mod tests {
         assert!(matches!(StorageError::from_u32(0x30), Some(StorageError::DecommissionAlreadyRunning)));
         assert!(matches!(StorageError::from_u32(0x40), Some(StorageError::RebalanceAlreadyRunning)));
         assert!(matches!(StorageError::from_u32(0x41), Some(StorageError::OperationCanceled)));
+        assert!(matches!(
+            StorageError::from_u32(0x42),
+            Some(StorageError::NamespaceLockQuorumUnavailable { .. })
+        ));
 
         // Test invalid code returns None
         assert!(StorageError::from_u32(0xFF).is_none());
     }
 
+    #[test]
+    fn test_storage_error_code_contract_matches_storage_api() {
+        assert_eq!(StorageError::DiskFull.to_u32(), StorageErrorCode::DiskFull.as_u32());
+        assert_eq!(
+            StorageError::ObjectNotFound("bucket".to_string(), "object".to_string()).to_u32(),
+            StorageErrorCode::ObjectNotFound.as_u32()
+        );
+        assert!(matches!(
+            StorageError::from_u32(StorageErrorCode::ErasureReadQuorum.as_u32()),
+            Some(StorageError::ErasureReadQuorum)
+        ));
+        assert!(matches!(
+            StorageError::from_u32(StorageErrorCode::NamespaceLockQuorumUnavailable.as_u32()),
+            Some(StorageError::NamespaceLockQuorumUnavailable { .. })
+        ));
+        assert!(StorageError::from_u32(0x2B).is_none());
+        assert!(StorageError::from_u32(0x2C).is_none());
+    }
+
     #[test]
     fn test_storage_error_partial_eq() {
         // Test IO error comparison

crates/ecstore/src/global.rs

@@ -242,7 +242,7 @@ pub async fn update_erasure_type(setup_type: SetupType) {
 //     }
 // }
 
-type TypeLocalDiskSetDrives = Vec<Vec<Vec<Option<DiskStore>>>>;
+pub(crate) type TypeLocalDiskSetDrives = Vec<Vec<Vec<Option<DiskStore>>>>;
 
 /// Set the global region
 ///

crates/ecstore/src/lib.rs

@@ -34,6 +34,7 @@ pub mod metrics_realtime;
 pub mod notification_sys;
 pub mod pools;
 pub mod rebalance;
+pub mod rio;
 pub mod rpc;
 pub mod set_disk;
 mod sets;
@@ -60,3 +61,12 @@ pub use global::{get_global_lock_client, get_global_lock_clients, set_global_loc
 
 pub use global::GLOBAL_Endpoints;
 pub use store_api::StorageAPI;
+
+#[cfg(test)]
+mod rio_tests {
+    #[test]
+    fn uses_expected_rio_backend() {
+        let expected = if cfg!(feature = "rio-v2") { "rio-v2" } else { "legacy-rio" };
+        assert_eq!(crate::rio::backend_name(), expected);
+    }
+}

crates/ecstore/src/metrics_realtime.rs

@@ -18,7 +18,8 @@ use rustfs_common::{GLOBAL_LOCAL_NODE_NAME, GLOBAL_RUSTFS_ADDR, heal_channel::Dr
 use rustfs_io_metrics::internode_metrics::global_internode_metrics;
 use rustfs_madmin::metrics::{
     DiskIOStats, DiskMetric, LastMinute as MadminLastMinute, NetDevLine, NetMetrics, RPCMetrics, RealtimeMetrics,
-    ScannerMetrics as MadminScannerMetrics, TimedAction as MadminTimedAction,
+    ScannerMetrics as MadminScannerMetrics, ScannerPacingPressureSnapshot as MadminScannerPacingPressureSnapshot,
+    ScannerSourceCycleSnapshot as MadminScannerSourceCycleSnapshot, TimedAction as MadminTimedAction,
 };
 use rustfs_utils::os::get_drive_stats;
 use serde::{Deserialize, Serialize};
@@ -99,6 +100,26 @@ fn to_madmin_scanner_metrics(metrics: rustfs_common::metrics::ScannerMetricsRepo
                 .collect(),
         },
         active_paths: metrics.active_paths,
+        last_cycle_partial_source: metrics.last_cycle_partial_source,
+        last_cycle_partial_source_code: metrics.last_cycle_partial_source_code,
+        pacing_pressure: MadminScannerPacingPressureSnapshot {
+            primary_pressure: metrics.pacing_pressure.primary_pressure,
+            current_queued_scans: metrics.pacing_pressure.current_queued_scans,
+            current_active_scans: metrics.pacing_pressure.current_active_scans,
+            last_cycle_budget_limited: metrics.pacing_pressure.last_cycle_budget_limited,
+            last_cycle_pause_observed: metrics.pacing_pressure.last_cycle_pause_observed,
+            last_cycle_throttle_sleep_ratio: metrics.pacing_pressure.last_cycle_throttle_sleep_ratio,
+            last_cycle_yield_ratio: metrics.pacing_pressure.last_cycle_yield_ratio,
+            last_cycle_total_pause_ratio: metrics.pacing_pressure.last_cycle_total_pause_ratio,
+        },
+        partial_cycles_by_source: metrics
+            .partial_cycles_by_source
+            .into_iter()
+            .map(|source| MadminScannerSourceCycleSnapshot {
+                source: source.source,
+                cycles: source.cycles,
+            })
+            .collect(),
     }
 }
 
@@ -343,4 +364,52 @@ mod test {
 
         metrics.reset_for_test();
     }
+
+    #[test]
+    fn scanner_metrics_mapping_preserves_partial_source_status() {
+        let scanner = to_madmin_scanner_metrics(rustfs_common::metrics::ScannerMetricsReport {
+            last_cycle_partial_source: "usage".to_string(),
+            last_cycle_partial_source_code: 1,
+            partial_cycles_by_source: vec![rustfs_common::metrics::ScannerSourceCycleSnapshot {
+                source: "usage".to_string(),
+                cycles: 2,
+            }],
+            ..Default::default()
+        });
+
+        assert_eq!(scanner.last_cycle_partial_source, "usage");
+        assert_eq!(scanner.last_cycle_partial_source_code, 1);
+        let usage = scanner
+            .partial_cycles_by_source
+            .iter()
+            .find(|source| source.source == "usage")
+            .expect("usage partial source should be mapped");
+        assert_eq!(usage.cycles, 2);
+    }
+
+    #[test]
+    fn scanner_metrics_mapping_preserves_pacing_pressure() {
+        let scanner = to_madmin_scanner_metrics(rustfs_common::metrics::ScannerMetricsReport {
+            pacing_pressure: rustfs_common::metrics::ScannerPacingPressureSnapshot {
+                primary_pressure: "cycle_budget".to_string(),
+                current_queued_scans: 4,
+                current_active_scans: 2,
+                last_cycle_budget_limited: true,
+                last_cycle_pause_observed: true,
+                last_cycle_throttle_sleep_ratio: 0.25,
+                last_cycle_yield_ratio: 0.05,
+                last_cycle_total_pause_ratio: 0.3,
+            },
+            ..Default::default()
+        });
+
+        assert_eq!(scanner.pacing_pressure.primary_pressure, "cycle_budget");
+        assert_eq!(scanner.pacing_pressure.current_queued_scans, 4);
+        assert_eq!(scanner.pacing_pressure.current_active_scans, 2);
+        assert!(scanner.pacing_pressure.last_cycle_budget_limited);
+        assert!(scanner.pacing_pressure.last_cycle_pause_observed);
+        assert_eq!(scanner.pacing_pressure.last_cycle_throttle_sleep_ratio, 0.25);
+        assert_eq!(scanner.pacing_pressure.last_cycle_yield_ratio, 0.05);
+        assert_eq!(scanner.pacing_pressure.last_cycle_total_pause_ratio, 0.3);
+    }
 }

crates/ecstore/src/notification_sys.rs

@@ -25,15 +25,37 @@ use lazy_static::lazy_static;
 use rustfs_madmin::health::{Cpus, MemInfo, OsInfo, Partitions, ProcInfo, SysConfig, SysErrors, SysServices};
 use rustfs_madmin::metrics::RealtimeMetrics;
 use rustfs_madmin::net::NetInfo;
-use rustfs_madmin::{ItemState, ServerProperties};
+use rustfs_madmin::{ItemState, ServerProperties, StorageInfo};
 use std::collections::hash_map::DefaultHasher;
 use std::future::Future;
 use std::hash::{Hash, Hasher};
-use std::sync::OnceLock;
+use std::sync::{Mutex, OnceLock};
 use std::time::{Duration, SystemTime};
 use tokio::time::timeout;
 use tracing::{error, warn};
 
+/// After this many consecutive admin-call failures, mark the peer as offline.
+const CONSECUTIVE_FAILURE_THRESHOLD: u32 = 3;
+
+/// Cached result from the last successful admin call to a peer.
+struct PeerAdminCache {
+    last_storage_info: Option<StorageInfo>,
+    last_server_info: Option<ServerProperties>,
+    storage_failures: u32,
+    server_failures: u32,
+}
+
+impl PeerAdminCache {
+    fn new() -> Self {
+        Self {
+            last_storage_info: None,
+            last_server_info: None,
+            storage_failures: 0,
+            server_failures: 0,
+        }
+    }
+}
+
 lazy_static! {
     pub static ref GLOBAL_NotificationSys: OnceLock<NotificationSys> = OnceLock::new();
 }
@@ -53,14 +75,17 @@ pub struct NotificationSys {
     pub peer_clients: Vec<Option<PeerRestClient>>,
     #[allow(dead_code)]
     pub all_peer_clients: Vec<Option<PeerRestClient>>,
+    peer_admin_caches: Vec<Mutex<PeerAdminCache>>,
 }
 
 impl NotificationSys {
     pub async fn new(eps: EndpointServerPools) -> Self {
         let (peer_clients, all_peer_clients) = PeerRestClient::new_clients(eps).await;
+        let peer_admin_caches = (0..peer_clients.len()).map(|_| Mutex::new(PeerAdminCache::new())).collect();
         Self {
             peer_clients,
             all_peer_clients,
+            peer_admin_caches,
         }
     }
 }
@@ -189,33 +214,89 @@ impl NotificationSys {
         join_all(futures).await
     }
 
+    pub async fn reload_dynamic_config(&self, sub_sys: &str) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter() {
+            let sub_sys = sub_sys.to_string();
+            futures.push(async move {
+                if let Some(client) = client {
+                    match client
+                        .signal_service(crate::rpc::SERVICE_SIGNAL_RELOAD_DYNAMIC, &sub_sys, false, SystemTime::UNIX_EPOCH)
+                        .await
+                    {
+                        Ok(_) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: None,
+                        },
+                        Err(e) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: Some(e),
+                        },
+                    }
+                } else {
+                    NotificationPeerErr {
+                        host: "".to_string(),
+                        err: Some(Error::other("peer is not reachable")),
+                    }
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
+    pub async fn refresh_config_snapshot(&self) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter() {
+            futures.push(async move {
+                if let Some(client) = client {
+                    match client
+                        .signal_service(crate::rpc::SERVICE_SIGNAL_REFRESH_CONFIG, "", false, SystemTime::UNIX_EPOCH)
+                        .await
+                    {
+                        Ok(_) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: None,
+                        },
+                        Err(e) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: Some(e),
+                        },
+                    }
+                } else {
+                    NotificationPeerErr {
+                        host: "".to_string(),
+                        err: Some(Error::other("peer is not reachable")),
+                    }
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
     pub async fn storage_info<S: StorageAPI>(&self, api: &S) -> rustfs_madmin::StorageInfo {
         let mut futures = Vec::with_capacity(self.peer_clients.len());
         let endpoints = get_global_endpoints();
-        let peer_timeout = Duration::from_secs(2); // Same timeout as server_info
+        let peer_timeout = Duration::from_secs(5);
 
-        for client in self.peer_clients.iter() {
+        for (idx, client) in self.peer_clients.iter().enumerate() {
             let endpoints = endpoints.clone();
+            let cache = self.peer_admin_caches.get(idx);
             futures.push(async move {
                 if let Some(client) = client {
                     let host = client.host.to_string();
-                    // Wrap in timeout to ensure we don't hang on dead peers
                     match timeout(peer_timeout, client.local_storage_info()).await {
-                        Ok(Ok(info)) => Some(info),
+                        Ok(Ok(info)) => {
+                            update_storage_info_cache(cache, &host, &info);
+                            Some(info)
+                        }
                         Ok(Err(err)) => {
                             warn!("peer {} storage_info failed: {}", host, err);
-                            Some(rustfs_madmin::StorageInfo {
-                                disks: get_offline_disks(&host, &endpoints),
-                                ..Default::default()
-                            })
+                            handle_peer_failure(cache, &host, &endpoints)
                         }
                         Err(_) => {
                             warn!("peer {} storage_info timed out after {:?}", host, peer_timeout);
                             client.evict_connection().await;
-                            Some(rustfs_madmin::StorageInfo {
-                                disks: get_offline_disks(&host, &endpoints),
-                                ..Default::default()
-                            })
+                            handle_peer_failure(cache, &host, &endpoints)
                         }
                     }
                 } else {
@@ -240,24 +321,27 @@ impl NotificationSys {
     pub async fn server_info(&self) -> Vec<ServerProperties> {
         let mut futures = Vec::with_capacity(self.peer_clients.len());
         let endpoints = get_global_endpoints();
-        let peer_timeout = Duration::from_secs(2);
+        let peer_timeout = Duration::from_secs(5);
 
-        for client in self.peer_clients.iter() {
+        for (idx, client) in self.peer_clients.iter().enumerate() {
             let endpoints = endpoints.clone();
+            let cache = self.peer_admin_caches.get(idx);
             futures.push(async move {
                 if let Some(client) = client {
                     let host = client.host.to_string();
                     match timeout(peer_timeout, client.server_info()).await {
-                        Ok(Ok(info)) => info,
+                        Ok(Ok(info)) => {
+                            update_server_info_cache(cache, &host, &info);
+                            info
+                        }
                         Ok(Err(err)) => {
                             warn!("peer {} server_info failed: {}", host, err);
-                            // client.server_info handles eviction internally on error, but fallback needed
-                            offline_server_properties(&host, &endpoints)
+                            handle_server_info_failure(cache, &host, &endpoints)
                         }
                         Err(_) => {
                             warn!("peer {} server_info timed out after {:?}", host, peer_timeout);
                             client.evict_connection().await;
-                            offline_server_properties(&host, &endpoints)
+                            handle_server_info_failure(cache, &host, &endpoints)
                         }
                     }
                 } else {
@@ -782,6 +866,113 @@ where
     }
 }
 
+/// Handle a peer failure for storage_info: return cached data if available,
+/// or mark offline only after consecutive failures exceed the threshold.
+fn handle_peer_failure(
+    cache: Option<&Mutex<PeerAdminCache>>,
+    host: &str,
+    endpoints: &EndpointServerPools,
+) -> Option<StorageInfo> {
+    let cache = cache?;
+
+    let mut c = match cache.lock() {
+        Ok(cache) => cache,
+        Err(poisoned) => {
+            warn!("peer {host} storage_info cache mutex poisoned");
+            poisoned.into_inner()
+        }
+    };
+    c.storage_failures += 1;
+
+    if let Some(ref cached) = c.last_storage_info
+        && c.storage_failures < CONSECUTIVE_FAILURE_THRESHOLD
+    {
+        return Some(cached.clone());
+    }
+
+    if c.storage_failures >= CONSECUTIVE_FAILURE_THRESHOLD {
+        return Some(StorageInfo {
+            disks: get_offline_disks(host, endpoints),
+            ..Default::default()
+        });
+    }
+
+    None
+}
+
+fn update_storage_info_cache(cache: Option<&Mutex<PeerAdminCache>>, host: &str, info: &StorageInfo) {
+    let Some(cache) = cache else {
+        return;
+    };
+
+    let mut c = match cache.lock() {
+        Ok(cache) => cache,
+        Err(poisoned) => {
+            warn!("peer {host} storage_info cache mutex poisoned");
+            poisoned.into_inner()
+        }
+    };
+    c.last_storage_info = Some(info.clone());
+    c.storage_failures = 0;
+}
+
+/// Handle a peer failure for server_info: return cached data if available,
+/// or mark offline only after consecutive failures exceed the threshold.
+fn handle_server_info_failure(
+    cache: Option<&Mutex<PeerAdminCache>>,
+    host: &str,
+    endpoints: &EndpointServerPools,
+) -> ServerProperties {
+    let Some(cache) = cache else {
+        return initializing_server_properties(host);
+    };
+
+    let mut c = match cache.lock() {
+        Ok(cache) => cache,
+        Err(poisoned) => {
+            warn!("peer {host} server_info cache mutex poisoned");
+            poisoned.into_inner()
+        }
+    };
+    c.server_failures += 1;
+
+    if let Some(ref cached) = c.last_server_info
+        && c.server_failures < CONSECUTIVE_FAILURE_THRESHOLD
+    {
+        return cached.clone();
+    }
+
+    if c.server_failures >= CONSECUTIVE_FAILURE_THRESHOLD {
+        return offline_server_properties(host, endpoints);
+    }
+
+    initializing_server_properties(host)
+}
+
+fn update_server_info_cache(cache: Option<&Mutex<PeerAdminCache>>, host: &str, info: &ServerProperties) {
+    let Some(cache) = cache else {
+        return;
+    };
+
+    let mut c = match cache.lock() {
+        Ok(cache) => cache,
+        Err(poisoned) => {
+            warn!("peer {host} server_info cache mutex poisoned");
+            poisoned.into_inner()
+        }
+    };
+    c.last_server_info = Some(info.clone());
+    c.server_failures = 0;
+}
+
+fn initializing_server_properties(host: &str) -> ServerProperties {
+    ServerProperties {
+        endpoint: host.to_string(),
+        state: ItemState::Initializing.to_string().to_owned(),
+        ..Default::default()
+    }
+}
+
 fn offline_server_properties(host: &str, endpoints: &EndpointServerPools) -> ServerProperties {
     ServerProperties {
         uptime: GLOBAL_BOOT_TIME
@@ -905,6 +1096,7 @@ mod tests {
         let sys = NotificationSys {
             peer_clients: vec![None],
             all_peer_clients: Vec::new(),
+            peer_admin_caches: vec![Mutex::new(PeerAdminCache::new())],
         };
 
         let err = sys
@@ -923,6 +1115,7 @@ mod tests {
         let sys = NotificationSys {
             peer_clients: vec![None],
             all_peer_clients: Vec::new(),
+            peer_admin_caches: vec![Mutex::new(PeerAdminCache::new())],
         };
 
         let results = sys.load_transition_tier_config().await;
@@ -931,4 +1124,243 @@ mod tests {
         assert!(results[0].err.is_some());
         assert!(results[0].err.as_ref().unwrap().to_string().contains("peer is not reachable"));
     }
+
+    // --- Tests for handle_peer_failure / handle_server_info_failure caching ---
+
+    #[test]
+    fn handle_peer_failure_first_failure_returns_none_when_no_cache() {
+        let cache = Mutex::new(PeerAdminCache::new());
+        let endpoints = EndpointServerPools::default();
+
+        let result = handle_peer_failure(Some(&cache), "peer-1", &endpoints);
+        assert!(result.is_none());
+        assert_eq!(cache.lock().unwrap().storage_failures, 1);
+    }
+
+    #[test]
+    fn handle_peer_failure_returns_cached_data_on_single_failure() {
+        let cached_info = StorageInfo {
+            disks: vec![rustfs_madmin::Disk {
+                endpoint: "disk-0".to_string(),
+                state: "ok".to_string(),
+                ..Default::default()
+            }],
+            ..Default::default()
+        };
+
+        let cache = Mutex::new(PeerAdminCache {
+            last_storage_info: Some(cached_info),
+            last_server_info: None,
+            storage_failures: 0,
+            server_failures: 0,
+        });
+        let endpoints = EndpointServerPools::default();
+
+        // First failure: should return cached data
+        let result = handle_peer_failure(Some(&cache), "peer-1", &endpoints);
+        let info = result.unwrap();
+        assert_eq!(info.disks.len(), 1);
+        assert_eq!(info.disks[0].state, "ok");
+        assert_eq!(cache.lock().unwrap().storage_failures, 1);
+    }
+
+    #[test]
+    fn handle_peer_failure_returns_offline_after_threshold_exceeded() {
+        let cached_info = StorageInfo {
+            disks: vec![rustfs_madmin::Disk {
+                endpoint: "disk-0".to_string(),
+                state: "ok".to_string(),
+                ..Default::default()
+            }],
+            ..Default::default()
+        };
+
+        let cache = Mutex::new(PeerAdminCache {
+            last_storage_info: Some(cached_info),
+            last_server_info: None,
+            storage_failures: CONSECUTIVE_FAILURE_THRESHOLD - 1,
+            server_failures: 0,
+        });
+        let endpoints = EndpointServerPools::default();
+
+        // This failure pushes us to the threshold => offline
+        let result = handle_peer_failure(Some(&cache), "peer-1", &endpoints);
+        assert!(result.is_some());
+        assert_eq!(cache.lock().unwrap().storage_failures, CONSECUTIVE_FAILURE_THRESHOLD);
+    }
+
+    #[test]
+    fn handle_server_info_failure_returns_cached_on_single_failure() {
+        let cached_props = ServerProperties {
+            endpoint: "peer-1".to_string(),
+            state: "online".to_string(),
+            ..Default::default()
+        };
+
+        let cache = Mutex::new(PeerAdminCache {
+            last_storage_info: None,
+            last_server_info: Some(cached_props),
+            storage_failures: 0,
+            server_failures: 0,
+        });
+        let endpoints = EndpointServerPools::default();
+
+        let result = handle_server_info_failure(Some(&cache), "peer-1", &endpoints);
+        assert_eq!(result.endpoint, "peer-1");
+        assert_eq!(result.state, "online");
+        assert_eq!(cache.lock().unwrap().server_failures, 1);
+    }
+
+    #[test]
+    fn handle_server_info_failure_returns_initializing_before_threshold_without_cache() {
+        let cache = Mutex::new(PeerAdminCache::new());
+        let endpoints = EndpointServerPools::default();
+
+        let result = handle_server_info_failure(Some(&cache), "peer-1", &endpoints);
+        assert_eq!(result.endpoint, "peer-1");
+        assert_eq!(result.state, ItemState::Initializing.to_string());
+        assert!(result.disks.is_empty());
+        assert_eq!(cache.lock().unwrap().server_failures, 1);
+    }
+
+    #[test]
+    fn handle_server_info_failure_returns_offline_after_threshold() {
+        let cached_props = ServerProperties {
+            endpoint: "peer-1".to_string(),
+            state: "online".to_string(),
+            ..Default::default()
+        };
+
+        let cache = Mutex::new(PeerAdminCache {
+            last_storage_info: None,
+            last_server_info: Some(cached_props),
+            storage_failures: 0,
+            server_failures: CONSECUTIVE_FAILURE_THRESHOLD - 1,
+        });
+        let endpoints = EndpointServerPools::default();
+
+        let result = handle_server_info_failure(Some(&cache), "peer-1", &endpoints);
+        assert_eq!(result.state, ItemState::Offline.to_string());
+        assert_eq!(cache.lock().unwrap().server_failures, CONSECUTIVE_FAILURE_THRESHOLD);
+    }
+
+    #[test]
+    fn success_resets_failure_counters_independently() {
+        let cache = Mutex::new(PeerAdminCache {
+            last_storage_info: None,
+            last_server_info: None,
+            storage_failures: 2,
+            server_failures: 2,
+        });
+
+        {
+            let mut c = cache.lock().unwrap();
+            c.last_storage_info = Some(StorageInfo::default());
+            c.storage_failures = 0;
+        }
+
+        let cache = cache.lock().unwrap();
+        assert_eq!(cache.storage_failures, 0);
+        assert_eq!(cache.server_failures, 2);
+    }
+
+    #[test]
+    fn storage_failures_do_not_affect_server_failures() {
+        let cache = Mutex::new(PeerAdminCache {
+            last_storage_info: Some(StorageInfo::default()),
+            last_server_info: Some(ServerProperties {
+                endpoint: "peer-1".to_string(),
+                state: "online".to_string(),
+                ..Default::default()
+            }),
+            storage_failures: CONSECUTIVE_FAILURE_THRESHOLD - 1,
+            server_failures: 0,
+        });
+        let endpoints = EndpointServerPools::default();
+
+        let storage_result = handle_peer_failure(Some(&cache), "peer-1", &endpoints);
+        assert!(storage_result.is_some());
+
+        let server_result = handle_server_info_failure(Some(&cache), "peer-1", &endpoints);
+        assert_eq!(server_result.state, "online");
+        assert_eq!(cache.lock().unwrap().server_failures, 1);
+    }
+
+    #[test]
+    fn poisoned_admin_cache_mutex_still_returns_fallbacks() {
+        let storage_cache = Mutex::new(PeerAdminCache::new());
+        let server_cache = Mutex::new(PeerAdminCache::new());
+        let endpoints = EndpointServerPools::default();
+
+        let _ = std::panic::catch_unwind(|| {
+            let _guard = storage_cache.lock().expect("test: poison storage cache mutex");
+            panic!("poison storage cache mutex");
+        });
+        let _ = std::panic::catch_unwind(|| {
+            let _guard = server_cache.lock().expect("test: poison server cache mutex");
+            panic!("poison server cache mutex");
+        });
+
+        let storage_result = handle_peer_failure(Some(&storage_cache), "peer-1", &endpoints);
+        assert!(storage_result.is_none());
+
+        let server_result = handle_server_info_failure(Some(&server_cache), "peer-1", &endpoints);
+        assert_eq!(server_result.endpoint, "peer-1");
+        assert_eq!(server_result.state, ItemState::Initializing.to_string());
+    }
+
+    #[test]
+    fn poisoned_admin_cache_recovers_on_success_and_resets_failures() {
+        let storage_cache = Mutex::new(PeerAdminCache {
+            last_storage_info: None,
+            last_server_info: None,
+            storage_failures: CONSECUTIVE_FAILURE_THRESHOLD - 1,
+            server_failures: 0,
+        });
+        let server_cache = Mutex::new(PeerAdminCache {
+            last_storage_info: None,
+            last_server_info: None,
+            storage_failures: 0,
+            server_failures: CONSECUTIVE_FAILURE_THRESHOLD - 1,
+        });
+        let endpoints = EndpointServerPools::default();
+
+        let _ = std::panic::catch_unwind(|| {
+            let _guard = storage_cache.lock().expect("test: poison storage cache mutex");
+            panic!("poison storage cache mutex");
+        });
+        let _ = std::panic::catch_unwind(|| {
+            let _guard = server_cache.lock().expect("test: poison server cache mutex");
+            panic!("poison server cache mutex");
+        });
+
+        update_storage_info_cache(
+            Some(&storage_cache),
+            "peer-1",
+            &StorageInfo {
+                disks: vec![rustfs_madmin::Disk {
+                    endpoint: "disk-0".to_string(),
+                    state: "ok".to_string(),
+                    ..Default::default()
+                }],
+                ..Default::default()
+            },
+        );
+        update_server_info_cache(
+            Some(&server_cache),
+            "peer-1",
+            &ServerProperties {
+                endpoint: "peer-1".to_string(),
+                state: "online".to_string(),
+                ..Default::default()
+            },
+        );
+
+        let storage_result = handle_peer_failure(Some(&storage_cache), "peer-1", &endpoints);
+        assert!(storage_result.is_some());
+        assert_eq!(storage_result.unwrap().disks[0].state, "ok");
+
+        let server_result = handle_server_info_failure(Some(&server_cache), "peer-1", &endpoints);
+        assert_eq!(server_result.state, "online");
+    }
 }

crates/ecstore/src/rio.rs

@@ -0,0 +1,767 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#[cfg(feature = "rio-v2")]
+pub use rustfs_rio_v2::*;
+
+#[cfg(not(feature = "rio-v2"))]
+pub use rustfs_rio::*;
+
+use bytes::Bytes;
+use rustfs_utils::CompressionAlgorithm;
+use std::str::FromStr;
+use tokio::io::AsyncRead;
+
+#[cfg(feature = "rio-v2")]
+const MINIO_S2_COMPRESSION_SCHEME: &str = "klauspost/compress/s2";
+#[cfg(feature = "rio-v2")]
+const ENCRYPTED_S2_PADDING_MULTIPLE: usize = 256;
+
+pub const fn backend_name() -> &'static str {
+    #[cfg(feature = "rio-v2")]
+    {
+        "rio-v2"
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        "legacy-rio"
+    }
+}
+
+pub fn compression_metadata_value(algorithm: CompressionAlgorithm) -> String {
+    #[cfg(feature = "rio-v2")]
+    {
+        let _ = algorithm;
+        MINIO_S2_COMPRESSION_SCHEME.to_string()
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        algorithm.to_string()
+    }
+}
+
+pub fn compression_scheme_to_algorithm(scheme: &str) -> std::io::Result<CompressionAlgorithm> {
+    #[cfg(feature = "rio-v2")]
+    if scheme.eq_ignore_ascii_case(MINIO_S2_COMPRESSION_SCHEME) {
+        // rio_v2 currently routes all compressed-object handling through the S2
+        // reader implementation, so the enum is only a placeholder token here.
+        return Ok(CompressionAlgorithm::default());
+    }
+
+    CompressionAlgorithm::from_str(scheme)
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ReadCompressionBackend {
+    Legacy,
+    V2,
+}
+
+pub fn compression_scheme_to_read_plan(scheme: &str) -> std::io::Result<(CompressionAlgorithm, ReadCompressionBackend)> {
+    #[cfg(feature = "rio-v2")]
+    if scheme.eq_ignore_ascii_case(MINIO_S2_COMPRESSION_SCHEME) {
+        return Ok((CompressionAlgorithm::default(), ReadCompressionBackend::V2));
+    }
+
+    Ok((CompressionAlgorithm::from_str(scheme)?, ReadCompressionBackend::Legacy))
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ReadEncryptionBackend {
+    Legacy,
+    V2,
+}
+
+pub fn compression_index_storage_bytes(index: &Index) -> Bytes {
+    #[cfg(feature = "rio-v2")]
+    {
+        minio_index_storage_bytes(index)
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        index.clone().into_vec()
+    }
+}
+
+pub fn decode_compression_index_bytes(bytes: &Bytes) -> Option<Index> {
+    #[cfg(feature = "rio-v2")]
+    {
+        if let Some(decoded) = decode_minio_index_bytes(bytes) {
+            return Some(decoded);
+        }
+    }
+
+    let mut decoded = Index::new();
+    if decoded.load(bytes.as_ref()).is_ok() {
+        return Some(decoded);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    {
+        let restored = restore_legacy_index_headers(bytes.as_ref());
+        let mut decoded = Index::new();
+        if decoded.load(&restored).is_ok() {
+            return Some(decoded);
+        }
+    }
+
+    None
+}
+
+pub fn compression_reader<R>(reader: R, algorithm: CompressionAlgorithm, encrypted: bool) -> CompressReader<R>
+where
+    R: AsyncRead + Unpin + Send + Sync,
+{
+    #[cfg(feature = "rio-v2")]
+    {
+        if encrypted {
+            return CompressReader::with_encrypted_padding(reader, algorithm);
+        }
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    let _ = encrypted;
+
+    CompressReader::new(reader, algorithm)
+}
+
+pub fn decompression_reader<R>(
+    reader: R,
+    algorithm: CompressionAlgorithm,
+    backend: ReadCompressionBackend,
+) -> Box<dyn AsyncRead + Unpin + Send + Sync>
+where
+    R: AsyncRead + Unpin + Send + Sync + 'static,
+{
+    #[cfg(feature = "rio-v2")]
+    {
+        match backend {
+            ReadCompressionBackend::Legacy => Box::new(rustfs_rio::DecompressReader::new(reader, algorithm)),
+            ReadCompressionBackend::V2 => Box::new(rustfs_rio_v2::DecompressReader::new(reader, algorithm)),
+        }
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        let _ = backend;
+        Box::new(rustfs_rio::DecompressReader::new(reader, algorithm))
+    }
+}
+
+pub fn decrypt_reader<R>(
+    reader: R,
+    key: [u8; 32],
+    base_nonce: [u8; 12],
+    backend: ReadEncryptionBackend,
+    sequence_number: u32,
+) -> Box<dyn AsyncRead + Unpin + Send + Sync>
+where
+    R: AsyncRead + Unpin + Send + Sync + 'static,
+{
+    #[cfg(feature = "rio-v2")]
+    {
+        match backend {
+            ReadEncryptionBackend::Legacy => Box::new(rustfs_rio::DecryptReader::new(reader, key, base_nonce)),
+            ReadEncryptionBackend::V2 => {
+                Box::new(rustfs_rio_v2::DecryptReader::new_with_sequence(reader, key, base_nonce, sequence_number))
+            }
+        }
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        let _ = (backend, sequence_number);
+        Box::new(rustfs_rio::DecryptReader::new(reader, key, base_nonce))
+    }
+}
+
+pub fn decrypt_reader_with_object_key<R>(
+    reader: R,
+    object_key: [u8; 32],
+    sequence_number: u32,
+) -> Box<dyn AsyncRead + Unpin + Send + Sync>
+where
+    R: AsyncRead + Unpin + Send + Sync + 'static,
+{
+    #[cfg(feature = "rio-v2")]
+    {
+        Box::new(rustfs_rio_v2::DecryptReader::new_with_object_key_and_sequence(
+            reader,
+            object_key,
+            sequence_number,
+        ))
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        let _ = sequence_number;
+        Box::new(rustfs_rio::DecryptReader::new(reader, object_key, [0u8; 12]))
+    }
+}
+
+pub fn decrypt_multipart_reader<R>(
+    reader: R,
+    key: [u8; 32],
+    base_nonce: [u8; 12],
+    multipart_parts: Vec<usize>,
+    backend: ReadEncryptionBackend,
+    sequence_number: u32,
+) -> Box<dyn AsyncRead + Unpin + Send + Sync>
+where
+    R: AsyncRead + Unpin + Send + Sync + 'static,
+{
+    #[cfg(feature = "rio-v2")]
+    {
+        match backend {
+            ReadEncryptionBackend::Legacy => {
+                Box::new(rustfs_rio::DecryptReader::new_multipart(reader, key, base_nonce, multipart_parts))
+            }
+            ReadEncryptionBackend::V2 => Box::new(rustfs_rio_v2::DecryptReader::new_multipart_with_sequence(
+                reader,
+                key,
+                base_nonce,
+                multipart_parts,
+                sequence_number,
+            )),
+        }
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        let _ = (backend, sequence_number);
+        Box::new(rustfs_rio::DecryptReader::new_multipart(reader, key, base_nonce, multipart_parts))
+    }
+}
+
+pub fn decrypt_multipart_reader_with_object_key<R>(
+    reader: R,
+    object_key: [u8; 32],
+    multipart_parts: Vec<usize>,
+    sequence_number: u32,
+) -> Box<dyn AsyncRead + Unpin + Send + Sync>
+where
+    R: AsyncRead + Unpin + Send + Sync + 'static,
+{
+    #[cfg(feature = "rio-v2")]
+    {
+        Box::new(rustfs_rio_v2::DecryptReader::new_multipart_with_object_key_and_sequence(
+            reader,
+            object_key,
+            multipart_parts,
+            sequence_number,
+        ))
+    }
+
+    #[cfg(not(feature = "rio-v2"))]
+    {
+        let _ = sequence_number;
+        Box::new(rustfs_rio::DecryptReader::new_multipart(reader, object_key, [0u8; 12], multipart_parts))
+    }
+}
+
+#[cfg(feature = "rio-v2")]
+fn restore_legacy_index_headers(bytes: &[u8]) -> Vec<u8> {
+    if bytes.is_empty() {
+        return Vec::new();
+    }
+
+    const S2_INDEX_HEADER: &[u8] = b"s2idx\x00";
+    const S2_INDEX_TRAILER: &[u8] = b"\x00xdi2s";
+    let mut restored = Vec::with_capacity(4 + S2_INDEX_HEADER.len() + bytes.len() + 4 + S2_INDEX_TRAILER.len());
+    restored.extend_from_slice(&[0x99, 0x2A, 0x4D, 0x18]);
+    restored.extend_from_slice(S2_INDEX_HEADER);
+    restored.extend_from_slice(bytes);
+
+    let total_size = (restored.len() + 4 + S2_INDEX_TRAILER.len()) as u32;
+    restored.extend_from_slice(&total_size.to_le_bytes());
+    restored.extend_from_slice(S2_INDEX_TRAILER);
+
+    let chunk_len = restored.len() - 4;
+    restored[1] = chunk_len as u8;
+    restored[2] = (chunk_len >> 8) as u8;
+    restored[3] = (chunk_len >> 16) as u8;
+    restored
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct WriteEncryption {
+    key_bytes: [u8; 32],
+    mode: WriteEncryptionMode,
+}
+
+#[derive(Debug, Clone, Copy)]
+enum WriteEncryptionMode {
+    SinglepartObjectKey,
+    Singlepart {
+        base_nonce: [u8; 12],
+    },
+    MultipartLegacy {
+        base_nonce: [u8; 12],
+        multipart_part_number: usize,
+    },
+    MultipartObjectKey {
+        multipart_part_number: u32,
+    },
+}
+
+impl WriteEncryption {
+    pub const fn singlepart_object_key(object_key: [u8; 32]) -> Self {
+        Self {
+            key_bytes: object_key,
+            mode: WriteEncryptionMode::SinglepartObjectKey,
+        }
+    }
+
+    pub const fn singlepart(key_bytes: [u8; 32], base_nonce: [u8; 12]) -> Self {
+        Self {
+            key_bytes,
+            mode: WriteEncryptionMode::Singlepart { base_nonce },
+        }
+    }
+
+    pub const fn multipart(key_bytes: [u8; 32], base_nonce: [u8; 12], multipart_part_number: usize) -> Self {
+        Self {
+            key_bytes,
+            mode: WriteEncryptionMode::MultipartLegacy {
+                base_nonce,
+                multipart_part_number,
+            },
+        }
+    }
+
+    pub const fn multipart_object_key(object_key: [u8; 32], multipart_part_number: u32) -> Self {
+        Self {
+            key_bytes: object_key,
+            mode: WriteEncryptionMode::MultipartObjectKey { multipart_part_number },
+        }
+    }
+}
+
+#[derive(Debug, Clone, Default)]
+pub struct WritePlan {
+    compression: Option<CompressionAlgorithm>,
+    encryption: Option<WriteEncryption>,
+}
+
+impl WritePlan {
+    pub const fn new() -> Self {
+        Self {
+            compression: None,
+            encryption: None,
+        }
+    }
+
+    pub const fn with_compression(mut self, algorithm: CompressionAlgorithm) -> Self {
+        self.compression = Some(algorithm);
+        self
+    }
+
+    pub const fn with_encryption(mut self, encryption: WriteEncryption) -> Self {
+        self.encryption = Some(encryption);
+        self
+    }
+
+    pub const fn is_passthrough(&self) -> bool {
+        self.compression.is_none() && self.encryption.is_none()
+    }
+
+    pub fn apply(self, mut reader: HashReader, actual_size: i64) -> std::io::Result<HashReader> {
+        let encrypted = self.encryption.is_some();
+        if let Some(algorithm) = self.compression {
+            reader = HashReader::from_reader(
+                compression_reader(reader, algorithm, encrypted),
+                HashReader::SIZE_PRESERVE_LAYER,
+                actual_size,
+                None,
+                None,
+                false,
+            )?;
+        }
+
+        if let Some(encryption) = self.encryption {
+            reader = match encryption.mode {
+                WriteEncryptionMode::SinglepartObjectKey => HashReader::from_reader(
+                    #[cfg(feature = "rio-v2")]
+                    EncryptReader::new_with_object_key(reader, encryption.key_bytes),
+                    #[cfg(not(feature = "rio-v2"))]
+                    EncryptReader::new(reader, encryption.key_bytes, [0u8; 12]),
+                    HashReader::SIZE_PRESERVE_LAYER,
+                    actual_size,
+                    None,
+                    None,
+                    false,
+                )?,
+                WriteEncryptionMode::Singlepart { base_nonce } => HashReader::from_reader(
+                    EncryptReader::new(reader, encryption.key_bytes, base_nonce),
+                    HashReader::SIZE_PRESERVE_LAYER,
+                    actual_size,
+                    None,
+                    None,
+                    false,
+                )?,
+                WriteEncryptionMode::MultipartLegacy {
+                    base_nonce,
+                    multipart_part_number,
+                } => HashReader::from_reader(
+                    EncryptReader::new_multipart(reader, encryption.key_bytes, base_nonce, multipart_part_number),
+                    HashReader::SIZE_PRESERVE_LAYER,
+                    actual_size,
+                    None,
+                    None,
+                    false,
+                )?,
+                WriteEncryptionMode::MultipartObjectKey { multipart_part_number } => HashReader::from_reader(
+                    #[cfg(feature = "rio-v2")]
+                    EncryptReader::new_multipart_with_object_key(reader, encryption.key_bytes, multipart_part_number),
+                    #[cfg(not(feature = "rio-v2"))]
+                    EncryptReader::new_multipart(reader, encryption.key_bytes, [0u8; 12], multipart_part_number as usize),
+                    HashReader::SIZE_PRESERVE_LAYER,
+                    actual_size,
+                    None,
+                    None,
+                    false,
+                )?,
+            };
+        }
+
+        Ok(reader)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use rustfs_utils::CompressionAlgorithm;
+    use std::io::Cursor;
+    use tokio::io::AsyncReadExt;
+
+    #[cfg(feature = "rio-v2")]
+    fn s2_chunk_types(stream: &[u8]) -> Vec<u8> {
+        let mut chunk_types = Vec::new();
+        let mut offset = 0usize;
+        while offset + 4 <= stream.len() {
+            let chunk_type = stream[offset];
+            let chunk_len =
+                (stream[offset + 1] as usize) | ((stream[offset + 2] as usize) << 8) | ((stream[offset + 3] as usize) << 16);
+            chunk_types.push(chunk_type);
+            offset += 4 + chunk_len;
+        }
+        chunk_types
+    }
+
+    #[tokio::test]
+    async fn write_plan_passthrough_keeps_plaintext() {
+        let plaintext = b"write-plan-plain".to_vec();
+        let reader = HashReader::from_stream(
+            Cursor::new(plaintext.clone()),
+            plaintext.len() as i64,
+            plaintext.len() as i64,
+            None,
+            None,
+            false,
+        )
+        .expect("create hash reader");
+
+        let mut reader = WritePlan::new()
+            .apply(reader, plaintext.len() as i64)
+            .expect("apply passthrough plan");
+
+        let mut actual = Vec::new();
+        reader.read_to_end(&mut actual).await.expect("read passthrough stream");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[tokio::test]
+    async fn write_plan_compress_then_encrypt_multipart_roundtrip() {
+        let plaintext = b"abcdefghijklmnopqrstuvwxyz".repeat(128);
+        let actual_size = plaintext.len() as i64;
+        let key_bytes = [0x5Au8; 32];
+        let base_nonce = [0xA5u8; 12];
+        let part_number = 7;
+
+        let reader = HashReader::from_stream(Cursor::new(plaintext.clone()), actual_size, actual_size, None, None, false)
+            .expect("create hash reader");
+
+        let mut transformed = WritePlan::new()
+            .with_compression(CompressionAlgorithm::default())
+            .with_encryption(WriteEncryption::multipart(key_bytes, base_nonce, part_number))
+            .apply(reader, actual_size)
+            .expect("apply transform plan");
+
+        let mut ciphertext = Vec::new();
+        transformed
+            .read_to_end(&mut ciphertext)
+            .await
+            .expect("read transformed ciphertext");
+
+        let decrypt_reader = DecryptReader::new_multipart(Cursor::new(ciphertext), key_bytes, base_nonce, vec![part_number]);
+        let mut decompressed = DecompressReader::new(Box::new(decrypt_reader), CompressionAlgorithm::default());
+
+        let mut actual = Vec::new();
+        decompressed
+            .read_to_end(&mut actual)
+            .await
+            .expect("decrypt and decompress transformed stream");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn write_plan_supports_singlepart_object_key_encryption_roundtrip() {
+        let plaintext = b"singlepart-object-key".repeat(512);
+        let actual_size = plaintext.len() as i64;
+        let object_key = [0x7Cu8; 32];
+
+        let reader = HashReader::from_stream(Cursor::new(plaintext.clone()), actual_size, actual_size, None, None, false)
+            .expect("create hash reader");
+
+        let mut transformed = WritePlan::new()
+            .with_encryption(WriteEncryption::singlepart_object_key(object_key))
+            .apply(reader, actual_size)
+            .expect("apply singlepart object-key plan");
+
+        let mut encrypted = Vec::new();
+        transformed
+            .read_to_end(&mut encrypted)
+            .await
+            .expect("read encrypted object-key stream");
+
+        let mut decrypted = DecryptReader::new_with_object_key(Cursor::new(encrypted), object_key);
+        let mut actual = Vec::new();
+        decrypted.read_to_end(&mut actual).await.expect("decrypt object-key stream");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn write_plan_supports_multipart_object_key_encryption_roundtrip() {
+        let plaintext = b"multipart-object-key-".repeat(4096);
+        let actual_size = plaintext.len() as i64;
+        let object_key = [0x2Du8; 32];
+        let part_number = 3u32;
+
+        let reader = HashReader::from_stream(Cursor::new(plaintext.clone()), actual_size, actual_size, None, None, false)
+            .expect("create hash reader");
+
+        let mut transformed = WritePlan::new()
+            .with_encryption(WriteEncryption::multipart_object_key(object_key, part_number))
+            .apply(reader, actual_size)
+            .expect("apply multipart object-key encryption");
+
+        let mut ciphertext = Vec::new();
+        transformed
+            .read_to_end(&mut ciphertext)
+            .await
+            .expect("read multipart object-key ciphertext");
+
+        let mut actual = Vec::new();
+        DecryptReader::new_multipart_with_object_key(Cursor::new(ciphertext), object_key, vec![part_number as usize])
+            .read_to_end(&mut actual)
+            .await
+            .expect("decrypt multipart object-key ciphertext");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn write_plan_rio_v2_compression_emits_s2_stream_and_seekable_index() {
+        let plaintext = b"rustfs-rio-v2-s2-".repeat(600_000);
+        let actual_size = plaintext.len() as i64;
+        let reader = HashReader::from_stream(Cursor::new(plaintext.clone()), actual_size, actual_size, None, None, false)
+            .expect("create hash reader");
+
+        let mut transformed = WritePlan::new()
+            .with_compression(CompressionAlgorithm::default())
+            .apply(reader, actual_size)
+            .expect("apply compression plan");
+
+        let mut compressed = Vec::new();
+        transformed
+            .read_to_end(&mut compressed)
+            .await
+            .expect("read compressed stream");
+
+        assert!(
+            compressed.starts_with(b"\xff\x06\x00\x00S2sTwO"),
+            "rio_v2 compressed stream must start with the S2 stream identifier"
+        );
+
+        let index = transformed
+            .try_get_index()
+            .cloned()
+            .expect("rio_v2 compressed stream should expose a compression index");
+        let (compressed_offset, uncompressed_offset) = index.find(2 * 1024 * 1024).expect("seek into compression index");
+
+        assert!(compressed_offset > 0, "expected a non-zero compressed offset for the second block");
+        assert!(uncompressed_offset > 0, "expected a non-zero uncompressed offset for the second block");
+
+        let mut decompressed = DecompressReader::new(Cursor::new(compressed), CompressionAlgorithm::default());
+        let mut actual = Vec::new();
+        decompressed.read_to_end(&mut actual).await.expect("decompress rio_v2 stream");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn write_plan_rio_v2_small_compression_skips_index_below_minio_threshold() {
+        let plaintext = b"rustfs-rio-v2-s2-".repeat(32_768);
+        let actual_size = plaintext.len() as i64;
+        let reader = HashReader::from_stream(Cursor::new(plaintext.clone()), actual_size, actual_size, None, None, false)
+            .expect("create hash reader");
+
+        let mut transformed = WritePlan::new()
+            .with_compression(CompressionAlgorithm::default())
+            .apply(reader, actual_size)
+            .expect("apply compression plan");
+
+        let mut compressed = Vec::new();
+        transformed
+            .read_to_end(&mut compressed)
+            .await
+            .expect("read compressed stream");
+
+        assert!(
+            transformed.try_get_index().is_none(),
+            "rio_v2 should match MinIO and skip compression indexes for small objects"
+        );
+
+        let mut decompressed = DecompressReader::new(Cursor::new(compressed), CompressionAlgorithm::default());
+        let mut actual = Vec::new();
+        decompressed.read_to_end(&mut actual).await.expect("decompress rio_v2 stream");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn rio_v2_singlepart_encrypt_decrypt_roundtrip_preserves_small_compressed_stream() {
+        let plaintext = b"abcdefghijklmnopqrstuvwxyz".to_vec();
+        let key_bytes = [0x33u8; 32];
+        let base_nonce = [0x55u8; 12];
+
+        let mut compressed = Vec::new();
+        CompressReader::new(Cursor::new(plaintext.clone()), CompressionAlgorithm::default())
+            .read_to_end(&mut compressed)
+            .await
+            .expect("compress plaintext");
+
+        let mut encrypted = Vec::new();
+        EncryptReader::new(Cursor::new(compressed), key_bytes, base_nonce)
+            .read_to_end(&mut encrypted)
+            .await
+            .expect("encrypt compressed stream");
+
+        let decrypt_reader = DecryptReader::new(Cursor::new(encrypted), key_bytes, base_nonce);
+        let mut decompressed = DecompressReader::new(Box::new(decrypt_reader), CompressionAlgorithm::default());
+        let mut actual = Vec::new();
+        decompressed
+            .read_to_end(&mut actual)
+            .await
+            .expect("decrypt and decompress small stream");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn rio_v2_compress_then_encrypt_adds_s2_padding_frames() {
+        let plaintext = b"padding-check-".repeat(4097);
+        let actual_size = plaintext.len() as i64;
+        let key_bytes = [0x1Bu8; 32];
+        let base_nonce = [0xC4u8; 12];
+
+        let reader = HashReader::from_stream(Cursor::new(plaintext.clone()), actual_size, actual_size, None, None, false)
+            .expect("create hash reader");
+
+        let mut transformed = WritePlan::new()
+            .with_compression(CompressionAlgorithm::default())
+            .with_encryption(WriteEncryption::singlepart(key_bytes, base_nonce))
+            .apply(reader, actual_size)
+            .expect("apply transform plan");
+
+        let mut ciphertext = Vec::new();
+        transformed
+            .read_to_end(&mut ciphertext)
+            .await
+            .expect("read transformed ciphertext");
+
+        let mut decrypted_compressed = Vec::new();
+        DecryptReader::new(Cursor::new(ciphertext), key_bytes, base_nonce)
+            .read_to_end(&mut decrypted_compressed)
+            .await
+            .expect("decrypt compressed stream");
+
+        assert_eq!(decrypted_compressed.len() % ENCRYPTED_S2_PADDING_MULTIPLE, 0);
+
+        let chunk_types = s2_chunk_types(&decrypted_compressed);
+        assert!(
+            chunk_types.contains(&0xfe),
+            "rio_v2 compressed+encrypted streams must include S2 padding frames before encryption"
+        );
+
+        let mut actual = Vec::new();
+        DecompressReader::new(Cursor::new(decrypted_compressed), CompressionAlgorithm::default())
+            .read_to_end(&mut actual)
+            .await
+            .expect("decompress padded stream");
+
+        assert_eq!(actual, plaintext);
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn rio_v2_decompress_reader_returns_bytes_on_first_read() {
+        let plaintext = b"abcdefghijklmnopqrstuvwxyz".to_vec();
+
+        let mut compressed = Vec::new();
+        CompressReader::new(Cursor::new(plaintext.clone()), CompressionAlgorithm::default())
+            .read_to_end(&mut compressed)
+            .await
+            .expect("compress plaintext");
+
+        let mut decompressor = DecompressReader::new(Cursor::new(compressed), CompressionAlgorithm::default());
+        let mut buf = [0u8; 64];
+        let n = decompressor.read(&mut buf).await.expect("read first decompressed chunk");
+
+        assert!(n > 0);
+        assert_eq!(&buf[..n], plaintext.as_slice());
+    }
+
+    #[cfg(feature = "rio-v2")]
+    #[tokio::test]
+    async fn rio_v2_decompress_reader_returns_bytes_on_first_large_read() {
+        let plaintext = b"abcdefghijklmnopqrstuvwxyz".to_vec();
+
+        let mut compressed = Vec::new();
+        CompressReader::new(Cursor::new(plaintext.clone()), CompressionAlgorithm::default())
+            .read_to_end(&mut compressed)
+            .await
+            .expect("compress plaintext");
+
+        let mut decompressor = DecompressReader::new(Cursor::new(compressed), CompressionAlgorithm::default());
+        let mut buf = [0u8; 8192];
+        let n = decompressor.read(&mut buf).await.expect("read first decompressed chunk");
+
+        assert!(n > 0);
+        assert_eq!(&buf[..n], plaintext.as_slice());
+    }
+}

crates/ecstore/src/rpc/mod.rs

@@ -29,7 +29,10 @@ pub use internode_data_transport::{
     InternodeDataTransport, InternodeDataTransportCapabilities, ReadStreamRequest, TcpHttpInternodeDataTransport,
     WalkDirStreamRequest, WriteStreamRequest, build_internode_data_transport, build_internode_data_transport_from_env,
 };
-pub use peer_rest_client::PeerRestClient;
+pub use peer_rest_client::{
+    PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG,
+    SERVICE_SIGNAL_RELOAD_DYNAMIC,
+};
 pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, RemotePeerS3Client, S3PeerSys};
 pub use remote_disk::RemoteDisk;
 pub use remote_locker::RemoteClient;

crates/ecstore/src/rpc/peer_rest_client.rs

@@ -57,6 +57,8 @@ use tracing::warn;
 pub const PEER_RESTSIGNAL: &str = "signal";
 pub const PEER_RESTSUB_SYS: &str = "sub-sys";
 pub const PEER_RESTDRY_RUN: &str = "dry-run";
+pub const SERVICE_SIGNAL_REFRESH_CONFIG: u64 = 1;
+pub const SERVICE_SIGNAL_RELOAD_DYNAMIC: u64 = 2;
 const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
 const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
 

crates/ecstore/src/rpc/peer_s3_client.rs

@@ -843,12 +843,13 @@ impl PeerS3Client for RemotePeerS3Client {
                 });
                 let response = client.make_bucket(request).await?.into_inner();
 
-                // TODO: deal with error
                 if !response.success {
                     return if let Some(err) = response.error {
                         Err(err.into())
                     } else {
-                        Err(Error::other(""))
+                        Err(Error::other(format!(
+                            "make_bucket({bucket}): peer returned failure without error details"
+                        )))
                     };
                 }
 
@@ -1046,6 +1047,44 @@ async fn clone_drives() -> Vec<Option<DiskStore>> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::disk::WalkDirOptions;
+    use crate::disk::disk_store::LocalDiskWrapper;
+    use crate::disk::endpoint::Endpoint;
+    use crate::disk::local::LocalDisk;
+    use crate::endpoints::{Endpoints, PoolEndpoints};
+    use crate::global::{GLOBAL_LOCAL_DISK_ID_MAP, GLOBAL_LOCAL_DISK_MAP, GLOBAL_LOCAL_DISK_SET_DRIVES};
+    use crate::store::init_local_disks;
+    use rustfs_filemeta::FileInfo;
+    use serial_test::serial;
+    use std::{
+        io,
+        pin::Pin,
+        task::{Context, Poll},
+    };
+    use tempfile::TempDir;
+    use tokio::io::AsyncWrite;
+
+    struct PendingWriter;
+
+    impl AsyncWrite for PendingWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
+            Poll::Pending
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
+
+    async fn reset_local_disk_globals() {
+        GLOBAL_LOCAL_DISK_MAP.write().await.clear();
+        GLOBAL_LOCAL_DISK_ID_MAP.write().await.clear();
+        GLOBAL_LOCAL_DISK_SET_DRIVES.write().await.clear();
+    }
 
     #[derive(Debug)]
     struct TestPeerS3Client {
@@ -1152,6 +1191,77 @@ mod tests {
         client.cancel_token.cancel();
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn local_get_bucket_info_survives_prior_walk_timeout() {
+        reset_local_disk_globals().await;
+
+        let temp_dir = TempDir::new().expect("create temp dir for local peer listing regression");
+        let disk_path = temp_dir.path().join("disk1");
+        std::fs::create_dir_all(&disk_path).expect("create disk path");
+
+        let mut endpoint = Endpoint::try_from(disk_path.to_str().expect("disk path to str")).expect("endpoint");
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+
+        let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 1,
+            endpoints: Endpoints::from(vec![endpoint.clone()]),
+            cmd_line: "local-get-bucket-info-survives-prior-walk-timeout".to_string(),
+            platform: "test".to_string(),
+        }]);
+
+        init_local_disks(endpoint_pools).await.expect("init local disks");
+
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+        let wrapper = crate::disk::Disk::Local(Box::new(LocalDiskWrapper::new(disk, false)));
+        let disk_store: DiskStore = Arc::new(wrapper);
+        let bucket = "test-bucket";
+        let object = "test-object";
+
+        disk_store.make_volume(bucket).await.expect("bucket should be created");
+
+        let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+        file_info.volume = bucket.to_string();
+        file_info.name = object.to_string();
+        file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+        file_info.erasure.index = 1;
+
+        disk_store
+            .write_metadata("", bucket, object, file_info)
+            .await
+            .expect("object metadata should be written");
+
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let mut writer = PendingWriter;
+            let walk_err = disk_store
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await
+                .expect_err("walk_dir should time out against a non-draining writer");
+
+            assert_eq!(walk_err, DiskError::Timeout);
+
+            let info = LocalPeerS3Client::new(None, Some(vec![0]))
+                .get_bucket_info(bucket, &BucketOptions::default())
+                .await
+                .expect("bucket info should still succeed after prior walk timeout");
+            assert_eq!(info.name, bucket);
+        })
+        .await;
+
+        reset_local_disk_globals().await;
+    }
+
     #[test]
     fn test_reduce_pool_write_quorum_uses_only_pool_participants() {
         let clients = vec![

crates/ecstore/src/rpc/remote_disk.rs

@@ -36,7 +36,7 @@ use metrics::counter;
 use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
 use rustfs_io_metrics::internode_metrics::{
     INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC,
-    global_internode_metrics,
+    INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, global_internode_metrics,
 };
 use rustfs_protos::evict_failed_connection;
 use rustfs_protos::proto_gen::node_service::RenamePartRequest;
@@ -74,6 +74,9 @@ enum FailureHealthAction {
     IgnoreFailure,
 }
 
+const REMOTE_DISK_OPEN_WRITE_MAX_ATTEMPTS: usize = 2;
+const REMOTE_DISK_OPEN_WRITE_RETRY_BACKOFF: Duration = Duration::from_millis(20);
+
 async fn copy_stream_with_buffer<R, W>(reader: &mut R, writer: &mut W, buffer_size: usize) -> io::Result<u64>
 where
     R: AsyncRead + Unpin,
@@ -110,6 +113,19 @@ pub struct RemoteDisk {
 }
 
 impl RemoteDisk {
+    fn is_retryable_walk_dir_error(err: &DiskError) -> bool {
+        if is_network_like_disk_error(err) {
+            return true;
+        }
+
+        let err_text = err.to_string().to_ascii_lowercase();
+        err_text.contains("httpreader stream error") || err_text.contains("error decoding response body")
+    }
+
+    fn is_retryable_open_write_error(err: &DiskError) -> bool {
+        err.is_retryable_internode_write_failure()
+    }
+
     pub(crate) async fn new(ep: &Endpoint, opt: &DiskOption, data_transport: Arc<dyn InternodeDataTransport>) -> Result<Self> {
         let addr = if let Some(port) = ep.url.port() {
             format!("{}://{}:{}", ep.url.scheme(), ep.url.host_str().unwrap(), port)
@@ -147,6 +163,50 @@ impl RemoteDisk {
         self.health.last_capacity_snapshot()
     }
 
+    async fn open_write_with_retry(&self, request: WriteStreamRequest) -> Result<FileWriter> {
+        let mut attempt = 1;
+        let mut last_retry_classification = None;
+        loop {
+            match self.data_transport.open_write(request.clone()).await {
+                Ok(writer) => {
+                    if attempt > 1
+                        && let Some(classification) = last_retry_classification
+                    {
+                        global_internode_metrics().record_retry_success_for_operation_and_backend(
+                            rustfs_io_metrics::internode_metrics::INTERNODE_OPERATION_PUT_FILE_STREAM,
+                            INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
+                            classification,
+                        );
+                    }
+                    return Ok(writer);
+                }
+                Err(err) if attempt < REMOTE_DISK_OPEN_WRITE_MAX_ATTEMPTS && Self::is_retryable_open_write_error(&err) => {
+                    if let Some(classification) = err.internode_http_error_kind() {
+                        let classification = classification.metric_label();
+                        global_internode_metrics().record_retry_for_operation_and_backend(
+                            rustfs_io_metrics::internode_metrics::INTERNODE_OPERATION_PUT_FILE_STREAM,
+                            INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
+                            classification,
+                        );
+                        last_retry_classification = Some(classification);
+                    }
+                    debug!(
+                        endpoint = %request.endpoint,
+                        volume = %request.volume,
+                        path = %request.path,
+                        append = request.append,
+                        size = request.size,
+                        attempt,
+                        "retrying remote open_write after retryable transport error"
+                    );
+                    tokio::time::sleep(REMOTE_DISK_OPEN_WRITE_RETRY_BACKOFF).await;
+                    attempt += 1;
+                }
+                Err(err) => return Err(err),
+            }
+        }
+    }
+
     pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
         self.health.record_capacity_probe(total, used, free);
     }
@@ -175,6 +235,10 @@ impl RemoteDisk {
         });
     }
 
+    fn mark_suspect_or_offline(&self, reason: &'static str) -> bool {
+        self.health.mark_failure(&self.endpoint, reason)
+    }
+
     /// Enable health monitoring after disk creation.
     /// Used to defer health checks until after startup format loading completes,
     /// so that remote peers have time to come online.
@@ -202,7 +266,10 @@ impl RemoteDisk {
         let mut interval = time::interval(get_drive_active_check_interval());
 
         // Perform basic connectivity check
-        if Self::perform_connectivity_check(&addr).await.is_err() && health.mark_offline(&endpoint, "connectivity_probe_failed") {
+        let initial_probe_ok = Self::perform_connectivity_check(&addr).await.is_ok();
+        if initial_probe_ok {
+            health.record_operation_success(&endpoint, "connectivity_probe_success");
+        } else if health.mark_failure(&endpoint, "connectivity_probe_failed") {
             warn!("Remote disk health check failed for {}: marking as faulty", addr);
 
             // Start recovery monitoring
@@ -245,7 +312,9 @@ impl RemoteDisk {
                     }
 
                     // Perform basic connectivity check
-                    if Self::perform_connectivity_check(&addr).await.is_err() && health.mark_offline(&endpoint, "connectivity_probe_failed") {
+                    if Self::perform_connectivity_check(&addr).await.is_ok() {
+                        health.record_operation_success(&endpoint, "connectivity_probe_success");
+                    } else if health.mark_failure(&endpoint, "connectivity_probe_failed") {
                         warn!("Remote disk health check failed for {}: marking as faulty", addr);
 
                         // Start recovery monitoring
@@ -286,7 +355,7 @@ impl RemoteDisk {
                             return;
                         }
                     } else {
-                        health.mark_offline(&endpoint, "connectivity_probe_failed");
+                        health.mark_failure(&endpoint, "connectivity_probe_failed");
                     }
                 }
             }
@@ -440,7 +509,11 @@ impl RemoteDisk {
     }
 
     async fn mark_faulty_and_evict(&self, reason: &'static str) {
-        if self.health.mark_offline(&self.endpoint, reason) {
+        let previous_state = self.runtime_state();
+        let transitioned_to_offline = self.mark_suspect_or_offline(reason);
+        let state = self.runtime_state();
+
+        if state != previous_state {
             self.spawn_recovery_monitor_if_needed();
             counter!(
                 "rustfs_drive_faulty_mark_total",
@@ -448,10 +521,17 @@ impl RemoteDisk {
                 "reason" => reason.to_string()
             )
             .increment(1);
-            warn!(
-                "Remote disk marked faulty after timeout: endpoint={}, addr={}, reason={}",
-                self.endpoint, self.addr, reason
-            );
+            if transitioned_to_offline {
+                warn!(
+                    "Remote disk marked faulty after timeout: endpoint={}, addr={}, reason={}",
+                    self.endpoint, self.addr, reason
+                );
+            } else {
+                warn!(
+                    "Remote disk marked suspect after timeout: endpoint={}, addr={}, reason={}, state={:?}",
+                    self.endpoint, self.addr, reason, state
+                );
+            }
             counter!(
                 "rustfs_drive_connection_evict_total",
                 "endpoint" => self.endpoint.to_string(),
@@ -1128,7 +1208,8 @@ impl DiskAPI for RemoteDisk {
     ) -> Result<RenameDataResp> {
         info!("rename_data {}/{}/{}/{}", self.addr, self.endpoint.to_string(), dst_volume, dst_path);
 
-        self.execute_with_timeout(
+        self.execute_with_timeout_for_op(
+            "rename_data",
             || async {
                 let file_info = serde_json::to_string(&fi)?;
                 let mut client = self
@@ -1195,26 +1276,65 @@ impl DiskAPI for RemoteDisk {
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
         info!("walk_dir {}", self.endpoint.to_string());
 
+        let disk = self.disk_ref().await;
+        let body = serde_json::to_vec(&opts)?;
+        let stall_timeout = get_drive_walkdir_stall_timeout();
+        let bucket = opts.bucket.clone();
+        let base_dir = opts.base_dir.clone();
+        let disk_for_log = disk.clone();
+        let timeout_duration = if opts.skip_total_timeout {
+            Duration::ZERO
+        } else {
+            get_drive_walkdir_timeout()
+        };
+
         self.execute_with_timeout_for_op_and_health_action(
             "walk_dir",
             || async {
-                let disk = self.disk_ref().await;
-                let opts = serde_json::to_vec(&opts)?;
-                let mut reader = self
-                    .data_transport
-                    .open_walk_dir(WalkDirStreamRequest {
-                        endpoint: self.endpoint.grid_host(),
-                        disk,
-                        body: opts,
-                        stall_timeout: Some(get_drive_walkdir_stall_timeout()),
-                    })
-                    .await?;
+                let mut last_err = None;
 
-                copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await?;
+                for attempt in 1..=2 {
+                    let mut reader = match self
+                        .data_transport
+                        .open_walk_dir(WalkDirStreamRequest {
+                            endpoint: self.endpoint.grid_host(),
+                            disk: disk.clone(),
+                            body: body.clone(),
+                            stall_timeout: Some(stall_timeout),
+                        })
+                        .await
+                    {
+                        Ok(reader) => reader,
+                        Err(err) => {
+                            if attempt == 1 && Self::is_retryable_walk_dir_error(&err) {
+                                warn!(
+                                    endpoint = %self.endpoint,
+                                    addr = %self.addr,
+                                    disk = %disk_for_log,
+                                    bucket = %bucket,
+                                    base_dir = %base_dir,
+                                    attempt,
+                                    stall_timeout_ms = stall_timeout.as_millis(),
+                                    error = %err,
+                                    "remote walk_dir returned retryable transport error; retrying"
+                                );
+                                last_err = Some(err);
+                                continue;
+                            }
 
-                Ok(())
+                            return Err(err);
+                        }
+                    };
+
+                    match copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await {
+                        Ok(_) => return Ok(()),
+                        Err(io_err) => return Err(DiskError::Io(io_err)),
+                    }
+                }
+
+                Err(last_err.unwrap_or_else(|| DiskError::other("walk_dir retry exhausted without captured error")))
             },
-            get_drive_walkdir_timeout(),
+            timeout_duration,
             FailureHealthAction::IgnoreFailure,
         )
         .await
@@ -1278,16 +1398,15 @@ impl DiskAPI for RemoteDisk {
             return Err(DiskError::FaultyDisk);
         }
         let disk = self.disk_ref().await;
-        self.data_transport
-            .open_write(WriteStreamRequest {
-                endpoint: self.endpoint.grid_host(),
-                disk,
-                volume: volume.to_string(),
-                path: path.to_string(),
-                append: true,
-                size: 0,
-            })
-            .await
+        self.open_write_with_retry(WriteStreamRequest {
+            endpoint: self.endpoint.grid_host(),
+            disk,
+            volume: volume.to_string(),
+            path: path.to_string(),
+            append: true,
+            size: 0,
+        })
+        .await
     }
 
     #[tracing::instrument(level = "debug", skip(self))]
@@ -1304,16 +1423,15 @@ impl DiskAPI for RemoteDisk {
             return Err(DiskError::FaultyDisk);
         }
         let disk = self.disk_ref().await;
-        self.data_transport
-            .open_write(WriteStreamRequest {
-                endpoint: self.endpoint.grid_host(),
-                disk,
-                volume: volume.to_string(),
-                path: path.to_string(),
-                append: false,
-                size: file_size,
-            })
-            .await
+        self.open_write_with_retry(WriteStreamRequest {
+            endpoint: self.endpoint.grid_host(),
+            disk,
+            volume: volume.to_string(),
+            path: path.to_string(),
+            append: false,
+            size: file_size,
+        })
+        .await
     }
 
     #[tracing::instrument(level = "debug", skip(self))]
@@ -1738,6 +1856,65 @@ mod tests {
         }
     }
 
+    #[derive(Debug)]
+    enum WalkDirTestStep {
+        Error(DiskError),
+        Data(Vec<u8>),
+        PartialDataThenError { data: Vec<u8>, error: io::Error },
+    }
+
+    #[derive(Debug, Clone, Default)]
+    struct RetryingWalkDirInternodeDataTransport {
+        calls: Arc<StdMutex<Vec<RecordedTransportCall>>>,
+        steps: Arc<StdMutex<Vec<WalkDirTestStep>>>,
+    }
+
+    impl RetryingWalkDirInternodeDataTransport {
+        fn with_steps(steps: Vec<WalkDirTestStep>) -> Self {
+            Self {
+                calls: Arc::new(StdMutex::new(Vec::new())),
+                steps: Arc::new(StdMutex::new(steps)),
+            }
+        }
+
+        fn calls(&self) -> Vec<RecordedTransportCall> {
+            self.calls.lock().expect("recorded transport calls lock poisoned").clone()
+        }
+
+        fn record(&self, call: RecordedTransportCall) {
+            self.calls.lock().expect("recorded transport calls lock poisoned").push(call);
+        }
+    }
+
+    #[derive(Debug, Clone)]
+    enum OpenWriteTestStep {
+        Error(DiskError),
+        Success,
+    }
+
+    #[derive(Debug, Clone, Default)]
+    struct RetryingOpenWriteInternodeDataTransport {
+        calls: Arc<StdMutex<Vec<RecordedTransportCall>>>,
+        steps: Arc<StdMutex<Vec<OpenWriteTestStep>>>,
+    }
+
+    impl RetryingOpenWriteInternodeDataTransport {
+        fn with_steps(steps: Vec<OpenWriteTestStep>) -> Self {
+            Self {
+                calls: Arc::new(StdMutex::new(Vec::new())),
+                steps: Arc::new(StdMutex::new(steps)),
+            }
+        }
+
+        fn calls(&self) -> Vec<RecordedTransportCall> {
+            self.calls.lock().expect("recorded transport calls lock poisoned").clone()
+        }
+
+        fn record(&self, call: RecordedTransportCall) {
+            self.calls.lock().expect("recorded transport calls lock poisoned").push(call);
+        }
+    }
+
     #[derive(Debug, Default)]
     struct EmptyTestReader;
 
@@ -1790,6 +1967,66 @@ mod tests {
         }
     }
 
+    #[async_trait::async_trait]
+    impl InternodeDataTransport for RetryingWalkDirInternodeDataTransport {
+        async fn open_read(&self, _request: ReadStreamRequest) -> Result<FileReader> {
+            panic!("open_read should not be used in walk_dir retry test");
+        }
+
+        async fn open_write(&self, _request: WriteStreamRequest) -> Result<FileWriter> {
+            panic!("open_write should not be used in walk_dir retry test");
+        }
+
+        async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader> {
+            self.record(RecordedTransportCall::WalkDir(request));
+            let step = self.steps.lock().expect("walk_dir retry steps lock poisoned").remove(0);
+            match step {
+                WalkDirTestStep::Error(err) => Err(err),
+                WalkDirTestStep::Data(data) => Ok(Box::new(Cursor::new(data))),
+                WalkDirTestStep::PartialDataThenError { data, error } => Ok(Box::new(PartialThenErrorReader {
+                    cursor: Cursor::new(data),
+                    error: Some(error),
+                })),
+            }
+        }
+
+        fn name(&self) -> &'static str {
+            "retrying-walk-dir"
+        }
+
+        fn capabilities(&self) -> InternodeDataTransportCapabilities {
+            InternodeDataTransportCapabilities::tcp_http()
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl InternodeDataTransport for RetryingOpenWriteInternodeDataTransport {
+        async fn open_read(&self, _request: ReadStreamRequest) -> Result<FileReader> {
+            panic!("open_read should not be used in open_write retry test");
+        }
+
+        async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter> {
+            self.record(RecordedTransportCall::Write(request));
+            let step = self.steps.lock().expect("open_write retry steps lock poisoned").remove(0);
+            match step {
+                OpenWriteTestStep::Error(err) => Err(err),
+                OpenWriteTestStep::Success => Ok(Box::new(SinkTestWriter)),
+            }
+        }
+
+        async fn open_walk_dir(&self, _request: WalkDirStreamRequest) -> Result<FileReader> {
+            panic!("open_walk_dir should not be used in open_write retry test");
+        }
+
+        fn name(&self) -> &'static str {
+            "retrying-open-write"
+        }
+
+        fn capabilities(&self) -> InternodeDataTransportCapabilities {
+            InternodeDataTransportCapabilities::tcp_http()
+        }
+    }
+
     async fn new_remote_disk_with_transport(data_transport: Arc<dyn InternodeDataTransport>) -> RemoteDisk {
         let endpoint = Endpoint {
             url: url::Url::parse("http://remote-node:9000/data/rustfs0").unwrap(),
@@ -1806,6 +2043,32 @@ mod tests {
         RemoteDisk::new(&endpoint, &disk_option, data_transport).await.unwrap()
     }
 
+    #[derive(Debug)]
+    struct PartialThenErrorReader {
+        cursor: Cursor<Vec<u8>>,
+        error: Option<io::Error>,
+    }
+
+    impl AsyncRead for PartialThenErrorReader {
+        fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
+            let filled_before = buf.filled().len();
+            match Pin::new(&mut self.cursor).poll_read(cx, buf) {
+                Poll::Ready(Ok(())) => {
+                    if buf.filled().len() > filled_before {
+                        return Poll::Ready(Ok(()));
+                    }
+
+                    if let Some(err) = self.error.take() {
+                        return Poll::Ready(Err(err));
+                    }
+
+                    Poll::Ready(Ok(()))
+                }
+                other => other,
+            }
+        }
+    }
+
     fn init_tracing(filter_level: Level) {
         INIT.call_once(|| {
             let _ = tracing_subscriber::fmt()
@@ -1951,20 +2214,39 @@ mod tests {
             disk_idx: 0,
         };
 
-        let disk_option = DiskOption {
-            cleanup: false,
-            health_check: true,
-        };
+        temp_env::async_with_vars(
+            [
+                (rustfs_config::ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS, Some("1")),
+                (rustfs_config::ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS, Some("1")),
+            ],
+            async {
+                let disk_option = DiskOption {
+                    cleanup: false,
+                    health_check: true,
+                };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
-            .await
-            .unwrap();
-        remote_disk.enable_health_check();
+                let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+                    .await
+                    .unwrap();
+                remote_disk.enable_health_check();
 
-        // wait for health check connect timeout
-        tokio::time::sleep(Duration::from_secs(6)).await;
-
-        assert!(!remote_disk.is_online().await);
+                // Wait out the initial success-grace window so the active probe loop
+                // actually attempts a connectivity check. Under the new
+                // suspect-first semantics we only need to prove that the drive
+                // transitions away from a clean Online state at least once.
+                tokio::time::sleep(SKIP_IF_SUCCESS_BEFORE + Duration::from_secs(2)).await;
+                assert!(
+                    remote_disk.offline_duration_secs().is_some(),
+                    "missing listener should transition the drive through suspect/offline tracking"
+                );
+                assert_ne!(
+                    remote_disk.runtime_state(),
+                    RuntimeDriveHealthState::Online,
+                    "missing listener should not remain in a clean Online state after probing"
+                );
+            },
+        )
+        .await;
     }
 
     #[tokio::test]
@@ -2112,6 +2394,47 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_remote_disk_create_file_retries_once_on_retryable_open_write_error() {
+        let transport = RetryingOpenWriteInternodeDataTransport::with_steps(vec![
+            OpenWriteTestStep::Error(DiskError::from(rustfs_rio::new_test_internode_http_io_error(
+                rustfs_rio::InternodeHttpErrorKind::ConnectionReset,
+            ))),
+            OpenWriteTestStep::Success,
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        rustfs_io_metrics::internode_metrics::global_internode_metrics().reset_for_test();
+
+        let _created = remote_disk
+            .create_file("orig-bucket", "bucket", "object/part.1", 4096)
+            .await
+            .expect("retryable open_write error should recover");
+
+        let calls = transport.calls();
+        assert_eq!(calls.len(), 2, "create_file should retry exactly once");
+        let snapshot = rustfs_io_metrics::internode_metrics::global_internode_metrics().snapshot();
+        assert_eq!(snapshot.outgoing_requests_total, 0);
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_append_file_does_not_retry_non_retryable_open_write_error() {
+        let transport = RetryingOpenWriteInternodeDataTransport::with_steps(vec![OpenWriteTestStep::Error(DiskError::from(
+            rustfs_rio::new_test_internode_http_io_error(rustfs_rio::InternodeHttpErrorKind::DnsResolutionFailed),
+        ))]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+
+        let err = match remote_disk.append_file("bucket", "object/part.2").await {
+            Ok(_) => panic!("non-retryable open_write error should be returned directly"),
+            Err(err) => err,
+        };
+
+        assert_eq!(
+            err.internode_http_error_kind(),
+            Some(rustfs_rio::InternodeHttpErrorKind::DnsResolutionFailed)
+        );
+        assert_eq!(transport.calls().len(), 1, "append_file should not retry non-retryable errors");
+    }
+
     #[tokio::test]
     async fn test_remote_disk_walk_dir_uses_configured_data_transport() {
         let transport = RecordingInternodeDataTransport::default();
@@ -2126,6 +2449,7 @@ mod tests {
             forward_to: None,
             limit: 10,
             disk_id: String::new(),
+            ..Default::default()
         };
         let expected_body = serde_json::to_vec(&opts).unwrap();
         let mut writer = Vec::new();
@@ -2145,6 +2469,99 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_preserves_skip_total_timeout_option() {
+        let transport = RecordingInternodeDataTransport::default();
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "prefix".to_string(),
+            recursive: true,
+            skip_total_timeout: true,
+            ..Default::default()
+        };
+        let mut writer = Vec::new();
+
+        remote_disk
+            .walk_dir(opts, &mut writer)
+            .await
+            .expect("walk_dir should be sent through configured data transport");
+
+        let calls = transport.calls();
+        assert_eq!(calls.len(), 1);
+        match &calls[0] {
+            RecordedTransportCall::WalkDir(request) => {
+                let sent_opts: WalkDirOptions =
+                    serde_json::from_slice(&request.body).expect("walk_dir request body should deserialize");
+                assert!(sent_opts.skip_total_timeout);
+                assert_eq!(request.stall_timeout, Some(get_drive_walkdir_stall_timeout()));
+            }
+            other => panic!("expected walk-dir transport call, got {other:?}"),
+        }
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_retries_once_on_retryable_transport_error() {
+        let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![
+            WalkDirTestStep::Error(DiskError::other("HttpReader stream error: error decoding response body")),
+            WalkDirTestStep::Data(b"walk-dir-retry-ok".to_vec()),
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "config/iam".to_string(),
+            recursive: true,
+            report_notfound: false,
+            filter_prefix: None,
+            forward_to: None,
+            limit: 10,
+            disk_id: String::new(),
+            ..Default::default()
+        };
+        let mut writer = Vec::new();
+
+        remote_disk
+            .walk_dir(opts, &mut writer)
+            .await
+            .expect("retryable walk_dir error should recover");
+
+        assert_eq!(writer, b"walk-dir-retry-ok");
+        assert_eq!(transport.calls().len(), 2, "walk_dir should retry exactly once");
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_does_not_retry_after_partial_stream_failure() {
+        let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![
+            WalkDirTestStep::PartialDataThenError {
+                data: b"partial-walk-dir".to_vec(),
+                error: io::Error::new(io::ErrorKind::ConnectionReset, "connection reset"),
+            },
+            WalkDirTestStep::Data(b"walk-dir-retry-ok".to_vec()),
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "config/iam".to_string(),
+            recursive: true,
+            report_notfound: false,
+            filter_prefix: None,
+            forward_to: None,
+            limit: 10,
+            disk_id: String::new(),
+            ..Default::default()
+        };
+        let mut writer = Vec::new();
+
+        let err = remote_disk
+            .walk_dir(opts, &mut writer)
+            .await
+            .expect_err("partial stream failure should be returned without retry");
+
+        assert!(matches!(err, DiskError::Io(ref io_err) if io_err.kind() == io::ErrorKind::ConnectionReset));
+        assert_eq!(writer, b"partial-walk-dir");
+        assert_eq!(transport.calls().len(), 1, "walk_dir should not retry after writing partial bytes");
+    }
+
     #[tokio::test]
     async fn test_remote_disk_endpoints_with_different_schemes() {
         let test_cases = vec![
@@ -2284,7 +2701,12 @@ mod tests {
             .expect_err("timeout should fail");
 
         assert!(err.to_string().contains("timeout"));
-        assert!(!remote_disk.is_online().await, "remote disk should be marked faulty after timeout");
+        assert!(remote_disk.is_online().await, "first timeout should keep the remote disk online");
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first timeout should move the remote disk into suspect state"
+        );
     }
 
     #[tokio::test]
@@ -2450,7 +2872,15 @@ mod tests {
             },
             std::io::ErrorKind::TimedOut
         );
-        assert!(!remote_disk.is_online().await, "timeout-like errors should mark remote disk faulty");
+        assert!(
+            remote_disk.is_online().await,
+            "first timeout-like error should keep the remote disk online"
+        );
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first timeout-like error should move the remote disk into suspect state"
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "timeout-like errors should evict cached connection"
@@ -2503,7 +2933,15 @@ mod tests {
             },
             std::io::ErrorKind::ConnectionRefused
         );
-        assert!(!remote_disk.is_online().await, "network-like errors should mark remote disk faulty");
+        assert!(
+            remote_disk.is_online().await,
+            "first network-like error should keep the remote disk online"
+        );
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first network-like error should move the remote disk into suspect state"
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "network-like errors should evict cached connection"

crates/ecstore/src/rpc/remote_locker.rs

@@ -14,18 +14,21 @@
 
 use crate::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
 use async_trait::async_trait;
+use bytes::Bytes;
 use rustfs_lock::{
     LockClient, LockError, LockInfo, LockRequest, LockResponse, LockStats, LockStatus, LockType, Result,
     types::{LockId, LockMetadata, LockPriority},
 };
 use rustfs_protos::proto_gen::node_service::{BatchGenerallyLockRequest, GenerallyLockRequest, PingRequest};
-use rustfs_protos::{evict_failed_connection, proto_gen::node_service::node_service_client::NodeServiceClient};
+use rustfs_protos::{
+    evict_failed_connection, models::PingBodyBuilder, proto_gen::node_service::node_service_client::NodeServiceClient,
+};
 use std::time::Duration;
 use tokio::time::timeout;
 use tonic::Request;
 use tonic::service::interceptor::InterceptedService;
 use tonic::transport::Channel;
-use tracing::{info, warn};
+use tracing::{debug, info, warn};
 
 /// Remote lock client implementation
 #[derive(Debug, Clone)]
@@ -42,6 +45,20 @@ impl RemoteClient {
         Self { addr: url.to_string() }
     }
 
+    fn build_ping_request() -> PingRequest {
+        let mut fbb = flatbuffers::FlatBufferBuilder::new();
+        let payload = fbb.create_vector(b"health-check");
+        let mut builder = PingBodyBuilder::new(&mut fbb);
+        builder.add_payload(payload);
+        let root = builder.finish();
+        fbb.finish(root, None);
+
+        PingRequest {
+            version: 1,
+            body: Bytes::copy_from_slice(fbb.finished_data()),
+        }
+    }
+
     /// Create a minimal LockRequest for unlock operations using only lock_id
     fn create_unlock_request(lock_id: &LockId) -> LockRequest {
         LockRequest {
@@ -64,16 +81,44 @@ impl RemoteClient {
             .map_err(|err| LockError::internal(format!("can not get client, err: {err}")))
     }
 
-    async fn evict_connection(&self, op: &'static str, reason: &str) {
-        warn!(
-            addr = %self.addr,
-            op,
-            reason,
-            "Evicting cached remote lock connection after RPC failure"
-        );
+    fn is_scanner_leader_lock(resource_summary: &str) -> bool {
+        resource_summary == ".rustfs.sys/leader.lock@latest"
+    }
+
+    async fn evict_connection(&self, op: &'static str, reason: &str, resource_summary: &str) {
+        if Self::is_scanner_leader_lock(resource_summary) {
+            debug!(
+                addr = %self.addr,
+                op,
+                reason,
+                resource_summary,
+                "Evicting cached remote lock connection for scanner leader-lock RPC failure"
+            );
+        } else {
+            warn!(
+                addr = %self.addr,
+                op,
+                reason,
+                resource_summary,
+                "Evicting cached remote lock connection after RPC failure"
+            );
+        }
         evict_failed_connection(&self.addr).await;
     }
 
+    fn summarize_resources(requests: &[LockRequest]) -> String {
+        const LIMIT: usize = 3;
+        let mut resources = requests
+            .iter()
+            .take(LIMIT)
+            .map(|request| request.resource.to_string())
+            .collect::<Vec<_>>();
+        if requests.len() > LIMIT {
+            resources.push(format!("... (+{} more)", requests.len() - LIMIT));
+        }
+        resources.join(", ")
+    }
+
     fn rpc_timeout(timeout_duration: Duration) -> Duration {
         if timeout_duration.is_zero() {
             Duration::from_millis(1)
@@ -86,39 +131,74 @@ impl RemoteClient {
         &self,
         op: &'static str,
         timeout_duration: Duration,
+        resource_summary: &str,
         future: F,
     ) -> std::result::Result<T, LockError>
     where
         F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
     {
-        let timeout_duration = Self::rpc_timeout(timeout_duration);
-        match timeout(timeout_duration, future).await {
+        let lock_timeout = Self::rpc_timeout(timeout_duration);
+        match timeout(lock_timeout, future).await {
             Ok(Ok(response)) => Ok(response),
             Ok(Err(err)) => {
                 let reason = err.to_string();
-                self.evict_connection(op, &reason).await;
+                if Self::is_scanner_leader_lock(resource_summary) {
+                    debug!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        tonic_code = ?err.code(),
+                        tonic_message = err.message(),
+                        "Remote lock RPC returned tonic error for scanner leader lock"
+                    );
+                } else {
+                    warn!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        tonic_code = ?err.code(),
+                        tonic_message = err.message(),
+                        "Remote lock RPC returned tonic error"
+                    );
+                }
+                self.evict_connection(op, &reason, resource_summary).await;
                 Err(LockError::internal(format!("{op} RPC failed: {reason}")))
             }
             Err(_) => {
-                let reason = format!("RPC timed out after {:?}", timeout_duration);
-                self.evict_connection(op, &reason).await;
-                Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), timeout_duration))
+                let reason = format!("RPC timed out after {:?}", lock_timeout);
+                if Self::is_scanner_leader_lock(resource_summary) {
+                    debug!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        "Remote lock RPC timed out for scanner leader lock"
+                    );
+                } else {
+                    warn!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        "Remote lock RPC timed out"
+                    );
+                }
+                self.evict_connection(op, &reason, resource_summary).await;
+                Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), lock_timeout))
             }
         }
     }
 
-    fn timeout_failure_response(request: &LockRequest) -> LockResponse {
-        LockResponse::failure("Lock acquisition timeout", request.acquire_timeout)
+    fn rpc_timeout_failure_response(request: &LockRequest, err: &LockError) -> LockResponse {
+        LockResponse::failure(format!("Remote lock RPC timed out: {err}"), request.acquire_timeout)
     }
 
     fn rpc_failure_response(_request: &LockRequest, err: &LockError) -> LockResponse {
         LockResponse::failure(format!("Remote lock RPC failed: {err}"), Duration::ZERO)
     }
 
-    fn timeout_failure_batch(requests: &[LockRequest]) -> Vec<LockResponse> {
-        requests.iter().map(Self::timeout_failure_response).collect()
-    }
-
     fn rpc_failure_batch(requests: &[LockRequest], err: &LockError) -> Vec<LockResponse> {
         requests
             .iter()
@@ -126,6 +206,13 @@ impl RemoteClient {
             .collect()
     }
 
+    fn rpc_timeout_failure_batch(requests: &[LockRequest], err: &LockError) -> Vec<LockResponse> {
+        requests
+            .iter()
+            .map(|request| Self::rpc_timeout_failure_response(request, err))
+            .collect()
+    }
+
     fn batch_rpc_timeout(requests: &[LockRequest]) -> Duration {
         requests
             .iter()
@@ -179,14 +266,18 @@ impl LockClient for RemoteClient {
     async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
         info!("remote acquire_exclusive for {}", request.resource);
         let mut client = self.get_client().await?;
+        let resource_summary = request.resource.to_string();
         let req = Request::new(GenerallyLockRequest {
             args: serde_json::to_string(&request)
                 .map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
         });
 
-        let resp = match self.execute_rpc("lock", request.acquire_timeout, client.lock(req)).await {
+        let resp = match self
+            .execute_rpc("lock", request.acquire_timeout, &resource_summary, client.lock(req))
+            .await
+        {
             Ok(resp) => resp.into_inner(),
-            Err(LockError::Timeout { .. }) => return Ok(Self::timeout_failure_response(request)),
+            Err(err @ LockError::Timeout { .. }) => return Ok(Self::rpc_timeout_failure_response(request, &err)),
             Err(err) => return Ok(Self::rpc_failure_response(request, &err)),
         };
 
@@ -212,6 +303,7 @@ impl LockClient for RemoteClient {
         }
 
         let mut client = self.get_client().await?;
+        let resource_summary = Self::summarize_resources(requests);
         let req = Request::new(BatchGenerallyLockRequest {
             args: requests
                 .iter()
@@ -222,11 +314,11 @@ impl LockClient for RemoteClient {
         });
 
         let resp = match self
-            .execute_rpc("lock_batch", Self::batch_rpc_timeout(requests), client.lock_batch(req))
+            .execute_rpc("lock_batch", Self::batch_rpc_timeout(requests), &resource_summary, client.lock_batch(req))
             .await
         {
             Ok(resp) => resp.into_inner(),
-            Err(LockError::Timeout { .. }) => return Ok(Self::timeout_failure_batch(requests)),
+            Err(err @ LockError::Timeout { .. }) => return Ok(Self::rpc_timeout_failure_batch(requests, &err)),
             Err(err) => return Ok(Self::rpc_failure_batch(requests, &err)),
         };
 
@@ -435,10 +527,7 @@ impl LockClient for RemoteClient {
             }
         };
 
-        let ping_req = Request::new(PingRequest {
-            version: 1,
-            body: bytes::Bytes::new(),
-        });
+        let ping_req = Request::new(Self::build_ping_request());
 
         match client.ping(ping_req).await {
             Ok(_) => {
@@ -511,7 +600,14 @@ mod tests {
             "remote lock RPC should honor request timeout"
         );
         assert!(!response.success, "timed out lock acquisition should fail");
-        assert_eq!(response.error.as_deref(), Some("Lock acquisition timeout"));
+        assert!(
+            response
+                .error
+                .as_deref()
+                .is_some_and(|error| error.contains("Remote lock RPC timed out")),
+            "expected remote RPC timeout marker, got {:?}",
+            response.error
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "timeout should evict cached connection"
@@ -539,7 +635,14 @@ mod tests {
         );
         assert_eq!(responses.len(), 1);
         assert!(!responses[0].success, "timed out batch lock acquisition should fail");
-        assert_eq!(responses[0].error.as_deref(), Some("Lock acquisition timeout"));
+        assert!(
+            responses[0]
+                .error
+                .as_deref()
+                .is_some_and(|error| error.contains("Remote lock RPC timed out")),
+            "expected remote RPC timeout marker, got {:?}",
+            responses[0].error
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "batch timeout should evict cached connection"