feat(scanner): add runtime scanner controls and status (#3203)

* feat(scanner): add runtime scanner controls and status

* fix(scanner): validate persisted scanner config

* docs(scanner): clarify start delay behavior

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

.agents/skills/code-change-verification/SKILL.md

@@ -41,6 +41,19 @@ Use this skill to review code changes consistently before merge, before release,
   - hidden coupling to shared helpers/constants/features
 - If a point is uncertain, mark it as an open question instead of guessing.
 
+#### Rust-specific checks (apply to all Rust changes)
+
+- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
+- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
+- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
+- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
+- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
+- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
+- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
+- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
+- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
+- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
+
 ### 4) Findings-first output
 - Order findings by severity:
   - P0: critical failure, security breach, or data loss risk

.agents/skills/rust-code-quality/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: rust-code-quality
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+---
+
+# Rust Code Quality Gate
+
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
+
+## Quick Start
+
+1. Identify changed `.rs` files.
+2. Run automated checks on changed files.
+3. Run manual review checklist on the diff.
+4. Report findings; block merge if P0/P1 issues exist.
+
+## Automated Checks
+
+Run these on every changed `.rs` file (excluding test modules):
+
+```bash
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
+
+# 2. Silent type truncation via `as` cast
+rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
+
+# 3. String as error type
+rg -n 'Result<.*String>' <changed-files> | grep -v test
+
+# 4. Box<dyn Error> in public APIs
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
+
+# 5. println/eprintln in production
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
+
+# 6. Ordering::Relaxed usage (verify each is intentional)
+rg -n 'Ordering::Relaxed' <changed-files>
+```
+
+## Manual Review Checklist
+
+For every Rust code change, verify:
+
+### Error Handling
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
+- [ ] No `Result<_, String>` in public API signatures
+- [ ] No `Box<dyn Error>` in public trait/struct methods
+- [ ] `Error::source()` is overridden when inner error is stored
+- [ ] Error messages are actionable (what failed, with what input)
+
+### Type Safety
+- [ ] No silent `as` truncation (negative→unsigned, large→small)
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
+
+### Concurrency
+- [ ] Lock acquisition order is documented when multiple locks are used
+- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
+- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
+
+### Memory and Performance
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
+
+### Recursion Safety
+- [ ] Recursive functions have a depth limit or use iterative traversal
+- [ ] Tree/cache traversals handle corrupted/cyclic input safely
+
+### Testing
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
+
+### Serde
+- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
+- [ ] `#[serde(default)]` not used on security-critical fields without validation
+
+### Code Hygiene
+- [ ] No `#![allow(dead_code)]` at crate root
+- [ ] No camelCase statics or Hungarian notation
+- [ ] New string literals don't duplicate existing constants
+
+## Severity Classification
+
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
+
+## Output Template
+
+```
+## Rust Code Quality Report
+
+### Automated Scan
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
+
+### Findings
+- [P1] `path:line` — description
+  - Fix: ...
+  - Validation: ...
+
+### Verdict
+PASS / BLOCKED (list blocking findings)
+```

.agents/skills/rust-code-quality/references/checklist.md

@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,8 +1,7 @@
 ---
 name: rustfs-release-version-bump
-description: Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation.
+description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
 ---
-
 # RustFS Release Version Bump
 
 Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).

.agents/skills/security-advisory-lessons/SKILL.md

@@ -70,6 +70,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
 - Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
 - Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
 
 ### Secrets, default credentials, and crypto
 - Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
@@ -86,6 +87,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
 - Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
 - RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
 
 ### Browser, CORS, and console surfaces
 - Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
@@ -115,5 +117,6 @@ Use these prompts while reviewing a diff:
 - Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
 - Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -38,6 +38,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
 - `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
 
 ### Secrets, defaults, and cryptographic misuse
 
@@ -55,6 +56,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
 - `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
 
 ### Browser, CORS, and console isolation
 
@@ -69,6 +71,13 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
+### Serde deserialization and input validation
+
+- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
+- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
+- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
+- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
+
 ## Useful Search Seeds
 
 Use these targeted searches when a diff touches security-sensitive code:
@@ -76,10 +85,12 @@ Use these targeted searches when a diff touches security-sensitive code:
 ```bash
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
 ## Minimum Regression Test Expectations
@@ -87,7 +98,8 @@ rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-
 - Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.docker/compose/docker-compose.cluster.local-build.yml

@@ -20,13 +20,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node1
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -47,13 +52,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node2
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -74,13 +84,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node3
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -101,13 +116,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node4
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"

.docker/compose/docker-compose.cluster.yaml

@@ -21,8 +21,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9003:9000" # Map port 9004 of the host to port 9000 of the container

.docker/observability/grafana/dashboards/rustfs.json

@@ -4551,7 +4551,7 @@
                   "index": 0,
                   "text": "INACTIVE"
                 },
-                "to": 1e-09
+                "to": 1e-9
               },
               "type": "range"
             }
@@ -5281,7 +5281,10 @@
       "id": 102,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5368,7 +5371,10 @@
       "id": 103,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5455,7 +5461,10 @@
       "id": 104,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5533,7 +5542,9 @@
       "id": 105,
       "options": {
         "legend": {
-          "calcs": ["mean"],
+          "calcs": [
+            "mean"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5629,7 +5640,10 @@
       "id": 106,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5725,7 +5739,10 @@
       "id": 107,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5803,7 +5820,10 @@
       "id": 108,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -8071,6 +8091,374 @@
         "x": 0,
         "y": 195
       },
+      "id": 150,
+      "panels": [],
+      "title": "Object Lock Diagnostics",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 196
+      },
+      "id": 151,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_acquire_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Acquires",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 196
+      },
+      "id": 152,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_hold_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Holds",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 204
+      },
+      "id": 153,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_acquire_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Acquire Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 204
+      },
+      "id": 154,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_hold_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Hold Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [
+            {
+              "options": {
+                "0": {
+                  "text": "Disabled"
+                },
+                "1": {
+                  "text": "Enabled"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": 0
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 212
+      },
+      "id": 155,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "percentChangeColorMode": "standard",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max(rustfs_object_lock_diag_enabled{job=~\"$job\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Diagnostics Enabled",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 213
+      },
       "id": 137,
       "panels": [],
       "title": "Debug / Raw Explorer",
@@ -8138,7 +8526,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 196
+        "y": 214
       },
       "id": 138,
       "options": {
@@ -8235,7 +8623,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 196
+        "y": 214
       },
       "id": 139,
       "options": {
@@ -8332,7 +8720,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 204
+        "y": 222
       },
       "id": 140,
       "options": {
@@ -8429,7 +8817,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 204
+        "y": 222
       },
       "id": 141,
       "options": {
@@ -8526,7 +8914,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 212
+        "y": 230
       },
       "id": 142,
       "options": {
@@ -8623,7 +9011,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 212
+        "y": 230
       },
       "id": 143,
       "options": {
@@ -8720,7 +9108,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 220
+        "y": 238
       },
       "id": 144,
       "options": {
@@ -8817,7 +9205,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 220
+        "y": 238
       },
       "id": 145,
       "options": {
@@ -8914,7 +9302,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 228
+        "y": 246
       },
       "id": 146,
       "options": {
@@ -9011,7 +9399,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 228
+        "y": 246
       },
       "id": 147,
       "options": {
@@ -9108,7 +9496,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 236
+        "y": 254
       },
       "id": 148,
       "options": {
@@ -9205,7 +9593,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 236
+        "y": 254
       },
       "id": 149,
       "options": {
@@ -9382,5 +9770,5 @@
   "timezone": "browser",
   "title": "RustFS",
   "uid": "rustfs-s3",
-  "version": 13
+  "version": 14
 }

.github/actions/setup/action.yml

@@ -54,12 +54,13 @@ runs:
           build-essential \
           pkg-config \
           libssl-dev \
+          unzip \
           protobuf-compiler
 
     - name: Install protoc
       uses: rustfs/setup-protoc@v3.0.1
       with:
-        version: "29.3"
+        version: "34.1"
         repo-token: ${{ github.token }}
 
     - name: Install flatc
@@ -83,7 +84,7 @@ runs:
       uses: taiki-e/install-action@cargo-zigbuild
 
     - name: Install cargo-nextest
-      uses: taiki-e/install-action@cargo-nextest
+      uses: taiki-e/install-action@nextest
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@v2

.github/dependabot.yml

@@ -40,6 +40,8 @@ updates:
         versions: [ "1.x" ]
       - dependency-name: "ratelimit"
         versions: [ "2.x" ]
+      - dependency-name: "pyroscope"
+        versions: [ "2.x" ]
     groups:
       s3s:
         update-types:

.github/s3tests/s3tests.conf

@@ -11,8 +11,8 @@
 # host set for RustFS - will be substituted via envsubst
 host = ${S3_HOST}
 
-# port for RustFS
-port = 9000
+# port for RustFS - will be substituted via envsubst
+port = ${S3_PORT}
 
 ## say "False" to disable TLS
 is_secure = False

.github/workflows/audit.yml

@@ -40,7 +40,7 @@ env:
 jobs:
   security-audit:
     name: Security Audit
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 15
     steps:
       - name: Checkout repository
@@ -65,7 +65,7 @@ jobs:
 
   dependency-review:
     name: Dependency Review
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: github.event_name == 'pull_request'
     permissions:
       contents: read
@@ -75,7 +75,7 @@ jobs:
         uses: actions/checkout@v6
 
       - name: Dependency Review
-        uses: actions/dependency-review-action@v4
+        uses: actions/dependency-review-action@v5
         with:
           fail-on-severity: moderate
-          comment-summary-in-pr: true
+          comment-summary-in-pr: always

.github/workflows/build.yml

@@ -73,7 +73,7 @@ jobs:
   # Build strategy check - determine build type based on trigger
   build-check:
     name: Build Strategy Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       build_type: ${{ steps.check.outputs.build_type }}
@@ -146,7 +146,7 @@ jobs:
   # Build RustFS binaries
   prepare-platform-matrix:
     name: Prepare Platform Matrix
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
       selected: ${{ steps.select.outputs.selected }}
@@ -169,7 +169,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":true,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -549,7 +549,7 @@ jobs:
     name: Build Summary
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Build completion summary
         shell: bash
@@ -601,7 +601,7 @@ jobs:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     outputs:
@@ -687,7 +687,7 @@ jobs:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
       actions: read
@@ -768,7 +768,7 @@ jobs:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/')
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Update latest.json
         env:
@@ -793,9 +793,10 @@ jobs:
 
           curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
           unzip "$OSSUTIL_ZIP"
-          mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+          OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
+          mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
           rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-          chmod +x /usr/local/bin/ossutil
+          chmod +x "$OSSUTIL_BIN"
 
           # Create latest.json
           cat > latest.json << EOF
@@ -809,7 +810,7 @@ jobs:
           EOF
 
           # Upload to OSS
-          ossutil cp latest.json oss://rustfs-version/latest.json --force
+          "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
 
           echo "✅ Updated latest.json for stable release $VERSION"
 
@@ -818,7 +819,7 @@ jobs:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     steps:

.github/workflows/ci.yml

@@ -78,7 +78,7 @@ jobs:
     permissions:
       actions: write
       contents: read
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_skip: ${{ steps.skip_check.outputs.should_skip }}
     steps:
@@ -95,7 +95,7 @@ jobs:
     name: Typos
     needs: skip-check
     if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
       - uses: dtolnay/rust-toolchain@stable
@@ -182,11 +182,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v6
 
-      - name: Clean up previous test run
-        run: |
-          rm -rf /tmp/rustfs
-          rm -f /tmp/rustfs.log
-
       - name: Download debug binary
         uses: actions/download-artifact@v7
         with:
@@ -209,14 +204,19 @@ jobs:
       - name: Run end-to-end tests
         run: |
           s3s-e2e --version
-          ./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          RUSTFS_TEST_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
+          RUSTFS_TEST_PORT="${RUSTFS_TEST_PORT}" \
+          RUSTFS_TEST_LOG="${RUN_ROOT}/rustfs.log" \
+          ./scripts/e2e-run.sh ./target/debug/rustfs "${RUN_ROOT}/data"
 
       - name: Upload test logs
         if: failure()
         uses: actions/upload-artifact@v6
         with:
           name: e2e-test-logs-${{ github.run_number }}
-          path: /tmp/rustfs.log
+          path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
   s3-implemented-tests:
@@ -240,10 +240,16 @@ jobs:
 
       - name: Run implemented s3-tests
         run: |
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
           DEPLOY_MODE=binary \
           RUSTFS_BINARY=./target/debug/rustfs \
           TEST_MODE=single \
           MAXFAIL=1 \
+          S3_PORT="${S3_PORT}" \
+          DATA_ROOT="${RUN_ROOT}" \
+          S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
           ./scripts/s3-tests/run.sh
 
       - name: Upload s3 test artifacts

.github/workflows/docker.yml

@@ -73,7 +73,7 @@ jobs:
   # Check if we should build Docker images
   build-check:
     name: Docker Build Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       should_push: ${{ steps.check.outputs.should_push }}
@@ -421,7 +421,7 @@ jobs:
     name: Docker Build Summary
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Docker build completion summary
         run: |

.github/workflows/e2e-s3tests.yml

@@ -40,8 +40,8 @@ on:
 
 env:
   # main user
-  S3_ACCESS_KEY: rustfs-ci-admin
-  S3_SECRET_KEY: rustfs-ci-secret
+  S3_ACCESS_KEY: rustfsadmin
+  S3_SECRET_KEY: rustfsadmin
   # alt user (must be different from main for many s3-tests)
   S3_ALT_ACCESS_KEY: rustfsalt
   S3_ALT_SECRET_KEY: rustfsalt

.github/workflows/helm-package.yml

@@ -31,7 +31,7 @@ permissions:
 
 jobs:
   build-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: |
       github.event_name == 'workflow_dispatch' ||
       (
@@ -99,7 +99,7 @@ jobs:
           retention-days: 1
 
   publish-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     needs: [ build-helm-package ]
     if: needs.build-helm-package.result == 'success'
 

.github/workflows/issue-translator.yml

@@ -25,7 +25,7 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubicloud-standard-4
+    runs-on: ubuntu-latest
     steps:
       - uses: usthe/issues-translate-action@v2.7
         with:

.github/workflows/nix.yml

@@ -38,41 +38,65 @@ concurrency:
   cancel-in-progress: true
 permissions:
   contents: read
+  id-token: write
 
 jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
     timeout-minutes: 60
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    permissions:
+      contents: read
+      id-token: write
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
 
       - name: Install Nix
-        uses: cachix/install-nix-action@v31
+        uses: DeterminateSystems/determinate-nix-action@v3.21.0
         with:
-          github_access_token: ${{ secrets.GITHUB_TOKEN }}
-          extra_nix_config: |
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          extra-conf: |
             experimental-features = nix-command flakes
+            max-jobs = 1
 
-      - name: Setup Magic Nix Cache
-        uses: DeterminateSystems/magic-nix-cache-action@v13
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@v3.21.0
 
-      - name: Setup Flake Checker
+      - name: Check Nix Flake Inputs
         uses: DeterminateSystems/flake-checker-action@v12
+        with:
+          fail-mode: true
+          ignore-missing-flake-lock: false
 
       - name: Verify Flake
         run: |
           echo "Checking flake structure and evaluation..."
           nix flake show
-          nix flake check --print-build-logs
+          for attempt in 1 2 3; do
+            if nix flake check --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix flake check failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Build RustFS
         run: |
           echo "Building the default package..."
-          nix build .#default --print-build-logs
+          for attempt in 1 2 3; do
+            if nix build .#default --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix build failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Test Binary
         run: |

AGENTS.md

@@ -103,6 +103,18 @@ Do not open a PR with code changes when the required checks fail.
 - Use environment variables or vault tooling for sensitive configuration.
 - For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
+## Serde Safety
+
+- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs).
+- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
+- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
+
+## Naming Conventions
+
+- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
+- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
+- If existing code violates naming conventions, do not widen the violation in new code. Fix opportunistically when touching the surrounding area.
+
 ## Scoped Guidance in This Repository
 
 - `.github/AGENTS.md`

Cargo.toml

@@ -47,6 +47,7 @@ members = [
     "crates/signer", # client signer
     "crates/targets", # Target-specific configurations and utilities
     "crates/trusted-proxies", # Trusted proxies management
+    "crates/tls-runtime", # Project-wide TLS runtime foundation
     "crates/utils", # Utility functions and helpers
     "crates/io-metrics", # Zero-copy metrics collection for performance analysis
     "crates/io-core", # Zero-copy core reader and writer implementations
@@ -59,7 +60,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.95.0"
-version = "1.0.0-beta.4"
+version = "1.0.0-beta.7"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -76,72 +77,73 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.4" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.4" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.4" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.4" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.4" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.4" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.4" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.4" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.4" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.4" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.4" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.4" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.4" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.4" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.4" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.4" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.4" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.4" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.4" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.4" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.4" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.4" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.4" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.4" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.4" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.4" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.4" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.4" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.4" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.4" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.4" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.4" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.4" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.4" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.4" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.4" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.7" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.7" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.7" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.7" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.7" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.7" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.7" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.7" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.7" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.7" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.7" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.7" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.7" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.7" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.7" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.7" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.7" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.7" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.7" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.7" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.7" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.7" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.7" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.7" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.7" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.7" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.7" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.7" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.7" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.7" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.7" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.7" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.7" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.7" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.7" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.7" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.7" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "2bad388283bc3ce48801fc2ffcd22445eb6f3d24" }
+mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-async-nats = "0.48.0"
+async-nats = "0.49.0"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
-pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
-lapin = { version = "4.7.4", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
-hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
+pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime"] }
+lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
+hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
-http = "1.4.0"
+http = "1.4.1"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
-reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
-socket2 = { version = "0.6.3", features = ["all"] }
+socket2 = { version = "0.6.4", features = ["all"] }
 tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
 tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.6", features = ["gzip"] }
+tonic = { version = "0.14.6", features = ["gzip", "deflate"] }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
@@ -158,11 +160,11 @@ quick-xml = "0.40.1"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
-serde_json = { version = "1.0.149", features = ["raw_value"] }
+serde_json = { version = "1.0.150", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
-aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
+aes-gcm = { version = "0.11.0-rc.4", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.8" }
 blake2 = "0.11.0-rc.6"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
@@ -183,13 +185,13 @@ zeroize = { version = "1.8.2", features = ["derive"] }
 # Time and Date
 chrono = { version = "0.4.44", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.24", features = ["serde"] }
+jiff = { version = "0.2.28", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Database
 deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
 tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
-tokio-postgres-rustls = "0.13"
+tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
 anyhow = "1.0.102"
@@ -197,14 +199,14 @@ arc-swap = "1.9.1"
 astral-tokio-tar = "0.6.2"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.16" }
+aws-config = { version = "1.8.18" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.8" }
+aws-sdk-s3 = { version = "1.135.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.13", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.9" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
-brotli = "8.0.2"
+brotli = "8.0.3"
 clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
@@ -253,22 +255,22 @@ rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.33.1", features = ["websocket"] }
-redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
+rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
+redis = { version = "1.2.2", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
 s3s = { git = "https://github.com/rustfs/s3s", rev = "507e1312b211c3ddc214b03875d6fabd15d22ed5", features = ["minio"] }
-serial_test = "3.4.0"
+serial_test = "3.5.0"
 shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
-snafu = "0.9.0"
+snafu = "0.9.1"
 snap = "1.1.1"
 starshard = { version = "2.2.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.39.2"
+sysinfo = "0.39.3"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -281,7 +283,7 @@ tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.23.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.23.2", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
@@ -296,18 +298,18 @@ dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.32.0" }
 opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
 opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.32.0", features = ["rt-tokio"] }
+opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
 opentelemetry-semantic-conventions = { version = "0.32.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.0.4", features = ["backend-pprof-rs"] }
+pyroscope = { version = "=2.0.5", features = ["backend-pprof-rs", "backend-jemalloc"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
 suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.8"
-russh = { version = "0.60.3", git = "https://github.com/Eugeny/russh", rev = "fc6e3ab4cd4338e94ae64e17aeed2acee9335e6b" }
-russh-sftp = "2.1.2"
+russh = { version = "0.61.1",features = ["serde"] }
+russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -322,7 +324,9 @@ tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profilin
 jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
 # pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
-# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
+# Use pprof-pyroscope-fork to match pyroscope 2.0.5's internal pprof dependency (v0.1500.4).
+# Cargo unifies them into a single instance, avoiding duplicate perf_signal_handler symbols.
+# NOTE: Do NOT upgrade pyroscope to >=2.0.6 — it vendors pprof-rs internally, causing symbol collision.
 pprof = { package = "pprof-pyroscope-fork", version = "0.1500.4", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]

Dockerfile.glibc

@@ -105,6 +105,8 @@ RUN groupadd -g 10001 rustfs && \
 
 ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_CONSOLE_ADDRESS=":9001" \
+    RUSTFS_ACCESS_KEY="rustfsadmin" \
+    RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \

INTERNODE_DATA_TRANSPORT_RFC.md

@@ -1,400 +0,0 @@
-# RFC: Pluggable Internode Data Transport
-
-> Status: draft
-> Last updated: 2026-05-19
-> Scope: internode data-path analysis, benchmark baseline, and transport boundary
-
-## Summary
-
-RustFS does not currently include RDMA, RoCE, InfiniBand, DPU, BlueField/DOCA,
-DPDK, SPDK, or SmartNIC offload support. The current distributed internode
-paths use TCP-based HTTP/gRPC transports:
-
-- `tonic` gRPC `NodeService` for most control, metadata, lock, health, and
-  peer operations.
-- HTTP streaming routes under `/rustfs/rpc/` for remote disk file streams.
-
-RDMA/RoCE is still a plausible future optimization for large internode disk
-data transfers, but it should not replace the whole internode RPC surface.
-The correct first step is to isolate the data plane, establish a TCP baseline,
-and introduce a pluggable transport boundary only around high-volume streams.
-
-## Goals
-
-- Document the current internode control plane and data plane.
-- Identify the existing transfer paths that could benefit from a future
-  high-throughput backend.
-- Define the minimum benchmark baseline required before transport changes.
-- Sketch a pluggable transport boundary that preserves the current TCP/HTTP
-  behavior as the default backend.
-- Reserve explicit boundaries for future RDMA/RoCE/InfiniBand work without
-  committing RustFS to a specific vendor stack.
-
-## Non-Goals
-
-- Implement RDMA, RoCE, InfiniBand, DPU, DOCA, DPDK, SPDK, or SmartNIC support.
-- Replace `tonic` gRPC for control-plane RPCs.
-- Redesign erasure coding, quorum handling, disk health tracking, or object
-  correctness semantics.
-- Require RDMA-capable hardware for default development, CI, or ordinary
-  RustFS deployments.
-
-## Current Internode Architecture
-
-### Server-side entry points
-
-The main HTTP server builds a hybrid service per connection:
-
-- `rustfs/src/server/http.rs` wires a `NodeServiceServer` for gRPC.
-- `rustfs/src/storage/rpc/InternodeRpcService` intercepts HTTP paths under
-  `/rustfs/rpc/`.
-- Other HTTP/S3 traffic continues through the normal S3 service.
-
-Compression logic already treats `/rustfs/rpc/` and `/rustfs/peer/` as internode
-RPC paths and skips normal response compression for them.
-
-### gRPC channel management
-
-`crates/protos/src/lib.rs` creates internode gRPC channels with `tonic`
-`Endpoint`:
-
-- connect timeout
-- TCP keepalive
-- HTTP/2 keepalive interval and timeout
-- request timeout
-- optional TLS configuration
-- global channel caching and failed-connection eviction
-
-This confirms the current gRPC transport is TCP/HTTP2-based.
-
-### NodeService layout
-
-`crates/protos/src/node.proto` defines one `NodeService` that mixes several
-classes of RPCs:
-
-- meta service: bucket and metadata operations
-- disk service: local/remote disk operations
-- lock service: distributed lock operations
-- peer rest service: node health, metrics, IAM/policy reload, rebalance,
-  profiling, events, and admin-style operations
-
-The service layout is practical today, but it is too broad to become an RDMA
-surface. A future high-throughput transport should target only disk data
-streams and keep this gRPC service as the control plane.
-
-## Control Plane vs Data Plane
-
-### Control plane
-
-These paths carry coordination, metadata, health, and administrative state.
-They should remain on gRPC/TCP:
-
-| Area | Client/server code | Examples | Notes |
-| --- | --- | --- | --- |
-| Bucket peer ops | `crates/ecstore/src/rpc/peer_s3_client.rs`, `rustfs/src/storage/rpc/bucket.rs` | `MakeBucket`, `ListBucket`, `DeleteBucket`, `GetBucketInfo`, `HealBucket` | Small metadata/control payloads. |
-| Locking | `crates/ecstore/src/rpc/remote_locker.rs`, `rustfs/src/storage/rpc/lock.rs` | `Lock`, `UnLock`, `Refresh`, batch lock/unlock | Latency-sensitive but not bulk data; correctness and timeout semantics matter more than transport bandwidth. |
-| Peer/admin state | `crates/ecstore/src/rpc/peer_rest_client.rs`, `rustfs/src/storage/rpc/health.rs`, `metrics.rs`, `event.rs` | `LocalStorageInfo`, `ServerInfo`, `GetMetrics`, `GetLiveEvents`, reload APIs, rebalance APIs | Operational control plane. |
-| Disk metadata/control | `crates/ecstore/src/rpc/remote_disk.rs`, `rustfs/src/storage/rpc/disk.rs` | `DiskInfo`, `ReadXL`, `ReadVersion`, `ReadMetadata`, `WriteMetadata`, `RenameFile`, `RenamePart`, `Delete*`, `VerifyFile`, `CheckParts` | Usually metadata, integrity checks, or namespace mutations. |
-| Connection health | `RemoteDisk`, `RemotePeerS3Client`, `PeerRestClient` | TCP connectivity probes and fault/recovery state | Must remain available even if an optional data backend is unavailable. |
-
-### Data plane candidates
-
-These paths move object shard bytes or stream potentially large disk data and
-are the only reasonable first candidates for a pluggable transport.
-
-| Priority | Path | Current client | Current server | Current transport | Why it matters |
-| --- | --- | --- | --- | --- | --- |
-| P0 | `read_file_stream` | `RemoteDisk::read_file_stream` | `handle_read_file` in `http_service.rs` | HTTP `GET /rustfs/rpc/read_file_stream` with a streaming response body | Main remote disk read stream used by bitrot readers and erasure reads. |
-| P0 | `put_file_stream` | `RemoteDisk::create_file` and `RemoteDisk::append_file` | `handle_put_file` in `http_service.rs` | HTTP `PUT /rustfs/rpc/put_file_stream` with a streaming request body | Main remote disk write stream used by bitrot writers and erasure writes. |
-| P1 | `walk_dir` | `RemoteDisk::walk_dir` | `handle_walk_dir` in `http_service.rs` | HTTP `GET /rustfs/rpc/walk_dir` with a streamed metadata listing | Can be high-volume during scans/healing, but it is metadata-oriented rather than object byte data. |
-| P1 | `ReadAll` / `WriteAll` | `RemoteDisk::read_all` / `write_all` | gRPC unary disk handlers | gRPC unary `bytes` payload | Moves bytes today, but should be measured before treating it as a high-throughput data path. |
-| P2 | proto `WriteStream` / `ReadAt` | currently not used | currently returns unimplemented | gRPC streaming definitions exist but are not implemented | Possible future API shape, not a current production path. |
-
-## Current Object Write Path
-
-For object PUTs in distributed erasure mode, the relevant flow is:
-
-1. Upper storage layers prepare object data and erasure metadata.
-2. `SetDisks` selects local and remote disks.
-3. `create_bitrot_writer` calls `disk.create_file(...)` for each shard writer.
-4. For a remote disk, `RemoteDisk::create_file` returns an `HttpWriter`.
-5. `HttpWriter` sends an HTTP `PUT` to `/rustfs/rpc/put_file_stream`.
-6. The remote node's `handle_put_file` opens the local file writer and copies
-   incoming body chunks into it.
-7. `Erasure::encode` writes shards through `MultiWriter` to all selected
-   writers while enforcing write quorum.
-
-This is the primary write data-plane candidate.
-
-## Current Object Read Path
-
-For object GETs and repair reads in distributed erasure mode, the relevant flow is:
-
-1. `SetDisks` prepares shard readers for the selected disks.
-2. `create_bitrot_reader` uses local zero-copy only when `disk.is_local()`.
-3. For a remote disk, it calls `disk.read_file_stream(...)`.
-4. `RemoteDisk::read_file_stream` returns an `HttpReader`.
-5. `HttpReader` sends an HTTP `GET` to `/rustfs/rpc/read_file_stream`.
-6. The remote node's `handle_read_file` opens the local disk stream and returns
-   it as an HTTP streaming body.
-7. The erasure decoder reads from the shard streams and reconstructs the object.
-
-This is the primary read data-plane candidate.
-
-## Existing Metrics and Benchmark Surface
-
-RustFS already has coarse internode metrics in `crates/common/src/internode_metrics.rs`:
-
-- sent bytes
-- received bytes
-- outgoing requests
-- incoming requests
-- errors
-- dial errors
-- average dial time
-
-These metrics are useful as a starting point, but they are not enough for a
-transport RFC. A transport benchmark needs route-level and operation-level
-measurements for at least:
-
-- `read_file_stream`
-- `put_file_stream`
-- `walk_dir`
-- gRPC `ReadAll` / `WriteAll`
-- gRPC control-plane request volume
-
-Existing benchmark assets:
-
-- `scripts/run_object_batch_bench.sh`
-- `scripts/run_object_batch_bench_enhanced.sh`
-- `scripts/run_object_batch_bench_abc.sh`
-- `scripts/run_four_node_cluster_failover_bench.sh`
-- `scripts/run_internode_transport_baseline.sh` (scenario matrix wrapper for local vs distributed TCP baseline artifacts)
-- Criterion benches under `crates/ecstore/benches/`
-
-These mostly cover S3/object workload or erasure coding performance. They do
-not yet isolate internode transport cost.
-
-## Required TCP Baseline
-
-Before adding a transport abstraction or any RDMA backend, collect a baseline
-for the current TCP/HTTP/gRPC implementation.
-
-### Topology
-
-Minimum:
-
-- 1-node local erasure deployment, to measure local disk and erasure overhead.
-- 4-node distributed erasure deployment, to measure internode overhead.
-
-Preferred:
-
-- Same host count and disk layout for every run.
-- Dedicated network interface or isolated VLAN.
-- Fixed CPU governor and no unrelated background load.
-- Recorded kernel version, NIC model, MTU, RustFS commit, Rust toolchain, and
-  benchmark tool versions.
-
-### Workloads
-
-| Workload | Sizes | Concurrency | Main signal |
-| --- | --- | --- | --- |
-| S3 PUT | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end write throughput and tail latency. |
-| S3 GET | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end read throughput and tail latency. |
-| Remote disk stream read | shard-sized ranges from `read_file_stream` | 1, 16, 64 | Isolated internode read path. |
-| Remote disk stream write | shard-sized writes through `put_file_stream` | 1, 16, 64 | Isolated internode write path. |
-| Healing / repair | missing disk or missing shard scenario | controlled | Rebuild throughput and read/write amplification. |
-| Scanner walk | large bucket/object namespace | controlled | Metadata streaming pressure, not primary RDMA target. |
-
-### Measurements
-
-Collect:
-
-- throughput in bytes/s and objects/s
-- p50, p95, p99, and max latency
-- CPU utilization per process and per core
-- memory RSS and allocation pressure where available
-- `rustfs_system_network_internode_*` metrics
-- TCP retransmits, socket errors, and NIC throughput
-- disk throughput and utilization
-- failure/retry/fallback counts
-
-The baseline should produce a machine-readable artifact, for example
-`target/bench/internode-transport/<timestamp>/summary.csv`, plus the exact
-commands and configuration used.
-
-### Baseline runner entry point
-
-Use `scripts/run_internode_transport_baseline.sh` to execute a reproducible
-S3 PUT/GET matrix against `local` and `distributed` scenarios and export:
-
-- `summary.csv` (throughput/latency summary per workload and object size)
-- `internode_metric_deltas.csv` (operation-level internode metric deltas when
-  `--metrics-url` is provided)
-
-## Transport Abstraction Proposal
-
-### Design principle
-
-Keep `NodeService` as the control plane. Introduce a separate data transport
-only below `RemoteDisk`, where remote disk byte streams are opened today.
-
-The first implementation should be a no-behavior-change TCP/HTTP backend that
-wraps the current `HttpReader`, `HttpWriter`, and `/rustfs/rpc/*` handlers.
-Only after that wrapper is benchmarked should an experimental RDMA/RoCE backend
-be considered.
-
-### Candidate boundary
-
-The narrowest useful boundary is remote disk stream transfer:
-
-```rust
-#[async_trait::async_trait]
-pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
-    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
-    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
-    async fn walk_dir(&self, request: WalkDirStreamRequest, writer: &mut dyn AsyncWrite) -> Result<()>;
-    fn capabilities(&self) -> InternodeTransportCapabilities;
-}
-```
-
-Initial request fields should mirror the current HTTP query parameters:
-
-- peer endpoint
-- disk reference
-- volume
-- path
-- offset
-- length
-- append/create mode
-- expected size
-- auth or transfer token material
-
-The initial TCP backend can keep the current signed HTTP URLs internally.
-
-### Integration point
-
-`RemoteDisk` should delegate only these methods to the data transport:
-
-- `read_file_stream`
-- `read_file_zero_copy` as a wrapper over `read_file_stream` unless the backend
-  supports a stronger zero-copy API
-- `append_file`
-- `create_file`
-- optionally `walk_dir`
-
-All other `RemoteDisk` methods should continue using the current gRPC client
-until measurements prove otherwise.
-
-### Capability model
-
-Avoid hard-coding RDMA assumptions into the generic interface. Use capabilities:
-
-- stream read
-- stream write
-- bounded range read
-- bidirectional streaming
-- registered memory support
-- scatter/gather support
-- zero-copy receive into caller-owned buffers
-- authenticated out-of-band transfer
-- transport fallback support
-
-The first TCP backend should report only capabilities that it actually provides.
-
-## TCP Fallback Requirements
-
-TCP/HTTP/gRPC must remain the default and required backend.
-
-Fallback rules:
-
-- If no explicit data transport is configured, use the current TCP/HTTP
-  implementation.
-- If an experimental backend fails initialization, either fail fast with a clear
-  error or fall back to TCP only when the configured policy allows fallback.
-- Runtime fallback must preserve object correctness and quorum semantics.
-- Fallback events must be logged and counted in metrics.
-- CI and local development must not require RDMA-capable hardware.
-
-Suggested future configuration shape:
-
-```text
-RUSTFS_INTERNODE_DATA_TRANSPORT=tcp
-RUSTFS_INTERNODE_DATA_TRANSPORT_FALLBACK=tcp
-```
-
-Do not add these settings until there is an implementation PR that uses them.
-
-## Future RDMA/RoCE/InfiniBand Boundary
-
-A future RDMA backend should be experimental and feature-gated. It should be
-designed as an optional data-plane backend, not as a replacement for the gRPC
-control plane.
-
-Required design areas:
-
-- peer capability discovery over the existing gRPC control plane
-- connection management and health mapping into existing disk fault handling
-- memory registration lifecycle and registration cache
-- buffer ownership, pinning, alignment, and lifetime rules
-- scatter/gather behavior for erasure shards
-- authentication and authorization for out-of-band data transfers
-- encryption/TLS-equivalent story or a documented deployment boundary
-- timeout, cancellation, retry, and fallback behavior
-- metrics for registration cost, transfer latency, bytes, queue depth, retries,
-  fallback, and errors
-- hardware and kernel compatibility matrix
-
-The first RDMA prototype should target `read_file_stream` and `put_file_stream`
-only. `walk_dir`, metadata RPCs, locks, admin RPCs, and bucket coordination
-should remain on gRPC unless a later benchmark identifies a specific bottleneck.
-
-## DPU, DOCA, DPDK, SPDK, and SmartNIC Notes
-
-These technologies should not drive the first abstraction:
-
-- DPU/BlueField/DOCA may become relevant for TLS, checksum, compression, or
-  storage/network offload, but they are vendor- and deployment-specific.
-- DPDK is a poor first fit because RustFS is currently an HTTP/S3 object store
-  and does not have a custom packet data plane.
-- SPDK may be relevant only if RustFS adds a raw block or NVMe-oriented local
-  storage backend. The current disk model is filesystem-based.
-- SmartNIC offload should be discussed only after the data-plane boundary and
-  baseline metrics show where CPU is spent.
-
-## Suggested PR Sequence
-
-1. Add this RFC and the current-path classification.
-2. Add route-level internode metrics for `/rustfs/rpc/read_file_stream`,
-   `/rustfs/rpc/put_file_stream`, `/rustfs/rpc/walk_dir`, and gRPC disk byte
-   calls.
-3. Add an internode transport benchmark harness that can run against a local
-   multi-node cluster and produce repeatable artifacts.
-4. Introduce an `InternodeDataTransport` wrapper with a TCP/HTTP backend that
-   preserves current behavior.
-5. Move `RemoteDisk` stream methods to the transport wrapper without changing
-   default behavior.
-6. Add an experimental feature-gated RDMA/RoCE backend only after the baseline
-   proves that internode byte transfer is a limiting factor.
-
-## Open Questions
-
-- Which production workload is the primary target: large-object throughput,
-  small-object tail latency, healing throughput, or rebalance throughput?
-- Should `ReadAll` and `WriteAll` stay as gRPC unary calls, or should large
-  payloads be redirected to the data transport?
-- Is `walk_dir` a metadata control stream or a secondary data-plane stream for
-  scanner/healing workloads?
-- What is the acceptable fallback policy for an explicitly configured
-  experimental backend?
-- How should an RDMA backend preserve authentication and encryption guarantees
-  currently provided by signed HTTP requests and TLS-capable gRPC/HTTP clients?
-- What hardware matrix is required before accepting a non-default RDMA backend?
-
-## Immediate Next Steps
-
-- Create a focused issue from this RFC.
-- Add route-level internode metrics before changing transport code.
-- Extend existing benchmark scripts or add a new script to isolate remote disk
-  stream read/write throughput.
-- Keep the first code PR behavior-preserving and TCP-only.

README.md

@@ -7,6 +7,7 @@
   <a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="Build and Push Docker Images" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
   <img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
   <img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
+  <a href="https://discord.gg/NcKBCEJp6P"><img alt="Discord" src="https://img.shields.io/badge/Discord-Join%20Chat-5865F2?logo=discord&logoColor=white" /></a>
   <a href="https://hellogithub.com/repository/rustfs/rustfs" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=b95bcb72bdc340b68f16fdf6790b7d5b&claim_uid=MsbvjYeLDKAH457&theme=small" alt="Featured｜HelloGitHub" /></a>
 </p>
 
@@ -115,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.7
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -295,6 +296,7 @@ If you have any questions or need assistance:
 - [Documentation](https://docs.rustfs.com) - The manual you should read
 - [Changelog](https://github.com/rustfs/rustfs/releases) - What we broke and fixed
 - [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Where the community lives
+- [Discord](https://discord.gg/NcKBCEJp6P) - Chat with the RustFS community
 
 ## Contact
 

README_ZH.md

@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.7
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

crates/AGENTS.md

@@ -14,6 +14,41 @@ Applies to all paths under `crates/`.
 - Keep integration tests under each crate's `tests/` directory.
 - Add regression tests for bug fixes and behavior changes.
 
+## Error Type Design
+
+- Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`.
+- Do not use `Box<dyn Error>` or `Box<dyn Error + Send + Sync>` in public trait methods or struct methods. Define a concrete error type with specific variants.
+- When implementing `std::error::Error`, always override `fn source()` if you store an inner error. Breaking the error chain makes debugging impossible.
+- Internal helpers that return `Result<_, String>` and are immediately wrapped via `.map_err(Error::other)` should return the actual error type directly.
+
+## Concurrency
+
+- Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths.
+- Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics).
+- When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff.
+- `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`.
+
+## Recursion Safety
+
+- Recursive tree/graph traversals must have a depth limit (e.g., `max_depth` counter) or use an iterative approach with an explicit `Vec` stack.
+- This applies to cache trees, directory walks, and any user-influenced hierarchy.
+- A corrupted or malicious input must not be able to overflow the thread stack.
+
+## Type Casting
+
+- Never use `as` for numeric conversions that may truncate or overflow. Use `try_into()` with explicit error handling, or clamp with `value.max(0) as usize` when the domain is bounded.
+- `f64 as usize` saturates but is fragile; clamp to `[0, usize::MAX as f64]` first.
+- Treat every `as` cast in a PR review as a potential bug; require justification.
+
+## Testing
+
+- Keep unit tests close to the module they test.
+- Keep integration tests under each crate's `tests/` directory.
+- Add regression tests for bug fixes and behavior changes.
+- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test.
+- In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input.
+
 ## Async and Performance
 
 - Keep async paths non-blocking.

crates/common/src/globals.rs

@@ -17,6 +17,7 @@
 use chrono::{DateTime, Utc};
 use std::collections::HashMap;
 use std::sync::LazyLock;
+use std::sync::atomic::{AtomicU64, Ordering};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -27,6 +28,7 @@ pub static GLOBAL_RUSTFS_ADDR: LazyLock<RwLock<String>> = LazyLock::new(|| RwLoc
 pub static GLOBAL_CONN_MAP: LazyLock<RwLock<HashMap<String, Channel>>> = LazyLock::new(|| RwLock::new(HashMap::new()));
 pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
+pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 /// Global initialization time of the RustFS node.
 pub static GLOBAL_INIT_TIME: LazyLock<RwLock<Option<DateTime<Utc>>>> = LazyLock::new(|| RwLock::new(None));
 
@@ -88,6 +90,16 @@ pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
+/// Set the global outbound TLS generation.
+pub fn set_global_outbound_tls_generation(generation: u64) {
+    GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
+}
+
+/// Get the global outbound TLS generation.
+pub fn get_global_outbound_tls_generation() -> u64 {
+    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
+}
+
 /// Evict a stale/dead connection from the global connection cache.
 /// This is critical for cluster recovery when a node dies unexpectedly (e.g., power-off).
 /// By removing the cached connection, subsequent requests will establish a fresh connection.

crates/common/src/heal_channel.rs

@@ -110,6 +110,25 @@ pub enum HealScanMode {
     Deep = 2,
 }
 
+impl HealScanMode {
+    pub const fn as_str(self) -> &'static str {
+        match self {
+            Self::Unknown => "unknown",
+            Self::Normal => "normal",
+            Self::Deep => "deep",
+        }
+    }
+
+    pub const fn from_u8(value: u8) -> Option<Self> {
+        match value {
+            0 => Some(Self::Unknown),
+            1 => Some(Self::Normal),
+            2 => Some(Self::Deep),
+            _ => None,
+        }
+    }
+}
+
 impl Serialize for HealScanMode {
     fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
     where
@@ -137,12 +156,7 @@ impl<'de> Deserialize<'de> for HealScanMode {
             where
                 E: serde::de::Error,
             {
-                match value {
-                    0 => Ok(HealScanMode::Unknown),
-                    1 => Ok(HealScanMode::Normal),
-                    2 => Ok(HealScanMode::Deep),
-                    _ => Err(E::custom(format!("invalid HealScanMode value: {value}"))),
-                }
+                HealScanMode::from_u8(value).ok_or_else(|| E::custom(format!("invalid HealScanMode value: {value}")))
             }
 
             fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>
@@ -260,9 +274,17 @@ pub enum HealChannelCommand {
         response_tx: oneshot::Sender<Result<HealAdmissionResult, String>>,
     },
     /// Query heal task status
-    Query { heal_path: String, client_token: String },
+    Query {
+        heal_path: String,
+        client_token: String,
+        response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
+    },
     /// Cancel heal task
-    Cancel { heal_path: String },
+    Cancel {
+        heal_path: String,
+        client_token: String,
+        response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
+    },
 }
 
 /// Heal request from admin to ahm
@@ -407,14 +429,36 @@ pub async fn send_heal_request(request: HealChannelRequest) -> Result<(), String
     }
 }
 
+async fn receive_heal_channel_response(
+    response_rx: oneshot::Receiver<Result<HealChannelResponse, String>>,
+) -> Result<HealChannelResponse, String> {
+    response_rx
+        .await
+        .map_err(|e| format!("Failed to receive heal channel response: {e}"))?
+}
+
 /// Send heal query request
-pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<(), String> {
-    send_heal_command(HealChannelCommand::Query { heal_path, client_token }).await
+pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    send_heal_command(HealChannelCommand::Query {
+        heal_path,
+        client_token,
+        response_tx,
+    })
+    .await?;
+    receive_heal_channel_response(response_rx).await
 }
 
 /// Send heal cancel request
-pub async fn cancel_heal_task(heal_path: String) -> Result<(), String> {
-    send_heal_command(HealChannelCommand::Cancel { heal_path }).await
+pub async fn cancel_heal_task(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    send_heal_command(HealChannelCommand::Cancel {
+        heal_path,
+        client_token,
+        response_tx,
+    })
+    .await?;
+    receive_heal_channel_response(response_rx).await
 }
 
 /// Create a new heal request

crates/config/README.md

@@ -70,6 +70,27 @@ Current guidance:
 - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` (deprecated alias for compatibility)
 - `RUSTFS_SCANNER_IDLE_MODE` (canonical)
 - `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical)
+
+## Health compatibility switches
+
+- `RUSTFS_HEALTH_ENDPOINT_ENABLE`
+  - controls canonical `/health`, `/health/live`, and `/health/ready` endpoint exposure.
+- `RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE`
+  - enables minimal payload mode for GET health responses (`status`, `ready` only).
+- `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
+  - TTL for readiness cache evaluation.
+- `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
+  - enables busy protection behavior for health probes.
+  - default is `false`.
+- `RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS`
+  - max active HTTP requests; health probes return `429` when active requests reach or exceed this value.
+  - `0` disables thresholding even if busy protection is enabled.
+- `RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE`
+  - enables KMS readiness enforcement for `/health/ready`.
+  - default is `false`.
 
 ## Drive timeout environment variables
 
@@ -83,6 +104,28 @@ Legacy compatibility fallback:
 - `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION`
   This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.
 
+Drive timeout health-action policy:
+- `RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION`
+  - `mark_failure` (default): timeout marks failure and may transition drive runtime state.
+  - `ignore_scanner`: timeout does not mark failure for scanner-sensitive operations (`walk_dir`, `read_metadata`, `list_dir`, `disk_info`).
+
+Drive timeout profile preset:
+- `RUSTFS_DRIVE_TIMEOUT_PROFILE`
+  - `default` (default): keep current timeout defaults.
+  - `high_latency`: use 60s default timeout for scanner-sensitive operations when no per-operation timeout override is set (`read_metadata`, `disk_info`, `list_dir`, `walk_dir`, `walk_dir_stall`).
+- Precedence:
+  - Explicit per-operation timeout env (`RUSTFS_DRIVE_*_TIMEOUT_SECS`) takes highest precedence.
+  - Then `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION` legacy fallback.
+  - Then the profile-derived default (`default` or `high_latency`).
+
+## Startup filesystem boundary policy
+
+- `RUSTFS_UNSUPPORTED_FS_POLICY` controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
+  - `warn` (default): log warning and continue startup.
+  - `fail`: abort startup with an error.
+
+RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard.
+
 ## 📄 License
 
 This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.

crates/config/src/audit/kafka.rs

@@ -21,10 +21,14 @@ pub const ENV_AUDIT_KAFKA_TLS_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_TLS_ENABLE";
 pub const ENV_AUDIT_KAFKA_TLS_CA: &str = "RUSTFS_AUDIT_KAFKA_TLS_CA";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_KAFKA_SASL_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_SASL_ENABLE";
+pub const ENV_AUDIT_KAFKA_SASL_MECHANISM: &str = "RUSTFS_AUDIT_KAFKA_SASL_MECHANISM";
+pub const ENV_AUDIT_KAFKA_SASL_USERNAME: &str = "RUSTFS_AUDIT_KAFKA_SASL_USERNAME";
+pub const ENV_AUDIT_KAFKA_SASL_PASSWORD: &str = "RUSTFS_AUDIT_KAFKA_SASL_PASSWORD";
 pub const ENV_AUDIT_KAFKA_QUEUE_DIR: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_DIR";
 pub const ENV_AUDIT_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 14] = &[
     ENV_AUDIT_KAFKA_ENABLE,
     ENV_AUDIT_KAFKA_BROKERS,
     ENV_AUDIT_KAFKA_TOPIC,
@@ -33,6 +37,10 @@ pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
     ENV_AUDIT_KAFKA_TLS_CA,
     ENV_AUDIT_KAFKA_TLS_CLIENT_CERT,
     ENV_AUDIT_KAFKA_TLS_CLIENT_KEY,
+    ENV_AUDIT_KAFKA_SASL_ENABLE,
+    ENV_AUDIT_KAFKA_SASL_MECHANISM,
+    ENV_AUDIT_KAFKA_SASL_USERNAME,
+    ENV_AUDIT_KAFKA_SASL_PASSWORD,
     ENV_AUDIT_KAFKA_QUEUE_DIR,
     ENV_AUDIT_KAFKA_QUEUE_LIMIT,
 ];
@@ -47,6 +55,10 @@ pub const AUDIT_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,

crates/config/src/constants/app.rs

@@ -155,11 +155,15 @@ pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
-/// Environment variable to explicitly allow public default root credentials.
+/// Environment variable controlling startup behavior when unsupported filesystem types are detected.
 ///
-/// This is intended for local development only. Production startup paths should
-/// provide non-default `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` values.
-pub const ENV_RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS: &str = "RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS";
+/// Accepted values:
+/// - "warn" (default): log a warning and continue startup
+/// - "fail": abort startup with an error
+pub const ENV_RUSTFS_UNSUPPORTED_FS_POLICY: &str = "RUSTFS_UNSUPPORTED_FS_POLICY";
+pub const RUSTFS_UNSUPPORTED_FS_POLICY_WARN: &str = "warn";
+pub const RUSTFS_UNSUPPORTED_FS_POLICY_FAIL: &str = "fail";
+pub const DEFAULT_RUSTFS_UNSUPPORTED_FS_POLICY: &str = RUSTFS_UNSUPPORTED_FS_POLICY_WARN;
 
 /// Environment variable for server OBS endpoint.
 pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";

crates/config/src/constants/drive.rs

@@ -47,6 +47,16 @@ pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;
 pub const ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS";
 pub const DEFAULT_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: u64 = 5;
 
+/// Timeout-to-health transition policy for drive operations.
+///
+/// Accepted values:
+/// - "mark_failure" (default): timeout marks failure and may transition runtime state.
+/// - "ignore_scanner": timeout does not mark failure for scanner-sensitive operations.
+pub const ENV_DRIVE_TIMEOUT_HEALTH_ACTION: &str = "RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION";
+pub const DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE: &str = "mark_failure";
+pub const DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER: &str = "ignore_scanner";
+pub const DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION: &str = DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE;
+
 /// Number of consecutive failures before a suspect drive is classified as offline.
 pub const ENV_DRIVE_SUSPECT_FAILURE_THRESHOLD: &str = "RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD";
 pub const DEFAULT_DRIVE_SUSPECT_FAILURE_THRESHOLD: u64 = 2;
@@ -66,3 +76,19 @@ pub const DEFAULT_DRIVE_OFFLINE_GRACE_PERIOD_SECS: u64 = 30;
 /// Duration in seconds after which a recovered drive is classified as long offline.
 pub const ENV_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: &str = "RUSTFS_DRIVE_LONG_OFFLINE_THRESHOLD_SECS";
 pub const DEFAULT_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: u64 = 172_800;
+
+/// Drive timeout profile preset.
+///
+/// Accepted values:
+/// - "default": keep current timeout defaults.
+/// - "high_latency": use a higher default timeout preset for scanner-sensitive and metadata operations.
+///
+/// Explicit per-operation overrides (`RUSTFS_DRIVE_*_TIMEOUT_SECS`) still take precedence.
+pub const ENV_DRIVE_TIMEOUT_PROFILE: &str = "RUSTFS_DRIVE_TIMEOUT_PROFILE";
+pub const DRIVE_TIMEOUT_PROFILE_DEFAULT: &str = "default";
+pub const DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY: &str = "high_latency";
+pub const DEFAULT_DRIVE_TIMEOUT_PROFILE: &str = DRIVE_TIMEOUT_PROFILE_DEFAULT;
+
+/// Timeout preset (seconds) used when `RUSTFS_DRIVE_TIMEOUT_PROFILE=high_latency`
+/// and no per-operation timeout override is provided.
+pub const DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS: u64 = 60;

crates/config/src/constants/health.rs

@@ -26,3 +26,20 @@ pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
 /// When enabled, only `status` and `ready` fields are returned.
 pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
 pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;
+
+/// Enable busy protection for health probes.
+/// When enabled with a positive request threshold, alias health probes may
+/// return 429 when active HTTP requests exceed the threshold.
+pub const ENV_HEALTH_COMPAT_BUSY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_CHECK_ENABLE: bool = false;
+
+/// Max active HTTP requests; alias health probes report busy (429) when active requests reach or exceed this value.
+/// Set to 0 to disable thresholding even when busy protection is enabled.
+pub const ENV_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: &str = "RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: usize = 0;
+
+/// Enable KMS readiness check for alias readiness probes.
+/// When enabled, `/health/ready` additionally requires KMS service to be
+/// in running state if a global KMS manager exists.
+pub const ENV_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: bool = false;

crates/config/src/constants/internode.rs

@@ -36,6 +36,12 @@ pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 10;
 pub const ENV_RUSTFS_INTERNODE_DATA_TRANSPORT: &str = "RUSTFS_INTERNODE_DATA_TRANSPORT";
 pub const DEFAULT_INTERNODE_DATA_TRANSPORT: &str = "tcp-http";
 
+/// Legacy alias for "tcp-http". Both values select the TCP/HTTP transport backend.
+pub const INTERNODE_DATA_TRANSPORT_TCP: &str = "tcp";
+
+/// Known internode transport backend names accepted by the config parser.
+pub const KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS: &[&str] = &[DEFAULT_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP];
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -64,5 +70,7 @@ mod tests {
         assert_eq!(ENV_INTERNODE_RPC_TIMEOUT_SECS, "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS");
         assert_eq!(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, "RUSTFS_INTERNODE_DATA_TRANSPORT");
         assert_eq!(DEFAULT_INTERNODE_DATA_TRANSPORT, "tcp-http");
+        assert_eq!(INTERNODE_DATA_TRANSPORT_TCP, "tcp");
+        assert_eq!(KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS, &["tcp-http", "tcp"]);
     }
 }

crates/config/src/constants/object.rs

@@ -281,6 +281,40 @@ pub const ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT: &str = "RUSTFS_OBJECT_LOCK_ACQUIRE_TI
 /// Default lock acquisition timeout: 5 seconds.
 pub const DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT: u64 = 5;
 
+/// Environment variable to enable object namespace lock diagnostics.
+///
+/// When enabled, RustFS emits slow lock acquisition and long lock hold
+/// warnings for object-level namespace locks. This is intended for
+/// production debugging of contention on hot object keys.
+///
+/// Default: false (disabled, can be overridden by `RUSTFS_OBJECT_LOCK_DIAG_ENABLE`).
+pub const ENV_OBJECT_LOCK_DIAG_ENABLE: &str = "RUSTFS_OBJECT_LOCK_DIAG_ENABLE";
+
+/// Default: object lock diagnostics are disabled.
+pub const DEFAULT_OBJECT_LOCK_DIAG_ENABLE: bool = false;
+
+/// Environment variable for the slow object lock acquisition threshold in milliseconds.
+///
+/// When a read or write namespace lock takes at least this long to acquire,
+/// RustFS emits a warning with the operation name and object key.
+///
+/// Default: 500 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS";
+
+/// Default slow object lock acquisition threshold: 500 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: u64 = 500;
+
+/// Environment variable for the long object lock hold threshold in milliseconds.
+///
+/// When a namespace lock guard is held for at least this long, RustFS emits
+/// a warning when the guard is dropped.
+///
+/// Default: 1000 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_HOLD_MS";
+
+/// Default long object lock hold threshold: 1000 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: u64 = 1000;
+
 // ============================================================================
 // I/O priority scheduling configuration
 // ============================================================================

crates/config/src/constants/oidc.rs

@@ -28,6 +28,7 @@ pub const OIDC_GROUPS_CLAIM: &str = "groups_claim";
 pub const OIDC_ROLES_CLAIM: &str = "roles_claim";
 pub const OIDC_EMAIL_CLAIM: &str = "email_claim";
 pub const OIDC_USERNAME_CLAIM: &str = "username_claim";
+pub const OIDC_HIDE_FROM_UI: &str = "hide_from_ui";
 
 // Environment variable names for OIDC
 pub const ENV_IDENTITY_OPENID_ENABLE: &str = "RUSTFS_IDENTITY_OPENID_ENABLE";
@@ -46,9 +47,10 @@ pub const ENV_IDENTITY_OPENID_GROUPS_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_GROUP
 pub const ENV_IDENTITY_OPENID_ROLES_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_ROLES_CLAIM";
 pub const ENV_IDENTITY_OPENID_EMAIL_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM";
 pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM";
+pub const ENV_IDENTITY_OPENID_HIDE_FROM_UI: &str = "RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI";
 
 /// List of all environment variable keys for an OIDC provider.
-pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
+pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
     ENV_IDENTITY_OPENID_ENABLE,
     ENV_IDENTITY_OPENID_CONFIG_URL,
     ENV_IDENTITY_OPENID_CLIENT_ID,
@@ -65,6 +67,7 @@ pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
     ENV_IDENTITY_OPENID_ROLES_CLAIM,
     ENV_IDENTITY_OPENID_EMAIL_CLAIM,
     ENV_IDENTITY_OPENID_USERNAME_CLAIM,
+    ENV_IDENTITY_OPENID_HIDE_FROM_UI,
 ];
 
 /// A list of all valid configuration keys for an OIDC provider.
@@ -85,6 +88,7 @@ pub const IDENTITY_OPENID_KEYS: &[&str] = &[
     OIDC_ROLES_CLAIM,
     OIDC_EMAIL_CLAIM,
     OIDC_USERNAME_CLAIM,
+    OIDC_HIDE_FROM_UI,
     crate::COMMENT_KEY,
 ];
 

crates/config/src/constants/runtime.rs

@@ -69,6 +69,10 @@ pub const DEFAULT_TRANSITION_QUEUE_CAPACITY: usize = 1000;
 pub const DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS: usize = 100;
 /// Test-only fault injection env var that forces the immediate transition enqueue timeout path.
 pub const ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT: &str = "RUSTFS_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT";
+/// Test-only fault injection env var that forces a number of IAM bootstrap failures.
+pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS";
+/// Test-only env var that overrides the deferred IAM retry interval in debug builds.
+pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
 /// Runtime env var controlling the transition worker count.
 pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
 /// Runtime env var controlling the absolute maximum transition workers.

crates/config/src/constants/scanner.rs

@@ -14,6 +14,76 @@
 
 use std::time::Duration;
 
+/// Scanner admin config subsystem name.
+pub const SCANNER_SUB_SYS: &str = "scanner";
+
+/// Scanner config key selecting the speed preset.
+pub const SCANNER_SPEED: &str = "speed";
+
+/// Scanner config key overriding the cycle interval in seconds.
+pub const SCANNER_CYCLE: &str = "cycle";
+
+/// Scanner config key setting the startup delay in seconds.
+///
+/// For compatibility, this also acts as the scanner cycle interval when
+/// `cycle` is unset.
+pub const SCANNER_START_DELAY: &str = "start_delay";
+
+/// Scanner config key capping one cycle's runtime in seconds.
+pub const SCANNER_CYCLE_MAX_DURATION: &str = "cycle_max_duration";
+
+/// Scanner config key capping objects processed by one cycle.
+pub const SCANNER_CYCLE_MAX_OBJECTS: &str = "cycle_max_objects";
+
+/// Scanner config key capping directories entered by one cycle.
+pub const SCANNER_CYCLE_MAX_DIRECTORIES: &str = "cycle_max_directories";
+
+/// Scanner config key setting the periodic bitrot scan cycle in seconds.
+pub const SCANNER_BITROT_CYCLE: &str = "bitrot_cycle";
+
+/// Scanner config key controlling whether scanner throttling is enabled.
+pub const SCANNER_IDLE_MODE: &str = "idle_mode";
+
+/// Scanner config key controlling scanner cache save timeout in seconds.
+pub const SCANNER_CACHE_SAVE_TIMEOUT: &str = "cache_save_timeout";
+
+/// Scanner config key capping concurrent scanner set tasks.
+pub const SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "max_concurrent_set_scans";
+
+/// Scanner config key capping concurrent scanner disk bucket walks per set.
+pub const SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "max_concurrent_disk_scans";
+
+/// Scanner config key controlling how often object loops yield.
+pub const SCANNER_YIELD_EVERY_N_OBJECTS: &str = "yield_every_n_objects";
+
+/// Scanner config key controlling object version count alerts.
+pub const SCANNER_ALERT_EXCESS_VERSIONS: &str = "alert_excess_versions";
+
+/// Scanner config key controlling retained version size alerts.
+pub const SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "alert_excess_version_size";
+
+/// Scanner config key controlling direct subfolder count alerts.
+pub const SCANNER_ALERT_EXCESS_FOLDERS: &str = "alert_excess_folders";
+
+/// Scanner config keys supported by the admin config subsystem.
+pub const SCANNER_KEYS: &[&str] = &[
+    SCANNER_SPEED,
+    SCANNER_CYCLE,
+    SCANNER_START_DELAY,
+    SCANNER_CYCLE_MAX_DURATION,
+    SCANNER_CYCLE_MAX_OBJECTS,
+    SCANNER_CYCLE_MAX_DIRECTORIES,
+    SCANNER_BITROT_CYCLE,
+    SCANNER_IDLE_MODE,
+    SCANNER_CACHE_SAVE_TIMEOUT,
+    SCANNER_MAX_CONCURRENT_SET_SCANS,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
+    SCANNER_ALERT_EXCESS_VERSIONS,
+    SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_FOLDERS,
+];
+
 /// Canonical environment variable name that specifies the scanner start delay in seconds.
 /// If set, this overrides the cycle interval derived from `RUSTFS_SCANNER_SPEED`.
 /// - Unit: seconds (u64).
@@ -31,6 +101,22 @@ pub const ENV_DATA_SCANNER_START_DELAY_SECS: &str = "RUSTFS_DATA_SCANNER_START_D
 /// - Example: `export RUSTFS_SCANNER_CYCLE=3600` (1 hour)
 pub const ENV_SCANNER_CYCLE: &str = "RUSTFS_SCANNER_CYCLE";
 
+/// Environment variable that caps one scanner cycle's runtime in seconds.
+/// A value of `0` disables the cycle runtime budget.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS=1800`
+pub const ENV_SCANNER_CYCLE_MAX_DURATION_SECS: &str = "RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS";
+
+/// Environment variable that caps objects processed by one scanner cycle.
+/// A value of `0` disables the per-cycle object budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_OBJECTS=1000000`
+pub const ENV_SCANNER_CYCLE_MAX_OBJECTS: &str = "RUSTFS_SCANNER_CYCLE_MAX_OBJECTS";
+
+/// Environment variable that caps directories entered by one scanner cycle.
+/// A value of `0` disables the per-cycle directory budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000`
+pub const ENV_SCANNER_CYCLE_MAX_DIRECTORIES: &str = "RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES";
+
 /// Environment variable that selects the scanner speed preset.
 /// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
 /// Controls the sleep factor, maximum sleep duration, and cycle interval.
@@ -40,6 +126,50 @@ pub const ENV_SCANNER_SPEED: &str = "RUSTFS_SCANNER_SPEED";
 /// Default scanner speed preset.
 pub const DEFAULT_SCANNER_SPEED: &str = "default";
 
+/// Default scanner cycle runtime budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS: u64 = 0;
+
+/// Default scanner per-cycle object budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_OBJECTS: u64 = 0;
+
+/// Default scanner per-cycle directory budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES: u64 = 0;
+
+/// Environment variable that specifies the periodic bitrot scan cycle in seconds.
+/// When set to `0`, `true`, `on`, or `yes`, every scanner cycle runs in deep mode.
+/// When set to `false`, `off`, `no`, or `disabled`, periodic deep scans are disabled.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_BITROT_CYCLE_SECS=2592000` (30 days)
+pub const ENV_SCANNER_BITROT_CYCLE_SECS: &str = "RUSTFS_SCANNER_BITROT_CYCLE_SECS";
+
+/// Default bitrot scan cycle used by the scanner.
+pub const DEFAULT_SCANNER_BITROT_CYCLE_SECS: u64 = 30 * 24 * 60 * 60;
+
+/// Environment variable that controls how many object versions trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS=100`
+pub const ENV_SCANNER_ALERT_EXCESS_VERSIONS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS";
+
+/// Default object version count that triggers scanner alerts.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS: u64 = 100;
+
+/// Environment variable that controls how many cumulative bytes across object versions trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE=1099511627776`
+pub const ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE";
+
+/// Default cumulative object version bytes that trigger scanner alerts.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE: u64 = 1024 * 1024 * 1024 * 1024;
+
+/// Environment variable that controls how many subfolders trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS=65538`
+pub const ENV_SCANNER_ALERT_EXCESS_FOLDERS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS";
+
+/// Default subfolder count that triggers scanner alerts.
+/// Allows Proxmox Backup Server's chunk namespace layout.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS: u64 = 65_538;
+
 /// Environment variable that controls whether the scanner sleeps between operations.
 /// When `true` (default), the scanner throttles itself. When `false`, it runs at full speed.
 /// - Example: `export RUSTFS_SCANNER_IDLE_MODE=false`
@@ -51,9 +181,38 @@ pub const ENV_SCANNER_IDLE_MODE: &str = "RUSTFS_SCANNER_IDLE_MODE";
 /// - Example: `export RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS=30`
 pub const ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS: &str = "RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS";
 
+/// Default scanner cache save timeout in seconds.
+pub const DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS: u64 = 30;
+
+/// Environment variable that caps concurrent scanner set tasks.
+/// A value of `0` keeps the existing topology-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS=2`
+pub const ENV_SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS";
+
+/// Environment variable that caps concurrent scanner disk bucket walks per set.
+/// A value of `0` keeps the existing disk-count-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS=1`
+pub const ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS";
+
+/// Environment variable that controls how often scanner object loops yield to the async runtime.
+/// A value of `0` disables this extra object-count yield.
+/// - Example: `export RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS=32`
+pub const ENV_SCANNER_YIELD_EVERY_N_OBJECTS: &str = "RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS";
+
 /// Default scanner idle mode.
 pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
 
+/// Default set scan concurrency budget.
+/// `0` means no additional limit beyond deployment topology.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 0;
+
+/// Default disk scan concurrency budget.
+/// `0` means no additional limit beyond available disks in the set.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 0;
+
+/// Default object interval for cooperative scanner yields.
+pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
+
 /// Compatibility flag kept for Patch 3 rollback windows.
 ///
 /// Inline scanner heal execution has been removed in favor of heal-candidate enqueue.
@@ -117,15 +276,20 @@ impl ScannerSpeed {
         }
     }
 
-    pub fn from_env_str(s: &str) -> Self {
+    pub fn parse_str(s: &str) -> Option<Self> {
         match s.trim().to_ascii_lowercase().as_str() {
-            "fastest" => Self::Fastest,
-            "fast" => Self::Fast,
-            "slow" => Self::Slow,
-            "slowest" => Self::Slowest,
-            _ => Self::Default,
+            "fastest" => Some(Self::Fastest),
+            "fast" => Some(Self::Fast),
+            "default" => Some(Self::Default),
+            "slow" => Some(Self::Slow),
+            "slowest" => Some(Self::Slowest),
+            _ => None,
         }
     }
+
+    pub fn from_env_str(s: &str) -> Self {
+        Self::parse_str(s).unwrap_or_default()
+    }
 }
 
 impl std::fmt::Display for ScannerSpeed {

crates/config/src/constants/targets.rs

@@ -49,6 +49,10 @@ pub const KAFKA_TLS_ENABLE: &str = "tls_enable";
 pub const KAFKA_TLS_CA: &str = "tls_ca";
 pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const KAFKA_SASL_ENABLE: &str = "sasl_enable";
+pub const KAFKA_SASL_MECHANISM: &str = "sasl_mechanism";
+pub const KAFKA_SASL_USERNAME: &str = "sasl_username";
+pub const KAFKA_SASL_PASSWORD: &str = "sasl_password";
 
 pub const AMQP_URL: &str = "url";
 pub const AMQP_EXCHANGE: &str = "exchange";

crates/config/src/notify/kafka.rs

@@ -22,6 +22,10 @@ pub const NOTIFY_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,
@@ -36,10 +40,14 @@ pub const ENV_NOTIFY_KAFKA_TLS_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_TLS_ENABLE";
 pub const ENV_NOTIFY_KAFKA_TLS_CA: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CA";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_KAFKA_SASL_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_SASL_ENABLE";
+pub const ENV_NOTIFY_KAFKA_SASL_MECHANISM: &str = "RUSTFS_NOTIFY_KAFKA_SASL_MECHANISM";
+pub const ENV_NOTIFY_KAFKA_SASL_USERNAME: &str = "RUSTFS_NOTIFY_KAFKA_SASL_USERNAME";
+pub const ENV_NOTIFY_KAFKA_SASL_PASSWORD: &str = "RUSTFS_NOTIFY_KAFKA_SASL_PASSWORD";
 pub const ENV_NOTIFY_KAFKA_QUEUE_DIR: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_DIR";
 pub const ENV_NOTIFY_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 14] = &[
     ENV_NOTIFY_KAFKA_ENABLE,
     ENV_NOTIFY_KAFKA_BROKERS,
     ENV_NOTIFY_KAFKA_TOPIC,
@@ -48,6 +56,10 @@ pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
     ENV_NOTIFY_KAFKA_TLS_CA,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY,
+    ENV_NOTIFY_KAFKA_SASL_ENABLE,
+    ENV_NOTIFY_KAFKA_SASL_MECHANISM,
+    ENV_NOTIFY_KAFKA_SASL_USERNAME,
+    ENV_NOTIFY_KAFKA_SASL_PASSWORD,
     ENV_NOTIFY_KAFKA_QUEUE_DIR,
     ENV_NOTIFY_KAFKA_QUEUE_LIMIT,
 ];

crates/credentials/Cargo.toml

@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 [package]
 name = "rustfs-credentials"
 edition.workspace = true
@@ -12,9 +26,11 @@ categories = ["web-programming", "development-tools", "data-structures", "securi
 
 [dependencies]
 base64-simd = { workspace = true }
+hmac = { workspace = true }
 rand = { workspace = true }
 serde = { workspace = true }
 serde_json.workspace = true
+sha2 = { workspace = true }
 time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
 
 [lints]

crates/credentials/src/credentials.rs

@@ -12,10 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::{DEFAULT_SECRET_KEY, ENV_RPC_SECRET, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+use crate::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ENV_RPC_SECRET, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+use base64_simd::URL_SAFE_NO_PAD;
+use hmac::{Hmac, KeyInit, Mac};
 use rand::{Rng, RngExt};
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
+use sha2::Sha256;
 use std::collections::HashMap;
 use std::env;
 use std::fmt;
@@ -33,7 +36,12 @@ pub static GLOBAL_RUSTFS_RPC_SECRET: OnceLock<String> = OnceLock::new();
 pub const RPC_SECRET_REQUIRED_MESSAGE: &str = "RPC authentication secret is not configured";
 
 /// Operator-facing guidance for configuring RPC authentication safely.
-pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str = "RUSTFS_RPC_SECRET must be set to a non-default value or RUSTFS_SECRET_KEY must be changed from the default for RPC authentication";
+pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str =
+    "RUSTFS_RPC_SECRET can be set explicitly; otherwise the RPC secret is derived from the active access/secret key pair";
+
+type HmacSha256 = Hmac<Sha256>;
+
+const RPC_SECRET_DERIVATION_CONTEXT: &[u8] = b"rustfs-rpc-secret:v1";
 
 /// Error type for credentials operations
 #[derive(Debug)]
@@ -217,37 +225,59 @@ pub fn gen_secret_key(length: usize) -> std::io::Result<String> {
     Ok(encoded)
 }
 
-/// Get the RPC authentication token from environment variable
+/// Get the RPC authentication token from the environment or derive it from the active credentials.
 ///
 /// # Returns
 /// * `String` - The RPC authentication token
 ///
-fn resolve_rpc_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> Option<String> {
-    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
-        return (secret != DEFAULT_SECRET_KEY).then(|| secret.to_string());
+fn normalize_rpc_secret(secret: &str) -> Option<String> {
+    let secret = secret.trim();
+    (!secret.is_empty() && secret != DEFAULT_SECRET_KEY && secret != DEFAULT_ACCESS_KEY).then(|| secret.to_string())
+}
+
+fn derive_rpc_secret(access_key: &str, secret_key: &str) -> Option<String> {
+    let access_key = access_key.trim();
+    let secret_key = secret_key.trim();
+
+    if access_key.is_empty() || secret_key.is_empty() {
+        return None;
     }
 
-    global_secret
-        .map(str::trim)
-        .filter(|secret| !secret.is_empty() && *secret != DEFAULT_SECRET_KEY)
-        .map(ToOwned::to_owned)
+    let mut mac = <HmacSha256 as KeyInit>::new_from_slice(secret_key.as_bytes()).expect("HMAC can take key of any size");
+    mac.update(RPC_SECRET_DERIVATION_CONTEXT);
+    mac.update(&[0]);
+    mac.update(access_key.as_bytes());
+
+    Some(URL_SAFE_NO_PAD.encode_to_string(mac.finalize().into_bytes()))
+}
+
+fn resolve_rpc_secret(env_secret: Option<&str>, global_access: Option<&str>, global_secret: Option<&str>) -> Option<String> {
+    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
+        return normalize_rpc_secret(secret);
+    }
+
+    match (global_access, global_secret) {
+        (Some(access_key), Some(secret_key)) => derive_rpc_secret(access_key, secret_key),
+        _ => None,
+    }
 }
 
 pub fn try_get_rpc_token() -> std::io::Result<String> {
     if let Some(secret) = GLOBAL_RUSTFS_RPC_SECRET.get() {
-        return resolve_rpc_secret(None, Some(secret)).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
+        return normalize_rpc_secret(secret).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
     }
 
     let env_secret = env::var(ENV_RPC_SECRET).ok();
+    let global_access = get_global_access_key_opt();
     let global_secret = get_global_secret_key_opt();
-    let secret = resolve_rpc_secret(env_secret.as_deref(), global_secret.as_deref())
+    let secret = resolve_rpc_secret(env_secret.as_deref(), global_access.as_deref(), global_secret.as_deref())
         .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE))?;
 
     match GLOBAL_RUSTFS_RPC_SECRET.set(secret.clone()) {
         Ok(()) => Ok(secret),
         Err(_) => GLOBAL_RUSTFS_RPC_SECRET
             .get()
-            .and_then(|stored| resolve_rpc_secret(None, Some(stored)))
+            .and_then(|stored| normalize_rpc_secret(stored))
             .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE)),
     }
 }
@@ -407,7 +437,7 @@ impl Credentials {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::{IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+    use crate::{DEFAULT_ACCESS_KEY, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
     use time::Duration;
 
     #[test]
@@ -510,11 +540,12 @@ mod tests {
         // If it hasn't already been initialized, the test automatically generates logic
         if get_global_action_cred().is_none() {
             init_global_action_credentials(None, None).ok();
-            let ak = get_global_access_key();
-            let sk = get_global_secret_key();
-            assert_eq!(ak.len(), 20);
-            assert_eq!(sk.len(), 32);
         }
+
+        let ak = get_global_access_key();
+        let sk = get_global_secret_key();
+        assert!(ak.len() >= 3);
+        assert!(sk.len() >= 8);
     }
 
     #[test]
@@ -528,10 +559,26 @@ mod tests {
     }
 
     #[test]
-    fn test_resolve_rpc_secret_rejects_default_fallback() {
-        assert!(resolve_rpc_secret(None, None).is_none());
-        assert!(resolve_rpc_secret(None, Some(DEFAULT_SECRET_KEY)).is_none());
-        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-global-secret")).is_none());
+    fn test_gen_access_key_length_and_charset() {
+        let err = gen_access_key(2).expect_err("length below 3 should fail");
+        assert_eq!(err.to_string(), "access key length is too short");
+
+        let key = gen_access_key(20).expect("access key should generate");
+        assert_eq!(key.len(), 20);
+        assert!(key.chars().all(|ch| ch.is_ascii_uppercase() || ch.is_ascii_digit()));
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_allows_default_credentials_for_derivation() {
+        assert!(resolve_rpc_secret(None, None, None).is_none());
+
+        let expected = derive_rpc_secret(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY).expect("secret should derive");
+        assert_eq!(
+            resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).as_deref(),
+            Some(expected.as_str())
+        );
+
+        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-access"), Some("custom-global-secret")).is_none());
     }
 
     #[test]
@@ -551,37 +598,64 @@ mod tests {
 
     #[test]
     fn test_resolve_rpc_secret_accepts_non_default_secret() {
-        assert_eq!(resolve_rpc_secret(Some("custom-rpc-secret"), None).as_deref(), Some("custom-rpc-secret"));
         assert_eq!(
-            resolve_rpc_secret(None, Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("custom-rpc-secret"), None, None).as_deref(),
+            Some("custom-rpc-secret")
+        );
+        let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
+        assert_eq!(
+            resolve_rpc_secret(None, Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
     }
 
     #[test]
     fn test_resolve_rpc_secret_trims_and_falls_back_from_blank_env() {
+        let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
         assert_eq!(
-            resolve_rpc_secret(Some("  custom-rpc-secret  "), None).as_deref(),
+            resolve_rpc_secret(Some("  custom-rpc-secret  "), None, None).as_deref(),
             Some("custom-rpc-secret")
         );
         assert_eq!(
-            resolve_rpc_secret(Some(""), Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some(""), Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
         assert_eq!(
-            resolve_rpc_secret(Some("  "), Some("  custom-global-secret  ")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("  "), Some("  custom-access  "), Some("  custom-global-secret  ")).as_deref(),
+            Some(expected.as_str())
         );
         assert_eq!(
-            resolve_rpc_secret(Some("  "), Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("  "), Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
     }
 
     #[test]
     fn test_resolve_rpc_secret_returns_none_for_trimmed_default_secret() {
         let padded_default_secret = format!("  {}  ", DEFAULT_SECRET_KEY);
-        assert!(resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-global-secret")).is_none());
+        assert!(
+            resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-access"), Some("custom-global-secret"))
+                .is_none()
+        );
+
+        let padded_default_access = format!("  {}  ", DEFAULT_ACCESS_KEY);
+        assert!(
+            resolve_rpc_secret(Some(padded_default_access.as_str()), Some("custom-access"), Some("custom-global-secret"))
+                .is_none()
+        );
+    }
+
+    #[test]
+    fn test_derive_rpc_secret_is_stable_and_not_plaintext() {
+        let first = derive_rpc_secret("custom-access", "custom-secret").expect("secret should derive");
+        let second = derive_rpc_secret("custom-access", "custom-secret").expect("secret should derive");
+
+        assert_eq!(first, second);
+        assert_ne!(first, "custom-access");
+        assert_ne!(first, "custom-secret");
+        assert_ne!(first, format!("{}{}", "custom-access", "custom-secret"));
+        assert!(!first.contains("custom-access"));
+        assert!(!first.contains("custom-secret"));
     }
 
     #[test]

crates/data-usage/src/data_usage.rs

@@ -688,15 +688,14 @@ impl DataUsageCache {
             return;
         }
 
-        let mut leaves = Vec::new();
+        let mut candidates = Vec::new();
         let mut remove = total - limit;
-        add(self, path, &mut leaves);
-        leaves.sort_by_key(|a| a.objects);
+        add(self, path, &mut candidates);
+        candidates.sort_by_key(|a| a.objects);
 
-        while remove > 0 && !leaves.is_empty() {
-            let Some(e) = leaves.first() else {
-                break;
-            };
+        let mut candidate_index = 0;
+        while remove > 0 && candidate_index < candidates.len() {
+            let e = &candidates[candidate_index];
             let candidate = e.path.clone();
             if candidate == *path && !compact_self {
                 break;
@@ -705,7 +704,7 @@ impl DataUsageCache {
             let mut flat = match self.size_recursive(&candidate.key()) {
                 Some(flat) => flat,
                 None => {
-                    leaves.remove(0);
+                    candidate_index += 1;
                     continue;
                 }
             };
@@ -714,8 +713,8 @@ impl DataUsageCache {
             self.delete_recursive(&candidate);
             self.replace_hashed(&candidate, &None, &flat);
 
-            remove -= removing;
-            leaves.remove(0);
+            remove = remove.saturating_sub(removing);
+            candidate_index += 1;
         }
     }
 
@@ -869,23 +868,24 @@ struct Inner {
     path: DataUsageHash,
 }
 
-fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, leaves: &mut Vec<Inner>) {
+fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, candidates: &mut Vec<Inner>) -> usize {
     let e = match data_usage_cache.cache.get(&path.key()) {
         Some(e) => e,
-        None => return,
+        None => return 0,
     };
-    if !e.children.is_empty() {
-        return;
-    }
-
-    let sz = data_usage_cache.size_recursive(&path.key()).unwrap_or_default();
-    leaves.push(Inner {
-        objects: sz.objects,
-        path: path.clone(),
-    });
+    let mut objects = e.objects;
     for ch in e.children.iter() {
-        add(data_usage_cache, &DataUsageHash(ch.clone()), leaves);
+        objects += add(data_usage_cache, &DataUsageHash(ch.clone()), candidates);
     }
+    // Collect internal nodes (with children) as compaction candidates.
+    // Leaf nodes have no children to remove, so compacting them is a no-op.
+    if !e.children.is_empty() {
+        candidates.push(Inner {
+            objects,
+            path: path.clone(),
+        });
+    }
+    objects
 }
 
 fn mark(duc: &DataUsageCache, entry: &DataUsageEntry, found: &mut HashSet<String>) {
@@ -1361,4 +1361,161 @@ mod tests {
         assert_eq!(child_entry.size, 30);
         assert_eq!(child_entry.objects, 3);
     }
+
+    // --- Tests for `add` and `reduce_children_of` (bug fixes) ---
+
+    /// Build a small tree: root -> child1 (leaf), child2 -> grandchild (leaf).
+    fn build_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let c1 = hash_path("bucket/a");
+        let c2 = hash_path("bucket/b");
+        let gc = hash_path("bucket/b/c");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(&root, &None, &DataUsageEntry::default());
+        cache.replace_hashed(
+            &c1,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                size: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &c2,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 2,
+                size: 20,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &gc,
+            &Some(c2.clone()),
+            &DataUsageEntry {
+                objects: 3,
+                size: 30,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    fn build_underflow_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let small = hash_path("bucket/small");
+        let small_a = hash_path("bucket/small/a");
+        let small_b = hash_path("bucket/small/b");
+        let large = hash_path("bucket/large");
+        let large_a = hash_path("bucket/large/a");
+        let large_b = hash_path("bucket/large/b");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(
+            &root,
+            &None,
+            &DataUsageEntry {
+                objects: 100,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&small, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &small_a,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &small_b,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&large, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &large_a,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &large_b,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    #[test]
+    fn test_add_collects_internal_nodes_as_compaction_candidates() {
+        let (cache, root) = build_test_tree();
+        let mut candidates = Vec::new();
+        add(&cache, &root, &mut candidates);
+
+        let mut paths: Vec<String> = candidates.iter().map(|l| l.path.key()).collect();
+        paths.sort();
+        assert_eq!(paths.len(), 2, "add() should find internal nodes with children");
+        assert!(paths.contains(&hash_path("bucket").key()));
+        assert!(paths.contains(&hash_path("bucket/b").key()));
+    }
+
+    #[test]
+    fn test_add_skips_leaf_node() {
+        let mut cache = DataUsageCache::default();
+        let h = hash_path("single-leaf");
+        cache.replace_hashed(
+            &h,
+            &None,
+            &DataUsageEntry {
+                objects: 5,
+                size: 50,
+                ..Default::default()
+            },
+        );
+
+        let mut candidates = Vec::new();
+        add(&cache, &h, &mut candidates);
+        assert!(candidates.is_empty(), "leaf node should not be a compaction candidate");
+    }
+
+    #[test]
+    fn test_reduce_children_of_compacts_internal_node() {
+        let (mut cache, root) = build_test_tree();
+        cache.reduce_children_of(&root, 2, false);
+
+        let entry_c2 = cache.find("bucket/b").unwrap();
+        assert!(entry_c2.compacted, "internal node 'bucket/b' should be compacted");
+        let entry_c1 = cache.find("bucket/a").unwrap();
+        assert!(!entry_c1.compacted, "leaf 'bucket/a' should not be compacted");
+        assert!(cache.find("bucket/b/c").is_none(), "grandchild should be removed");
+    }
+
+    #[test]
+    fn test_reduce_children_of_usize_underflow_saturates() {
+        let (mut cache, root) = build_underflow_test_tree();
+
+        // total children=6, limit=5, remove=1. The smallest candidate removes
+        // two descendants, so plain subtraction would underflow and compact the
+        // next candidate too.
+        cache.reduce_children_of(&root, 5, false);
+
+        assert!(cache.find("bucket/small").is_some_and(|entry| entry.compacted));
+        assert!(cache.find("bucket/small/a").is_none());
+        assert!(cache.find("bucket/small/b").is_none());
+        assert!(cache.find("bucket/large").is_some_and(|entry| !entry.compacted));
+        assert!(cache.find("bucket/large/a").is_some());
+        assert!(cache.find("bucket/large/b").is_some());
+    }
 }

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -0,0 +1,212 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Erasure-set healing regression tests.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, execute_awscurl, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use serial_test::serial;
+    use std::collections::HashSet;
+    use std::path::{Path, PathBuf};
+    use tokio::time::{Duration, sleep};
+    use tracing::info;
+
+    fn has_file_under(path: &Path) -> bool {
+        let Ok(entries) = std::fs::read_dir(path) else {
+            return false;
+        };
+
+        for entry in entries.filter_map(Result::ok) {
+            let path = entry.path();
+            if path.is_dir() {
+                if has_file_under(&path) {
+                    return true;
+                }
+            } else {
+                return true;
+            }
+        }
+
+        false
+    }
+
+    fn object_metadata_exists_on_disk(disk: &Path, bucket: &str, key: &str) -> bool {
+        disk.join(bucket).join(key).join("xl.meta").is_file()
+    }
+
+    async fn assert_object_body(env: &RustFSTestEnvironment, bucket: &str, key: &str, expected: &[u8]) {
+        let client = env.create_s3_client();
+        let response = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("GET should succeed during/after heal");
+        let body = response.body.collect().await.expect("GET body should collect").into_bytes();
+        assert_eq!(body.as_ref(), expected, "object body changed for {key}");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_admin_deep_heal_rebuilds_cleared_disk_in_single_node_erasure_set() {
+        init_logging();
+        info!("Discussion #2964: admin deep heal should rebuild a wiped disk in a 4-disk single-node erasure set");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        let root = PathBuf::from(env.temp_dir.clone());
+        let disk0 = root.join("disk0");
+        let disk1 = root.join("disk1");
+        let disk2 = root.join("disk2");
+        let disk3 = root.join("disk3");
+        for disk in [&disk0, &disk1, &disk2, &disk3] {
+            std::fs::create_dir_all(disk).expect("disk directory should be created");
+        }
+
+        // The test helper always appends env.temp_dir as the final storage path.
+        // Point it at disk3 and pass the other three disks explicitly.
+        env.temp_dir = disk3.to_string_lossy().to_string();
+        let disk0_arg = disk0.to_string_lossy().to_string();
+        let disk1_arg = disk1.to_string_lossy().to_string();
+        let disk2_arg = disk2.to_string_lossy().to_string();
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true")],
+        )
+        .await
+        .expect("Failed to start 4-disk RustFS");
+
+        let client = env.create_s3_client();
+        let bucket = "heal-cleared-disk-regression";
+        let target_object_count = std::env::var("RUSTFS_HEAL_REBUILD_OBJECT_COUNT")
+            .ok()
+            .and_then(|value| value.parse::<usize>().ok())
+            .unwrap_or(4)
+            .max(4);
+        let heal_timeout_secs = std::env::var("RUSTFS_HEAL_REBUILD_TIMEOUT_SECS")
+            .ok()
+            .and_then(|value| value.parse::<u64>().ok())
+            .unwrap_or(60);
+
+        let mut objects: Vec<(String, Vec<u8>, &'static str)> = vec![
+            (
+                "中文/报告-0001.json".to_string(),
+                "{\"message\":\"hello 中文\"}".as_bytes().to_vec(),
+                "application/json",
+            ),
+            (
+                "english/images/photo-0002.jpg".to_string(),
+                vec![0xff, 0xd8, 0xff, 0x00, 0x42, 0x24],
+                "image/jpeg",
+            ),
+            (
+                "mixed/空 格 + symbols @#%.txt".to_string(),
+                b"text object with spaces and symbols".to_vec(),
+                "text/plain; charset=utf-8",
+            ),
+            (
+                "bin/archive-0004.bin".to_string(),
+                (0..=255).collect::<Vec<u8>>(),
+                "application/octet-stream",
+            ),
+        ];
+        for index in objects.len()..target_object_count {
+            objects.push((
+                format!("bulk/prefix-{}/object-{index:04}.txt", index % 17),
+                format!("bulk object {index}: heal regression payload").into_bytes(),
+                "text/plain; charset=utf-8",
+            ));
+        }
+
+        let object_keys = objects.iter().map(|(key, _, _)| key.clone()).collect::<Vec<_>>();
+        let mut remaining_rebuild_keys: HashSet<String> = object_keys.iter().cloned().collect();
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("bucket create should succeed");
+        for (key, body, content_type) in &objects {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .content_type(*content_type)
+                .body(ByteStream::from(body.clone()))
+                .send()
+                .await
+                .expect("PUT should succeed");
+        }
+
+        assert!(has_file_under(&disk0), "disk0 should contain object shards before wipe");
+        env.stop_server();
+
+        std::fs::remove_dir_all(&disk0).expect("disk0 wipe should succeed");
+        std::fs::create_dir_all(&disk0).expect("disk0 should be recreated empty");
+        assert!(!has_file_under(&disk0), "disk0 must be empty before restart");
+
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true")],
+        )
+        .await
+        .expect("Failed to restart 4-disk RustFS after disk wipe");
+        // The helper's Drop cleanup removes env.temp_dir. Reset it to the parent
+        // directory after server startup so all four disk directories are cleaned
+        // without manually deleting a path Drop will also try to remove.
+        env.temp_dir = root.to_string_lossy().to_string();
+
+        let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
+        let heal_url = format!("{}/rustfs/admin/v3/heal/{}?forceStart=true", env.url, bucket);
+        execute_awscurl(&heal_url, "POST", Some(heal_body), &env.access_key, &env.secret_key)
+            .await
+            .expect("admin deep heal should be accepted");
+
+        for _ in 0..heal_timeout_secs {
+            if !remaining_rebuild_keys.is_empty() {
+                let mut rebuilt = Vec::new();
+                for key in &remaining_rebuild_keys {
+                    if object_metadata_exists_on_disk(&disk0, bucket, key) {
+                        rebuilt.push(key.clone());
+                    }
+                }
+                for key in rebuilt {
+                    let _ = remaining_rebuild_keys.remove(&key);
+                }
+            }
+
+            if remaining_rebuild_keys.is_empty() {
+                for (key, body, _) in &objects {
+                    assert_object_body(&env, bucket, key, body).await;
+                }
+
+                env.stop_server();
+                for key in &object_keys {
+                    assert!(
+                        object_metadata_exists_on_disk(&disk0, bucket, key),
+                        "wiped disk should contain rebuilt xl.meta for {key}"
+                    );
+                }
+                return;
+            }
+
+            sleep(Duration::from_secs(1)).await;
+        }
+
+        panic!("admin deep heal did not rebuild all files on the wiped disk within timeout");
+    }
+}

crates/e2e_test/src/kms/README.md

@@ -191,7 +191,7 @@ RUST_LOG=debug cargo test test_local_kms_end_to_end -- --nocapture
 4. **Monitor ports**
    ```bash
    netstat -an | grep 9050
-   curl http://127.0.0.1:9050/minio/health/ready
+   curl http://127.0.0.1:9050/health/ready
    ```
 
 ## 📊 Coverage

crates/e2e_test/src/lib.rs

@@ -63,6 +63,10 @@ mod archive_download_integrity_test;
 #[cfg(test)]
 mod list_objects_v2_pagination_test;
 
+// Regression test for Issue #3107: mc mirror small-bucket listing must not time out.
+#[cfg(test)]
+mod mc_mirror_small_bucket_test;
+
 // Policy variables tests
 #[cfg(test)]
 mod policy;
@@ -114,6 +118,9 @@ mod head_object_range_test;
 #[cfg(test)]
 mod head_object_consistency_test;
 
+#[cfg(test)]
+mod heal_erasure_disk_rebuild_test;
+
 #[cfg(test)]
 mod copy_object_metadata_test;
 
@@ -139,4 +146,7 @@ mod replication_extension_test;
 #[cfg(test)]
 mod snowball_auto_extract_test;
 
+#[cfg(test)]
+mod namespace_lock_quorum_test;
+
 pub mod tls_gen;

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -612,4 +612,421 @@ mod tests {
 
         env.stop_server();
     }
+
+    /// Regression test: delimiter listing must not produce false-positive truncation
+    /// when many raw keys collapse into a small number of CommonPrefixes.
+    ///
+    /// Scenario: 30 objects across 3 directories + 2 direct files under prefix.
+    /// With max_keys=1000, all 5 visible results (3 prefixes + 2 objects) fit in one
+    /// page, so IsTruncated must be false even though raw entry count is much larger.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_delimiter_collapsed_prefix_no_false_truncation() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 delimiter collapsed-prefix no false truncation");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-collapsed-prefix";
+        let prefix = "data/";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create objects under 3 subdirectories (10 each = 30 raw keys)
+        let dirs = ["alpha/", "beta/", "gamma/"];
+        let mut expected_keys = Vec::new();
+        for dir in &dirs {
+            for i in 0..10 {
+                let key = format!("{prefix}{dir}file{i:02}.txt");
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                expected_keys.push(key);
+            }
+        }
+        // Add 2 direct objects under prefix
+        for name in ["readme.txt", "config.json"] {
+            let key = format!("{prefix}{name}");
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(&key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put object");
+            expected_keys.push(key);
+        }
+
+        // List with delimiter="/", max_keys large enough to cover all visible results
+        // Visible: 3 common prefixes + 2 direct objects = 5
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix(prefix)
+            .delimiter("/")
+            .max_keys(1000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        let listed_keys: Vec<String> = output
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key())
+            .map(|k| k.to_string())
+            .collect();
+        let listed_prefixes: Vec<String> = output
+            .common_prefixes()
+            .iter()
+            .filter_map(|cp| cp.prefix())
+            .map(|p| p.to_string())
+            .collect();
+
+        // KEY ASSERTION: IsTruncated must be false because all visible results fit within max_keys
+        let is_truncated = output.is_truncated().unwrap_or(false);
+        assert!(
+            !is_truncated,
+            "BUG: IsTruncated should be false when all visible results ({} objects + {} prefixes = {}) fit within max_keys (1000)",
+            listed_keys.len(),
+            listed_prefixes.len(),
+            listed_keys.len() + listed_prefixes.len()
+        );
+
+        assert!(
+            output.next_continuation_token().is_none(),
+            "NextContinuationToken should be None when IsTruncated is false"
+        );
+
+        // All 32 objects must be covered (listed_keys + keys under listed_prefixes)
+        let total_raw_count = expected_keys.len();
+        let keys_under_prefixes: usize = listed_prefixes
+            .iter()
+            .map(|p| {
+                let dir = p.trim_end_matches('/');
+                expected_keys.iter().filter(|k| k.starts_with(&format!("{dir}/"))).count()
+            })
+            .sum();
+        let covered = listed_keys.len() + keys_under_prefixes;
+
+        assert_eq!(
+            covered,
+            total_raw_count,
+            "Collapsed-prefix listing must cover all {} objects, got {} (keys={}, under_prefixes={})",
+            total_raw_count,
+            covered,
+            listed_keys.len(),
+            keys_under_prefixes
+        );
+
+        info!(
+            "Collapsed-prefix test passed: {} objects covered ({} keys + {} prefixes), IsTruncated=false",
+            covered,
+            listed_keys.len(),
+            listed_prefixes.len()
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: delimiter listing pagination must traverse all keys
+    /// even when the visible result count per page is much smaller than the
+    /// raw entry count (due to CommonPrefix collapse).
+    ///
+    /// Scenario: 1000 objects across 100 directories, paginated with max_keys=50.
+    /// Each page returns up to 50 CommonPrefixes. The server must correctly set
+    /// IsTruncated and provide a valid continuation token across all pages.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_delimiter_small_page_traverses_all() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 delimiter small page traverses all keys");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-delimiter-small-page";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create 1000 objects: 100 directories with 10 files each
+        let mut all_keys = Vec::new();
+        let dirs: Vec<String> = (0..100).map(|i| format!("dir-{:03}/", i)).collect();
+        for dir in &dirs {
+            for i in 0..10 {
+                let key = format!("{dir}file{:02}.txt", i);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                all_keys.push(key);
+            }
+        }
+
+        // Paginate with delimiter="/" and max_keys=50
+        // Visible per page: up to 50 CommonPrefixes
+        let mut listed_keys = Vec::new();
+        let mut listed_prefixes = Vec::new();
+        let mut continuation_token: Option<String> = None;
+        let mut page_count = 0;
+        let mut last_page_is_truncated: bool;
+
+        loop {
+            let mut request = client.list_objects_v2().bucket(bucket).delimiter("/").max_keys(50);
+
+            if let Some(token) = continuation_token.take() {
+                request = request.continuation_token(token);
+            }
+
+            let output = request.send().await.expect("Failed to list objects");
+            last_page_is_truncated = output.is_truncated().unwrap_or(false);
+
+            for obj in output.contents() {
+                if let Some(key) = obj.key() {
+                    listed_keys.push(key.to_string());
+                }
+            }
+            for cp in output.common_prefixes() {
+                if let Some(p) = cp.prefix() {
+                    listed_prefixes.push(p.to_string());
+                }
+            }
+
+            page_count += 1;
+
+            if last_page_is_truncated {
+                continuation_token = output.next_continuation_token().map(|s| s.to_string());
+                assert!(
+                    continuation_token.is_some(),
+                    "BUG: NextContinuationToken must be present when IsTruncated is true"
+                );
+            } else {
+                break;
+            }
+
+            if page_count > 20 {
+                panic!("Too many pages, possible infinite loop in delimiter pagination");
+            }
+        }
+
+        // Last page must have IsTruncated=false
+        assert!(
+            !last_page_is_truncated,
+            "BUG: Last page must have IsTruncated=false after all results returned"
+        );
+
+        // Verify all objects are covered via listed prefixes
+        let keys_under_prefixes: HashSet<String> = all_keys
+            .iter()
+            .filter(|k| {
+                listed_prefixes
+                    .iter()
+                    .any(|p| k.starts_with(&format!("{}/", p.trim_end_matches('/'))))
+            })
+            .cloned()
+            .collect();
+        let listed_set: HashSet<String> = listed_keys.iter().cloned().collect();
+        let covered: HashSet<String> = listed_set.union(&keys_under_prefixes).cloned().collect();
+        let expected_set: HashSet<String> = all_keys.iter().cloned().collect();
+
+        assert_eq!(
+            covered,
+            expected_set,
+            "Delimiter pagination must cover all {} objects, missing: {:?}",
+            expected_set.difference(&covered).count(),
+            expected_set.difference(&covered)
+        );
+
+        info!(
+            "Delimiter small-page test passed: {} objects covered in {} pages",
+            covered.len(),
+            page_count
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: raw entries exceed MaxKeys but all collapse into fewer
+    /// visible CommonPrefixes — IsTruncated must be false because there are no
+    /// more visible results beyond the collapsed prefixes.
+    ///
+    /// Scenario: 10 directories with 200 files each = 2000 raw keys.
+    /// With MaxKeys=1000, the raw entry count exceeds MaxKeys (disk_has_more=true),
+    /// but after delimiter collapse only 10 CommonPrefixes are visible (10 < 1000).
+    /// IsTruncated must be false since there are no additional visible results.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_raw_exceeds_maxkeys_but_visible_below() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 raw > MaxKeys but visible < MaxKeys after collapse");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-raw-exceeds-visible-below";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create 10 directories × 200 files = 2000 raw keys
+        let dir_count = 10;
+        let files_per_dir = 200;
+        let mut all_keys = Vec::new();
+        for d in 0..dir_count {
+            let dir = format!("dir-{:02}/", d);
+            for f in 0..files_per_dir {
+                let key = format!("{}file{:03}.txt", dir, f);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                all_keys.push(key);
+            }
+        }
+
+        // List with delimiter="/", max_keys=1000
+        // Visible: 10 CommonPrefixes (dir-00/ .. dir-09/) = 10 visible << 1000 MaxKeys
+        // But raw entry count (2000+1001 requested) exceeds MaxKeys
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .delimiter("/")
+            .max_keys(1000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        let listed_keys: Vec<String> = output
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key())
+            .map(|k| k.to_string())
+            .collect();
+        let listed_prefixes: Vec<String> = output
+            .common_prefixes()
+            .iter()
+            .filter_map(|cp| cp.prefix())
+            .map(|p| p.to_string())
+            .collect();
+
+        // KEY ASSERTION: IsTruncated must be false because visible results (10)
+        // are far below MaxKeys (1000), even though raw entries exceeded MaxKeys.
+        let is_truncated = output.is_truncated().unwrap_or(false);
+        assert!(
+            !is_truncated,
+            "BUG: IsTruncated should be false when visible results ({} objects + {} prefixes = {}) < MaxKeys (1000), even if raw entries exceeded MaxKeys",
+            listed_keys.len(),
+            listed_prefixes.len(),
+            listed_keys.len() + listed_prefixes.len()
+        );
+
+        assert!(
+            output.next_continuation_token().is_none(),
+            "NextContinuationToken should be None when IsTruncated is false"
+        );
+
+        // All objects must be covered by listed prefixes
+        assert_eq!(
+            listed_prefixes.len(),
+            dir_count,
+            "Expected {} prefixes, got {}",
+            dir_count,
+            listed_prefixes.len()
+        );
+        for d in 0..dir_count {
+            let prefix = format!("dir-{:02}/", d);
+            assert!(listed_prefixes.contains(&prefix), "Missing prefix: {}", prefix);
+        }
+
+        info!(
+            "Raw-exceeds-visible test passed: {} raw keys → {} prefixes, IsTruncated=false",
+            all_keys.len(),
+            listed_prefixes.len()
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: pagination with MaxKeys exceeding S3 limit (1000) and delimiter.
+    /// The server caps MaxKeys to 1000. With delimiter="/", many raw keys collapse into
+    /// few CommonPrefixes. Visible results (prefixes) < capped MaxKeys → IsTruncated=false.
+    ///
+    /// This complements test_list_objects_v2_max_keys_above_limit_returns_token which
+    /// tests the non-delimiter case.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_maxkeys_above_limit_with_delimiter() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 MaxKeys above limit with delimiter");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-maxkeys-limit-delimiter";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // 12 dirs × 100 files = 1200 raw keys
+        let dir_count = 12;
+        let files_per_dir = 100;
+        for d in 0..dir_count {
+            for f in 0..files_per_dir {
+                let key = format!("dir-{:02}/file{:03}.txt", d, f);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+            }
+        }
+
+        // With delimiter: 12 CommonPrefixes visible, all fit within capped 1000
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .delimiter("/")
+            .max_keys(2000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        assert_eq!(
+            output.common_prefixes().len(),
+            dir_count,
+            "Expected {} prefixes, got {}",
+            dir_count,
+            output.common_prefixes().len()
+        );
+        assert_eq!(output.max_keys(), Some(1000));
+        // 12 visible < 1000 capped MaxKeys → not truncated
+        assert!(
+            !output.is_truncated().unwrap_or(false),
+            "BUG: IsTruncated should be false when visible ({}) < capped MaxKeys (1000)",
+            output.common_prefixes().len()
+        );
+
+        info!("MaxKeys above limit with delimiter test passed");
+
+        env.stop_server();
+    }
 }

crates/e2e_test/src/mc_mirror_small_bucket_test.rs

@@ -0,0 +1,124 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, RustFSTestEnvironment};
+use serial_test::serial;
+use std::path::Path;
+use std::process::Command;
+use std::time::Duration;
+use tokio::fs;
+use tokio::time::timeout;
+use tracing::info;
+
+const BUCKET: &str = "ddd";
+const OBJECT_COUNT: usize = 484;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn create_issue_3107_fixture(root: &Path) -> TestResult {
+    for idx in 0..OBJECT_COUNT {
+        let object_path = root.join("ddd").join("requirements").join(format!("file-{idx:04}.txt"));
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, format!("issue-3107 payload {idx}\n").repeat(128)).await?;
+    }
+
+    for dir in ["aaa", "ccc", "ccc-package"] {
+        let object_path = root.join(dir).join("marker.txt");
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, dir.as_bytes()).await?;
+    }
+
+    Ok(())
+}
+
+fn mc_available() -> bool {
+    Command::new("mc")
+        .arg("--version")
+        .output()
+        .is_ok_and(|output| output.status.success())
+}
+
+fn run_mc(args: &[&str]) -> TestResult {
+    let output = Command::new("mc").args(args).output()?;
+    if !output.status.success() {
+        return Err(format!(
+            "mc {:?} failed: status={:?}, stdout={}, stderr={}",
+            args,
+            output.status.code(),
+            String::from_utf8_lossy(&output.stdout),
+            String::from_utf8_lossy(&output.stderr)
+        )
+        .into());
+    }
+    Ok(())
+}
+
+fn count_files(root: &Path) -> usize {
+    walkdir::WalkDir::new(root)
+        .into_iter()
+        .filter_map(Result::ok)
+        .filter(|entry| entry.file_type().is_file())
+        .count()
+}
+
+#[tokio::test]
+#[serial]
+async fn test_mc_mirror_small_bucket_completes_without_list_timeout() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting issue #3107 mc mirror regression test");
+    if !mc_available() {
+        info!("Skipping issue #3107 mc mirror regression test because mc is not installed");
+        return Ok(());
+    }
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    env.create_test_bucket(BUCKET).await?;
+
+    let alias = format!("rustfs-issue-3107-{}", uuid::Uuid::new_v4());
+    run_mc(&["alias", "set", &alias, &env.url, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY])?;
+
+    let fixture = Path::new(&env.temp_dir).join("issue-3107-fixture");
+    let backup = Path::new(&env.temp_dir).join("issue-3107-backup");
+    fs::create_dir_all(&fixture).await?;
+    fs::create_dir_all(&backup).await?;
+    create_issue_3107_fixture(&fixture).await?;
+
+    let bucket_target = format!("{alias}/{BUCKET}");
+    let source_path = fixture.to_string_lossy().to_string();
+    run_mc(&["mirror", "--overwrite", &source_path, &bucket_target])?;
+
+    let backup_path = backup.join("ddd-backup");
+    let backup_path_string = backup_path.to_string_lossy().to_string();
+    timeout(
+        Duration::from_secs(20),
+        tokio::task::spawn_blocking({
+            let bucket_target = bucket_target.clone();
+            let backup_path_string = backup_path_string.clone();
+            move || run_mc(&["mirror", "--overwrite", &bucket_target, &backup_path_string])
+        }),
+    )
+    .await
+    .map_err(|_| "mc mirror timed out while listing/comparing a small bucket")???;
+
+    let mirrored_count = count_files(&backup_path);
+    assert_eq!(
+        mirrored_count,
+        OBJECT_COUNT + 3,
+        "mc mirror should copy every object from the issue #3107 small bucket fixture"
+    );
+
+    run_mc(&["alias", "remove", &alias])?;
+    Ok(())
+}

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -0,0 +1,122 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::RustFSTestClusterEnvironment;
+use aws_sdk_s3::Client;
+use aws_sdk_s3::error::SdkError;
+use bytes::Bytes;
+use serial_test::serial;
+use std::sync::Arc;
+use tokio::sync::Barrier;
+use tracing::{info, warn};
+
+const BUCKET: &str = "namespace-lock-quorum-bucket";
+const KEY: &str = "thumb/79/concurrent-overwrite.jpg";
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .body(Bytes::from(payload).into())
+        .send()
+        .await
+        .map(|_| ())
+        .map_err(|err| format_s3_error(err, writer_id))
+}
+
+fn format_s3_error(err: SdkError<aws_sdk_s3::operation::put_object::PutObjectError>, writer_id: usize) -> String {
+    match err {
+        SdkError::ServiceError(service_err) => {
+            let err = service_err.err();
+            let code = err.meta().code().unwrap_or("<unknown>");
+            let message = err.meta().message().unwrap_or("<empty>");
+            format!("writer {writer_id} returned service error {code}: {message}")
+        }
+        other => format!("writer {writer_id} returned SDK error: {other:?}"),
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting namespace lock quorum regression test with auto cluster");
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    // Keep the regression focused on false quorum-loss errors, not ordinary lock
+    // wait exhaustion under a heavily contended same-key overwrite workload.
+    cluster.set_env("RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT", "20");
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let clients = cluster.create_all_clients()?;
+    let first_payload = b"initial object contents".to_vec();
+    put_object(clients[0].clone(), first_payload, 0).await?;
+
+    let writer_count = clients.len() * 2;
+    let barrier = Arc::new(Barrier::new(writer_count));
+    let mut handles = Vec::with_capacity(writer_count);
+
+    for writer_id in 0..writer_count {
+        let client = clients[writer_id % clients.len()].clone();
+        let barrier = barrier.clone();
+        let payload = format!("replacement payload from writer {writer_id:02}").into_bytes();
+        handles.push(tokio::spawn(async move {
+            barrier.wait().await;
+            put_object(client, payload, writer_id).await
+        }));
+    }
+
+    let mut failures = Vec::new();
+    for handle in handles {
+        match handle.await {
+            Ok(Ok(())) => {}
+            Ok(Err(err)) => failures.push(err),
+            Err(err) => failures.push(format!("writer task join failed: {err}")),
+        }
+    }
+
+    if !failures.is_empty() {
+        for failure in &failures {
+            warn!("concurrent overwrite failure: {}", failure);
+        }
+    }
+
+    assert!(
+        failures.is_empty(),
+        "concurrent overwrites must not surface namespace lock quorum failures: {failures:#?}"
+    );
+
+    let body = clients[0]
+        .get_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .send()
+        .await?
+        .body
+        .collect()
+        .await?
+        .into_bytes();
+    let body = std::str::from_utf8(&body)?;
+    assert!(
+        body.starts_with("replacement payload from writer "),
+        "final object body should be one of the successful overwrite payloads, got {body:?}"
+    );
+
+    clients[0].delete_object().bucket(BUCKET).key(KEY).send().await?;
+    Ok(())
+}

crates/e2e_test/src/snowball_auto_extract_test.rs

@@ -70,6 +70,36 @@ mod tests {
         Ok(builder.into_inner().await?.into_inner())
     }
 
+    fn build_archive_with_parent_dir_entry(victim_bucket: &str) -> Vec<u8> {
+        let path = format!("../{victim_bucket}/evil-injected.txt");
+        let data = b"injected-body";
+        let mut header = [0u8; 512];
+
+        header[..path.len()].copy_from_slice(path.as_bytes());
+        header[100..108].copy_from_slice(b"0000644\0");
+        header[108..116].copy_from_slice(b"0000000\0");
+        header[116..124].copy_from_slice(b"0000000\0");
+        let size = format!("{:011o}\0", data.len());
+        header[124..136].copy_from_slice(size.as_bytes());
+        header[136..148].copy_from_slice(b"00000000000\0");
+        header[148..156].fill(b' ');
+        header[156] = b'0';
+        header[257..263].copy_from_slice(b"ustar\0");
+        header[263..265].copy_from_slice(b"00");
+
+        let checksum: u32 = header.iter().map(|byte| *byte as u32).sum();
+        let checksum = format!("{:06o}\0 ", checksum);
+        header[148..156].copy_from_slice(checksum.as_bytes());
+
+        let mut archive = Vec::new();
+        archive.extend_from_slice(&header);
+        archive.extend_from_slice(data);
+        let padding = (512 - (data.len() % 512)) % 512;
+        archive.extend(std::iter::repeat_n(0, padding));
+        archive.extend_from_slice(&[0u8; 1024]);
+        archive
+    }
+
     #[tokio::test]
     #[serial]
     async fn snowball_auto_extract_supports_minio_prefix_and_directory_markers() -> Result<(), Box<dyn Error + Send + Sync>> {
@@ -273,6 +303,49 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn snowball_auto_extract_rejects_parent_dir_entry_without_cross_bucket_write()
+    -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(vec![]).await?;
+
+        let client = env.create_s3_client();
+        let attacker_bucket = "snowball-traversal-source";
+        let victim_bucket = "snowball-traversal-victim";
+        let archive = build_archive_with_parent_dir_entry(victim_bucket);
+
+        client.create_bucket().bucket(attacker_bucket).send().await?;
+        client.create_bucket().bucket(victim_bucket).send().await?;
+
+        let err = client
+            .put_object()
+            .bucket(attacker_bucket)
+            .key("fixture.tar")
+            .metadata("Snowball-Auto-Extract", "true")
+            .body(ByteStream::from(archive))
+            .send()
+            .await
+            .expect_err("parent directory archive entry should be rejected");
+        let service_err = err.into_service_error();
+        assert_eq!(service_err.code(), Some("InvalidArgument"));
+
+        let victim_err = client
+            .head_object()
+            .bucket(victim_bucket)
+            .key("evil-injected.txt")
+            .send()
+            .await
+            .expect_err("rejected archive entry must not write into the victim bucket");
+        let victim_service_err = victim_err.into_service_error();
+        assert_eq!(victim_service_err.code(), Some("NotFound"));
+
+        env.stop_server();
+        Ok(())
+    }
+
     #[tokio::test]
     #[serial]
     async fn snowball_auto_extract_prefers_exact_minio_prefix_over_suffix_fallback() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/ecstore/AGENTS.md

@@ -14,6 +14,30 @@ Applies to `crates/ecstore/`.
 - Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
 - Benchmark-sensitive changes should include measurable rationale.
 
+## Allocation Discipline in Hot Paths
+
+- Do not implement `Clone` on structs with >5 heap-allocated fields without considering `Arc` for heavy fields.
+- Before cloning a struct in a loop or per-request path, check if a reference or `Cow` would suffice.
+- When a struct contains a large buffer (e.g., erasure coding block), wrap it in `Arc` and pass by reference rather than cloning.
+- Use `&str` and `Cow<str>` instead of `String` for temporary computations (header parsing, signature building, path manipulation).
+- Use `HashMap::with_capacity()` / `Vec::with_capacity()` when the size is known or estimable.
+
+## Lock Ordering
+
+- When a function acquires multiple locks, document the acquisition order in a comment.
+- Never acquire the same set of locks in different orders across code paths — this is a deadlock.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters.
+
+## Recursion Safety
+
+- Recursive tree traversals (cache trees, directory walks) must have a depth limit or use iterative traversal with an explicit `Vec` stack.
+- `flatten()`, `delete_recursive()`, `copy_with_children()`, `total_children_rec()`, `mark()` — all must handle deep or corrupted trees safely.
+
+## Dead Code
+
+- Do not add `#![allow(dead_code)]` at the crate root. If code is unused, remove it or gate it behind a feature flag.
+- Each `#[allow(dead_code)]` annotation must have a comment explaining why the code is kept.
+
 ## Cross-Module Coordination
 
 - Validate behavior impacts on:

crates/ecstore/Cargo.toml

@@ -38,6 +38,7 @@ rustfs-filemeta.workspace = true
 rustfs-utils = { workspace = true, features = ["full"] }
 rustfs-rio.workspace = true
 rustfs-signer.workspace = true
+rustfs-tls-runtime.workspace = true
 rustfs-checksums.workspace = true
 rustfs-config = { workspace = true, features = ["constants", "notify", "audit"] }
 rustfs-credentials = { workspace = true }
@@ -91,6 +92,7 @@ hyper.workspace = true
 hyper-util.workspace = true
 hyper-rustls.workspace = true
 rustls.workspace = true
+rustls-pki-types.workspace = true
 tokio = { workspace = true, features = ["io-util", "sync", "signal","io-uring"] }
 tonic.workspace = true
 xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] }
@@ -129,7 +131,7 @@ aws-smithy-http-client.workspace = true
 metrics = { workspace = true }
 
 [dev-dependencies]
-tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
+tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true, features = ["async_closure"] }
 tracing-subscriber = { workspace = true }

crates/ecstore/docs/internode-transport/internode-transport-adapter-rfc.md

@@ -0,0 +1,484 @@
+# RFC: Internode Transport Adapter Boundary
+
+> Status: draft
+> Last updated: 2026-05-22
+> Scope: OSS internode data-plane adapter analysis, benchmark baseline, and
+> transport boundary
+
+## Summary
+
+The current distributed internode paths use TCP-based HTTP/gRPC transports:
+
+- `tonic` gRPC `NodeService` for most control, metadata, lock, health, and
+  peer operations.
+- HTTP streaming routes under `/rustfs/rpc/` for remote disk file streams.
+
+This document frames the existing work as an OSS `InternodeDataTransport`
+adapter boundary. The adapter keeps RustFS data-plane logic separate from the
+concrete transport backend while preserving the current TCP/HTTP behavior as
+the default implementation.
+
+Current implementation status:
+
+- `InternodeDataTransport` exists in
+  `crates/ecstore/src/rpc/internode_data_transport.rs`.
+- The default and only production backend is `tcp-http`; `tcp` is accepted as
+  an alias.
+- `RUSTFS_INTERNODE_DATA_TRANSPORT` selects the backend. Blank or unset values
+  use `tcp-http`; invalid values fail closed.
+- `RemoteDisk::read_file_stream`, `RemoteDisk::create_file`,
+  `RemoteDisk::append_file`, and `RemoteDisk::walk_dir` delegate to the
+  transport.
+- `NodeService` gRPC remains the internode control plane and continues to carry
+  metadata/control operations.
+
+Related design notes in this directory:
+
+- `transport-capabilities.md`
+- `transport-buffer-lifecycle.md`
+- `transport-buffer-contract.md`
+- `transport-fallback-and-selection.md`
+- `transport-metrics-and-baseline.md`
+
+## Open-source Scope
+
+The OSS scope is:
+
+- define a clear `InternodeDataTransport` adapter boundary;
+- keep `tcp-http` as the default backend;
+- keep existing TCP/HTTP behavior unchanged;
+- keep internode data-plane behavior observable through metrics and baseline
+  tooling;
+- document buffer ownership, fallback, and capability expectations for
+  maintainable transport code;
+- avoid adding dependencies or backend implementations.
+
+The OSS scope is not:
+
+- adding another transport backend;
+- replacing the gRPC control plane;
+- adding benchmark plans for another transport;
+- adding runtime plugin loading;
+- changing object correctness semantics.
+
+## Goals
+
+- Document the current internode control plane and data plane.
+- Identify the existing transfer paths covered by the
+  `InternodeDataTransport` adapter and the paths that remain on gRPC.
+- Define the minimum benchmark baseline required before transport changes.
+- Sketch a pluggable transport boundary that preserves the current TCP/HTTP
+  behavior as the default backend.
+- Document backend-neutral capability, fallback, buffer ownership, and
+  observability expectations.
+
+## Non-Goals
+
+- Implement another transport backend.
+- Replace `tonic` gRPC for control-plane RPCs.
+- Redesign erasure coding, quorum handling, disk health tracking, or object
+  correctness semantics.
+- Change default development, CI, or ordinary RustFS deployment requirements.
+
+## Current Internode Architecture
+
+### Server-side entry points
+
+The main HTTP server builds a hybrid service per connection:
+
+- `rustfs/src/server/http.rs` wires a `NodeServiceServer` for gRPC.
+- `rustfs/src/storage/rpc/InternodeRpcService` intercepts HTTP paths under
+  `/rustfs/rpc/`.
+- Other HTTP/S3 traffic continues through the normal S3 service.
+
+Compression logic already treats `/rustfs/rpc/` and `/rustfs/peer/` as internode
+RPC paths and skips normal response compression for them.
+
+### gRPC channel management
+
+`crates/protos/src/lib.rs` creates internode gRPC channels with `tonic`
+`Endpoint`:
+
+- connect timeout
+- TCP keepalive
+- HTTP/2 keepalive interval and timeout
+- request timeout
+- optional TLS configuration
+- global channel caching and failed-connection eviction
+
+This confirms the current gRPC transport is TCP/HTTP2-based.
+
+### NodeService layout
+
+`crates/protos/src/node.proto` defines one `NodeService` that mixes several
+classes of RPCs:
+
+- meta service: bucket and metadata operations
+- disk service: local/remote disk operations
+- lock service: distributed lock operations
+- peer rest service: node health, metrics, IAM/policy reload, rebalance,
+  profiling, events, and admin-style operations
+
+The service layout is practical today, but it is too broad to become the
+transport adapter surface. A pluggable data transport should target only disk
+data streams and keep this gRPC service as the control plane.
+
+## Control Plane vs Data Plane
+
+### Control plane
+
+These paths carry coordination, metadata, health, and administrative state.
+They should remain on gRPC/TCP:
+
+| Area | Client/server code | Examples | Notes |
+| --- | --- | --- | --- |
+| Bucket peer ops | `crates/ecstore/src/rpc/peer_s3_client.rs`, `rustfs/src/storage/rpc/bucket.rs` | `MakeBucket`, `ListBucket`, `DeleteBucket`, `GetBucketInfo`, `HealBucket` | Small metadata/control payloads. |
+| Locking | `crates/ecstore/src/rpc/remote_locker.rs`, `rustfs/src/storage/rpc/lock.rs` | `Lock`, `UnLock`, `Refresh`, batch lock/unlock | Latency-sensitive but not bulk data; correctness and timeout semantics matter more than transport bandwidth. |
+| Peer/admin state | `crates/ecstore/src/rpc/peer_rest_client.rs`, `rustfs/src/storage/rpc/health.rs`, `metrics.rs`, `event.rs` | `LocalStorageInfo`, `ServerInfo`, `GetMetrics`, `GetLiveEvents`, reload APIs, rebalance APIs | Operational control plane. |
+| Disk metadata/control | `crates/ecstore/src/rpc/remote_disk.rs`, `rustfs/src/storage/rpc/disk.rs` | `DiskInfo`, `ReadXL`, `ReadVersion`, `ReadMetadata`, `WriteMetadata`, `RenameFile`, `RenamePart`, `Delete*`, `VerifyFile`, `CheckParts` | Usually metadata, integrity checks, or namespace mutations. |
+| Connection health | `RemoteDisk`, `RemotePeerS3Client`, `PeerRestClient` | TCP connectivity probes and fault/recovery state | Must remain available even if an optional data backend is unavailable. |
+
+### Data plane candidates
+
+These paths move object shard bytes or stream potentially large disk data and
+are the only reasonable first candidates for a pluggable transport.
+
+| Priority | Path | Current client | Current server | Current transport | Why it matters |
+| --- | --- | --- | --- | --- | --- |
+| P0 | `read_file_stream` | `RemoteDisk::read_file_stream` | `handle_read_file` in `http_service.rs` | HTTP `GET /rustfs/rpc/read_file_stream` with a streaming response body | Main remote disk read stream used by bitrot readers and erasure reads. |
+| P0 | `put_file_stream` | `RemoteDisk::create_file` and `RemoteDisk::append_file` | `handle_put_file` in `http_service.rs` | HTTP `PUT /rustfs/rpc/put_file_stream` with a streaming request body | Main remote disk write stream used by bitrot writers and erasure writes. |
+| P1 | `walk_dir` | `RemoteDisk::walk_dir` | `handle_walk_dir` in `http_service.rs` | HTTP `GET /rustfs/rpc/walk_dir` with a streamed metadata listing | Can be high-volume during scans/healing, but it is metadata-oriented rather than object byte data. |
+| P1 | `ReadAll` / `WriteAll` | `RemoteDisk::read_all` / `write_all` | gRPC unary disk handlers | gRPC unary `bytes` payload | Moves bytes today, but should be measured before treating it as a high-throughput data path. |
+| P2 | proto `WriteStream` / `ReadAt` | currently not used | currently returns unimplemented | gRPC streaming definitions exist but are not implemented | Declared proto shape, not a current production path. |
+
+## P1 Data Path Inventory
+
+Classification:
+
+- Covered by `InternodeDataTransport`: `RemoteDisk` opens the transfer through
+  the transport abstraction.
+- Still direct TCP/HTTP/gRPC: bytes move over a fixed internode protocol
+  outside the transport abstraction.
+- Metadata/control-plane only: payloads are expected to be small metadata,
+  namespace, lock, health, or admin messages.
+- Not relevant: declared or test-only paths that are not current production
+  data paths.
+
+### Covered by `InternodeDataTransport`
+
+| Path | Owner references | Server references | Classification | Notes |
+| --- | --- | --- | --- | --- |
+| Remote shard read stream | `crates/ecstore/src/rpc/remote_disk.rs::RemoteDisk::read_file_stream`; `crates/ecstore/src/rpc/internode_data_transport.rs::InternodeDataTransport::open_read`; `crates/ecstore/src/bitrot.rs::create_bitrot_reader` | `rustfs/src/storage/rpc/http_service.rs::handle_read_file` | Covered by `InternodeDataTransport` | Object GET, repair reads, and erasure decode use this path for remote shard bytes. |
+| Remote shard write stream | `RemoteDisk::create_file`; `RemoteDisk::append_file`; `InternodeDataTransport::open_write`; `crates/ecstore/src/bitrot.rs::create_bitrot_writer` | `rustfs/src/storage/rpc/http_service.rs::handle_put_file` | Covered by `InternodeDataTransport` | Object PUT and multipart part upload use this path for remote shard bytes. |
+| Remote namespace walk stream | `RemoteDisk::walk_dir`; `InternodeDataTransport::open_walk_dir`; `crates/ecstore/src/cache_value/metacache_set.rs` walk producers | `rustfs/src/storage/rpc/http_service.rs::handle_walk_dir` | Covered by `InternodeDataTransport` | High-volume listing/scanner/heal metadata stream. It is not object byte data, but it is a large internode stream. |
+| Remote zero-copy read fallback | `RemoteDisk::read_file_zero_copy` | same as remote shard read stream | Covered by `InternodeDataTransport` through `read_file_stream` | The remote path buffers the stream into `Bytes`; true zero-copy is not guaranteed for remote disks. |
+
+### Still Direct TCP/HTTP/gRPC
+
+| Path | Owner references | Server references | Classification | Notes |
+| --- | --- | --- | --- | --- |
+| `ReadAll` | `RemoteDisk::read_all`; `crates/ecstore/src/store_init.rs`; heal resume metadata readers | `rustfs/src/storage/rpc/disk.rs::handle_read_all` | Still direct gRPC | Unary `bytes` response. Currently used mostly for metadata/config files; measure before moving. |
+| `WriteAll` | `RemoteDisk::write_all`; `crates/ecstore/src/store_init.rs`; heal resume metadata writers | `rustfs/src/storage/rpc/disk.rs::handle_write_all` | Still direct gRPC | Unary `bytes` request. Currently used mostly for metadata/config/checkpoint writes. |
+| `ReadMultiple` | `RemoteDisk::read_multiple`; `crates/ecstore/src/set_disk/read.rs::read_multiple_files` | `rustfs/src/storage/rpc/disk.rs::handle_read_multiple` | Still direct gRPC | Returns multiple small file payloads, usually metadata/listing support. Could become large with many entries. |
+| `ReadParts` | `RemoteDisk::read_parts`; `crates/ecstore/src/set_disk/read.rs::read_parts`; multipart list/complete paths | `rustfs/src/storage/rpc/disk.rs::handle_read_parts` | Still direct gRPC | Encoded `ObjectPartInfo` metadata, not object data. |
+| `RenamePart` | `RemoteDisk::rename_part`; `crates/ecstore/src/set_disk/write.rs::rename_part` | `rustfs/src/storage/rpc/disk.rs::handle_rename_part` | Still direct gRPC | Carries part metadata while committing multipart data already written through stream writers. |
+| `ListDir` | `RemoteDisk::list_dir`; multipart/lifecycle metadata listing callers | `rustfs/src/storage/rpc/disk.rs::handle_list_dir` | Still direct gRPC | Directory name listing, metadata/control-plane unless measured otherwise. |
+| Legacy gRPC `WalkDir` | `rustfs/src/storage/rpc/node_service.rs::NodeService::walk_dir` | same file | Still direct gRPC | Server implementation remains, but current `RemoteDisk::walk_dir` uses HTTP through the transport. Keep until callers are audited or compatibility policy is set. |
+
+### Metadata/control-plane only
+
+| Area | Owner references | Classification | Notes |
+| --- | --- | --- | --- |
+| Disk metadata and namespace mutations | `RemoteDisk::{read_metadata,write_metadata,update_metadata,read_version,read_xl,rename_data,rename_file,delete*,verify_file,check_parts,disk_info}` | Metadata/control-plane only | These remain on gRPC by design. |
+| Peer/bucket/admin operations | `crates/ecstore/src/rpc/{peer_s3_client.rs,peer_rest_client.rs,remote_locker.rs}` and matching `rustfs/src/storage/rpc/*` handlers | Metadata/control-plane only | Not candidates for a data-plane backend without separate measurements. |
+| Store init and format operations | `crates/ecstore/src/store_init.rs` | Metadata/control-plane only | Uses `ReadAll`/`WriteAll` for small format/config objects. |
+| Heal orchestration | `crates/heal/src/heal/storage.rs` and `crates/ecstore/src/set_disk.rs::heal_object` | Metadata/control-plane plus covered data reads | Heal object data reads go through `get_object_reader` and then covered shard streams; resume/checkpoint metadata uses direct gRPC disk metadata calls. |
+
+### Not Relevant Current Paths
+
+| Path | Owner references | Classification | Notes |
+| --- | --- | --- | --- |
+| Proto `Write` | `crates/protos/src/node.proto`; `rustfs/src/storage/rpc/disk.rs::handle_write` | Not relevant | Handler is unimplemented. |
+| Proto `WriteStream` | `crates/protos/src/node.proto`; `rustfs/src/storage/rpc/node_service.rs::write_stream` | Not relevant | Returns `unimplemented`. |
+| Proto `ReadAt` | `crates/protos/src/node.proto`; `rustfs/src/storage/rpc/node_service.rs::read_at` | Not relevant | Returns `unimplemented`. |
+| E2E reliant gRPC helpers | `crates/e2e_test/src/reliant/*` | Not relevant | Test harnesses, not production internode data-path callers. |
+
+### Current Limitations
+
+| Risk | Limitation |
+| --- | --- |
+| Medium | `ReadAll` and `WriteAll` still carry unary `bytes` over gRPC. They appear metadata-oriented today, but there is no size threshold or routing policy. |
+| Medium | `ReadMultiple` can aggregate many metadata files into one gRPC response. |
+| Low | Legacy gRPC `WalkDir` remains implemented while `RemoteDisk::walk_dir` uses HTTP through the transport. |
+| Medium | Remote `read_file_zero_copy` is a buffered read over the transport, not a remote zero-copy contract. |
+| Medium | Server-side TCP HTTP route handling is outside the client-side trait. |
+
+## Current Object Write Path
+
+For object PUTs in distributed erasure mode, the relevant flow is:
+
+1. Upper storage layers prepare object data and erasure metadata.
+2. `SetDisks` selects local and remote disks.
+3. `create_bitrot_writer` calls `disk.create_file(...)` for each shard writer.
+4. For a remote disk, `RemoteDisk::create_file` delegates to
+   `InternodeDataTransport::open_write`.
+5. `HttpWriter` sends an HTTP `PUT` to `/rustfs/rpc/put_file_stream`.
+6. The remote node's `handle_put_file` opens the local file writer and copies
+   incoming body chunks into it.
+7. `Erasure::encode` writes shards through `MultiWriter` to all selected
+   writers while enforcing write quorum.
+
+This is the primary write data-plane candidate.
+
+## Current Object Read Path
+
+For object GETs and repair reads in distributed erasure mode, the relevant flow is:
+
+1. `SetDisks` prepares shard readers for the selected disks.
+2. `create_bitrot_reader` uses local zero-copy only when `disk.is_local()`.
+3. For a remote disk, it calls `disk.read_file_stream(...)`.
+4. `RemoteDisk::read_file_stream` delegates to
+   `InternodeDataTransport::open_read`.
+5. `HttpReader` sends an HTTP `GET` to `/rustfs/rpc/read_file_stream`.
+6. The remote node's `handle_read_file` opens the local disk stream and returns
+   it as an HTTP streaming body.
+7. The erasure decoder reads from the shard streams and reconstructs the object.
+
+This is the primary read data-plane candidate.
+
+## Existing Metrics and Benchmark Surface
+
+RustFS already has coarse internode metrics in `crates/io-metrics/src/internode_metrics.rs`:
+
+- sent bytes
+- received bytes
+- outgoing requests
+- incoming requests
+- errors
+- dial errors
+- average dial time
+
+These metrics are useful as a starting point. For backend comparisons, the
+relevant route-level and operation-level dimensions are:
+
+- `read_file_stream`
+- `put_file_stream`
+- `walk_dir`
+- gRPC `ReadAll` / `WriteAll`
+- gRPC control-plane request volume
+
+Existing benchmark assets:
+
+- `scripts/run_object_batch_bench.sh`
+- `scripts/run_object_batch_bench_enhanced.sh`
+- `scripts/run_object_batch_bench_abc.sh`
+- `scripts/run_four_node_cluster_failover_bench.sh`
+- `scripts/run_internode_transport_baseline.sh` (scenario matrix wrapper for local vs distributed TCP baseline artifacts)
+- Criterion benches under `crates/ecstore/benches/`
+
+These mostly cover S3/object workload or erasure coding performance. They do
+not yet isolate internode transport cost.
+
+## Required TCP Baseline
+
+Before changing internode data transport behavior or comparing a non-default
+backend, collect a baseline for the current TCP/HTTP/gRPC implementation.
+
+### Topology
+
+Minimum:
+
+- 1-node local erasure deployment, to measure local disk and erasure overhead.
+- 4-node distributed erasure deployment, to measure internode overhead.
+
+Preferred:
+
+- Same host count and disk layout for every run.
+- Dedicated network interface or isolated VLAN.
+- Fixed CPU governor and no unrelated background load.
+- Recorded kernel version, NIC model, MTU, RustFS commit, Rust toolchain, and
+  benchmark tool versions.
+
+### Workloads
+
+| Workload | Sizes | Concurrency | Main signal |
+| --- | --- | --- | --- |
+| S3 PUT | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end write throughput and tail latency. |
+| S3 GET | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end read throughput and tail latency. |
+| Remote disk stream read | shard-sized ranges from `read_file_stream` | 1, 16, 64 | Isolated internode read path. |
+| Remote disk stream write | shard-sized writes through `put_file_stream` | 1, 16, 64 | Isolated internode write path. |
+| Healing / repair | missing disk or missing shard scenario | controlled | Rebuild throughput and read/write amplification. |
+| Scanner walk | large bucket/object namespace | controlled | Metadata streaming pressure, not the primary object-byte transport path. |
+
+### Measurements
+
+Collect:
+
+- throughput in bytes/s and objects/s
+- p50, p95, p99, and max latency
+- CPU utilization per process and per core
+- memory RSS and allocation pressure where available
+- `rustfs_system_network_internode_*` metrics
+- TCP retransmits, socket errors, and NIC throughput
+- disk throughput and utilization
+- failure/retry/fallback counts
+
+The baseline should produce a machine-readable artifact, for example
+`target/bench/internode-transport/<timestamp>/summary.csv`, plus the exact
+commands and configuration used.
+
+### Baseline runner entry point
+
+Use `scripts/run_internode_transport_baseline.sh` to execute a reproducible
+S3 PUT/GET matrix against `local` and `distributed` scenarios and export:
+
+- `summary.csv` (throughput/latency summary per workload and object size)
+- `internode_metric_deltas.csv` (operation-level internode metric deltas when
+  `--metrics-url` is provided)
+
+See `transport-metrics-and-baseline.md` for current metric names, labels,
+operation values, baseline inputs, and baseline artifact fields.
+
+## Transport Abstraction Proposal
+
+### Design principle
+
+Keep `NodeService` as the control plane. Introduce a separate data transport
+only below `RemoteDisk`, where remote disk byte streams are opened today.
+
+The first implementation should be a no-behavior-change TCP/HTTP backend that
+wraps the current `HttpReader`, `HttpWriter`, and `/rustfs/rpc/*` handlers.
+Non-default backend work should not proceed until the default wrapper is
+measured and adapter gaps are documented.
+
+### Candidate boundary
+
+The current boundary is remote disk stream transfer:
+
+```rust
+#[async_trait::async_trait]
+pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
+    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
+    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
+    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
+    fn name(&self) -> &'static str;
+    fn capabilities(&self) -> InternodeDataTransportCapabilities;
+}
+```
+
+Initial request fields should mirror the current HTTP query parameters:
+
+- peer endpoint
+- disk reference
+- volume
+- path
+- offset
+- length
+- append/create mode
+- expected size
+- optional stall timeout for long-running listing streams
+
+The initial TCP backend can keep the current signed HTTP URLs internally.
+
+### Integration point
+
+`RemoteDisk` delegates only these methods to the data transport:
+
+- `read_file_stream`
+- `read_file_zero_copy` as the current wrapper over `read_file_stream`
+- `append_file`
+- `create_file`
+- `walk_dir`
+
+All other `RemoteDisk` methods continue using the current gRPC client
+in this adapter scope.
+
+### Capability model
+
+Avoid hard-coding transport-specific assumptions into the generic interface.
+The current conservative capability fields are:
+
+- streaming read
+- streaming write
+- streaming walk-dir
+- ordered delivery
+- maximum transfer size
+- fallback support
+
+The TCP/HTTP backend should report only capabilities that it actually provides.
+
+## TCP Fallback Requirements
+
+TCP/HTTP/gRPC must remain the default and required backend.
+
+Fallback rules:
+
+- If no explicit data transport is configured, use the current TCP/HTTP
+  implementation.
+- The current accepted values for `RUSTFS_INTERNODE_DATA_TRANSPORT` are
+  `tcp-http` and the `tcp` alias. Empty and unset values use `tcp-http`.
+- Invalid configured values fail closed with an error that includes the env var
+  name and invalid value.
+- Unsupported configured backends fail closed during transport construction.
+- Runtime fallback must preserve object correctness and quorum semantics.
+- Fallback events must be logged and counted in metrics.
+
+Do not add fallback settings until there is an implementation PR that uses them.
+
+## Baseline Validation Commands
+
+Dry-run command:
+
+```bash
+scripts/run_internode_transport_baseline.sh \
+  --access-key minioadmin \
+  --secret-key minioadmin \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --sizes 4KiB,1MiB \
+  --concurrencies 1 \
+  --duration 10s \
+  --dry-run
+```
+
+Real TCP baseline command with metrics:
+
+```bash
+RUSTFS_INTERNODE_DATA_TRANSPORT=tcp-http \
+scripts/run_internode_transport_baseline.sh \
+  --access-key "$RUSTFS_ACCESS_KEY" \
+  --secret-key "$RUSTFS_SECRET_KEY" \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --metrics-url http://127.0.0.1:9000/metrics \
+  --out-dir target/bench/internode-transport/manual-run
+```
+
+Expected artifacts:
+
+- `run_manifest.txt`
+- `summary.csv`
+- `internode_metric_deltas.csv` when `--metrics-url` is provided
+
+The baseline validates the default TCP/HTTP path only. It must not be used to
+claim support or performance for any other transport.
+
+## Adapter Constraints
+
+The current adapter boundary has these constraints:
+
+- `tcp-http` is the default and only OSS backend.
+- Backend selection is explicit and fail-closed.
+- The gRPC control plane remains responsible for metadata, health, locks, and
+  coordination.
+- Transport errors must preserve existing disk health, quorum, timeout, and
+  integrity semantics.
+- Metrics must identify the selected backend and operation without high-cardinality
+  labels.
+
+The current `RemoteDisk::walk_dir` stream is routed through the adapter.
+Metadata RPCs, locks, admin RPCs, bucket coordination, and the legacy gRPC
+`WalkDir` handler remain outside the current data-plane boundary.
+
+## Out of Scope
+
+This RFC does not add a plugin system, split the adapter into a separate crate,
+add accepted backend values, or implement a new transport backend.

crates/ecstore/docs/internode-transport/transport-buffer-contract.md

@@ -0,0 +1,91 @@
+# Internode Transport Buffer Contract
+
+Status: design note only. This document defines a backend-neutral buffer
+ownership and lifecycle contract for the `InternodeDataTransport` adapter. It
+does not implement a new backend and does not change production behavior.
+
+## Open-source Scope
+
+The open-source RustFS path keeps `tcp-http` as the default internode data
+transport. This document defines adapter contracts only:
+
+- no additional production backend is introduced;
+- no dependency is added;
+- no new accepted production backend value is added;
+- RustFS core data-plane logic remains independent of the concrete transport
+  implementation.
+
+## Current Adapter Surface
+
+The current data-plane surface is byte-stream based:
+
+| Current path | Current API shape | Current ownership |
+| --- | --- | --- |
+| Remote read stream | `InternodeDataTransport::open_read(...) -> FileReader` | Backend returns boxed `AsyncRead`; callers provide temporary `ReadBuf` storage per poll. |
+| Remote write stream | `InternodeDataTransport::open_write(...) -> FileWriter` | Callers pass borrowed `&[u8]` slices into boxed `AsyncWrite`; the backend owns any async body staging. |
+| Walk-dir stream | `InternodeDataTransport::open_walk_dir(...) -> FileReader` | Same boxed stream model as read, with a small serialized request body. |
+
+This API is correct for the current TCP/HTTP backend. The adapter contract
+describes current ownership boundaries without assuming implementation details
+outside `TcpHttpInternodeDataTransport`.
+
+## Buffer Ownership Model
+
+| Buffer role | Allocator | Lifetime owner | Transport state | TCP/HTTP behavior |
+| --- | --- | --- | --- | --- |
+| Send buffer | Caller or RustFS-owned pool | Caller until the writer copies or accepts bytes for the HTTP body | Existing HTTP body staging | Copy into the existing `AsyncWrite` path when the writer cannot use the borrowed slice directly. |
+| Receive buffer | Caller-provided storage | Reader while filling; caller after `poll_read` returns | Existing HTTP response body chunks | Copy from `AsyncRead` into caller storage as today. |
+| Control metadata | RustFS caller | Caller/request object | Not buffer-managed by the data-plane backend | Serialize into HTTP/gRPC/control-plane messages. |
+| Fallback staging | TCP/HTTP backend | TCP/HTTP backend | Existing `HttpReader`/`HttpWriter` buffers | Existing buffering semantics. |
+
+The current writer must not retain borrowed caller slices beyond the write call.
+When bytes must outlive the call, they are copied into owned HTTP body chunks.
+
+This contract does not claim zero-copy behavior. The current TCP/HTTP path
+documents where copies occur.
+
+## Compatibility Contract
+
+The current stream API remains the OSS compatibility contract:
+
+```rust
+#[async_trait::async_trait]
+pub trait InternodeDataTransport {
+    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
+    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
+    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
+}
+```
+
+## Current Adapter Contract
+
+| Area | Required contract |
+| --- | --- |
+| Ownership | Define when caller-owned bytes are copied or accepted by the transport. |
+| Completion | Return from stream operations only when bytes are accepted or an error is reported. |
+| Staging | Keep staging behavior inside the TCP/HTTP implementation. |
+| Size limits | Report any RustFS-visible `max_transfer_size`; TCP/HTTP currently reports none. |
+| Ordering | Preserve ordered byte-stream semantics. |
+| Copy accounting | Document known copy boundaries and avoid unmeasured zero-copy claims. |
+
+## Current API Limitations
+
+| Current API | Current limitation |
+| --- | --- |
+| `FileReader = Box<dyn AsyncRead + Send + Sync + Unpin>` | `AsyncRead` exposes temporary caller `ReadBuf` storage. |
+| `FileWriter = Box<dyn AsyncWrite + Send + Sync + Unpin>` | `AsyncWrite::poll_write` receives borrowed `&[u8]` that cannot outlive the poll. |
+| `HttpWriter` | The async HTTP body must own `Bytes`, so borrowed write buffers are copied into `BytesMut` or `Bytes`. |
+| `write_body_chunks_to_writer` | Server-side HTTP body chunks are copied into `BytesMut` before local disk write. |
+| Erasure encode output | Encoded shards are represented as `Vec<Bytes>` and written through `AsyncWrite`. |
+| Erasure decode input | Shard reads allocate `Vec<u8>` buffers before decode. |
+
+These limitations do not block the current `tcp-http` backend.
+
+## Adapter Stability
+
+`InternodeDataTransport` should keep RustFS core data-plane logic separate from
+the concrete transport implementation. The trait and `tcp-http` backend remain
+inside `ecstore`.
+
+This PR does not perform a crate split, add runtime loading, introduce a plugin
+system, add a backend value, or implement a new transport backend.

crates/ecstore/docs/internode-transport/transport-buffer-lifecycle.md

@@ -0,0 +1,122 @@
+# Internode Buffer Lifecycle and Copy Count
+
+Status: P1-D analysis only. This document records the current TCP/HTTP
+internode data path and the ownership boundaries that matter for the
+backend-neutral `InternodeDataTransport` adapter. It does not implement a new
+backend or change production behavior.
+
+## Open-source Scope
+
+The OSS scope is:
+
+- define buffer ownership and copy-count behavior for the current
+  `InternodeDataTransport` adapter;
+- keep `tcp-http` as the default backend;
+- keep existing TCP/HTTP behavior unchanged;
+- document copy hotspots and ownership gaps for maintainable transport code;
+- avoid adding dependencies or backend implementations.
+
+The OSS scope is not:
+
+- adding another transport backend;
+- replacing the current TCP/HTTP path;
+- adding benchmark plans for another transport;
+- changing object correctness semantics.
+
+## Scope
+
+The covered paths are the large internode data-plane calls currently routed
+through `InternodeDataTransport`:
+
+| Path | Entry | Transport owner | Server owner |
+| --- | --- | --- | --- |
+| Read stream | `RemoteDisk::read_file_stream` in `crates/ecstore/src/rpc/remote_disk.rs` | `TcpHttpInternodeDataTransport::open_read` in `crates/ecstore/src/rpc/internode_data_transport.rs`, `HttpReader` in `crates/rio/src/http_reader.rs` | `handle_read_file` in `rustfs/src/storage/rpc/http_service.rs` |
+| Write stream | `RemoteDisk::create_file`, `RemoteDisk::append_file` in `crates/ecstore/src/rpc/remote_disk.rs` | `TcpHttpInternodeDataTransport::open_write` in `crates/ecstore/src/rpc/internode_data_transport.rs`, `HttpWriter` in `crates/rio/src/http_reader.rs` | `handle_put_file` in `rustfs/src/storage/rpc/http_service.rs` |
+| Walk dir stream | `RemoteDisk::walk_dir` in `crates/ecstore/src/rpc/remote_disk.rs` | `TcpHttpInternodeDataTransport::open_walk_dir`, `HttpReader` | `handle_walk_dir` in `rustfs/src/storage/rpc/http_service.rs` |
+
+Object read/write/heal callers enter these streams through
+`create_bitrot_reader` and `create_bitrot_writer` in
+`crates/ecstore/src/bitrot.rs`. Erasure decode and encode then move data
+through `ParallelReader` in `crates/ecstore/src/erasure_coding/decode.rs` and
+`MultiWriter` in `crates/ecstore/src/erasure_coding/encode.rs`.
+
+## Read Stream
+
+| Step | Owner | Buffer type | Copy? | Reason |
+| --- | --- | --- | --- | --- |
+| Build request | `RemoteDisk::read_file_stream` | `String` fields in `ReadStreamRequest` | Yes | Volume, path, endpoint, and disk references are copied into an owned request before async transport dispatch. This is metadata, not payload. |
+| Select transport | `TcpHttpInternodeDataTransport::open_read` | URL `String`, `HeaderMap` | Yes | URL and auth headers are HTTP control data. No object bytes are copied here. |
+| Open local file on server | `handle_read_file`, `LocalDisk::read_file_stream` | `FileCacheReclaimReader` boxed as `FileReader` | No payload copy | The server owns an async file reader positioned at the requested offset. |
+| File to HTTP body | `read_file_body_stream` | `ReaderStream<AsyncRead>` yielding `Bytes` | Yes | `ReaderStream::with_capacity` reads from the file into chunk buffers. This is the file-to-network buffer materialization point. |
+| Length limiting | `rustfs_utils::net::bytes_stream` | `Bytes` | Usually no | `Bytes::truncate` adjusts the chunk view when the last chunk exceeds the requested length. It does not copy the retained prefix. |
+| HTTP receive | `HttpReader::with_capacity_and_stall_timeout` | `reqwest::Response::bytes_stream()` yielding `Bytes` | Network stack dependent | The user-level object is `Bytes`; any kernel/TLS/hyper copy is below the current RustFS abstraction. |
+| Stream to caller buffer | `HttpReader::poll_read` | `StreamReader<Stream<Item = Bytes>>`, caller `ReadBuf` | Yes | `StreamReader` exposes `AsyncRead`, so it copies bytes from each `Bytes` chunk into the caller-provided `ReadBuf`. |
+| Bitrot verification | `BitrotReader::read` | caller `&mut [u8]`, `hash_buf: Vec<u8>` | No additional payload copy | The bitrot reader reads hash bytes into `hash_buf` and payload bytes directly into the supplied output slice. Hash calculation reads the slice. |
+| Erasure shard read | `ParallelReader::read` | `Vec<u8>` per shard | Yes | Each shard read allocates `vec![0u8; shard_size]`; data is filled there before decode/reconstruction. |
+| Object response write | `write_data_blocks` | slices of shard `Vec<u8>` | No extra staging copy | Decoded data block slices are written to the target writer with `write_all`; the target may copy internally. |
+| Remote zero-copy helper | `RemoteDisk::read_file_zero_copy` | `Vec<u8>` then `Bytes` | Yes | The remote implementation reads the full stream into a `Vec` and converts it into `Bytes`. It is a convenience fallback, not network zero-copy. |
+
+## Write Stream
+
+| Step | Owner | Buffer type | Copy? | Reason |
+| --- | --- | --- | --- | --- |
+| Build writer request | `RemoteDisk::create_file`, `RemoteDisk::append_file` | `String` fields in `WriteStreamRequest` | Yes | Volume, path, endpoint, and disk references are copied into an owned request. This is metadata. |
+| Select transport | `TcpHttpInternodeDataTransport::open_write` | URL `String`, `HeaderMap` | Yes | URL and auth headers are HTTP control data. No object bytes are copied here. |
+| Erasure encode input | `Erasure::encode` in `encode.rs` | reusable `Vec<u8>` sized to `block_size` | Yes | `rustfs_utils::read_full` fills a block buffer from the source reader before encoding. |
+| Erasure encode output | `Erasure::encode_data` caller in `encode.rs` | `Vec<Bytes>` per encoded block | Yes | Encoding creates shard `Bytes` for data and parity blocks before queueing them to writers. |
+| Multi-writer fanout | `MultiWriter::write` | borrowed `Bytes` shards | No additional fanout copy | The writer fanout passes borrowed `Bytes` references to each `BitrotWriterWrapper`. |
+| Bitrot write | `BitrotWriter::write` | shard `&[u8]`, checksum bytes | Yes for checksum bytes | The payload slice is passed to the inner writer, while checksum bytes are generated and written before payload when enabled. |
+| Client HTTP writer buffer | `HttpWriter::poll_write` and `poll_write_vectored` | `BytesMut` pending chunk or `Bytes::copy_from_slice` | Yes | Small writes are coalesced with `BytesMut::extend_from_slice`; large single writes still copy into an owned `Bytes` because the async body must outlive the caller's borrowed buffer. |
+| Client channel to reqwest | `HttpWriter::poll_send_pending_chunk`, `ReceiverStream` | `Bytes` | No | `BytesMut::split().freeze()` transfers owned chunk storage to `Bytes`; the mpsc channel and stream move the `Bytes` handle. |
+| HTTP receive body on server | `handle_put_file` | `Incoming::into_data_stream()` yielding `Bytes` | Network stack dependent | The server receives owned `Bytes` chunks from hyper. |
+| Server body coalescing | `write_body_chunks_to_writer` | `BytesMut` sized to `DEFAULT_READ_BUFFER_SIZE` | Yes | Each incoming `Bytes` chunk is copied into `pending` before writing to the local file writer. This normalizes chunk size but adds a full payload copy. |
+| Local file write | `LocalDisk::create_file`, `LocalDisk::append_file`, `FileCacheReclaimWriter` | `&[u8]` into `tokio::fs::File` | Kernel dependent | RustFS passes slices to Tokio file writes. Kernel page-cache copies are below the RustFS abstraction. |
+
+## Request and Serialization Boundaries
+
+| Boundary | Owner | Buffer type | Copy? | Notes |
+| --- | --- | --- | --- | --- |
+| Read/write query parameters | `build_read_file_stream_url`, `build_put_file_stream_url` | URL-encoded `String` | Yes | Metadata only. It includes disk, volume, path, offset, length, append, and size. |
+| Auth headers | `build_auth_headers` callers | `HeaderMap` | Yes | Metadata only. This is currently tied to HTTP request construction. |
+| Walk dir request | `RemoteDisk::walk_dir`, `open_walk_dir`, `handle_walk_dir` | JSON `Vec<u8>` body, collected `Bytes` on server | Yes | Walk dir is a streamed response but its request body is serialized JSON control data. |
+| gRPC read/write-all | `RemoteDisk::read_all`, `RemoteDisk::write_all`, `NodeService::{handle_read_all,handle_write_all}` | Prost `Bytes`/message bodies | Yes | These paths are still gRPC byte paths, not `InternodeDataTransport`; they matter for metrics and inventory but are outside this P1-D stream-copy count. |
+
+## Hotspots
+
+| Rank | Hotspot | Impact | Reason |
+| --- | --- | --- | --- |
+| 1 | `HttpWriter::poll_write` and `poll_write_vectored` | High on write path | Every borrowed caller buffer is copied into owned `BytesMut` or `Bytes` before it can be sent by an async HTTP body. |
+| 2 | `write_body_chunks_to_writer` | High on write path | The server copies every received `Bytes` chunk into a coalescing `BytesMut` before local disk write. |
+| 3 | `ParallelReader::read` shard buffers | High on read path | Each shard read allocates and fills a `Vec<u8>` before decode can proceed. This is also where degraded reads wait on quorum. |
+| 4 | `ReaderStream::with_capacity` plus `StreamReader` | Medium on read path | Server file reads create `Bytes` chunks, then client `AsyncRead` copies those chunks into the caller's `ReadBuf`. |
+| 5 | `Erasure::encode` block and shard materialization | Medium on write path | Source data is first read into a block `Vec<u8>`, then encoded into per-shard `Bytes`. This is necessary for the current erasure API. |
+| 6 | `RemoteDisk::read_file_zero_copy` | Medium when used | Remote zero-copy reads buffer the whole stream into memory. The name does not mean zero-copy over the network. |
+| 7 | URL/query/header/JSON serialization | Low | Metadata copies are small and not on the large payload hot path. |
+
+## Adapter Ownership Gaps
+
+1. `FileReader` and `FileWriter` are boxed `AsyncRead`/`AsyncWrite` trait
+   objects. They expose borrowed buffers per poll, not stable backend-owned
+   regions, transfer handles, or explicit completion ownership.
+2. `InternodeDataTransport` currently returns stream traits only. Its
+   capabilities advertise that TCP/HTTP does not require backend-specific
+   buffer registration and is not a zero-copy candidate, but there is no
+   backend API to pass stable backend-managed buffers.
+3. `HttpWriter` must own outgoing chunks because the async request body outlives
+   the caller's borrowed `&[u8]`. A lower-copy backend would need a different
+   lifetime contract or an owned buffer pool.
+4. Server write handling normalizes all incoming body chunks into a new
+   `BytesMut`. Avoiding that copy would require passing incoming `Bytes` or
+   backend-owned receive buffers directly into the disk/bitrot writer contract.
+5. Erasure decode owns shard `Vec<u8>` buffers and write-back happens through
+   `AsyncWrite`. A lower-copy backend would need explicit ownership of shard
+   buffers across decode, reconstruction, and network completion.
+6. Erasure encode materializes `Vec<Bytes>` blocks before fanout. A
+   backend that can send multiple stable slices would need an encode output
+   representation that can be transferred without repacking.
+7. The HTTP auth and URL construction boundary is part of the current TCP/HTTP
+   backend. A non-HTTP backend would need equivalent peer authentication and
+   disk addressing without assuming URL query parameters.
+8. Local disk zero-copy exists only for local reads via `read_file_zero_copy`.
+   Remote disks deliberately fall back to network streaming and full-buffer
+   collection for the zero-copy helper.

crates/ecstore/docs/internode-transport/transport-capabilities.md

@@ -0,0 +1,67 @@
+# Internode Transport Capabilities
+
+Status: design note for backend-neutral capability reporting. This document
+does not add a backend or transport crate.
+
+## Open-source Scope
+
+The OSS scope is:
+
+- define honest capability reporting for the `InternodeDataTransport` adapter;
+- keep `tcp-http` as the default backend;
+- keep existing TCP/HTTP behavior unchanged;
+- document the capability fields needed for maintainable transport code;
+- avoid adding dependencies or backend implementations.
+
+The OSS scope is not:
+
+- adding another transport backend;
+- replacing the current TCP/HTTP path;
+- adding benchmark plans for another transport;
+- changing object correctness semantics.
+
+## Purpose
+
+`InternodeDataTransportCapabilities` describes what a backend can honestly do
+for RustFS internode data-plane transfers. The fields describe observable
+adapter behavior without naming a specific transport implementation.
+
+The capability report is descriptive. It does not select a backend, negotiate
+with peers, or weaken object correctness semantics.
+
+## Capability Fields
+
+| Field | Meaning |
+| --- | --- |
+| `streaming_read` | The backend can open a remote disk reader for `read_file_stream`. |
+| `streaming_write` | The backend can open a remote disk writer for `create_file` or `append_file`. |
+| `streaming_walk_dir` | The backend can stream `walk_dir` responses. |
+| `ordered_delivery` | Bytes for each opened transfer are delivered in order. |
+| `max_transfer_size` | Optional RustFS-level cap for a single transfer. `None` means no additional cap beyond the backend/protocol/runtime limits. |
+| `fallback_supported` | The backend can participate in the behavior-preserving TCP fallback path. |
+
+## TCP/HTTP Backend
+
+The default TCP/HTTP backend reports only capabilities it actually provides:
+
+| Field | TCP/HTTP value | Reason |
+| --- | --- | --- |
+| `streaming_read` | `true` | `HttpReader` streams `/rustfs/rpc/read_file_stream` responses. |
+| `streaming_write` | `true` | `HttpWriter` streams `/rustfs/rpc/put_file_stream` request bodies. |
+| `streaming_walk_dir` | `true` | `HttpReader` streams `/rustfs/rpc/walk_dir` responses. |
+| `ordered_delivery` | `true` | Each HTTP request body or response body is consumed as an ordered byte stream. |
+| `max_transfer_size` | `None` | RustFS does not impose an extra per-transfer cap at the capability layer. |
+| `fallback_supported` | `true` | TCP/HTTP is the behavior-preserving default and fallback path. |
+
+## Capability Compatibility
+
+Any new capability field should describe observable RustFS behavior without
+assuming a specific transport implementation:
+
+| Capability shape | Interpretation |
+| --- | --- |
+| `max_transfer_size=Some(n)` | The backend has a RustFS-visible transfer size ceiling and callers must split larger transfers or use fallback behavior. |
+| `ordered_delivery=false` | The backend cannot be used behind the current stream API without an ordering or reassembly layer. |
+
+Unsupported or mismatched capabilities must not silently change quorum,
+integrity verification, retry, or timeout semantics.

crates/ecstore/docs/internode-transport/transport-fallback-and-selection.md

@@ -0,0 +1,135 @@
+# Internode Transport Fallback and Backend Selection Model
+
+Status: design note only. This document defines backend-neutral selection,
+fallback, failure handling, negotiation, security, and observability rules for
+the `InternodeDataTransport` adapter. It does not implement a new backend and
+does not change production behavior.
+
+## Open-source Scope
+
+The open-source RustFS path keeps `tcp-http` as the default internode data
+transport. This document defines adapter contracts only:
+
+- no additional production backend is introduced;
+- no dependency is added;
+- no new accepted production backend value is added;
+- RustFS core data-plane logic remains independent of the concrete transport
+  implementation.
+
+## Static Backend Selection
+
+Static config is the first selection model. Existing accepted values remain:
+
+| Config value | Meaning |
+| --- | --- |
+| unset | Use default TCP/HTTP backend. |
+| `tcp-http` | Use default TCP/HTTP backend. |
+| `tcp` | Alias for `tcp-http`. |
+| any unsupported value | Fail closed with a diagnostic naming `RUSTFS_INTERNODE_DATA_TRANSPORT` and the invalid value. |
+
+Unknown backend values must fail closed. Unsupported backend values must fail
+closed. Any additional backend value must be explicitly added and must not
+silently replace `tcp-http`.
+
+Backend selection must expose an observable backend identity for metrics, logs,
+and benchmark interpretation. The default and fallback path remains `tcp-http`.
+
+## Fallback Contract
+
+Fallback must be explicit and observable. Silent fallback is not allowed for
+benchmark or production interpretation because it hides which backend moved the
+payload.
+
+| Condition | Default behavior | Explicit fallback behavior | Observability |
+| --- | --- | --- | --- |
+| Unsupported configured backend | Fail closed during transport construction. | Fall back only when a separately configured policy explicitly allows unsupported-backend fallback. | Error includes config key and invalid value; fallback event is counted when fallback is enabled. |
+| Peer does not support selected backend | Fail before payload transfer. | Use TCP/HTTP only when both local policy and peer policy allow it. | Count peer mismatch and selected fallback backend. |
+| Capability mismatch | Fail before payload transfer. | Use TCP/HTTP only if it satisfies the operation and policy allows fallback. | Record missing capability names or a low-cardinality reason. |
+| Connection setup failure | Fail the operation. | Retry on TCP/HTTP only when fallback is allowed and no payload bytes have transferred. | Count setup failure, retry, fallback backend, and fallback result. |
+| Partial transfer failure | Fail the operation and let existing object/quorum logic decide retry behavior. | Do not silently resume on another backend unless the transfer protocol can prove byte range, checksum, and idempotency boundaries. | Count partial failure with bytes completed. |
+| Max transfer size exceeded | Fail before payload transfer or split at a higher layer. | Use TCP/HTTP if policy allows and TCP has no RustFS-level cap. | Record rejected size and selected backend. |
+| Auth or encryption mismatch | Fail closed. | No fallback unless the fallback path satisfies the same or stronger security requirements. | Security failure metric and audit log entry. |
+
+Fallback settings should not be added until there is an implementation that
+uses them. A backend must define failure behavior before production use.
+
+## Dynamic Negotiation Boundary
+
+Dynamic negotiation is not implemented by this PR. If it is added later, it
+belongs on the existing gRPC control plane. Data transfer must start only after
+both peers agree on:
+
+| Negotiated item | Required property |
+| --- | --- |
+| Backend name | Both peers know the backend and have it enabled. |
+| Capability set | Required capabilities match the operation. |
+| Max transfer size | The selected operation fits or is split before transfer starts. |
+| Buffer rules | Both peers agree on staging and ownership rules. |
+| Completion semantics | Both peers agree when a transfer is considered complete and when buffers may be reused. |
+| Security mode | Authentication and encryption requirements are satisfied before any out-of-band transfer. |
+| Fallback policy | Both peers agree whether TCP/HTTP fallback is allowed for this operation. |
+
+Negotiation must not silently downgrade security or bypass existing disk
+health, quorum, timeout, and integrity semantics.
+
+## Failure Handling Requirements
+
+| Failure mode | Requirement |
+| --- | --- |
+| Invalid config | Fail closed with `RUSTFS_INTERNODE_DATA_TRANSPORT` and the invalid value. |
+| Backend disabled | Fail closed with the selected backend name and the missing enablement condition. |
+| Backend unavailable | Fail closed with an actionable diagnostic; do not silently use TCP/HTTP. |
+| Peer mismatch | Fail before payload transfer unless explicit fallback is configured. |
+| Connection failure | Fail the operation and record setup failure; fallback only if policy allows and no payload bytes moved. |
+| Completion failure | Return an operation error and release backend-owned resources. |
+| Timeout | Return an operation error and preserve existing disk health and quorum semantics. |
+| Partial transfer | Do not silently resume on another backend without a safe byte-range/checksum/idempotency proof. |
+| Unsupported operation | Return a clear unsupported-operation error. |
+
+## Security Requirements
+
+- Backend selection must preserve peer authentication.
+- Fallback must not weaken encryption or authorization.
+- Partial transfers must not bypass bitrot verification or erasure quorum
+  handling.
+- Any adapter implementation must preserve the same request authority, disk,
+  volume, path, and operation binding as the current TCP/HTTP path.
+
+## Metrics and Observability Requirements
+
+Metrics and logs must use low-cardinality labels or metadata:
+
+- selected backend;
+- requested backend;
+- fallback backend, when used;
+- operation name;
+- success/failure;
+- transferred bytes;
+- setup failure count;
+- partial transfer failure count;
+- capability mismatch count;
+- fallback decision count.
+
+Adapter implementations must not add high-cardinality labels such as object
+names, full paths, full URLs, peer-specific dynamic strings, memory addresses,
+or buffer identifiers.
+
+## TCP/HTTP Compatibility
+
+The `tcp-http` backend remains the default and behavior-preserving path. It
+uses ordinary byte streams, does not require backend-specific buffer
+registration, and remains suitable as the fallback path when an explicit
+fallback policy exists.
+
+Any adapter implementation must not change the correctness semantics of
+object writes, object reads, healing, bitrot verification, erasure quorum,
+timeouts, or disk health handling.
+
+## Adapter Stability
+
+`InternodeDataTransport` should keep RustFS core data-plane logic separate from
+the concrete transport implementation. The trait and `tcp-http` backend remain
+inside `ecstore`.
+
+This PR does not perform a crate split, add runtime loading, introduce a plugin
+system, add a backend value, or implement a new transport backend.

crates/ecstore/docs/internode-transport/transport-metrics-and-baseline.md

@@ -0,0 +1,143 @@
+# Internode Transport Metrics and Baseline
+
+Status: current OSS behavior. This document describes the metrics and baseline
+runner used to observe the existing `tcp-http` internode transport adapter. It
+does not add a backend, change runtime behavior, or make performance claims.
+
+## Scope
+
+The current OSS adapter scope is:
+
+- `tcp-http` is the default and only supported internode data transport backend;
+- `tcp` is a legacy alias for `tcp-http`;
+- unsupported backend values fail closed during transport construction;
+- metrics and baseline artifacts are used to understand the current data-plane
+  behavior.
+
+## Operation Metrics
+
+Operation-level internode metrics are defined in
+`crates/io-metrics/src/internode_metrics.rs`.
+
+| Metric | Labels | Meaning |
+| --- | --- | --- |
+| `rustfs_system_network_internode_operation_sent_bytes_total` | `operation`, `backend` | Bytes sent for a known internode operation. |
+| `rustfs_system_network_internode_operation_recv_bytes_total` | `operation`, `backend` | Bytes received for a known internode operation. |
+| `rustfs_system_network_internode_operation_requests_outgoing_total` | `operation`, `backend` | Outgoing requests opened for a known internode operation. |
+| `rustfs_system_network_internode_operation_requests_incoming_total` | `operation`, `backend` | Incoming requests handled for a known internode operation. |
+| `rustfs_system_network_internode_operation_errors_total` | `operation`, `backend` | Errors observed for a known internode operation. |
+
+Current operation label values are:
+
+| Operation | Transport path | Notes |
+| --- | --- | --- |
+| `read_file_stream` | HTTP `/rustfs/rpc/read_file_stream` | Remote disk read stream opened by `InternodeDataTransport::open_read`. |
+| `put_file_stream` | HTTP `/rustfs/rpc/put_file_stream` | Remote disk write stream opened by `InternodeDataTransport::open_write`. |
+| `walk_dir` | HTTP `/rustfs/rpc/walk_dir` | Remote walk-dir stream opened by `InternodeDataTransport::open_walk_dir`. |
+| `grpc_read_all` | gRPC `ReadAll` | Unary bytes response outside the adapter. |
+| `grpc_write_all` | gRPC `WriteAll` | Unary bytes request outside the adapter. |
+
+Current backend label values are:
+
+| Backend | Meaning |
+| --- | --- |
+| `tcp-http` | Current HTTP stream backend for adapter-routed operations. |
+| `grpc` | Current gRPC path for `ReadAll` and `WriteAll`. |
+| `unknown` | Aggregate internode metric path when an operation-specific backend is not available. |
+
+The `backend` label reflects the current instrumentation source, for example
+the `tcp-http` stream path, gRPC path, or aggregate unknown path. It does not
+imply additional supported transport backends.
+
+Metric labels must stay low cardinality. Do not add object names, full paths,
+full URLs, peer-specific dynamic strings, request IDs, or other per-request
+values as labels.
+
+## Current Instrumentation Points
+
+| Area | File | Current signal |
+| --- | --- | --- |
+| Outgoing HTTP stream requests | `crates/rio/src/http_reader.rs` | Outgoing request count, sent bytes for writer bodies, received bytes for reader bodies, and errors for known `/rustfs/rpc/` operations. |
+| Incoming HTTP stream requests | `rustfs/src/storage/rpc/http_service.rs` | Incoming request count, sent bytes for read and walk streams, received bytes for put streams, and route errors. |
+| gRPC `ReadAll` / `WriteAll` | `rustfs/src/storage/rpc/disk.rs` | Incoming request count, sent/received bytes, and errors with the `grpc` backend label. |
+
+Known gaps:
+
+- Operation-level metrics do not currently expose latency histograms.
+- The baseline runner reads Prometheus text output only when `--metrics-url` is
+  provided.
+- Metrics are useful for attribution, but they do not replace end-to-end
+  correctness tests or object benchmark output.
+
+## Baseline Runner
+
+Use `scripts/run_internode_transport_baseline.sh` to run the current S3 PUT/GET
+matrix against one or more configured endpoints.
+
+Required inputs:
+
+- `--access-key`
+- `--secret-key`
+
+Common optional inputs:
+
+- `--tool warp|s3bench`
+- `--scenarios name=url,...`
+- `--sizes 4KiB,1MiB,...`
+- `--concurrencies 1,16,...`
+- `--duration 90s`
+- `--metrics-url http://host:port/metrics`
+- `--out-dir target/bench/internode-transport/manual-run`
+- `--dry-run`
+
+Dry-run example:
+
+```bash
+scripts/run_internode_transport_baseline.sh \
+  --access-key minioadmin \
+  --secret-key minioadmin \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --sizes 4KiB,1MiB \
+  --concurrencies 1 \
+  --duration 10s \
+  --dry-run
+```
+
+Real baseline example with metrics:
+
+```bash
+RUSTFS_INTERNODE_DATA_TRANSPORT=tcp-http \
+scripts/run_internode_transport_baseline.sh \
+  --access-key "$RUSTFS_ACCESS_KEY" \
+  --secret-key "$RUSTFS_SECRET_KEY" \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --metrics-url http://127.0.0.1:9000/metrics \
+  --out-dir target/bench/internode-transport/manual-run
+```
+
+The real baseline requires a running RustFS deployment and working benchmark
+tool. Do not invent or copy synthetic benchmark numbers into PR descriptions.
+
+## Output Artifacts
+
+The baseline runner writes:
+
+| File | Created when | Contents |
+| --- | --- | --- |
+| `run_manifest.txt` | Always | Timestamp, git commit, dirty flag, Rust compiler version, kernel string, scenario matrix, tool, workload settings, metrics URL, and redacted credentials. |
+| `summary.csv` | Always | `scenario`, `endpoint`, `workload`, `concurrency`, `size`, `status`, `throughput`, `requests_per_sec`, `avg_latency`, `error_count`, `log_file`, and `run_dir`. |
+| `internode_metric_deltas.csv` | When `--metrics-url` is set | `scenario`, `workload`, `concurrency`, `size`, metric name, operation, backend, before value, after value, and delta. |
+
+In dry-run mode, the runner still creates the manifest and CSV headers, and the
+underlying object benchmark command is printed with credentials redacted.
+
+## Interpretation
+
+Use baseline artifacts to record and inspect the current `tcp-http` adapter
+behavior across commits, environments, and scenario matrices. A baseline run is
+valid only for the exact environment and command recorded in
+`run_manifest.txt`.
+
+Baseline artifacts should state whether metrics were collected. When
+`internode_metric_deltas.csv` is absent, benchmark output cannot attribute
+internode operation deltas from Prometheus metrics.

crates/ecstore/docs/s3-table/s3-table-concept-model.md

@@ -0,0 +1,130 @@
+# RustFS S3 Table Core Concepts
+
+This document keeps only the core concepts for adding table catalog capability
+to RustFS. The direction is to build an Iceberg REST Catalog-compatible layer
+on top of existing RustFS object storage.
+
+## Concept Model
+
+```text
+TableBucket
+  Warehouse
+    Namespace
+      TableIdentifier
+        TableMetadataLocation
+        TableMetadata
+          Schema
+          Snapshot
+          Manifest
+          DataFile
+```
+
+## TableBucket
+
+`TableBucket` is a normal RustFS bucket enabled for table catalog use. It is not
+a new physical bucket type.
+
+RustFS can mark it with internal bucket metadata such as `table-bucket.json`.
+That marker is an implementation detail, not a public compatibility API.
+
+## Warehouse
+
+`Warehouse` is the catalog storage root exposed to Iceberg clients. In the first
+RustFS model, a warehouse maps to a table-enabled bucket.
+
+Clients should interact with the warehouse through catalog APIs, not by knowing
+RustFS internal metadata paths.
+
+## Namespace
+
+`Namespace` groups tables inside a warehouse. It is a first-class catalog
+resource, not a raw object prefix.
+
+Namespace validation must be conservative: no path traversal, encoded
+separators, empty segments, control characters, or ambiguous normalization.
+
+## TableIdentifier
+
+`TableIdentifier` names one table inside a warehouse and namespace.
+
+```text
+warehouse + namespace + table name
+```
+
+It is resolved by the catalog layer to internal metadata objects. It should not
+be treated as an S3 object key.
+
+## TableMetadataLocation
+
+`TableMetadataLocation` points to the current Iceberg table metadata file.
+
+It is the main mutable table pointer. Only catalog commit code may update it.
+Updates must use conflict detection so stale writers cannot overwrite newer
+table state.
+
+## TableMetadata
+
+`TableMetadata` is the Iceberg metadata JSON referenced by the current metadata
+location.
+
+RustFS should preserve Iceberg metadata exactly where possible. Full parsing can
+be added incrementally for fields RustFS must enforce, such as schemas,
+snapshots, manifest lists, table location, and properties.
+
+## Schema
+
+`Schema` describes table columns.
+
+RustFS does not need to interpret every schema detail at the beginning, but it
+must preserve schema metadata without corrupting unknown Iceberg fields.
+
+## Snapshot
+
+`Snapshot` represents an immutable table state at a point in time.
+
+Catalog commits advance the current metadata to a new snapshot. Maintenance
+features such as rollback and snapshot expiration can be added later.
+
+## Manifest
+
+`Manifest` and manifest lists describe table data-file additions, deletions, and
+references.
+
+RustFS should store and protect these as catalog-owned metadata. Full manifest
+parsing can be deferred until validation or maintenance requires it.
+
+## DataFile
+
+`DataFile` is a table data object, commonly Parquet, Avro, or ORC.
+
+Data files live in object storage. The catalog metadata decides which data files
+belong to a table snapshot.
+
+## Reserved Metadata Boundary
+
+Catalog-owned metadata should live under a reserved internal prefix, for
+example:
+
+```text
+.rustfs-table/
+  warehouses/
+    <warehouse-id>/
+      namespaces/
+        <namespace-id>/
+          tables/
+            <table-id>/
+              current.json
+              metadata/
+              manifests/
+```
+
+Ordinary S3 mutating APIs must not create, overwrite, copy into, delete, or
+mutate objects under this reserved prefix for table-enabled buckets.
+
+## Catalog Boundary
+
+S3 object APIs remain object APIs.
+
+Namespace, table, metadata pointer, and commit operations belong to a separate
+catalog layer. Public compatibility should target Iceberg REST Catalog semantics
+rather than exposing RustFS internal metadata objects directly.

crates/ecstore/src/admin_server_info.rs

@@ -17,7 +17,7 @@ use crate::error::{Error, Result};
 use crate::rpc::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
 use crate::{
     disk::endpoint::Endpoint,
-    global::{GLOBAL_BOOT_TIME, GLOBAL_Endpoints},
+    global::{GLOBAL_BOOT_TIME, GLOBAL_Endpoints, get_global_deployment_id},
     new_object_layer_fn,
     notification_sys::get_global_notification_sys,
     store_api::StorageAPI,
@@ -291,7 +291,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
         domain: None,
         region: None,
         sqs_arn: None,
-        deployment_id: None,
+        deployment_id: get_global_deployment_id(),
         buckets: Some(buckets),
         objects: Some(objects),
         versions: Some(versions),
@@ -393,3 +393,21 @@ pub fn get_commit_id() -> String {
 
     format!("{}@{}", build::COMMIT_DATE_3339, ver)
 }
+
+#[cfg(test)]
+mod tests {
+    use serial_test::serial;
+
+    use crate::global::get_global_deployment_id;
+
+    use super::get_server_info;
+
+    #[serial]
+    #[tokio::test]
+    async fn server_info_includes_global_deployment_id() {
+        let expected_deployment_id = get_global_deployment_id();
+        let info = get_server_info(false).await;
+
+        assert_eq!(info.deployment_id, expected_deployment_id);
+    }
+}

crates/ecstore/src/batch_processor.rs

@@ -274,10 +274,10 @@ mod tests {
         assert!(successes.len() >= 2);
     }
 
-    #[tokio::test]
+    #[tokio::test(start_paused = true)]
     async fn test_batch_processor_quorum_returns_before_slow_tail() {
         let processor = AsyncBatchProcessor::new(4);
-        let started = std::time::Instant::now();
+        let started = tokio::time::Instant::now();
 
         let tasks: Vec<_> = [(10_u64, Ok(1_i32)), (15, Ok(2)), (250, Ok(3))]
             .into_iter()
@@ -295,10 +295,10 @@ mod tests {
         assert!(started.elapsed() < Duration::from_millis(100));
     }
 
-    #[tokio::test]
+    #[tokio::test(start_paused = true)]
     async fn test_batch_processor_quorum_fails_once_quorum_becomes_impossible() {
         let processor = AsyncBatchProcessor::new(4);
-        let started = std::time::Instant::now();
+        let started = tokio::time::Instant::now();
 
         let tasks: Vec<_> = vec![
             (10_u64, Ok(1_i32)),

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -48,8 +48,8 @@ use rustfs_common::heal_channel::rep_has_active_rules;
 use rustfs_common::metrics::{IlmAction, Metrics};
 use rustfs_config::{
     DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
-    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, ENV_TRANSITION_QUEUE_CAPACITY,
-    ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
+    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TRANSITION_QUEUE_CAPACITY, ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS,
+    ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
 };
 use rustfs_data_usage::TierStats;
 use rustfs_filemeta::{
@@ -75,6 +75,8 @@ use time::OffsetDateTime;
 use tokio::select;
 use tokio::sync::mpsc::{Receiver, Sender};
 use tokio::sync::{RwLock, mpsc};
+use tokio::task::JoinHandle;
+use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, info, warn};
 use uuid::Uuid;
 use xxhash_rust::xxh64;
@@ -145,7 +147,7 @@ fn is_immediate_transition_source(src: &LcEventSrc) -> bool {
 
 #[cfg(any(test, debug_assertions))]
 fn should_force_immediate_transition_enqueue_timeout() -> bool {
-    env::var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
+    env::var(rustfs_config::ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
         .ok()
         .is_some_and(|value| value == "1")
 }
@@ -587,12 +589,16 @@ impl ExpiryOp for TransitionTask {
     }
 }
 
+struct TransitionWorker {
+    cancel: CancellationToken,
+    handle: JoinHandle<()>,
+}
+
 pub struct TransitionState {
     transition_tx: A_Sender<Option<TransitionTask>>,
     transition_rx: A_Receiver<Option<TransitionTask>>,
     pub num_workers: AtomicI64,
-    kill_tx: A_Sender<()>,
-    kill_rx: A_Receiver<()>,
+    workers: Mutex<Vec<TransitionWorker>>,
     transition_queue_capacity: usize,
     transition_queue_send_timeout: StdDuration,
     active_tasks: AtomicI64,
@@ -621,13 +627,11 @@ impl TransitionState {
         let capacity = capacity.max(1);
         let queue_send_timeout = resolve_transition_queue_send_timeout();
         let (tx1, rx1) = bounded(capacity);
-        let (tx2, rx2) = bounded(1);
         Arc::new(Self {
             transition_tx: tx1,
             transition_rx: rx1,
             num_workers: AtomicI64::new(0),
-            kill_tx: tx2,
-            kill_rx: rx2,
+            workers: Mutex::new(Vec::new()),
             transition_queue_capacity: capacity,
             transition_queue_send_timeout: queue_send_timeout,
             active_tasks: AtomicI64::new(0),
@@ -849,10 +853,12 @@ impl TransitionState {
         Self::counter_value(&self.compensation_running_tasks)
     }
 
-    pub async fn worker(api: Arc<ECStore>) {
+    async fn worker_with_cancel(api: Arc<ECStore>, cancel_token: CancellationToken) {
         loop {
             select! {
-                _ = GLOBAL_TransitionState.kill_rx.recv() => {
+                biased;
+
+                _ = cancel_token.cancelled() => {
                     return;
                 }
                 task = GLOBAL_TransitionState.transition_rx.recv() => {
@@ -952,33 +958,45 @@ impl TransitionState {
         }
         // Allow environment override of maximum workers
         let absolute_max = resolve_transition_workers_absolute_max();
-        n = std::cmp::min(n, absolute_max);
+        n = n.clamp(0, absolute_max);
 
-        let previous_num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        let mut num_workers = previous_num_workers;
-        while num_workers < n {
+        Self::resize_workers_to(api, n, requested, absolute_max);
+    }
+
+    fn resize_workers_to(api: Arc<ECStore>, n: i64, requested: i64, absolute_max: i64) {
+        let target = n as usize;
+        let mut workers = GLOBAL_TransitionState.workers.lock().unwrap();
+        let tracked_workers = workers.len();
+        workers.retain(|worker| !worker.handle.is_finished());
+        let pruned_finished_workers = tracked_workers.saturating_sub(workers.len());
+        let previous_num_workers = workers.len() as i64;
+
+        while workers.len() < target {
             let clone_api = api.clone();
-            tokio::spawn(async move {
-                TransitionState::worker(clone_api).await;
+            let cancel = CancellationToken::new();
+            let worker_cancel = cancel.clone();
+            let handle = tokio::spawn(async move {
+                TransitionState::worker_with_cancel(clone_api, worker_cancel).await;
             });
-            num_workers += 1;
-            GLOBAL_TransitionState.num_workers.fetch_add(1, Ordering::SeqCst);
+            workers.push(TransitionWorker { cancel, handle });
         }
 
-        let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        while num_workers > n {
-            let worker = GLOBAL_TransitionState.kill_tx.clone();
-            let _ = worker.send(()).await;
-            num_workers -= 1;
-            GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
+        while workers.len() > target {
+            if let Some(worker) = workers.pop() {
+                worker.cancel.cancel();
+            }
         }
 
+        let current_workers = workers.len() as i64;
+        GLOBAL_TransitionState.num_workers.store(current_workers, Ordering::SeqCst);
+
         info!(
             requested_transition_workers = requested,
             effective_transition_workers = n,
             absolute_max_workers = absolute_max,
             previous_transition_workers = previous_num_workers,
-            current_transition_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst),
+            current_transition_workers = current_workers,
+            pruned_finished_transition_workers = pruned_finished_workers,
             "transition workers updated"
         );
     }
@@ -1850,11 +1868,11 @@ pub async fn put_restore_opts(
             ..Default::default()
         });
     }
-    for (k, v) in &oi.user_defined {
+    for (k, v) in oi.user_defined.iter() {
         meta.insert(k.to_string(), v.clone());
     }
     if !oi.user_tags.is_empty() {
-        meta.insert(AMZ_OBJECT_TAGGING.to_string(), oi.user_tags.clone());
+        meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
     }
     let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
     meta.insert(
@@ -1885,7 +1903,7 @@ impl LifecycleOps for ObjectInfo {
     fn to_lifecycle_opts(&self) -> lifecycle::ObjectOpts {
         lifecycle::ObjectOpts {
             name: self.name.clone(),
-            user_tags: self.user_tags.clone(),
+            user_tags: (*self.user_tags).clone(),
             version_id: self.version_id,
             mod_time: self.mod_time,
             size: self.size as usize,
@@ -2264,7 +2282,8 @@ mod tests {
         lifecycle_rule_has_date_expiration, lifecycle_version_purge_state_from_completed_targets,
         mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate, replication_state_for_delete,
         resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
-        should_defer_date_expiry_for_recent_config_update, should_reuse_lifecycle_delete_replication_state,
+        resolve_transition_workers_absolute_max, should_defer_date_expiry_for_recent_config_update,
+        should_reuse_lifecycle_delete_replication_state,
     };
     use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
     use crate::bucket::metadata_sys;
@@ -2655,15 +2674,45 @@ mod tests {
         })
         .await;
 
-        let current_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        if original_workers > 0 {
-            TransitionState::update_workers(ecstore, original_workers).await;
-        } else {
-            for _ in 0..current_workers {
-                let _ = GLOBAL_TransitionState.kill_tx.send(()).await;
-                GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
-            }
-        }
+        let absolute_max = resolve_transition_workers_absolute_max();
+        TransitionState::resize_workers_to(ecstore, original_workers, original_workers, absolute_max);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn transition_worker_resize_cancels_removed_workers_directly() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let original_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        let absolute_max = resolve_transition_workers_absolute_max();
+
+        TransitionState::resize_workers_to(ecstore.clone(), 0, 0, absolute_max);
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 0);
+
+        TransitionState::resize_workers_to(ecstore.clone(), 2, 2, absolute_max);
+        let worker_tokens = {
+            let workers = GLOBAL_TransitionState.workers.lock().unwrap();
+            assert_eq!(workers.len(), 2);
+            workers.iter().map(|worker| worker.cancel.clone()).collect::<Vec<_>>()
+        };
+
+        TransitionState::resize_workers_to(ecstore.clone(), 1, 1, absolute_max);
+
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 1);
+        assert_eq!(worker_tokens.iter().filter(|token| token.is_cancelled()).count(), 1);
+
+        let remaining_token = {
+            let workers = GLOBAL_TransitionState.workers.lock().unwrap();
+            assert_eq!(workers.len(), 1);
+            let token = workers[0].cancel.clone();
+            assert!(!token.is_cancelled());
+            token
+        };
+
+        TransitionState::resize_workers_to(ecstore.clone(), 0, 0, absolute_max);
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 0);
+        assert!(remaining_token.is_cancelled());
+
+        TransitionState::resize_workers_to(ecstore, original_workers, original_workers, absolute_max);
     }
 
     #[test]
@@ -2994,6 +3043,52 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[ignore = "requires isolated global object layer state"]
+    #[serial]
+    async fn ecstore_new_succeeds_on_fresh_local_volumes() {
+        let test_base_dir = format!("/tmp/rustfs_ecstore_empty_boot_{}", Uuid::new_v4());
+        let temp_dir = PathBuf::from(&test_base_dir);
+        if temp_dir.exists() {
+            fs::remove_dir_all(&temp_dir).await.ok();
+        }
+        fs::create_dir_all(&temp_dir).await.unwrap();
+
+        let disk_paths = vec![
+            temp_dir.join("disk1"),
+            temp_dir.join("disk2"),
+            temp_dir.join("disk3"),
+            temp_dir.join("disk4"),
+        ];
+
+        for disk_path in &disk_paths {
+            fs::create_dir_all(disk_path).await.unwrap();
+        }
+
+        let mut endpoints = Vec::new();
+        for (i, disk_path) in disk_paths.iter().enumerate() {
+            let mut endpoint = Endpoint::try_from(disk_path.to_str().unwrap()).unwrap();
+            endpoint.set_pool_index(0);
+            endpoint.set_set_index(0);
+            endpoint.set_disk_index(i);
+            endpoints.push(endpoint);
+        }
+
+        let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 4,
+            endpoints: Endpoints::from(endpoints),
+            cmd_line: "fresh-boot-test".to_string(),
+            platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
+        }]);
+
+        crate::store::init_local_disks(endpoint_pools.clone()).await.unwrap();
+
+        let ecstore = ECStore::new("127.0.0.1:0".parse().unwrap(), endpoint_pools, CancellationToken::new()).await;
+        assert!(ecstore.is_ok(), "fresh local ECStore boot should succeed, got {ecstore:?}");
+    }
+
     #[tokio::test]
     #[serial]
     async fn stale_multipart_cleanup_uses_default_expiry_without_lifecycle() {
@@ -3138,6 +3233,76 @@ mod tests {
         assert!(!parts.user_defined.contains_key(RUSTFS_MULTIPART_OBJECT_KEY));
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn repeated_upload_part_overwrites_previous_part_state() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let bucket = format!("multipart-overwrite-{}", Uuid::new_v4().simple());
+        let object = "overwrite/object.txt";
+        create_test_bucket(&ecstore, &bucket).await;
+
+        let upload = ecstore
+            .new_multipart_upload(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect("multipart upload should be created");
+
+        let mut first = PutObjReader::from_vec(vec![1, 2, 3]);
+        let first_part = ecstore
+            .put_object_part(&bucket, object, &upload.upload_id, 1, &mut first, &ObjectOptions::default())
+            .await
+            .expect("first multipart part should be uploaded");
+
+        let mut second = PutObjReader::from_vec(vec![4, 5, 6, 7]);
+        let second_part = ecstore
+            .put_object_part(&bucket, object, &upload.upload_id, 1, &mut second, &ObjectOptions::default())
+            .await
+            .expect("second multipart part should overwrite the previous part");
+
+        assert_ne!(
+            first_part.etag, second_part.etag,
+            "the overwrite path should persist the latest part metadata rather than reusing stale state"
+        );
+
+        let parts = ecstore
+            .list_object_parts(
+                &bucket,
+                object,
+                &upload.upload_id,
+                None,
+                crate::set_disk::MAX_PARTS_COUNT,
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("multipart parts should be readable after overwrite");
+
+        assert_eq!(parts.parts.len(), 1, "only the latest version of part 1 should remain visible");
+        assert_eq!(parts.parts[0].part_num, 1);
+        assert_eq!(parts.parts[0].etag, second_part.etag);
+        assert_eq!(parts.parts[0].size, second_part.size);
+        assert_eq!(parts.parts[0].actual_size, second_part.actual_size);
+
+        let completed = ecstore
+            .complete_multipart_upload(
+                &bucket,
+                object,
+                &upload.upload_id,
+                vec![crate::store_api::CompletePart {
+                    part_num: 1,
+                    etag: second_part.etag.clone(),
+                    checksum_crc32: None,
+                    checksum_crc32c: None,
+                    checksum_sha1: None,
+                    checksum_sha256: None,
+                    checksum_crc64nvme: None,
+                }],
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("complete multipart upload should succeed with the latest overwritten part");
+
+        assert_eq!(completed.size, second_part.size as i64);
+    }
+
     #[tokio::test]
     #[serial]
     async fn cleanup_removes_empty_multipart_sha_dirs() {

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -34,7 +34,8 @@ const ERR_LIFECYCLE_NO_RULE: &str = "Lifecycle configuration should have at leas
 const ERR_LIFECYCLE_DUPLICATE_ID: &str = "Rule ID must be unique. Found same ID for more than one rule";
 const _ERR_XML_NOT_WELL_FORMED: &str =
     "The XML you provided was not well-formed or did not validate against our published schema";
-const ERR_LIFECYCLE_BUCKET_LOCKED: &str = "ExpiredObjectDeleteMarker is not allowed on a bucket with Object Lock enabled";
+const ERR_LIFECYCLE_BUCKET_LOCKED: &str =
+    "ExpiredObjectAllVersions element and DelMarkerExpiration action cannot be used on an object locked bucket";
 const ERR_LIFECYCLE_TOO_MANY_RULES: &str = "Lifecycle configuration should have at most 1000 rules";
 const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "Lifecycle expiration days must not be negative";
 const ERR_LIFECYCLE_INVALID_NONCURRENT_EXPIRATION_DAYS: &str = "Lifecycle noncurrent expiration days must not be negative";
@@ -320,14 +321,15 @@ impl Lifecycle for BucketLifecycleConfiguration {
             r.validate()?;
             if let Some(object_lock_enabled) = lr.object_lock_enabled.as_ref()
                 && object_lock_enabled.as_str() == ObjectLockEnabled::ENABLED
-                && let Some(expiration) = r.expiration.as_ref()
             {
-                // Object Lock + ExpiredObjectDeleteMarker conflict
-                if expiration.expired_object_delete_marker.is_some_and(|v| v) {
-                    return Err(std::io::Error::other(ERR_LIFECYCLE_BUCKET_LOCKED));
+                if let Some(expiration) = r.expiration.as_ref() {
+                    // Object Lock + ExpiredObjectAllVersions conflict (MinIO extension)
+                    if expiration.expired_object_all_versions.is_some_and(|v| v) {
+                        return Err(std::io::Error::other(ERR_LIFECYCLE_BUCKET_LOCKED));
+                    }
                 }
-                // Object Lock + ExpiredObjectAllVersions conflict (MinIO extension)
-                if expiration.expired_object_all_versions.is_some_and(|v| v) {
+                // Object Lock + DelMarkerExpiration conflict
+                if r.del_marker_expiration.is_some() {
                     return Err(std::io::Error::other(ERR_LIFECYCLE_BUCKET_LOCKED));
                 }
             }
@@ -879,7 +881,7 @@ impl ObjectOpts {
     pub fn from_object_info(oi: &ObjectInfo) -> Self {
         Self {
             name: oi.name.clone(),
-            user_tags: oi.user_tags.clone(),
+            user_tags: (*oi.user_tags).clone(),
             mod_time: oi.mod_time,
             size: oi.size as usize,
             version_id: oi.version_id,
@@ -892,7 +894,7 @@ impl ObjectOpts {
             restore_expires: oi.restore_expires,
             versioned: false,
             version_suspended: false,
-            user_defined: oi.user_defined.clone(),
+            user_defined: (*oi.user_defined).clone(),
             version_purge_status: oi.version_purge_status.clone(),
             replication_status: oi.replication_status.clone(),
         }
@@ -2060,11 +2062,11 @@ mod tests {
         assert_eq!(event_after.due, Some(future_date));
     }
 
-    // --- TASK-002 tests: Object Lock + ExpiredObjectDeleteMarker conflict ---
+    // --- TASK-002 tests: Object Lock + ExpiredObjectDeleteMarker compatibility ---
 
     #[tokio::test]
     #[serial]
-    async fn validate_rejects_expired_object_delete_marker_on_locked_bucket() {
+    async fn validate_allows_expired_object_delete_marker_on_locked_bucket() {
         let lc = BucketLifecycleConfiguration {
             expiry_updated_at: None,
             rules: vec![LifecycleRule {
@@ -2089,8 +2091,9 @@ mod tests {
             ..Default::default()
         };
 
-        let err = lc.validate(&locked_config).await.unwrap_err();
-        assert_eq!(err.to_string(), ERR_LIFECYCLE_BUCKET_LOCKED);
+        lc.validate(&locked_config)
+            .await
+            .expect("expected validation to pass for ExpiredObjectDeleteMarker on locked bucket");
     }
 
     #[tokio::test]
@@ -2154,6 +2157,68 @@ mod tests {
             .expect("expected days-based expiration to pass on locked bucket");
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn validate_rejects_del_marker_expiration_on_locked_bucket() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: Some(s3s::dto::DelMarkerExpiration { days: Some(1) }),
+                filter: None,
+                id: Some("test-rule".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let locked_config = ObjectLockConfiguration {
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            ..Default::default()
+        };
+
+        let err = lc.validate(&locked_config).await.unwrap_err();
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_BUCKET_LOCKED);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn validate_rejects_zero_day_del_marker_expiration_on_locked_bucket() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: Some(s3s::dto::DelMarkerExpiration { days: Some(0) }),
+                filter: None,
+                id: Some("test-rule".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let locked_config = ObjectLockConfiguration {
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            ..Default::default()
+        };
+
+        let err = lc.validate(&locked_config).await.unwrap_err();
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_BUCKET_LOCKED);
+    }
+
     // --- TASK-003 tests: Round up to next UTC processing boundary ---
 
     #[test]

crates/ecstore/src/bucket/metadata.rs

@@ -244,6 +244,8 @@ pub const BUCKET_ACCELERATE_CONFIG: &str = "accelerate.xml";
 pub const BUCKET_REQUEST_PAYMENT_CONFIG: &str = "request-payment.xml";
 pub const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = "public-access-block.xml";
 pub const BUCKET_ACL_CONFIG: &str = "bucket-acl.json";
+pub const BUCKET_TABLE_CONFIG: &str = "table-bucket.json";
+pub const BUCKET_TABLE_RESERVED_PREFIX: &str = ".rustfs-table";
 
 #[derive(Debug, Clone)]
 pub struct BucketMetadata {
@@ -268,6 +270,7 @@ pub struct BucketMetadata {
     pub request_payment_config_xml: Vec<u8>,
     pub public_access_block_config_xml: Vec<u8>,
     pub bucket_acl_config_json: Vec<u8>,
+    pub table_bucket_config_json: Vec<u8>,
 
     pub policy_config_updated_at: OffsetDateTime,
     pub object_lock_config_updated_at: OffsetDateTime,
@@ -287,6 +290,7 @@ pub struct BucketMetadata {
     pub request_payment_config_updated_at: OffsetDateTime,
     pub public_access_block_config_updated_at: OffsetDateTime,
     pub bucket_acl_config_updated_at: OffsetDateTime,
+    pub table_bucket_config_updated_at: OffsetDateTime,
 
     pub new_field_updated_at: OffsetDateTime,
 
@@ -334,6 +338,7 @@ impl Default for BucketMetadata {
             request_payment_config_xml: Default::default(),
             public_access_block_config_xml: Default::default(),
             bucket_acl_config_json: Default::default(),
+            table_bucket_config_json: Default::default(),
             policy_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             object_lock_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             encryption_config_updated_at: OffsetDateTime::UNIX_EPOCH,
@@ -352,6 +357,7 @@ impl Default for BucketMetadata {
             request_payment_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             public_access_block_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             bucket_acl_config_updated_at: OffsetDateTime::UNIX_EPOCH,
+            table_bucket_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             new_field_updated_at: OffsetDateTime::UNIX_EPOCH,
             policy_config: Default::default(),
             notification_config: Default::default(),
@@ -397,6 +403,10 @@ impl BucketMetadata {
         self.lock_enabled || (self.versioning_config.as_ref().is_some_and(|v| v.enabled()))
     }
 
+    pub fn table_bucket_enabled(&self) -> bool {
+        !self.table_bucket_config_json.is_empty()
+    }
+
     /// Decode from msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn decode_from<R: Read>(&mut self, rd: &mut R) -> Result<()> {
         let mut fields = rmp::decode::read_map_len(rd)?;
@@ -447,6 +457,7 @@ impl BucketMetadata {
                     self.public_access_block_config_xml = read_msgp_bin(rd)?
                 }
                 "BucketAclConfigJSON" | "BucketAclConfigJson" => self.bucket_acl_config_json = read_msgp_bin(rd)?,
+                "TableBucketConfigJSON" | "TableBucketConfigJson" => self.table_bucket_config_json = read_msgp_bin(rd)?,
                 "CorsConfigUpdatedAt" => self.cors_config_updated_at = read_msgp_time_value(rd)?,
                 "LoggingConfigUpdatedAt" => self.logging_config_updated_at = read_msgp_time_value(rd)?,
                 "WebsiteConfigUpdatedAt" => self.website_config_updated_at = read_msgp_time_value(rd)?,
@@ -454,6 +465,7 @@ impl BucketMetadata {
                 "RequestPaymentConfigUpdatedAt" => self.request_payment_config_updated_at = read_msgp_time_value(rd)?,
                 "PublicAccessBlockConfigUpdatedAt" => self.public_access_block_config_updated_at = read_msgp_time_value(rd)?,
                 "BucketAclConfigUpdatedAt" => self.bucket_acl_config_updated_at = read_msgp_time_value(rd)?,
+                "TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
                 other => {
                     tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
                     skip_msgp_value(rd)?;
@@ -466,8 +478,8 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Map size: MinIO fields (25) + RustFS extensions (14)
-        let map_len: u32 = 39;
+        // Map size: MinIO fields (25) + RustFS extensions (16)
+        let map_len: u32 = 41;
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -523,6 +535,7 @@ impl BucketMetadata {
         write_bin_field(wr, "RequestPaymentConfigXML", &self.request_payment_config_xml)?;
         write_bin_field(wr, "PublicAccessBlockConfigXML", &self.public_access_block_config_xml)?;
         write_bin_field(wr, "BucketAclConfigJSON", &self.bucket_acl_config_json)?;
+        write_bin_field(wr, "TableBucketConfigJSON", &self.table_bucket_config_json)?;
         rmp::encode::write_str(wr, "CorsConfigUpdatedAt")?;
         write_msgp_time(wr, self.cors_config_updated_at)?;
         rmp::encode::write_str(wr, "LoggingConfigUpdatedAt")?;
@@ -537,6 +550,8 @@ impl BucketMetadata {
         write_msgp_time(wr, self.public_access_block_config_updated_at)?;
         rmp::encode::write_str(wr, "BucketAclConfigUpdatedAt")?;
         write_msgp_time(wr, self.bucket_acl_config_updated_at)?;
+        rmp::encode::write_str(wr, "TableBucketConfigUpdatedAt")?;
+        write_msgp_time(wr, self.table_bucket_config_updated_at)?;
 
         Ok(())
     }
@@ -632,6 +647,9 @@ impl BucketMetadata {
         if self.bucket_acl_config_updated_at == OffsetDateTime::UNIX_EPOCH {
             self.bucket_acl_config_updated_at = self.created
         }
+        if self.table_bucket_config_updated_at == OffsetDateTime::UNIX_EPOCH {
+            self.table_bucket_config_updated_at = self.created
+        }
     }
 
     pub fn update_config(&mut self, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
@@ -709,6 +727,10 @@ impl BucketMetadata {
                 self.bucket_acl_config_json = data;
                 self.bucket_acl_config_updated_at = updated;
             }
+            BUCKET_TABLE_CONFIG => {
+                self.table_bucket_config_json = data;
+                self.table_bucket_config_updated_at = updated;
+            }
             _ => return Err(Error::other(format!("config file not found : {config_file}"))),
         }
 
@@ -1028,6 +1050,10 @@ mod test {
         bm.bucket_acl_config_json = bucket_acl.as_bytes().to_vec();
         bm.bucket_acl_config_updated_at = OffsetDateTime::now_utc();
 
+        let table_bucket_marker = r#"{"enabled":true}"#;
+        bm.table_bucket_config_json = table_bucket_marker.as_bytes().to_vec();
+        bm.table_bucket_config_updated_at = OffsetDateTime::now_utc();
+
         // Test serialization
         let buf = bm.marshal_msg().unwrap();
         assert!(!buf.is_empty(), "Serialized buffer should not be empty");
@@ -1049,6 +1075,7 @@ mod test {
         assert_eq!(bm.quota_config_json, deserialized_bm.quota_config_json);
         assert_eq!(bm.public_access_block_config_xml, deserialized_bm.public_access_block_config_xml);
         assert_eq!(bm.bucket_acl_config_json, deserialized_bm.bucket_acl_config_json);
+        assert_eq!(bm.table_bucket_config_json, deserialized_bm.table_bucket_config_json);
         assert_eq!(bm.object_lock_config_xml, deserialized_bm.object_lock_config_xml);
         assert_eq!(bm.notification_config_xml, deserialized_bm.notification_config_xml);
         assert_eq!(bm.replication_config_xml, deserialized_bm.replication_config_xml);
@@ -1100,6 +1127,11 @@ mod test {
             bm.bucket_targets_config_meta_updated_at.unix_timestamp(),
             deserialized_bm.bucket_targets_config_meta_updated_at.unix_timestamp()
         );
+        assert_eq!(
+            bm.table_bucket_config_updated_at.unix_timestamp(),
+            deserialized_bm.table_bucket_config_updated_at.unix_timestamp()
+        );
+        assert!(deserialized_bm.table_bucket_enabled());
 
         // Test that the serialized data contains expected content
         let buf_str = String::from_utf8_lossy(&buf);
@@ -1116,6 +1148,20 @@ mod test {
         println!("   - Serialized buffer size: {} bytes", buf.len());
     }
 
+    #[test]
+    fn table_bucket_marker_tracks_config_presence() {
+        let mut bm = BucketMetadata::new("table-bucket");
+        assert!(!bm.table_bucket_enabled());
+
+        bm.update_config(BUCKET_TABLE_CONFIG, br#"{"enabled":true}"#.to_vec())
+            .unwrap();
+        assert!(bm.table_bucket_enabled());
+        assert!(!bm.table_bucket_config_json.is_empty());
+
+        bm.update_config(BUCKET_TABLE_CONFIG, Vec::new()).unwrap();
+        assert!(!bm.table_bucket_enabled());
+    }
+
     /// After policy deletion (policy_config_json cleared), parse_policy_config sets policy_config to None.
     #[test]
     fn test_parse_policy_config_clears_cache_when_json_empty() {

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -1079,7 +1079,7 @@ pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc:
         target_statuses: tgt_statuses,
         target_purge_statuses: purge_statuses,
         replication_timestamp: tm,
-        user_tags: oi.user_tags,
+        user_tags: (*oi.user_tags).clone(),
         checksum: None,
         retry_count: 0,
         event_type: "".to_string(),

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -929,18 +929,18 @@ impl ReplicationResyncer {
 }
 
 fn heal_should_use_check_replicate_delete(oi: &ObjectInfo) -> bool {
-    oi.delete_marker || (!oi.replication_status.is_empty() && oi.replication_status != ReplicationStatusType::Failed)
+    oi.delete_marker || !oi.version_purge_status.is_empty()
 }
 
 pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationConfig) -> ReplicateObjectInfo {
     let mut oi = oi.clone();
-    let mut user_defined = oi.user_defined.clone();
+    let mut user_defined = (*oi.user_defined).clone();
 
     if let Some(rc) = rcfg.config.as_ref()
         && !rc.role.is_empty()
     {
-        if !oi.replication_status.is_empty() {
-            oi.replication_status_internal = Some(format!("{}={};", rc.role, oi.replication_status.as_str()));
+        if !oi.version_purge_status.is_empty() {
+            oi.version_purge_status_internal = Some(format!("{}={};", rc.role, oi.version_purge_status.as_str()));
         }
 
         if !oi.replication_status.is_empty() {
@@ -982,7 +982,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
             &oi.name,
             MustReplicateOptions::new(
                 &user_defined,
-                oi.user_tags.clone(),
+                (*oi.user_tags).clone(),
                 ReplicationStatusType::Empty,
                 ReplicationType::Heal,
                 ObjectOptions::default(),
@@ -1020,7 +1020,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
         target_purge_statuses,
         replication_timestamp: None,
         ssec: false, // TODO: add ssec support
-        user_tags: oi.user_tags.clone(),
+        user_tags: (*oi.user_tags).clone(),
         checksum: oi.checksum.clone(),
         retry_count: 0,
     }
@@ -1158,7 +1158,7 @@ impl ReplicationConfig {
             return self.resync_internal(oi, dsc, status);
         }
 
-        let mut user_defined = oi.user_defined.clone();
+        let mut user_defined = (*oi.user_defined).clone();
         user_defined.remove(AMZ_BUCKET_REPLICATION_STATUS);
 
         let dsc = must_replicate(
@@ -1166,7 +1166,7 @@ impl ReplicationConfig {
             &oi.name,
             MustReplicateOptions::new(
                 &user_defined,
-                oi.user_tags.clone(),
+                (*oi.user_tags).clone(),
                 ReplicationStatusType::Empty,
                 ReplicationType::ExistingObject,
                 ObjectOptions::default(),
@@ -1296,7 +1296,7 @@ impl MustReplicateOptions {
     }
 
     pub fn from_object_info(oi: &ObjectInfo, op_type: ReplicationType, opts: ObjectOptions) -> Self {
-        Self::new(&oi.user_defined, oi.user_tags.clone(), oi.replication_status.clone(), op_type, opts)
+        Self::new(&oi.user_defined, (*oi.user_tags).clone(), oi.replication_status.clone(), op_type, opts)
     }
 
     pub fn replication_status(&self) -> ReplicationStatusType {
@@ -1411,7 +1411,7 @@ fn delete_replication_object_opts(dobj: &ObjectToDelete, oi: &ObjectInfo) -> Obj
     ObjectOpts {
         name: dobj.object_name.clone(),
         ssec: is_ssec_encrypted(&oi.user_defined),
-        user_tags: oi.user_tags.clone(),
+        user_tags: (*oi.user_tags).clone(),
         delete_marker: oi.delete_marker,
         version_id: dobj.version_id,
         op_type: ReplicationType::Delete,
@@ -2995,7 +2995,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             mod_time: self.mod_time,
             version_id: self.version_id,
             size: self.size,
-            user_tags: self.user_tags.clone(),
+            user_tags: Arc::new(self.user_tags.clone()),
             actual_size: self.actual_size,
             replication_status_internal: self.replication_status_internal.clone(),
             replication_status: self.replication_status.clone(),
@@ -3221,7 +3221,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
     }
 
     // Use case-insensitive lookup for headers
-    let lk_map = object_info.user_defined.clone();
+    let lk_map = &*object_info.user_defined;
 
     if let Some(lang) = lk_map.lookup(CONTENT_LANGUAGE) {
         put_op.content_language = lang.to_string();
@@ -3500,7 +3500,7 @@ fn get_replication_action(oi1: &ObjectInfo, oi2: &HeadObjectOutput, op_type: Rep
 
     // compare metadata on both maps to see if meta is identical
     let mut compare_meta1 = HashMap::new();
-    for (k, v) in &oi1.user_defined {
+    for (k, v) in oi1.user_defined.iter() {
         let mut found = false;
         for prefix in &compare_keys {
             if strings_has_prefix_fold(k, prefix) {
@@ -3742,7 +3742,7 @@ mod tests {
     }
 
     #[test]
-    fn test_heal_should_use_check_replicate_delete_pending_uses_delete_path() {
+    fn test_heal_should_use_check_replicate_delete_pending_non_delete_marker_uses_must_replicate_path() {
         let oi = ObjectInfo {
             bucket: "b".to_string(),
             name: "obj".to_string(),
@@ -3751,8 +3751,8 @@ mod tests {
             ..Default::default()
         };
         assert!(
-            heal_should_use_check_replicate_delete(&oi),
-            "Pending (non-Failed) status with non-empty replication uses check_replicate_delete path"
+            !heal_should_use_check_replicate_delete(&oi),
+            "Pending non-delete-marker object must use must_replicate path to evaluate current target set"
         );
     }
 
@@ -3771,6 +3771,36 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_heal_should_use_check_replicate_delete_version_purge_status_uses_delete_path() {
+        let oi = ObjectInfo {
+            bucket: "b".to_string(),
+            name: "obj".to_string(),
+            delete_marker: false,
+            version_purge_status: VersionPurgeStatusType::Pending,
+            ..Default::default()
+        };
+        assert!(
+            heal_should_use_check_replicate_delete(&oi),
+            "Version purge entries must use check_replicate_delete path"
+        );
+    }
+
+    #[test]
+    fn test_heal_should_use_check_replicate_delete_completed_non_delete_marker_uses_must_replicate_path() {
+        let oi = ObjectInfo {
+            bucket: "b".to_string(),
+            name: "obj".to_string(),
+            delete_marker: false,
+            replication_status: ReplicationStatusType::Completed,
+            ..Default::default()
+        };
+        assert!(
+            !heal_should_use_check_replicate_delete(&oi),
+            "Completed non-delete-marker object must use must_replicate path so new targets can be evaluated"
+        );
+    }
+
     #[test]
     fn test_delete_replication_object_opts_marks_replica_deletes() {
         let dobj = ObjectToDelete {
@@ -4051,6 +4081,32 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn test_get_heal_replicate_object_info_maps_version_purge_status_for_role() {
+        let role = "arn:rustfs:replication::target:bucket";
+        let oi = ObjectInfo {
+            bucket: "test-bucket".to_string(),
+            name: "key".to_string(),
+            delete_marker: false,
+            version_purge_status: VersionPurgeStatusType::Pending,
+            version_id: Some(Uuid::nil()),
+            mod_time: Some(OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+        let rcfg = ReplicationConfig::new(
+            Some(ReplicationConfiguration {
+                role: role.to_string(),
+                rules: vec![],
+            }),
+            None,
+        );
+        let roi = get_heal_replicate_object_info(&oi, &rcfg).await;
+
+        assert_eq!(roi.replication_status_internal, None);
+        assert_eq!(roi.version_purge_status_internal.as_deref(), Some(format!("{role}=PENDING;").as_str()));
+        assert_eq!(roi.target_purge_statuses.get(role), Some(&VersionPurgeStatusType::Pending));
+    }
+
     #[tokio::test]
     async fn test_cancel_marks_only_matching_bucket_target_token() {
         let resyncer = ReplicationResyncer::new().await;

crates/ecstore/src/cache_value/metacache_set.rs

@@ -442,10 +442,13 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 {
                     finished_fn(&errs).await;
                 }
-                if errs.iter().flatten().any(|err| *err == DiskError::Timeout) {
-                    return Err(DiskError::Timeout);
-                }
-
+                // All remaining readers are either at EOF or failed. Earlier logic
+                // returned Timeout here for even a single stalled drive, despite
+                // `has_err` being within the tolerated drive-failure budget. That
+                // makes small distributed listings fail once healthy quorum readers
+                // reach EOF but one remote walk stream is slow/stalled. Only the
+                // `has_err > opts.disks.len() - opts.min_disks` branch above should
+                // turn tolerated reader failures into request failures.
                 // error!("list_path_raw: at_eof + has_err == readers.len() break {:?}", &errs);
                 break;
             }
@@ -486,13 +489,23 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
         return Err(err);
     }
 
+    // The merge consumer can finish successfully before every producer finishes
+    // (for example after reaching EOF quorum while a tolerated drive is stalled,
+    // or after the requested listing limit is satisfied). Cancel remaining walk
+    // jobs before joining them so list calls do not wait for slow remote streams.
+    cancel_rx.cancel();
+
     let results = join_all(jobs).await;
     let mut job_errs = Vec::new();
     for result in results {
         match result {
             Ok(Ok(())) => {}
             Ok(Err(err)) => {
-                error!("list_path_raw producer err {:?}", err);
+                if matches!(err, DiskError::FileNotFound | DiskError::VolumeNotFound) {
+                    warn!("list_path_raw producer missing path {:?}", err);
+                } else {
+                    error!("list_path_raw producer err {:?}", err);
+                }
                 job_errs.push(err);
             }
             Err(err) => {
@@ -541,18 +554,34 @@ mod tests {
             CancellationToken::new(),
             ListPathRawOptions {
                 disks: vec![None, None],
-                min_disks: 1,
+                min_disks: 2,
                 test_reader_behaviors: vec![TestReaderBehavior::Stall, TestReaderBehavior::Eof],
                 peek_timeout: Some(Duration::from_millis(20)),
                 ..Default::default()
             },
         )
         .await
-        .expect_err("stalled reader should make listing fail explicitly");
+        .expect_err("stalled reader should fail when read quorum cannot be met");
 
         assert_eq!(err, DiskError::Timeout);
     }
 
+    #[tokio::test]
+    async fn list_path_raw_tolerates_stalled_reader_after_quorum_eof() {
+        list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None, None, None],
+                min_disks: 2,
+                test_reader_behaviors: vec![TestReaderBehavior::Eof, TestReaderBehavior::Eof, TestReaderBehavior::Stall],
+                peek_timeout: Some(Duration::from_millis(20)),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("listing should complete when healthy quorum reached EOF and only a tolerated drive stalled");
+    }
+
     #[tokio::test]
     async fn list_path_raw_returns_timeout_when_producer_fails_after_partial_entry() {
         let seen = Arc::new(Mutex::new(Vec::new()));

crates/ecstore/src/client/transition_api.rs

@@ -51,6 +51,7 @@ use md5::Md5;
 use rand::{Rng, RngExt};
 use rustfs_config::MAX_S3_CLIENT_RESPONSE_SIZE;
 use rustfs_rio::HashReader;
+use rustfs_tls_runtime::{load_global_outbound_tls_state, record_tls_generation};
 use rustfs_utils::HashAlgorithm;
 use rustfs_utils::{
     net::get_endpoint_url,
@@ -58,6 +59,8 @@ use rustfs_utils::{
         DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, MAX_RETRY, RetryTimer, is_http_status_retryable, is_s3code_retryable,
     },
 };
+use rustls_pki_types::PrivateKeyDer;
+use rustls_pki_types::pem::PemObject;
 use s3s::S3ErrorCode;
 use s3s::dto::Owner;
 use s3s::dto::ReplicationStatus;
@@ -152,24 +155,11 @@ pub enum BucketLookupType {
     BucketLookupPath,
 }
 
-fn load_root_store_from_tls_path() -> Option<rustls::RootCertStore> {
-    // Load the root certificate bundle from the path specified by the
-    // RUSTFS_TLS_PATH environment variable.
-    let tp = rustfs_utils::get_env_str(rustfs_config::ENV_RUSTFS_TLS_PATH, rustfs_config::DEFAULT_RUSTFS_TLS_PATH);
-    // If no TLS path is configured, do not fall back to a CA bundle in the current directory.
-    if tp.is_empty() {
-        return None;
-    }
-    let ca = std::path::Path::new(&tp).join(rustfs_config::RUSTFS_CA_CERT);
-    if !ca.exists() {
-        return None;
-    }
-
-    let der_list = rustfs_utils::load_cert_bundle_der_bytes(ca.to_str().unwrap_or_default()).ok()?;
+fn build_root_store_from_der_list(der_list: Vec<Vec<u8>>) -> Option<rustls::RootCertStore> {
     let mut store = rustls::RootCertStore::empty();
     for der in der_list {
         if let Err(e) = store.add(der.into()) {
-            warn!("Warning: failed to add certificate from '{}' to root store: {e}", ca.display());
+            warn!("Warning: failed to add certificate to root store: {e}");
         }
     }
     Some(store)
@@ -199,18 +189,38 @@ where
     })
 }
 
-fn build_tls_config() -> Result<rustls::ClientConfig, std::io::Error> {
-    with_rustls_init_guard(|| {
-        let config = if let Some(store) = load_root_store_from_tls_path() {
-            rustls::ClientConfig::builder()
-                .with_root_certificates(store)
-                .with_no_client_auth()
-        } else {
-            rustls::ClientConfig::builder().with_native_roots()?.with_no_client_auth()
-        };
+async fn build_tls_config() -> Result<rustls::ClientConfig, std::io::Error> {
+    with_rustls_init_guard(|| Ok(()))?;
 
-        Ok(config)
-    })
+    let outbound_tls = load_global_outbound_tls_state().await;
+    record_tls_generation("ecstore_transition_client", outbound_tls.generation.0);
+    let builder = if let Some(root_ca_pem) = outbound_tls.root_ca_pem.as_ref() {
+        let mut reader = std::io::BufReader::new(root_ca_pem.as_slice());
+        let certs_der = rustls_pki_types::CertificateDer::pem_reader_iter(&mut reader)
+            .collect::<Result<Vec<_>, _>>()
+            .map_err(|e| std::io::Error::other(format!("failed to parse published root CA PEM: {e}")))?;
+
+        let root_store = build_root_store_from_der_list(certs_der.into_iter().map(|cert| cert.to_vec()).collect::<Vec<_>>())
+            .ok_or_else(|| std::io::Error::other("published outbound root CA material could not build root store"))?;
+        rustls::ClientConfig::builder().with_root_certificates(root_store)
+    } else {
+        rustls::ClientConfig::builder().with_native_roots()?
+    };
+
+    let config = if let Some(identity) = outbound_tls.mtls_identity.as_ref() {
+        let certs = rustls_pki_types::CertificateDer::pem_reader_iter(&mut std::io::BufReader::new(identity.cert_pem.as_slice()))
+            .collect::<Result<Vec<_>, _>>()
+            .map_err(|e| std::io::Error::other(format!("failed to parse published client cert PEM: {e}")))?;
+        let key = PrivateKeyDer::from_pem_reader(&mut std::io::BufReader::new(identity.key_pem.as_slice()))
+            .map_err(|e| std::io::Error::other(format!("failed to parse published client key PEM: {e}")))?;
+        builder
+            .with_client_auth_cert(certs, key)
+            .map_err(|e| std::io::Error::other(format!("failed to build client mTLS identity: {e}")))?
+    } else {
+        builder.with_no_client_auth()
+    };
+
+    Ok(config)
 }
 
 impl TransitionClient {
@@ -234,7 +244,7 @@ impl TransitionClient {
 
         let endpoint_url = get_endpoint_url(endpoint, opts.secure)?;
 
-        let tls = build_tls_config()?;
+        let tls = build_tls_config().await?;
 
         let https = hyper_rustls::HttpsConnectorBuilder::new()
             .with_tls_config(tls)
@@ -1374,9 +1384,7 @@ pub struct CreateBucketConfiguration {
 
 #[cfg(test)]
 mod tests {
-    use super::{
-        build_tls_config, load_root_store_from_tls_path, signer_error_to_io_error, validate_header_values, with_rustls_init_guard,
-    };
+    use super::{build_tls_config, signer_error_to_io_error, validate_header_values, with_rustls_init_guard};
     use http::{HeaderMap, HeaderValue};
 
     #[test]
@@ -1395,22 +1403,6 @@ mod tests {
         assert!(outcome.is_ok(), "TLS config creation should not panic");
     }
 
-    /// When RUSTFS_TLS_PATH is not set, `load_root_store_from_tls_path` must return `None`
-    /// (i.e. it must not silently look for a CA bundle in the current working directory).
-    #[test]
-    fn tls_path_unset_returns_none() {
-        let result = temp_env::with_var_unset(rustfs_config::ENV_RUSTFS_TLS_PATH, || load_root_store_from_tls_path());
-        assert!(result.is_none(), "expected None when RUSTFS_TLS_PATH is unset, but got a root store");
-    }
-
-    /// When RUSTFS_TLS_PATH is set to an empty string, `load_root_store_from_tls_path` must
-    /// return `None` to avoid accidentally trusting a CA bundle in the current directory.
-    #[test]
-    fn tls_path_empty_returns_none() {
-        let result = temp_env::with_var(rustfs_config::ENV_RUSTFS_TLS_PATH, Some(""), || load_root_store_from_tls_path());
-        assert!(result.is_none(), "expected None when RUSTFS_TLS_PATH is empty, but got a root store");
-    }
-
     /// Installing the rustls crypto provider when one is already set must not panic or return
     /// an error that surfaces to callers (the race-safe `get_default` check guards the install).
     #[test]

crates/ecstore/src/config/com.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::config::{Config, GLOBAL_STORAGE_CLASS, KVS, audit, notify, oidc, storageclass};
+use crate::config::{Config, KVS, audit, notify, oidc, set_global_storage_class, storageclass};
 use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result};
 use crate::global::is_first_cluster_node_local;
@@ -1164,11 +1164,8 @@ async fn apply_dynamic_config_for_sub_sys<S: StorageAPI>(cfg: &mut Config, api:
         for (i, count) in set_drive_counts.iter().enumerate() {
             match storageclass::lookup_config(&kvs, *count) {
                 Ok(res) => {
-                    if i == 0
-                        && GLOBAL_STORAGE_CLASS.get().is_none()
-                        && let Err(r) = GLOBAL_STORAGE_CLASS.set(res)
-                    {
-                        error!("GLOBAL_STORAGE_CLASS.set failed {:?}", r);
+                    if i == 0 {
+                        set_global_storage_class(res);
                     }
                 }
                 Err(err) => {
@@ -1358,7 +1355,7 @@ mod tests {
                 size: self.data.len() as i64,
                 actual_size: self.data.len() as i64,
                 is_dir: false,
-                user_defined: HashMap::new(),
+                user_defined: Arc::new(HashMap::new()),
                 parity_blocks: 0,
                 data_blocks: 0,
                 version_id: None,
@@ -1366,8 +1363,8 @@ mod tests {
                 transitioned_object: Default::default(),
                 restore_ongoing: false,
                 restore_expires: None,
-                user_tags: String::new(),
-                parts: Vec::new(),
+                user_tags: Arc::new(String::new()),
+                parts: Arc::new(Vec::new()),
                 is_latest: true,
                 content_type: Some("application/json".to_string()),
                 content_encoding: None,
@@ -2638,4 +2635,87 @@ mod tests {
         assert_eq!(object_info.bucket, crate::disk::RUSTFS_META_BUCKET);
         assert_eq!(object_info.name, "config/test.json");
     }
+
+    #[test]
+    fn test_encode_decode_roundtrip_preserves_storage_class() {
+        use crate::config::STORAGE_CLASS_SUB_SYS;
+
+        // Create a config with custom storage class values
+        let mut cfg = Config::new();
+        let kvs = storage_class_kvs_mut(&mut cfg);
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        kvs.insert("rrs".to_string(), "EC:2".to_string());
+        kvs.insert("optimize".to_string(), "availability".to_string());
+
+        // Encode to external format (what save_server_config produces)
+        let encoded = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+
+        // Verify the encoded data uses "storageclass" (no underscore)
+        let v: serde_json::Value = serde_json::from_slice(&encoded).expect("should be valid json");
+        assert!(v.get("storageclass").is_some(), "encoded should have 'storageclass' key");
+        assert!(v.get("storage_class").is_none(), "encoded should NOT have 'storage_class' key");
+
+        // Decode back (what load_server_config_from_store produces)
+        let decoded = decode_server_config_blob(&encoded).expect("decode should succeed");
+
+        // Verify the decoded config has "storage_class" (with underscore) subsystem
+        let kvs = decoded
+            .get_value(STORAGE_CLASS_SUB_SYS, crate::config::DEFAULT_DELIMITER)
+            .expect("decoded config should have storage_class subsystem");
+        assert_eq!(kvs.get("standard"), "EC:4", "standard should be EC:4");
+        assert_eq!(kvs.get("rrs"), "EC:2", "rrs should be EC:2");
+        assert_eq!(kvs.get("optimize"), "availability", "optimize should be availability");
+    }
+
+    #[test]
+    fn test_set_then_load_preserves_storage_class_values() {
+        use crate::config::STORAGE_CLASS_SUB_SYS;
+
+        // Step 1: Start with a fresh config (simulates initial server state)
+        let mut cfg = Config::new();
+
+        // Step 2: Apply set directives (simulates "mc admin config set storage_class standard=EC:4 rrs=EC:2")
+        let kvs = cfg
+            .0
+            .entry(STORAGE_CLASS_SUB_SYS.to_string())
+            .or_default()
+            .entry(DEFAULT_DELIMITER.to_string())
+            .or_default();
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        kvs.insert("rrs".to_string(), "EC:2".to_string());
+        cfg.set_defaults();
+
+        // Step 3: Save (encode to external format)
+        let encoded = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+
+        // Step 4: Load (decode from external format)
+        let loaded = decode_server_config_blob(&encoded).expect("decode should succeed");
+
+        // Step 5: Verify the values are preserved
+        let loaded_kvs = loaded
+            .get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("should have storage_class");
+        assert_eq!(loaded_kvs.get("standard"), "EC:4");
+        assert_eq!(loaded_kvs.get("rrs"), "EC:2");
+
+        // Step 6: Verify the subsystem is accessible by name (what render_selected_config does)
+        let targets = loaded
+            .0
+            .get(STORAGE_CLASS_SUB_SYS)
+            .expect("storage_class subsystem should exist");
+        let target_kvs = targets.get(DEFAULT_DELIMITER).expect("default target should exist");
+        assert_eq!(target_kvs.get("standard"), "EC:4");
+        assert_eq!(target_kvs.get("rrs"), "EC:2");
+    }
+
+    #[test]
+    fn test_config_unmarshal_fails_on_external_format() {
+        // This verifies that the external "storageclass" format cannot be
+        // mistakenly parsed by Config::unmarshal (which expects internal format).
+        // If unmarshal succeeds, the config would have "storageclass" instead of
+        // "storage_class", causing render_selected_config to fail.
+        let external_json = r#"{"version":"33","storageclass":{"standard":"EC:4","rrs":"EC:2"}}"#;
+        let result = Config::unmarshal(external_json.as_bytes());
+        assert!(result.is_err(), "Config::unmarshal should fail on external format, but got: {:?}", result);
+    }
 }

crates/ecstore/src/config/mod.rs

@@ -18,6 +18,7 @@ pub mod com;
 pub mod heal;
 mod notify;
 mod oidc;
+mod scanner;
 pub mod storageclass;
 
 use crate::error::Result;
@@ -37,11 +38,12 @@ use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::sync::LazyLock;
-use std::sync::{Arc, OnceLock};
+use std::sync::{Arc, OnceLock, RwLock};
 
-pub static GLOBAL_STORAGE_CLASS: LazyLock<OnceLock<storageclass::Config>> = LazyLock::new(OnceLock::new);
+pub static GLOBAL_STORAGE_CLASS: LazyLock<RwLock<storageclass::Config>> =
+    LazyLock::new(|| RwLock::new(storageclass::Config::default()));
 pub static DEFAULT_KVS: LazyLock<OnceLock<HashMap<String, KVS>>> = LazyLock::new(OnceLock::new);
-pub static GLOBAL_SERVER_CONFIG: LazyLock<OnceLock<Config>> = LazyLock::new(OnceLock::new);
+pub static GLOBAL_SERVER_CONFIG: LazyLock<RwLock<Option<Config>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_CONFIG_SYS: LazyLock<ConfigSys> = LazyLock::new(ConfigSys::new);
 
 pub static RUSTFS_CONFIG_PREFIX: &str = "config";
@@ -63,14 +65,30 @@ impl ConfigSys {
 
         lookup_configs(&mut cfg, api).await;
 
-        let _ = GLOBAL_SERVER_CONFIG.set(cfg);
+        set_global_server_config(cfg);
 
         Ok(())
     }
 }
 
 pub fn get_global_server_config() -> Option<Config> {
-    GLOBAL_SERVER_CONFIG.get().cloned()
+    GLOBAL_SERVER_CONFIG.read().ok().and_then(|guard| (*guard).clone())
+}
+
+pub fn set_global_server_config(cfg: Config) {
+    if let Ok(mut guard) = GLOBAL_SERVER_CONFIG.write() {
+        *guard = Some(cfg);
+    }
+}
+
+pub fn get_global_storage_class() -> Option<storageclass::Config> {
+    GLOBAL_STORAGE_CLASS.read().ok().map(|guard| (*guard).clone())
+}
+
+pub fn set_global_storage_class(cfg: storageclass::Config) {
+    if let Ok(mut guard) = GLOBAL_STORAGE_CLASS.write() {
+        *guard = cfg;
+    }
 }
 
 pub async fn init_global_config_sys(api: Arc<ECStore>) -> Result<()> {
@@ -237,6 +255,7 @@ pub fn init() {
     let mut kvs = HashMap::new();
     // Load storageclass default configuration
     kvs.insert(STORAGE_CLASS_SUB_SYS.to_owned(), storageclass::DEFAULT_KVS.clone());
+    kvs.insert(rustfs_config::SCANNER_SUB_SYS.to_owned(), scanner::DEFAULT_KVS.clone());
     // New: Loading default configurations for notify_webhook and notify_mqtt
     // Referring subsystem names through constants to improve the readability and maintainability of the code
     kvs.insert(NOTIFY_WEBHOOK_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_WEBHOOK_KVS.clone());
@@ -262,3 +281,38 @@ pub fn init() {
     // Register all default configurations
     register_default_kvs(kvs)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use rustfs_config::{DEFAULT_DELIMITER, DEFAULT_SCANNER_SPEED, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_SPEED, SCANNER_SUB_SYS};
+
+    #[test]
+    fn global_server_config_set_and_get_roundtrip() {
+        init();
+        let mut cfg = Config::new();
+        let mut kvs = KVS::new();
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        cfg.0
+            .insert(STORAGE_CLASS_SUB_SYS.to_string(), HashMap::from([("_".to_string(), kvs)]));
+
+        set_global_server_config(cfg.clone());
+        let loaded = get_global_server_config().expect("global config should be set");
+        let sc_kvs = loaded
+            .get_value(STORAGE_CLASS_SUB_SYS, "_")
+            .expect("storage_class should exist");
+        assert_eq!(sc_kvs.get("standard"), "EC:4");
+    }
+
+    #[test]
+    fn scanner_defaults_are_registered_for_admin_config() {
+        init();
+        let cfg = Config::new();
+        let scanner_kvs = cfg
+            .get_value(SCANNER_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("scanner defaults should exist");
+
+        assert_eq!(scanner_kvs.get(SCANNER_SPEED), DEFAULT_SCANNER_SPEED);
+        assert_eq!(scanner_kvs.get(SCANNER_CYCLE_MAX_OBJECTS), "0");
+    }
+}

crates/ecstore/src/config/scanner.rs

@@ -0,0 +1,106 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::config::{KV, KVS};
+use rustfs_config::{
+    DEFAULT_SCANNER_BITROT_CYCLE_SECS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES,
+    DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_IDLE_MODE,
+    DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_SPEED,
+    DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE,
+    SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_IDLE_MODE,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_SPEED, SCANNER_START_DELAY,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
+};
+use std::sync::LazyLock;
+
+pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: SCANNER_SPEED.to_owned(),
+            value: DEFAULT_SCANNER_SPEED.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_START_DELAY.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_DURATION.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_OBJECTS.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_OBJECTS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_DIRECTORIES.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_BITROT_CYCLE.to_owned(),
+            value: DEFAULT_SCANNER_BITROT_CYCLE_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_IDLE_MODE.to_owned(),
+            value: DEFAULT_SCANNER_IDLE_MODE.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CACHE_SAVE_TIMEOUT.to_owned(),
+            value: DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_MAX_CONCURRENT_SET_SCANS.to_owned(),
+            value: DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_MAX_CONCURRENT_DISK_SCANS.to_owned(),
+            value: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_YIELD_EVERY_N_OBJECTS.to_owned(),
+            value: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_VERSIONS.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_VERSION_SIZE.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_FOLDERS.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS.to_string(),
+            hidden_if_empty: false,
+        },
+    ])
+});

crates/ecstore/src/config/storageclass.rs

@@ -101,13 +101,13 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
 });
 
 // StorageClass - holds storage class information
-#[derive(Serialize, Deserialize, Debug, Default)]
+#[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct StorageClass {
     parity: usize,
 }
 
 // Config storage class configuration
-#[derive(Serialize, Deserialize, Debug, Default)]
+#[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Config {
     standard: StorageClass,
     rrs: StorageClass,
@@ -205,7 +205,7 @@ pub fn lookup_config(kvs: &KVS, set_drive_count: usize) -> Result<Config> {
             if let Ok(ssc_str) = env::var(RRS_ENV) {
                 ssc_str
             } else {
-                kvs.get(RRS)
+                kvs.get(CLASS_RRS)
             }
         };
 
@@ -336,3 +336,36 @@ pub fn validate_parity_inner(ss_parity: usize, rrs_parity: usize, set_drive_coun
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn lookup_config_reads_rrs_from_class_rrs_key() {
+        // Regression: kvs.get(RRS) used RRS="REDUCED_REDUNDANCY" instead of
+        // CLASS_RRS="rrs", so admin-written RRS values were never read back.
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
+        kvs.insert(CLASS_RRS.to_string(), "EC:2".to_string());
+
+        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        assert_eq!(cfg.standard.parity, 4, "standard parity should be 4");
+        assert_eq!(cfg.rrs.parity, 2, "rrs parity should be 2");
+    }
+
+    #[test]
+    fn lookup_config_ignores_redundancy_key_name() {
+        // Ensure the old key name "REDUCED_REDUNDANCY" is NOT read.
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
+        kvs.insert(RRS.to_string(), "EC:2".to_string());
+
+        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        assert_eq!(cfg.standard.parity, 4);
+        assert_eq!(
+            cfg.rrs.parity, DEFAULT_RRS_PARITY,
+            "rrs should fall back to default when CLASS_RRS key is absent"
+        );
+    }
+}

crates/ecstore/src/data_movement.rs

@@ -94,7 +94,7 @@ fn data_movement_new_multipart_opts(object_info: &ObjectInfo, src_pool_idx: usiz
     ObjectOptions {
         versioned: object_info.version_id.is_some(),
         version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
-        user_defined: object_info.user_defined.clone(),
+        user_defined: (*object_info.user_defined).clone(),
         preserve_etag: object_info.etag.clone(),
         src_pool_idx,
         data_movement: true,
@@ -120,7 +120,7 @@ fn data_movement_put_object_opts(object_info: &ObjectInfo, src_pool_idx: usize)
         data_movement: true,
         version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
         mod_time: object_info.mod_time,
-        user_defined: object_info.user_defined.clone(),
+        user_defined: (*object_info.user_defined).clone(),
         preserve_etag: object_info.etag.clone(),
         ..Default::default()
     }
@@ -341,7 +341,7 @@ mod tests {
         let object_info = ObjectInfo {
             version_id: Some(version_id),
             etag: Some("etag-value".to_string()),
-            user_defined: std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())]),
+            user_defined: Arc::new(std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())])),
             ..Default::default()
         };
 
@@ -382,7 +382,7 @@ mod tests {
             version_id: Some(version_id),
             mod_time: Some(OffsetDateTime::UNIX_EPOCH),
             etag: Some("etag-value".to_string()),
-            user_defined: std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())]),
+            user_defined: Arc::new(std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())])),
             ..Default::default()
         };
 

crates/ecstore/src/data_usage.rs

@@ -48,7 +48,14 @@ const DATA_USAGE_BLOOM_NAME: &str = ".bloomcycle.bin";
 pub const DATA_USAGE_CACHE_NAME: &str = ".usage-cache.bin";
 const DATA_USAGE_CACHE_TTL_SECS: u64 = 30;
 
-type UsageMemoryCache = Arc<RwLock<HashMap<String, (u64, SystemTime)>>>;
+#[derive(Debug, Clone)]
+struct CachedBucketUsage {
+    usage: BucketUsageInfo,
+    refreshed_at: SystemTime,
+    usage_updated_at: SystemTime,
+}
+
+type UsageMemoryCache = Arc<RwLock<HashMap<String, CachedBucketUsage>>>;
 type CacheUpdating = Arc<RwLock<bool>>;
 
 static USAGE_MEMORY_CACHE: OnceLock<UsageMemoryCache> = OnceLock::new();
@@ -403,21 +410,90 @@ pub async fn compute_bucket_usage(store: Arc<ECStore>, bucket_name: &str) -> Res
     Ok(usage)
 }
 
-/// Fast in-memory increment for immediate quota consistency
-pub async fn increment_bucket_usage_memory(bucket: &str, size_increment: u64) {
+async fn ensure_bucket_usage_cached(bucket: &str) {
+    let cache = memory_cache().read().await;
+    if cache.contains_key(bucket) {
+        return;
+    }
+    drop(cache);
+
+    update_usage_cache_if_needed().await;
+}
+
+fn cached_bucket_usage_from_backend(usage: BucketUsageInfo, updated_at: SystemTime) -> CachedBucketUsage {
+    CachedBucketUsage {
+        usage,
+        refreshed_at: SystemTime::now(),
+        usage_updated_at: updated_at,
+    }
+}
+
+fn cached_bucket_usage_now(usage: BucketUsageInfo) -> CachedBucketUsage {
+    let now = SystemTime::now();
+    CachedBucketUsage {
+        usage,
+        refreshed_at: now,
+        usage_updated_at: now,
+    }
+}
+
+fn data_usage_info_updated_at(data_usage_info: &DataUsageInfo) -> SystemTime {
+    data_usage_info.last_update.unwrap_or(SystemTime::UNIX_EPOCH)
+}
+
+/// Fast in-memory update for immediate quota and admin usage consistency.
+pub async fn record_bucket_object_write_memory(bucket: &str, previous_current_size: Option<u64>, new_size: u64) {
+    ensure_bucket_usage_cached(bucket).await;
+
     let mut cache = memory_cache().write().await;
-    let current = cache.entry(bucket.to_string()).or_insert_with(|| (0, SystemTime::now()));
-    current.0 += size_increment;
-    current.1 = SystemTime::now();
+    let entry = cache
+        .entry(bucket.to_string())
+        .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));
+
+    match previous_current_size {
+        Some(previous_size) => {
+            entry.usage.size = entry.usage.size.saturating_sub(previous_size).saturating_add(new_size);
+        }
+        None => {
+            entry.usage.size = entry.usage.size.saturating_add(new_size);
+            entry.usage.objects_count = entry.usage.objects_count.saturating_add(1);
+            entry.usage.versions_count = entry.usage.versions_count.saturating_add(1);
+        }
+    }
+
+    let now = SystemTime::now();
+    entry.refreshed_at = now;
+    entry.usage_updated_at = now;
+}
+
+/// Fast in-memory increment for immediate quota consistency.
+pub async fn increment_bucket_usage_memory(bucket: &str, size_increment: u64) {
+    record_bucket_object_write_memory(bucket, None, size_increment).await;
+}
+
+/// Fast in-memory update for successful object deletes.
+pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
+    ensure_bucket_usage_cached(bucket).await;
+
+    let mut cache = memory_cache().write().await;
+    let entry = cache
+        .entry(bucket.to_string())
+        .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));
+
+    entry.usage.size = entry.usage.size.saturating_sub(deleted_size);
+    if removed_current_object {
+        entry.usage.objects_count = entry.usage.objects_count.saturating_sub(1);
+        entry.usage.versions_count = entry.usage.versions_count.saturating_sub(1);
+    }
+
+    let now = SystemTime::now();
+    entry.refreshed_at = now;
+    entry.usage_updated_at = now;
 }
 
 /// Fast in-memory decrement for immediate quota consistency
 pub async fn decrement_bucket_usage_memory(bucket: &str, size_decrement: u64) {
-    let mut cache = memory_cache().write().await;
-    if let Some(current) = cache.get_mut(bucket) {
-        current.0 = current.0.saturating_sub(size_decrement);
-        current.1 = SystemTime::now();
-    }
+    record_bucket_object_delete_memory(bucket, size_decrement, size_decrement > 0).await;
 }
 
 /// Get bucket usage from in-memory cache
@@ -425,7 +501,7 @@ pub async fn get_bucket_usage_memory(bucket: &str) -> Option<u64> {
     update_usage_cache_if_needed().await;
 
     let cache = memory_cache().read().await;
-    cache.get(bucket).map(|(usage, _)| *usage)
+    cache.get(bucket).map(|cached| cached.usage.size)
 }
 
 async fn update_usage_cache_if_needed() {
@@ -434,7 +510,7 @@ async fn update_usage_cache_if_needed() {
     let now = SystemTime::now();
 
     let cache = memory_cache().read().await;
-    let earliest_timestamp = cache.values().map(|(_, ts)| *ts).min();
+    let earliest_timestamp = cache.values().map(|cached| cached.refreshed_at).min();
     drop(cache);
 
     let age = match earliest_timestamp {
@@ -461,8 +537,12 @@ async fn update_usage_cache_if_needed() {
                 && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
             {
                 let mut cache = cache_clone.write().await;
+                let usage_updated_at = data_usage_info_updated_at(&data_usage_info);
                 for (bucket_name, bucket_usage) in data_usage_info.buckets_usage.iter() {
-                    cache.insert(bucket_name.clone(), (bucket_usage.size, SystemTime::now()));
+                    cache.insert(
+                        bucket_name.clone(),
+                        cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at),
+                    );
                 }
             }
             let mut updating = updating_clone.write().await;
@@ -488,8 +568,12 @@ async fn update_usage_cache_if_needed() {
         && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
     {
         let mut cache = memory_cache().write().await;
+        let usage_updated_at = data_usage_info_updated_at(&data_usage_info);
         for (bucket_name, bucket_usage) in data_usage_info.buckets_usage.iter() {
-            cache.insert(bucket_name.clone(), (bucket_usage.size, SystemTime::now()));
+            cache.insert(
+                bucket_name.clone(),
+                cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at),
+            );
         }
     }
 
@@ -497,15 +581,62 @@ async fn update_usage_cache_if_needed() {
     *updating = false;
 }
 
+pub async fn replace_bucket_usage_memory_from_info(data_usage_info: &DataUsageInfo) {
+    let usage_updated_at = data_usage_info_updated_at(data_usage_info);
+    let mut cache = memory_cache().write().await;
+    let mut next_cache = HashMap::new();
+
+    for (bucket, bucket_usage) in data_usage_info.buckets_usage.iter() {
+        if let Some(existing) = cache.get(bucket)
+            && existing.usage_updated_at > usage_updated_at
+        {
+            next_cache.insert(bucket.clone(), existing.clone());
+            continue;
+        }
+
+        next_cache.insert(bucket.clone(), cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at));
+    }
+
+    for (bucket, existing) in cache.iter() {
+        if !data_usage_info.buckets_usage.contains_key(bucket) && existing.usage_updated_at > usage_updated_at {
+            next_cache.insert(bucket.clone(), existing.clone());
+        }
+    }
+
+    *cache = next_cache;
+}
+
+pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageInfo) {
+    let cache = memory_cache().read().await;
+    if cache.is_empty() {
+        return;
+    }
+
+    let persisted_update = data_usage_info.last_update;
+    let mut changed = false;
+
+    for (bucket, cached) in cache.iter() {
+        if persisted_update.is_some_and(|persisted| cached.usage_updated_at <= persisted) {
+            continue;
+        }
+
+        data_usage_info.buckets_usage.insert(bucket.clone(), cached.usage.clone());
+        data_usage_info.bucket_sizes.insert(bucket.clone(), cached.usage.size);
+        changed = true;
+    }
+
+    if changed {
+        data_usage_info.buckets_count = data_usage_info.buckets_usage.len() as u64;
+        data_usage_info.calculate_totals();
+    }
+}
+
 /// Sync memory cache with backend data (called by scanner)
 pub async fn sync_memory_cache_with_backend() -> Result<(), Error> {
     if let Some(store) = crate::global::GLOBAL_OBJECT_API.get() {
         match load_data_usage_from_backend(store.clone()).await {
             Ok(data_usage_info) => {
-                let mut cache = memory_cache().write().await;
-                for (bucket, bucket_usage) in data_usage_info.buckets_usage.iter() {
-                    cache.insert(bucket.clone(), (bucket_usage.size, SystemTime::now()));
-                }
+                replace_bucket_usage_memory_from_info(&data_usage_info).await;
             }
             Err(e) => {
                 debug!("Failed to sync memory cache with backend: {}", e);
@@ -687,6 +818,32 @@ pub async fn save_data_usage_cache(cache: &DataUsageCache, name: &str) -> crate:
 mod tests {
     use super::*;
     use rustfs_data_usage::BucketUsageInfo;
+    use serial_test::serial;
+
+    async fn clear_usage_memory_cache_for_test() {
+        memory_cache().write().await.clear();
+        *cache_updating().write().await = false;
+    }
+
+    fn data_usage_info_for_test(bucket: &str, objects_count: u64, size: u64, last_update: SystemTime) -> DataUsageInfo {
+        let mut info = DataUsageInfo {
+            last_update: Some(last_update),
+            ..Default::default()
+        };
+        info.buckets_usage.insert(
+            bucket.to_string(),
+            BucketUsageInfo {
+                objects_count,
+                versions_count: objects_count,
+                size,
+                ..Default::default()
+            },
+        );
+        info.bucket_sizes.insert(bucket.to_string(), size);
+        info.buckets_count = info.buckets_usage.len() as u64;
+        info.calculate_totals();
+        info
+    }
 
     fn aggregate_for_test(
         inputs: Vec<(DiskUsageStatus, Result<Option<LocalUsageSnapshot>, Error>)>,
@@ -772,4 +929,101 @@ mod tests {
         assert_eq!(aggregated.buckets_count, 1);
         assert_eq!(aggregated.buckets_usage.get("bucket-a").map(|b| (b.objects_count, b.size)), Some((3, 42)));
     }
+
+    #[tokio::test]
+    #[serial]
+    async fn memory_overlay_reflects_recent_delete_before_scanner_persists() {
+        clear_usage_memory_cache_for_test().await;
+
+        let persisted = data_usage_info_for_test("bucket-a", 1, 42, SystemTime::now() - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&persisted).await;
+        record_bucket_object_delete_memory("bucket-a", 42, true).await;
+
+        let mut response = persisted.clone();
+        apply_bucket_usage_memory_overlay(&mut response).await;
+
+        assert_eq!(response.objects_total_count, 0);
+        assert_eq!(response.objects_total_size, 0);
+        assert_eq!(
+            response
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((0, 0))
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn memory_overlay_preserves_object_count_for_overwrite() {
+        clear_usage_memory_cache_for_test().await;
+
+        let persisted = data_usage_info_for_test("bucket-a", 1, 10, SystemTime::now() - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&persisted).await;
+        record_bucket_object_write_memory("bucket-a", Some(10), 20).await;
+
+        let mut response = persisted.clone();
+        apply_bucket_usage_memory_overlay(&mut response).await;
+
+        assert_eq!(response.objects_total_count, 1);
+        assert_eq!(response.objects_total_size, 20);
+        assert_eq!(
+            response
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((1, 20))
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn memory_overlay_does_not_replace_newer_persisted_usage() {
+        clear_usage_memory_cache_for_test().await;
+
+        let now = SystemTime::now();
+        let old_persisted = data_usage_info_for_test("bucket-a", 1, 42, now - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&old_persisted).await;
+        record_bucket_object_delete_memory("bucket-a", 42, true).await;
+
+        let mut newer_persisted = data_usage_info_for_test("bucket-a", 2, 84, now + Duration::from_secs(10));
+        apply_bucket_usage_memory_overlay(&mut newer_persisted).await;
+
+        assert_eq!(newer_persisted.objects_total_count, 2);
+        assert_eq!(newer_persisted.objects_total_size, 84);
+        assert_eq!(
+            newer_persisted
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((2, 84))
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn scanner_sync_preserves_newer_memory_update() {
+        clear_usage_memory_cache_for_test().await;
+
+        let now = SystemTime::now();
+        let old_persisted = data_usage_info_for_test("bucket-a", 1, 42, now - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&old_persisted).await;
+        record_bucket_object_delete_memory("bucket-a", 42, true).await;
+
+        let scanner_snapshot = data_usage_info_for_test("bucket-a", 1, 42, now - Duration::from_secs(5));
+        replace_bucket_usage_memory_from_info(&scanner_snapshot).await;
+
+        let mut response = scanner_snapshot.clone();
+        apply_bucket_usage_memory_overlay(&mut response).await;
+
+        assert_eq!(response.objects_total_count, 0);
+        assert_eq!(response.objects_total_size, 0);
+        assert_eq!(
+            response
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((0, 0))
+        );
+    }
 }

crates/ecstore/src/disk/disk_store.rs

@@ -27,6 +27,8 @@ use crate::global::GLOBAL_LOCAL_DISK_ID_MAP;
 use bytes::Bytes;
 use metrics::counter;
 use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
+#[cfg(not(test))]
+use std::sync::OnceLock;
 use std::{
     path::PathBuf,
     sync::{
@@ -50,10 +52,54 @@ enum TimeoutHealthAction {
     IgnoreFailure,
 }
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum TimeoutHealthPolicy {
+    MarkFailure,
+    IgnoreScanner,
+}
+
+impl TimeoutHealthPolicy {
+    fn parse(raw: &str) -> Option<Self> {
+        match raw.trim().to_ascii_lowercase().as_str() {
+            rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE => Some(Self::MarkFailure),
+            rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER => Some(Self::IgnoreScanner),
+            _ => None,
+        }
+    }
+
+    fn scanner_timeout_health_action(self) -> TimeoutHealthAction {
+        match self {
+            Self::MarkFailure => TimeoutHealthAction::MarkFailure,
+            Self::IgnoreScanner => TimeoutHealthAction::IgnoreFailure,
+        }
+    }
+}
+
 pub const ENV_RUSTFS_DRIVE_ACTIVE_MONITORING: &str = "RUSTFS_DRIVE_ACTIVE_MONITORING";
 pub const DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING: bool = true;
 pub const SKIP_IF_SUCCESS_BEFORE: Duration = Duration::from_secs(5);
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum DriveTimeoutProfile {
+    Default,
+    HighLatency,
+}
+
+impl DriveTimeoutProfile {
+    fn parse(raw: &str) -> Option<Self> {
+        match raw.trim().to_ascii_lowercase().as_str() {
+            rustfs_config::DRIVE_TIMEOUT_PROFILE_DEFAULT => Some(Self::Default),
+            rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY => Some(Self::HighLatency),
+            _ => None,
+        }
+    }
+}
+
+#[cfg(not(test))]
+static DRIVE_TIMEOUT_PROFILE_CACHE: OnceLock<DriveTimeoutProfile> = OnceLock::new();
+#[cfg(not(test))]
+static DRIVE_TIMEOUT_HEALTH_POLICY_CACHE: OnceLock<TimeoutHealthPolicy> = OnceLock::new();
+
 lazy_static::lazy_static! {
     static ref TEST_DATA: Bytes = Bytes::from(vec![42u8; 2048]);
     static ref TEST_BUCKET: String = ".rustfs.sys/tmp".to_string();
@@ -66,10 +112,39 @@ pub fn get_max_timeout_duration() -> Duration {
     ))
 }
 
-fn get_drive_timeout_duration(env_key: &str, default_secs: u64) -> Duration {
+fn resolve_drive_timeout_profile_from_env() -> DriveTimeoutProfile {
+    let raw = rustfs_utils::get_env_str(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, rustfs_config::DEFAULT_DRIVE_TIMEOUT_PROFILE);
+    if let Some(profile) = DriveTimeoutProfile::parse(&raw) {
+        return profile;
+    }
+    warn!(
+        env = rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE,
+        value = %raw,
+        default = rustfs_config::DEFAULT_DRIVE_TIMEOUT_PROFILE,
+        "Invalid drive timeout profile; falling back to default"
+    );
+    DriveTimeoutProfile::parse(rustfs_config::DEFAULT_DRIVE_TIMEOUT_PROFILE).unwrap_or(DriveTimeoutProfile::Default)
+}
+
+fn get_drive_timeout_profile() -> DriveTimeoutProfile {
+    #[cfg(test)]
+    {
+        resolve_drive_timeout_profile_from_env()
+    }
+    #[cfg(not(test))]
+    {
+        *DRIVE_TIMEOUT_PROFILE_CACHE.get_or_init(resolve_drive_timeout_profile_from_env)
+    }
+}
+
+fn get_drive_timeout_duration(env_key: &str, default_secs: u64, high_latency_secs: Option<u64>) -> Duration {
+    let fallback_default = match (get_drive_timeout_profile(), high_latency_secs) {
+        (DriveTimeoutProfile::HighLatency, Some(secs)) => secs,
+        _ => default_secs,
+    };
     Duration::from_secs(
         rustfs_utils::get_env_opt_u64_with_aliases(env_key, &[rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION])
-            .unwrap_or(default_secs),
+            .unwrap_or(fallback_default),
     )
 }
 
@@ -77,6 +152,7 @@ pub fn get_drive_metadata_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -84,6 +160,7 @@ pub fn get_drive_disk_info_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_DISK_INFO_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_DISK_INFO_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -91,6 +168,7 @@ pub fn get_drive_list_dir_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_LIST_DIR_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_LIST_DIR_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -98,6 +176,7 @@ pub fn get_drive_walkdir_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -105,6 +184,7 @@ pub fn get_drive_walkdir_stall_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -122,6 +202,34 @@ pub fn get_drive_active_check_timeout() -> Duration {
     ))
 }
 
+fn resolve_drive_timeout_health_policy_from_env() -> TimeoutHealthPolicy {
+    let raw = rustfs_utils::get_env_str(
+        rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+        rustfs_config::DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION,
+    );
+    if let Some(policy) = TimeoutHealthPolicy::parse(&raw) {
+        return policy;
+    }
+    warn!(
+        env = rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+        value = %raw,
+        default = rustfs_config::DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION,
+        "Invalid drive timeout health action policy; falling back to default"
+    );
+    TimeoutHealthPolicy::parse(rustfs_config::DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION).unwrap_or(TimeoutHealthPolicy::MarkFailure)
+}
+
+fn get_drive_timeout_health_policy() -> TimeoutHealthPolicy {
+    #[cfg(test)]
+    {
+        resolve_drive_timeout_health_policy_from_env()
+    }
+    #[cfg(not(test))]
+    {
+        *DRIVE_TIMEOUT_HEALTH_POLICY_CACHE.get_or_init(resolve_drive_timeout_health_policy_from_env)
+    }
+}
+
 /// DiskHealthTracker tracks the health status of a disk.
 /// Similar to Go's diskHealthTracker.
 #[derive(Debug)]
@@ -470,6 +578,8 @@ pub struct LocalDiskWrapper {
     cancel_token: CancellationToken,
     /// Disk ID for stale checking
     disk_id: Arc<RwLock<Option<Uuid>>>,
+    /// Timeout policy for scanner-sensitive operations, loaded once on wrapper initialization.
+    timeout_health_policy: TimeoutHealthPolicy,
 }
 
 impl LocalDiskWrapper {
@@ -486,6 +596,7 @@ impl LocalDiskWrapper {
             health_check: health_check && env_health_check,
             cancel_token: CancellationToken::new(),
             disk_id: Arc::new(RwLock::new(None)),
+            timeout_health_policy: get_drive_timeout_health_policy(),
         };
         record_drive_runtime_state(&wrapper.disk.endpoint(), RuntimeDriveHealthState::Online);
         wrapper
@@ -511,6 +622,10 @@ impl LocalDiskWrapper {
         self.health.record_capacity_probe(total, used, free);
     }
 
+    fn scanner_timeout_health_action(&self) -> TimeoutHealthAction {
+        self.timeout_health_policy.scanner_timeout_health_action()
+    }
+
     #[cfg(test)]
     pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
         self.health.force_runtime_state_for_test(state);
@@ -558,6 +673,7 @@ impl LocalDiskWrapper {
     /// Monitor disk writability periodically
     async fn monitor_disk_writable(disk: Arc<LocalDisk>, health: Arc<DiskHealthTracker>, cancel_token: CancellationToken) {
         let mut interval = time::interval(get_drive_active_check_interval());
+        let active_check_timeout = get_drive_active_check_timeout();
 
         loop {
             tokio::select! {
@@ -596,7 +712,7 @@ impl LocalDiskWrapper {
                         &test_obj,
                         &TEST_DATA,
                         true,
-                        get_drive_active_check_timeout(),
+                        active_check_timeout,
                     )
                     .await
                     .is_err()
@@ -691,6 +807,7 @@ impl LocalDiskWrapper {
     /// Monitor disk status and try to bring it back online
     async fn monitor_disk_status(disk: Arc<LocalDisk>, health: Arc<DiskHealthTracker>, cancel_token: CancellationToken) {
         let check_every = get_drive_returning_probe_interval();
+        let active_check_timeout = get_drive_active_check_timeout();
 
         let mut interval = time::interval(check_every);
 
@@ -711,7 +828,7 @@ impl LocalDiskWrapper {
                         &test_obj,
                         &TEST_DATA,
                         false,
-                        get_drive_active_check_timeout(),
+                        active_check_timeout,
                     )
                     .await
                     {
@@ -893,10 +1010,11 @@ impl LocalDiskWrapper {
 #[async_trait::async_trait]
 impl DiskAPI for LocalDiskWrapper {
     async fn read_metadata(&self, volume: &str, path: &str) -> Result<Bytes> {
-        self.track_disk_health_with_op(
+        self.track_disk_health_with_op_and_timeout_action(
             "read_metadata",
             || async { self.disk.read_metadata(volume, path).await },
             get_drive_metadata_timeout(),
+            self.scanner_timeout_health_action(),
         )
         .await
     }
@@ -973,7 +1091,7 @@ impl DiskAPI for LocalDiskWrapper {
             return Err(DiskError::FaultyDisk);
         }
 
-        self.track_disk_health_with_op(
+        self.track_disk_health_with_op_and_timeout_action(
             "disk_info",
             || async {
                 let result = self.disk.disk_info(opts).await?;
@@ -987,6 +1105,7 @@ impl DiskAPI for LocalDiskWrapper {
                 Ok(result)
             },
             get_drive_disk_info_timeout(),
+            self.scanner_timeout_health_action(),
         )
         .await
     }
@@ -1021,6 +1140,7 @@ impl DiskAPI for LocalDiskWrapper {
             "walk_dir",
             || async { self.disk.walk_dir(opts, wr).await },
             get_drive_walkdir_timeout(),
+            // Listing/scanner backpressure should fail only the current walk, not poison drive health.
             TimeoutHealthAction::IgnoreFailure,
         )
         .await
@@ -1122,7 +1242,8 @@ impl DiskAPI for LocalDiskWrapper {
         dst_volume: &str,
         dst_path: &str,
     ) -> Result<RenameDataResp> {
-        self.track_disk_health(
+        self.track_disk_health_with_op(
+            "rename_data",
             || async { self.disk.rename_data(src_volume, src_path, fi, dst_volume, dst_path).await },
             get_max_timeout_duration(),
         )
@@ -1130,10 +1251,11 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn list_dir(&self, origvolume: &str, volume: &str, dir_path: &str, count: i32) -> Result<Vec<String>> {
-        self.track_disk_health_with_op(
+        self.track_disk_health_with_op_and_timeout_action(
             "list_dir",
             || async { self.disk.list_dir(origvolume, volume, dir_path, count).await },
             get_drive_list_dir_timeout(),
+            self.scanner_timeout_health_action(),
         )
         .await
     }
@@ -1256,14 +1378,48 @@ mod tests {
     fn drive_metadata_timeout_uses_default_when_unset() {
         temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
             temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
-                assert_eq!(
-                    get_drive_metadata_timeout(),
-                    Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS)
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, || {
+                    assert_eq!(
+                        get_drive_metadata_timeout(),
+                        Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS)
+                    );
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_metadata_timeout_uses_high_latency_profile_when_unset() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                temp_env::with_var(
+                    rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE,
+                    Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY),
+                    || {
+                        assert_eq!(
+                            get_drive_metadata_timeout(),
+                            Duration::from_secs(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS)
+                        );
+                    },
                 );
             });
         });
     }
 
+    #[test]
+    fn drive_metadata_timeout_invalid_profile_falls_back_to_default() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                temp_env::with_var(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, Some("invalid"), || {
+                    assert_eq!(
+                        get_drive_metadata_timeout(),
+                        Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS)
+                    );
+                });
+            });
+        });
+    }
+
     #[test]
     fn drive_metadata_timeout_uses_legacy_fallback_when_canonical_unset() {
         temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
@@ -1398,6 +1554,58 @@ mod tests {
 
     #[tokio::test]
     async fn walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure() {
+        temp_env::async_with_vars(
+            [
+                (rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1")),
+                (
+                    rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+                    Some(rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER),
+                ),
+            ],
+            async {
+                let dir = tempfile::tempdir().expect("temp dir should be created");
+                let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8"))
+                    .expect("endpoint should parse");
+                let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+                let wrapper = LocalDiskWrapper::new(disk, false);
+                let bucket = "test-bucket";
+                let object = "test-object";
+
+                wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+                let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+                file_info.volume = bucket.to_string();
+                file_info.name = object.to_string();
+                file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+                file_info.erasure.index = 1;
+
+                wrapper
+                    .write_metadata("", bucket, object, file_info)
+                    .await
+                    .expect("object metadata should be written");
+
+                let mut writer = PendingWriter;
+                let result = wrapper
+                    .walk_dir(
+                        WalkDirOptions {
+                            bucket: bucket.to_string(),
+                            recursive: true,
+                            ..Default::default()
+                        },
+                        &mut writer,
+                    )
+                    .await;
+
+                assert_eq!(result.expect_err("walk_dir should time out"), DiskError::Timeout);
+                assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+                assert!(!wrapper.health.is_faulty());
+            },
+        )
+        .await;
+    }
+
+    #[tokio::test]
+    async fn walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure_by_default() {
         temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
             let dir = tempfile::tempdir().expect("temp dir should be created");
             let endpoint =
@@ -1439,6 +1647,58 @@ mod tests {
         .await;
     }
 
+    #[tokio::test]
+    async fn walk_dir_timeout_does_not_break_followup_stat_volume() {
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+            let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+            let wrapper = LocalDiskWrapper::new(disk, false);
+            let bucket = "test-bucket";
+            let object = "test-object";
+
+            wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+            let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+            file_info.volume = bucket.to_string();
+            file_info.name = object.to_string();
+            file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+            file_info.erasure.index = 1;
+
+            wrapper
+                .write_metadata("", bucket, object, file_info)
+                .await
+                .expect("object metadata should be written");
+
+            let mut writer = PendingWriter;
+            let walk_err = wrapper
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await
+                .expect_err("walk_dir should time out");
+
+            assert_eq!(walk_err, DiskError::Timeout);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+
+            let info = wrapper
+                .stat_volume(bucket)
+                .await
+                .expect("follow-up bucket stat should still succeed after walk timeout");
+            assert_eq!(info.name, bucket);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+        })
+        .await;
+    }
+
     #[tokio::test]
     async fn default_timeout_marks_drive_failure() {
         let dir = tempfile::tempdir().expect("temp dir should be created");
@@ -1462,6 +1722,40 @@ mod tests {
         assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Suspect);
     }
 
+    #[test]
+    #[serial_test::serial]
+    fn drive_timeout_health_policy_defaults_to_mark_failure() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION, || {
+            let policy = get_drive_timeout_health_policy();
+            assert_eq!(policy, TimeoutHealthPolicy::MarkFailure);
+            assert_eq!(policy.scanner_timeout_health_action(), TimeoutHealthAction::MarkFailure);
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn drive_timeout_health_policy_respects_ignore_scanner() {
+        temp_env::with_var(
+            rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+            Some(rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER),
+            || {
+                let policy = get_drive_timeout_health_policy();
+                assert_eq!(policy, TimeoutHealthPolicy::IgnoreScanner);
+                assert_eq!(policy.scanner_timeout_health_action(), TimeoutHealthAction::IgnoreFailure);
+            },
+        );
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn drive_timeout_health_policy_invalid_value_falls_back_to_default() {
+        temp_env::with_var(rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION, Some("invalid"), || {
+            let policy = get_drive_timeout_health_policy();
+            assert_eq!(policy, TimeoutHealthPolicy::MarkFailure);
+            assert_eq!(policy.scanner_timeout_health_action(), TimeoutHealthAction::MarkFailure);
+        });
+    }
+
     #[test]
     fn reset_for_store_init_retry_clears_faulty_and_back_online() {
         let endpoint = Endpoint::try_from("/tmp/reset-store-init-retry").expect("endpoint should parse");

crates/ecstore/src/disk/endpoint.rs

@@ -19,6 +19,23 @@ use std::{fmt::Display, path::Path};
 use tracing::debug;
 use url::{ParseError, Url};
 
+#[cfg(windows)]
+pub(crate) fn windows_fallback_local_path(
+    path: &str,
+    canonicalize_error: &std::io::Error,
+    context: &'static str,
+) -> std::io::Result<std::path::PathBuf> {
+    let absolute = Path::new(path).absolutize()?.to_path_buf();
+    tracing::warn!(
+        path = %path,
+        canonicalize_error = %canonicalize_error,
+        resolved = ?absolute,
+        context = context,
+        "using windows fallback path resolution for local endpoint"
+    );
+    Ok(absolute)
+}
+
 /// enum for endpoint type.
 #[derive(PartialEq, Eq, Debug)]
 pub enum EndpointType {

crates/ecstore/src/disk/fs.rs

@@ -161,33 +161,39 @@ pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
     fs::create_dir_all(path.as_ref()).await
 }
 
+fn is_dir_error(e: &io::Error) -> bool {
+    e.raw_os_error() == Some(libc::EISDIR)
+        || e.kind() == io::ErrorKind::IsADirectory
+        // macOS: remove_file on a directory returns EPERM
+        || (cfg!(target_os = "macos") && e.raw_os_error() == Some(libc::EPERM))
+}
+
 #[tracing::instrument(level = "debug", skip_all)]
 pub async fn remove(path: impl AsRef<Path>) -> io::Result<()> {
-    let meta = fs::metadata(path.as_ref()).await?;
-    if meta.is_dir() {
-        fs::remove_dir(path.as_ref()).await
-    } else {
-        fs::remove_file(path.as_ref()).await
+    // Try remove_file first; fall back to remove_dir if it's a directory
+    match fs::remove_file(path.as_ref()).await {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => fs::remove_dir(path.as_ref()).await,
+        Err(e) => Err(e),
     }
 }
 
 pub async fn remove_all(path: impl AsRef<Path>) -> io::Result<()> {
-    let meta = fs::metadata(path.as_ref()).await?;
-    if meta.is_dir() {
-        fs::remove_dir_all(path.as_ref()).await
-    } else {
-        fs::remove_file(path.as_ref()).await
+    // Try remove_file first; fall back to remove_dir_all if it's a directory
+    match fs::remove_file(path.as_ref()).await {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => fs::remove_dir_all(path.as_ref()).await,
+        Err(e) => Err(e),
     }
 }
 
 #[tracing::instrument(level = "debug", skip_all)]
 pub fn remove_std(path: impl AsRef<Path>) -> io::Result<()> {
-    let path = path.as_ref();
-    let meta = std::fs::metadata(path)?;
-    if meta.is_dir() {
-        std::fs::remove_dir(path)
-    } else {
-        std::fs::remove_file(path)
+    // Try remove_file first; fall back to remove_dir if it's a directory
+    match std::fs::remove_file(path.as_ref()) {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => std::fs::remove_dir(path.as_ref()),
+        Err(e) => Err(e),
     }
 }
 

crates/ecstore/src/disk/local.rs

@@ -26,7 +26,7 @@ use crate::disk::{
     format::FormatV3,
     fs::{O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, lstat, lstat_std, remove, remove_all_std, remove_std, rename},
     os,
-    os::{check_path_length, is_empty_dir, is_root_disk, rename_all},
+    os::{check_path_length, is_empty_dir, is_root_disk, rename_all, rename_all_ignore_missing_source},
 };
 use crate::erasure_coding::bitrot_verify;
 use crate::global::{GLOBAL_IsErasureSD, GLOBAL_RootDiskThreshold};
@@ -57,14 +57,15 @@ use std::{
 use time::OffsetDateTime;
 use tokio::fs::{self, File};
 use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWrite, AsyncWriteExt, ErrorKind};
-use tokio::sync::RwLock;
-use tokio::time::interval;
+use tokio::sync::{Notify, RwLock};
+use tokio::time::{Instant, interval_at, timeout};
 use tracing::{debug, error, info, warn};
 use uuid::Uuid;
 
 const DELETED_OBJECTS_CLEANUP_INTERVAL: Duration = Duration::from_secs(60 * 5);
 const STALE_TMP_OBJECT_EXPIRY: Duration = Duration::from_secs(24 * 60 * 60);
 const RUSTFS_META_TMP_OLD_BUCKET: &str = ".rustfs.sys/tmp-old";
+const STARTUP_CLEANUP_WAIT_TIMEOUT: Duration = Duration::from_secs(2);
 
 #[derive(Debug, Clone)]
 pub struct FormatInfo {
@@ -331,6 +332,8 @@ pub struct LocalDisk {
     // pub format_data: Mutex<Vec<u8>>,
     // pub format_file_info: Mutex<Option<Metadata>>,
     // pub format_last_check: Mutex<Option<OffsetDateTime>>,
+    startup_cleanup_ready: Arc<AtomicU32>,
+    startup_cleanup_notify: Arc<Notify>,
     exit_signal: Option<tokio::sync::broadcast::Sender<()>>,
 }
 
@@ -353,24 +356,72 @@ impl Debug for LocalDisk {
     }
 }
 
+/// Resolve the local disk root path from an endpoint path.
+///
+/// Tries `canonicalize` first (fast path). On Windows, if canonicalization reports
+/// `NotFound` for paths that may still be valid mount roots, falls back to
+/// `absolutize` + metadata check to accept valid local directory roots that
+/// don't support full canonicalization.
+fn resolve_local_disk_root(ep_path: &str) -> Result<PathBuf> {
+    match rustfs_utils::canonicalize(ep_path) {
+        Ok(path) => Ok(path),
+        Err(err) => {
+            if err.kind() != ErrorKind::NotFound {
+                return Err(to_file_error(err).into());
+            }
+
+            #[cfg(windows)]
+            {
+                // On Windows, canonicalize can fail for ZFS volumes, junction points,
+                // subst drives, and other non-standard filesystem mounts. Try a fallback
+                // path resolution using absolutize + metadata check.
+                let absolute = match crate::disk::endpoint::windows_fallback_local_path(ep_path, &err, "local disk root") {
+                    Ok(path) => path,
+                    Err(_) => {
+                        return Err(DiskError::VolumeNotFound);
+                    }
+                };
+
+                match std::fs::metadata(&absolute) {
+                    Ok(metadata) => {
+                        if !metadata.is_dir() {
+                            return Err(DiskError::DiskNotDir);
+                        }
+                        return Ok(absolute);
+                    }
+                    Err(meta_err) => {
+                        if meta_err.kind() == ErrorKind::NotFound {
+                            return Err(DiskError::VolumeNotFound);
+                        }
+                        return Err(to_file_error(meta_err).into());
+                    }
+                }
+            }
+
+            #[cfg(not(windows))]
+            {
+                Err(DiskError::VolumeNotFound)
+            }
+        }
+    }
+}
+
 impl LocalDisk {
     pub async fn new(ep: &Endpoint, cleanup: bool) -> Result<Self> {
         debug!("Creating local disk");
-        // Use optimized path resolution instead of absolutize() for better performance
-        // Use dunce::canonicalize instead of std::fs::canonicalize to avoid UNC paths on Windows
-        let root = match rustfs_utils::canonicalize(ep.get_file_path()) {
-            Ok(path) => path,
-            Err(e) => {
-                if e.kind() == ErrorKind::NotFound {
-                    return Err(DiskError::VolumeNotFound);
-                }
-                return Err(to_file_error(e).into());
-            }
-        };
+        let root = resolve_local_disk_root(&ep.get_file_path())?;
 
         ensure_data_usage_layout(&root).await.map_err(DiskError::from)?;
 
-        if cleanup && let Err(err) = Self::cleanup_tmp_on_startup(&root).await {
+        let startup_cleanup_ready = Arc::new(AtomicU32::new(u32::from(!cleanup)));
+        let startup_cleanup_notify = Arc::new(Notify::new());
+
+        if cleanup
+            && let Err(err) =
+                Self::cleanup_tmp_on_startup(&root, startup_cleanup_ready.clone(), startup_cleanup_notify.clone()).await
+        {
+            startup_cleanup_ready.store(1, Ordering::Release);
+            startup_cleanup_notify.notify_waiters();
             warn!(root = ?root, error = ?err, "failed to cleanup temporary data during disk startup");
         }
 
@@ -466,6 +517,8 @@ impl LocalDisk {
             // format_last_check: Mutex::new(format_last_check),
             path_cache: Arc::new(ParkingLotRwLock::new(HashMap::with_capacity(2048))),
             current_dir: Arc::new(OnceLock::new()),
+            startup_cleanup_ready,
+            startup_cleanup_notify,
             exit_signal: None,
         };
         let (info, _root) = get_disk_info(root).await?;
@@ -497,7 +550,8 @@ impl LocalDisk {
     }
 
     async fn cleanup_deleted_objects_loop(root: PathBuf, mut exit_rx: tokio::sync::broadcast::Receiver<()>) {
-        let mut interval = interval(DELETED_OBJECTS_CLEANUP_INTERVAL);
+        let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);
         loop {
             tokio::select! {
                 _ = interval.tick() => {
@@ -525,12 +579,18 @@ impl LocalDisk {
         root.join(meta_path)
     }
 
-    async fn cleanup_tmp_on_startup(root: &Path) -> Result<()> {
+    async fn cleanup_tmp_on_startup(
+        root: &Path,
+        startup_cleanup_ready: Arc<AtomicU32>,
+        startup_cleanup_notify: Arc<Notify>,
+    ) -> Result<()> {
         let tmp_path = Self::meta_path(root, RUSTFS_META_TMP_BUCKET);
         let tmp_old_path = Self::meta_path(root, RUSTFS_META_TMP_OLD_BUCKET).join(Uuid::new_v4().to_string());
 
         rename_all(&tmp_path, &tmp_old_path, root).await?;
 
+        tokio::fs::create_dir_all(Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET)).await?;
+
         let tmp_old_root = Self::meta_path(root, RUSTFS_META_TMP_OLD_BUCKET);
         tokio::spawn(async move {
             if let Err(err) = tokio::fs::remove_dir_all(&tmp_old_root).await
@@ -538,12 +598,35 @@ impl LocalDisk {
             {
                 warn!(path = ?tmp_old_root, error = ?err, "failed to remove old temporary data");
             }
+            startup_cleanup_ready.store(1, Ordering::Release);
+            startup_cleanup_notify.notify_waiters();
         });
 
-        tokio::fs::create_dir_all(Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET)).await?;
         Ok(())
     }
 
+    async fn wait_for_startup_cleanup(&self) {
+        if self.startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+            return;
+        }
+
+        if wait_for_startup_cleanup_signal(
+            self.startup_cleanup_ready.as_ref(),
+            self.startup_cleanup_notify.as_ref(),
+            STARTUP_CLEANUP_WAIT_TIMEOUT,
+        )
+        .await
+        {
+            debug!(disk = %self.endpoint, "startup cleanup barrier released before walk_dir");
+        } else {
+            warn!(
+                disk = %self.endpoint,
+                timeout_ms = STARTUP_CLEANUP_WAIT_TIMEOUT.as_millis(),
+                "startup cleanup barrier timed out; continuing walk_dir"
+            );
+        }
+    }
+
     async fn cleanup_stale_tmp_objects(root: PathBuf) -> Result<()> {
         Self::cleanup_stale_tmp_objects_with_expiry(root, STALE_TMP_OBJECT_EXPIRY).await
     }
@@ -846,19 +929,20 @@ impl LocalDisk {
         // }
 
         let err = if recursive {
-            rename_all(delete_path, trash_path, self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?)
+            rename_all_ignore_missing_source(delete_path, trash_path, self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?)
                 .await
                 .err()
         } else {
-            rename(&delete_path, &trash_path)
-                .await
-                .map_err(|e| to_file_error(e).into())
-                .err()
+            match rename(&delete_path, &trash_path).await {
+                Ok(()) => None,
+                Err(err) if err.kind() == ErrorKind::NotFound => None,
+                Err(err) => Some(to_file_error(err).into()),
+            }
         };
 
         if immediate_purge || delete_path.to_string_lossy().ends_with(SLASH_SEPARATOR) {
             let trash_path2 = self.get_object_path(RUSTFS_META_TMP_DELETED_BUCKET, Uuid::new_v4().to_string().as_str())?;
-            let _ = rename_all(
+            let _ = rename_all_ignore_missing_source(
                 encode_dir_object(delete_path.to_string_lossy().as_ref()),
                 trash_path2,
                 self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?,
@@ -1163,26 +1247,45 @@ impl LocalDisk {
     }
     // write_all_internal do write file
     async fn write_all_internal(&self, file_path: &Path, data: InternalBuf<'_>, sync: bool, skip_parent: &Path) -> Result<()> {
-        let flags = O_CREATE | O_WRONLY | O_TRUNC;
-
-        let mut f = {
-            if sync {
-                // TODO: support sync
-                self.open_file(file_path, flags, skip_parent).await?
-            } else {
-                self.open_file(file_path, flags, skip_parent).await?
-            }
+        let skip_parent = if skip_parent.as_os_str().is_empty() {
+            self.root.as_path()
+        } else {
+            skip_parent
         };
 
         match data {
             InternalBuf::Ref(buf) => {
+                let mut f = self.open_file(file_path, O_CREATE | O_WRONLY | O_TRUNC, skip_parent).await?;
                 f.write_all(buf).await.map_err(to_file_error)?;
             }
             InternalBuf::Owned(buf) => {
-                f.write_all(buf.as_ref()).await.map_err(to_file_error)?;
+                let path = file_path.to_path_buf();
+                if let Some(parent) = path.parent()
+                    && parent != skip_parent
+                {
+                    os::make_dir_all(parent, skip_parent).await?;
+                }
+
+                tokio::task::spawn_blocking(move || {
+                    let mut f = std::fs::OpenOptions::new()
+                        .create(true)
+                        .write(true)
+                        .truncate(true)
+                        .open(path)
+                        .map_err(to_file_error)?;
+
+                    std::io::Write::write_all(&mut f, buf.as_ref()).map_err(to_file_error)?;
+
+                    Ok::<(), std::io::Error>(())
+                })
+                .await
+                .map_err(DiskError::from)??;
             }
         }
 
+        // Keep existing durability contract: this path intentionally ignores `sync`.
+        let _ = sync;
+
         Ok(())
     }
 
@@ -1993,11 +2096,7 @@ impl DiskAPI for LocalDisk {
             let meta_op = match lstat_std(&src_file_path).map_err(|e| to_file_error(e).into()) {
                 Ok(meta) => Some(meta),
                 Err(e) => {
-                    if e != DiskError::FileNotFound {
-                        return Err(e);
-                    }
-
-                    None
+                    return Err(e);
                 }
             };
 
@@ -2009,10 +2108,22 @@ impl DiskAPI for LocalDisk {
             }
 
             remove_std(&dst_file_path).map_err(to_file_error)?;
+        } else {
+            let meta = lstat_std(&src_file_path).map_err(|e| -> DiskError { to_file_error(e).into() })?;
+            if meta.is_dir() {
+                warn!("rename_part src is dir {:?}", &src_file_path);
+                return Err(DiskError::FileAccessDenied);
+            }
         }
 
         rename_all(&src_file_path, &dst_file_path, &dst_volume_dir).await?;
 
+        let dst_meta = lstat_std(&dst_file_path).map_err(|e| -> DiskError { to_file_error(e).into() })?;
+        if src_is_dir != dst_meta.is_dir() {
+            warn!("rename_part dst type changed after rename {:?}", &dst_file_path);
+            return Err(DiskError::FileAccessDenied);
+        }
+
         self.write_all(dst_volume, format!("{dst_path}.meta").as_str(), meta).await?;
 
         if let Some(parent) = src_file_path.parent() {
@@ -2348,6 +2459,8 @@ impl DiskAPI for LocalDisk {
     // FIXME: TODO: io.writer TODO cancel
     #[tracing::instrument(level = "debug", skip(self, wr))]
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
+        self.wait_for_startup_cleanup().await;
+
         let volume_dir = self.get_bucket_path(&opts.bucket)?;
 
         if !skip_access_checks(&opts.bucket)
@@ -2485,75 +2598,57 @@ impl DiskAPI for LocalDisk {
         check_path_length(src_file_path.to_string_lossy().to_string().as_str())?;
         check_path_length(dst_file_path.to_string_lossy().to_string().as_str())?;
 
-        // Read the previous xl.meta
-
-        let has_dst_buf = match super::fs::read_file(&dst_file_path).await {
-            Ok(res) => Some(res),
-            Err(e) => {
-                let e: DiskError = to_file_error(e).into();
-
-                if e != DiskError::FileNotFound {
-                    return Err(e);
-                }
-
-                None
-            }
-        };
-
-        let mut xlmeta = FileMeta::new();
-
-        if let Some(dst_buf) = has_dst_buf.as_ref()
-            && FileMeta::is_xl2_v1_format(dst_buf)
-            && let Ok(nmeta) = FileMeta::load(dst_buf)
-        {
-            xlmeta = nmeta
-        }
-
-        let mut skip_parent = dst_volume_dir.clone();
-        if has_dst_buf.as_ref().is_some()
-            && let Some(parent) = dst_file_path.parent()
-        {
-            skip_parent = parent.to_path_buf();
-        }
-
-        // TODO: Healing
-
-        let version_id = fi.version_id.unwrap_or_default();
-        let search_version_id = Some(version_id);
         let no_inline = fi.data.is_none() && fi.size > 0;
 
-        // Check if there's an existing version with the same version_id that has a data_dir to clean up
-        // Reuse one metadata scan to find the version data_dir and determine whether it is shared.
-        let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(search_version_id);
-        if let Some(old_data_dir) = has_old_data_dir.as_ref() {
-            let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
-        }
-
-        xlmeta.add_version(fi)?;
-
-        if xlmeta.versions.len() <= 10 {
-            // TODO: Sign
-        }
-
-        if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
-            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
-            let meta_skip_parent = if no_inline {
-                src_file_parent
-            } else {
-                src_volume_dir.as_path()
+        if no_inline {
+            // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
+            let has_dst_buf = match super::fs::read_file(&dst_file_path).await {
+                Ok(res) => Some(res),
+                Err(e) => {
+                    let e: DiskError = to_file_error(e).into();
+                    if e != DiskError::FileNotFound {
+                        return Err(e);
+                    }
+                    None
+                }
             };
+
+            let mut xlmeta = FileMeta::new();
+            if let Some(dst_buf) = has_dst_buf.as_ref()
+                && FileMeta::is_xl2_v1_format(dst_buf)
+                && let Ok(nmeta) = FileMeta::load(dst_buf)
+            {
+                xlmeta = nmeta
+            }
+
+            let mut skip_parent = dst_volume_dir.clone();
+            if has_dst_buf.as_ref().is_some()
+                && let Some(parent) = dst_file_path.parent()
+            {
+                skip_parent = parent.to_path_buf();
+            }
+
+            let version_id = fi.version_id.unwrap_or_default();
+            let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
+            if let Some(old_data_dir) = has_old_data_dir.as_ref() {
+                let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
+            }
+            xlmeta.add_version(fi)?;
             let new_dst_buf = xlmeta.marshal_msg()?;
 
+            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
             self.write_all_private(
                 src_volume,
-                format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(),
+                &format!("{}/{}", &src_path, STORAGE_FORMAT_FILE),
                 new_dst_buf.into(),
                 true,
-                meta_skip_parent,
+                src_file_parent,
             )
             .await?;
 
-            if no_inline && let Err(err) = rename_all(&src_data_path, &dst_data_path, &skip_parent).await {
+            if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref()
+                && let Err(err) = rename_all(src_data_path, dst_data_path, &skip_parent).await
+            {
                 let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
                 info!(
                     "rename all failed src_data_path: {:?}, dst_data_path: {:?}, err: {:?}",
@@ -2561,19 +2656,21 @@ impl DiskAPI for LocalDisk {
                 );
                 return Err(err);
             }
-        } else {
-            let new_dst_buf = xlmeta.marshal_msg()?;
-            self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
-                .await?;
-        }
 
-        if let Some(old_data_dir) = has_old_data_dir {
-            // preserve current xl.meta inside the oldDataDir.
-            if let Some(dst_buf) = has_dst_buf
+            if let Err(err) = rename_all(&src_file_path, &dst_file_path, &skip_parent).await {
+                if let Some((_, dst_data_path)) = has_data_dir_path.as_ref() {
+                    let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
+                }
+                info!("rename all failed err: {:?}", err);
+                return Err(err);
+            }
+
+            if let Some(old_data_dir) = has_old_data_dir
+                && let Some(dst_buf) = has_dst_buf
                 && let Err(err) = self
                     .write_all_private(
                         dst_volume,
-                        format!("{}/{}/{}", &dst_path, &old_data_dir.to_string(), STORAGE_FORMAT_FILE).as_str(),
+                        &format!("{}/{}/{}", &dst_path, &old_data_dir.to_string(), STORAGE_FORMAT_FILE),
                         dst_buf.into(),
                         true,
                         &skip_parent,
@@ -2583,30 +2680,106 @@ impl DiskAPI for LocalDisk {
                 info!("write_all_private failed err: {:?}", err);
                 return Err(err);
             }
-        }
 
-        if let Err(err) = rename_all(&src_file_path, &dst_file_path, &skip_parent).await {
-            if let Some((_, dst_data_path)) = has_data_dir_path.as_ref() {
-                let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
+            if let Some(src_file_path_parent) = src_file_path.parent() {
+                if src_volume != super::RUSTFS_META_MULTIPART_BUCKET {
+                    let _ = remove_std(src_file_path_parent);
+                } else {
+                    let _ = self
+                        .delete_file(&dst_volume_dir, &src_file_path_parent.to_path_buf(), true, false)
+                        .await;
+                }
             }
-            info!("rename all failed err: {:?}", err);
-            return Err(err);
-        }
 
-        if let Some(src_file_path_parent) = src_file_path.parent() {
-            if src_volume != super::RUSTFS_META_MULTIPART_BUCKET {
-                let _ = remove_std(src_file_path_parent);
+            Ok(RenameDataResp {
+                old_data_dir: has_old_data_dir,
+                sign: None,
+            })
+        } else {
+            // Inline: merge read + parse + write + rename into single spawn_blocking
+            let src = src_file_path.clone();
+            let dst = dst_file_path.clone();
+            let cleanup_path = if src_volume == super::RUSTFS_META_MULTIPART_BUCKET {
+                src_file_path.parent().map(|p| p.to_path_buf())
             } else {
-                let _ = self
-                    .delete_file(&dst_volume_dir, &src_file_path_parent.to_path_buf(), true, false)
-                    .await;
-            }
-        }
+                None
+            };
 
-        Ok(RenameDataResp {
-            old_data_dir: has_old_data_dir,
-            sign: None, // TODO:
-        })
+            let (old_data_dir, _dst_buf) = tokio::task::spawn_blocking(move || {
+                // Read existing xl.meta
+                let has_dst_buf = match std::fs::read(&dst) {
+                    Ok(buf) => Some(Bytes::from(buf)),
+                    Err(e) if e.kind() == std::io::ErrorKind::NotFound => None,
+                    Err(e) => return Err(to_file_error(e)),
+                };
+
+                let mut xlmeta = FileMeta::new();
+                if let Some(ref buf) = has_dst_buf
+                    && FileMeta::is_xl2_v1_format(buf)
+                    && let Ok(nmeta) = FileMeta::load(buf)
+                {
+                    xlmeta = nmeta
+                }
+
+                let version_id = fi.version_id.unwrap_or_default();
+                let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
+                if let Some(d) = old_data_dir.as_ref() {
+                    let _ = xlmeta.data.remove_two(version_id, *d);
+                }
+                xlmeta.add_version(fi)?;
+                let new_buf = xlmeta.marshal_msg()?;
+
+                // Write new xl.meta + rename
+                if let Some(parent) = src.parent() {
+                    std::fs::create_dir_all(parent)?;
+                }
+                let mut f = std::fs::OpenOptions::new()
+                    .create(true)
+                    .write(true)
+                    .truncate(true)
+                    .open(&src)?;
+                std::io::Write::write_all(&mut f, &new_buf)?;
+                match std::fs::rename(&src, &dst) {
+                    Ok(()) => Ok(()),
+                    Err(err) if err.kind() == std::io::ErrorKind::NotFound && !src.exists() => Ok(()),
+                    Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+                        if let Some(parent) = dst.parent() {
+                            std::fs::create_dir_all(parent)?;
+                        }
+                        std::fs::rename(&src, &dst).map_err(to_file_error)?;
+                        Ok(())
+                    }
+                    Err(err) => Err(to_file_error(err)),
+                }?;
+
+                if let Some(old_dir) = old_data_dir.as_ref()
+                    && let Some(ref buf) = has_dst_buf
+                    && let Some(dst_parent) = dst.parent()
+                {
+                    let old_path = dst_parent.join(old_dir.to_string()).join(STORAGE_FORMAT_FILE);
+                    if let Some(old_parent) = old_path.parent() {
+                        std::fs::create_dir_all(old_parent)?;
+                    }
+                    std::fs::write(&old_path, buf).map_err(to_file_error)?;
+                }
+
+                Ok::<(Option<uuid::Uuid>, Option<Bytes>), std::io::Error>((old_data_dir, has_dst_buf))
+            })
+            .await
+            .map_err(DiskError::from)??;
+
+            // Cleanup
+            if let Some(ref cleanup) = cleanup_path {
+                let _ = self.delete_file(&dst_volume_dir, cleanup, true, false).await;
+            } else if let Some(parent) = src_file_path.parent() {
+                let _ = remove_std(parent);
+            }
+
+            Ok(RenameDataResp {
+                old_data_dir,
+                sign: None,
+            })
+        }
     }
 
     #[tracing::instrument(skip(self))]
@@ -3078,6 +3251,31 @@ impl DiskAPI for LocalDisk {
     }
 }
 
+async fn wait_for_startup_cleanup_signal(
+    startup_cleanup_ready: &AtomicU32,
+    startup_cleanup_notify: &Notify,
+    wait_timeout: Duration,
+) -> bool {
+    if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+        return true;
+    }
+
+    timeout(wait_timeout, async {
+        loop {
+            if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+                return;
+            }
+            let notified = startup_cleanup_notify.notified();
+            if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+                return;
+            }
+            notified.await;
+        }
+    })
+    .await
+    .is_ok()
+}
+
 #[tracing::instrument]
 async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInfo, bool)> {
     let drive_path = drive_path.to_string_lossy().to_string();
@@ -3130,7 +3328,9 @@ mod test {
         fs::create_dir_all(leftover.parent().unwrap()).await.unwrap();
         fs::write(&leftover, b"temporary").await.unwrap();
 
-        LocalDisk::cleanup_tmp_on_startup(dir.path()).await.unwrap();
+        LocalDisk::cleanup_tmp_on_startup(dir.path(), Arc::new(AtomicU32::new(0)), Arc::new(Notify::new()))
+            .await
+            .unwrap();
 
         assert!(!tmp.join("leftover").exists());
         assert!(LocalDisk::meta_path(dir.path(), RUSTFS_META_TMP_DELETED_BUCKET).exists());
@@ -3186,6 +3386,54 @@ mod test {
         assert!(entries.next_entry().await.unwrap().is_none());
     }
 
+    #[tokio::test(start_paused = true)]
+    async fn cleanup_loop_interval_does_not_tick_immediately() {
+        let start_at = tokio::time::Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);
+
+        assert!(tokio::time::timeout(Duration::from_secs(1), interval.tick()).await.is_err());
+
+        tokio::time::advance(DELETED_OBJECTS_CLEANUP_INTERVAL).await;
+        interval.tick().await;
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn startup_cleanup_barrier_waits_for_notification() {
+        let ready = Arc::new(AtomicU32::new(0));
+        let notify = Arc::new(Notify::new());
+
+        let wait = tokio::spawn({
+            let ready = ready.clone();
+            let notify = notify.clone();
+            async move { wait_for_startup_cleanup_signal(ready.as_ref(), notify.as_ref(), Duration::from_secs(2)).await }
+        });
+
+        tokio::task::yield_now().await;
+        assert!(!wait.is_finished());
+
+        ready.store(1, Ordering::Release);
+        notify.notify_waiters();
+
+        assert!(wait.await.unwrap());
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn startup_cleanup_barrier_times_out() {
+        let ready = Arc::new(AtomicU32::new(0));
+        let notify = Arc::new(Notify::new());
+
+        let wait = tokio::spawn({
+            let ready = ready.clone();
+            let notify = notify.clone();
+            async move { wait_for_startup_cleanup_signal(ready.as_ref(), notify.as_ref(), Duration::from_secs(2)).await }
+        });
+
+        tokio::task::yield_now().await;
+        tokio::time::advance(Duration::from_secs(2)).await;
+
+        assert!(!wait.await.unwrap());
+    }
+
     #[tokio::test]
     async fn test_scan_dir_includes_nested_object_dirs() {
         use rustfs_filemeta::MetacacheReader;

crates/ecstore/src/disk/os.rs

@@ -140,36 +140,55 @@ pub async fn rename_all(
     Ok(())
 }
 
+#[tracing::instrument(level = "debug", skip_all)]
+pub async fn rename_all_ignore_missing_source(
+    src_file_path: impl AsRef<Path>,
+    dst_file_path: impl AsRef<Path>,
+    base_dir: impl AsRef<Path>,
+) -> Result<()> {
+    match reliable_rename_inner(src_file_path, dst_file_path.as_ref(), base_dir, false).await {
+        Ok(()) => Ok(()),
+        Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(()),
+        Err(err) => Err(to_file_error(err).into()),
+    }
+}
+
 async fn reliable_rename(
     src_file_path: impl AsRef<Path>,
     dst_file_path: impl AsRef<Path>,
     base_dir: impl AsRef<Path>,
+) -> io::Result<()> {
+    reliable_rename_inner(src_file_path, dst_file_path, base_dir, true).await
+}
+
+async fn reliable_rename_inner(
+    src_file_path: impl AsRef<Path>,
+    dst_file_path: impl AsRef<Path>,
+    base_dir: impl AsRef<Path>,
+    warn_on_failure: bool,
 ) -> io::Result<()> {
     if let Some(parent) = dst_file_path.as_ref().parent()
         && !file_exists(parent)
     {
-        // info!("reliable_rename reliable_mkdir_all parent: {:?}", parent);
         reliable_mkdir_all(parent, base_dir.as_ref()).await?;
     }
 
     let mut i = 0;
     loop {
         if let Err(e) = super::fs::rename_std(src_file_path.as_ref(), dst_file_path.as_ref()) {
-            if e.kind() == io::ErrorKind::NotFound {
-                break;
-            }
-
             if i == 0 {
                 i += 1;
                 continue;
             }
-            warn!(
-                "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
-                src_file_path.as_ref(),
-                dst_file_path.as_ref(),
-                base_dir.as_ref(),
-                e
-            );
+            if warn_on_failure {
+                warn!(
+                    "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
+                    src_file_path.as_ref(),
+                    dst_file_path.as_ref(),
+                    base_dir.as_ref(),
+                    e
+                );
+            }
             return Err(e);
         }
 
@@ -249,3 +268,36 @@ pub async fn os_mkdir_all(dir_path: impl AsRef<Path>, base_dir: impl AsRef<Path>
 pub fn file_exists(path: impl AsRef<Path>) -> bool {
     std::fs::metadata(path.as_ref()).map(|_| true).unwrap_or(false)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tempfile::tempdir;
+
+    #[tokio::test]
+    async fn rename_all_missing_source_returns_file_not_found() {
+        let temp_dir = tempdir().expect("create temp dir");
+        let src = temp_dir.path().join("missing");
+        let dst = temp_dir.path().join("dst");
+
+        let err = rename_all(&src, &dst, temp_dir.path())
+            .await
+            .expect_err("missing source must fail");
+
+        assert!(matches!(err, DiskError::FileNotFound));
+        assert!(!dst.exists());
+    }
+
+    #[tokio::test]
+    async fn rename_all_ignore_missing_source_returns_ok() {
+        let temp_dir = tempdir().expect("create temp dir");
+        let src = temp_dir.path().join("missing");
+        let dst = temp_dir.path().join("dst");
+
+        rename_all_ignore_missing_source(&src, &dst, temp_dir.path())
+            .await
+            .expect("missing cleanup source must be ignored");
+
+        assert!(!dst.exists());
+    }
+}

crates/ecstore/src/endpoints.rs

@@ -683,14 +683,46 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
     }
 
     let mut device_paths = BTreeMap::<String, BTreeSet<String>>::new();
+    #[cfg(not(windows))]
     let mut missing_paths = Vec::new();
 
     for path in &local_paths {
         let canonical = match rustfs_utils::canonicalize(path) {
             Ok(path) => path,
             Err(err) if err.kind() == ErrorKind::NotFound => {
-                missing_paths.push(path.clone());
-                continue;
+                // On Windows, canonicalize can fail for ZFS volumes, junction points,
+                // subst drives, and other non-standard mounts. Try absolutize as fallback.
+                #[cfg(windows)]
+                {
+                    match crate::disk::endpoint::windows_fallback_local_path(path, &err, "disk independence validation") {
+                        Ok(absolute) => {
+                            let abs_path = absolute.to_string_lossy().into_owned();
+                            match rustfs_utils::os::get_physical_device_ids(&abs_path) {
+                                Ok(ids) => {
+                                    for device_id in ids {
+                                        device_paths.entry(device_id).or_default().insert(abs_path.clone());
+                                    }
+                                }
+                                Err(device_err) => {
+                                    return Err(Error::other(format!(
+                                        "failed to inspect physical disk for local endpoint '{abs_path}' after fallback path resolution: {device_err}"
+                                    )));
+                                }
+                            }
+                            continue;
+                        }
+                        Err(fallback_err) => {
+                            return Err(Error::other(format!(
+                                "failed to resolve local endpoint path '{path}' for disk validation: {err}; fallback resolution failed: {fallback_err}"
+                            )));
+                        }
+                    }
+                }
+                #[cfg(not(windows))]
+                {
+                    missing_paths.push(path.clone());
+                    continue;
+                }
             }
             Err(err) => {
                 return Err(Error::other(format!(
@@ -708,6 +740,7 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
         }
     }
 
+    #[cfg(not(windows))]
     if !missing_paths.is_empty() {
         warn!(
             missing_paths = ?missing_paths,

crates/ecstore/src/erasure_coding/encode.rs

@@ -209,6 +209,13 @@ impl Erasure {
     where
         R: AsyncRead + Send + Sync + Unpin + 'static,
     {
+        if self.block_size == 0 {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "erasure block_size must be non-zero",
+            ));
+        }
+
         // Bound queued encoded blocks by memory budget to avoid per-request spikes.
         let expanded_block_bytes = self.shard_size().saturating_mul(self.total_shard_count());
         let max_inflight_bytes = rustfs_utils::get_env_usize(
@@ -223,10 +230,20 @@ impl Erasure {
             let mut total = 0;
             let mut buf = vec![0u8; block_size];
             loop {
-                match rustfs_utils::read_full(&mut reader, &mut buf).await {
-                    Ok(n) if n > 0 => {
+                match rustfs_utils::read_full_or_eof(&mut reader, &mut buf).await {
+                    Ok(Some(n)) => {
+                        debug_assert!(n > 0, "non-zero block_size prevents zero-length reads");
                         total += n;
-                        let res = self.encode_data(&buf[..n])?;
+                        let erasure = self.clone();
+                        let encode_buf = std::mem::take(&mut buf);
+                        let (res, returned_buf) = tokio::task::spawn_blocking(move || {
+                            let res = erasure.encode_data(&encode_buf[..n]);
+                            (res, encode_buf)
+                        })
+                        .await
+                        .map_err(|err| std::io::Error::other(format!("EC encode task failed: {err}")))?;
+                        buf = returned_buf;
+                        let res = res?;
                         let queued_bytes = queued_block_bytes(&res);
                         rustfs_io_metrics::add_ec_encode_inflight_bytes(queued_bytes);
                         if let Err(err) = tx.send(res).await {
@@ -234,7 +251,7 @@ impl Erasure {
                             return Err(std::io::Error::other(format!("Failed to send encoded data : {err}")));
                         }
                     }
-                    Ok(_) => {
+                    Ok(None) => {
                         break;
                     }
                     Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
@@ -244,7 +261,7 @@ impl Erasure {
                         {
                             return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()));
                         }
-                        break;
+                        return Err(e);
                     }
                     Err(e) => {
                         return Err(e);
@@ -285,13 +302,42 @@ impl Erasure {
         writers.shutdown().await?;
         Ok((reader, total))
     }
+
+    /// Fast path for small inline objects: skip tokio::spawn + mpsc channel.
+    /// Reads all data, encodes directly, writes shards sequentially.
+    pub async fn encode_inline_small<R>(
+        self: Arc<Self>,
+        mut reader: R,
+        writers: &mut [Option<BitrotWriterWrapper>],
+        quorum: usize,
+    ) -> std::io::Result<(R, usize)>
+    where
+        R: AsyncRead + Send + Sync + Unpin,
+    {
+        use tokio::io::AsyncReadExt;
+
+        let mut buf = Vec::with_capacity(self.block_size);
+        let total = reader.read_to_end(&mut buf).await?;
+
+        if total == 0 {
+            return Ok((reader, 0));
+        }
+
+        let shards = self.encode_data(&buf)?;
+        let mut mw = MultiWriter::new(writers, quorum);
+        mw.write(shards).await?;
+        mw.shutdown().await?;
+        Ok((reader, total))
+    }
 }
 
 #[cfg(test)]
 mod tests {
     use super::*;
     use crate::erasure_coding::{BitrotWriterWrapper, CustomWriter};
+    use rustfs_rio::HardLimitReader;
     use rustfs_utils::HashAlgorithm;
+    use std::io::Cursor;
     use std::pin::Pin;
     use std::sync::{Arc, Mutex};
     use std::task::{Context, Poll};
@@ -348,6 +394,103 @@ mod tests {
         assert!(!committed.lock().unwrap().is_empty());
     }
 
+    #[tokio::test]
+    async fn encode_returns_unexpected_eof_for_truncated_limited_reader() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed);
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 16));
+        let truncated = HardLimitReader::new(Cursor::new(b"short".to_vec()), 10);
+
+        let err = match erasure.encode(truncated, &mut writers, 1).await {
+            Ok(_) => panic!("truncated input must fail"),
+            Err(err) => err,
+        };
+
+        assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
+    }
+
+    #[tokio::test]
+    async fn encode_rejects_zero_block_size() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed);
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"payload".to_vec()));
+        let err = erasure
+            .encode(reader, &mut writers, 1)
+            .await
+            .expect_err("zero block size must be rejected");
+
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
+        assert!(err.to_string().contains("block_size"));
+    }
+
+    /// encode_inline_small: empty reader returns (reader, 0) without writing to any shard.
+    #[tokio::test]
+    async fn encode_inline_small_empty_stream_returns_zero() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed.clone());
+        // 1 data shard, 0 parity shards, block_size = 16
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 16));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(Vec::<u8>::new()));
+        let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, 1).await.unwrap();
+
+        assert_eq!(total, 0);
+        // No shutdown was called, so nothing should be committed
+        assert!(committed.lock().unwrap().is_empty());
+    }
+
+    /// encode_inline_small: small payload is encoded into the correct number of shards
+    /// and each writer receives data after shutdown.
+    #[tokio::test]
+    async fn encode_inline_small_payload_writes_all_shards() {
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const TOTAL_SHARDS: usize = DATA_SHARDS + PARITY_SHARDS;
+        const BLOCK_SIZE: usize = 64;
+
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..TOTAL_SHARDS).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+
+        let mut writers: Vec<Option<BitrotWriterWrapper>> = committed
+            .iter()
+            .map(|c| {
+                Some(BitrotWriterWrapper::new(
+                    CustomWriter::new_tokio_writer(DeferredCommitWriter::new(c.clone())),
+                    BLOCK_SIZE / DATA_SHARDS,
+                    HashAlgorithm::HighwayHash256S,
+                ))
+            })
+            .collect();
+
+        let payload = b"hello inline small";
+        let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
+        let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, DATA_SHARDS).await.unwrap();
+
+        assert_eq!(total, payload.len());
+        // All shards must have received data (shutdown flushed the bitrot header + shard bytes)
+        for (i, c) in committed.iter().enumerate() {
+            assert!(!c.lock().unwrap().is_empty(), "shard {i} should have received data");
+        }
+    }
+
     #[test]
     fn encode_channel_capacity_never_returns_zero() {
         assert_eq!(encode_channel_capacity(0, 1024), 1);

crates/ecstore/src/erasure_coding/erasure.rs

@@ -106,7 +106,7 @@ impl LegacyReedSolomonEncoder {
         Ok(())
     }
 
-    fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+    fn reconstruct_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         let shard_len = shards
             .iter()
             .find_map(|s| s.as_ref().map(|v| v.len()))
@@ -172,6 +172,15 @@ impl LegacyReedSolomonEncoder {
 
         Ok(())
     }
+
+    fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        self.reconstruct_data(shards)?;
+        self.encode_parity(shards)
+    }
+
+    fn encode_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        encode_parity_shards(shards, self.data_shards, self.parity_shards, |shards| self.encode(shards))
+    }
 }
 
 /// Reed-Solomon encoder using reed-solomon-erasure
@@ -224,8 +233,8 @@ impl ReedSolomonEncoder {
         }
     }
 
-    /// Reconstruct missing shards.
-    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+    /// Reconstruct missing data shards.
+    pub fn reconstruct_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         if let Some(ref rs) = self.encoder {
             rs.reconstruct_data(shards)
                 .map_err(|e| io::Error::other(format!("Reed-Solomon reconstruct failed: {e:?}")))
@@ -233,6 +242,58 @@ impl ReedSolomonEncoder {
             Ok(())
         }
     }
+
+    /// Reconstruct missing data shards and regenerate parity shards.
+    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        self.reconstruct_data(shards)?;
+        self.encode_parity(shards)
+    }
+
+    fn encode_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        encode_parity_shards(shards, self.data_shards, self.parity_shards, |shards| self.encode(shards))
+    }
+}
+
+fn encode_parity_shards<F>(shards: &mut [Option<Vec<u8>>], data_shards: usize, parity_shards: usize, encode: F) -> io::Result<()>
+where
+    F: FnOnce(SmallVec<[&mut [u8]; 16]>) -> io::Result<()>,
+{
+    let expected_shards = data_shards + parity_shards;
+    if shards.len() != expected_shards {
+        return Err(io::Error::other(format!(
+            "invalid shard count: got {}, expected {}",
+            shards.len(),
+            expected_shards
+        )));
+    }
+
+    let shard_len = shards
+        .iter()
+        .find_map(|s| s.as_ref().map(Vec::len))
+        .ok_or_else(|| io::Error::other("No valid shards found for parity encoding"))?;
+
+    for shard in shards.iter_mut().skip(data_shards) {
+        if shard.is_none() {
+            *shard = Some(vec![0; shard_len]);
+        }
+    }
+
+    let mut shard_refs: SmallVec<[&mut [u8]; 16]> = SmallVec::new();
+    for (index, shard) in shards.iter_mut().enumerate() {
+        let shard = shard
+            .as_mut()
+            .ok_or_else(|| io::Error::other(format!("missing shard {index} after data reconstruction")))?;
+        if shard.len() != shard_len {
+            return Err(io::Error::other(format!(
+                "inconsistent shard length at index {index}: got {}, expected {}",
+                shard.len(),
+                shard_len
+            )));
+        }
+        shard_refs.push(shard.as_mut_slice());
+    }
+
+    encode(shard_refs)
 }
 
 /// Erasure coding utility for data reliability using Reed-Solomon codes.
@@ -247,7 +308,6 @@ impl ReedSolomonEncoder {
 /// - `encoder`: Optional ReedSolomon encoder instance.
 /// - `block_size`: Block size for each shard.
 /// - `_id`: Unique identifier for the erasure instance.
-/// - `_buf`: Internal buffer for block operations.
 ///
 /// # Example
 /// ```ignore
@@ -265,7 +325,6 @@ pub struct Erasure {
     pub block_size: usize,
     uses_legacy: bool,
     _id: Uuid,
-    _buf: Vec<u8>,
 }
 
 impl Default for Erasure {
@@ -278,7 +337,6 @@ impl Default for Erasure {
             block_size: 0,
             uses_legacy: false,
             _id: Uuid::nil(),
-            _buf: vec![],
         }
     }
 }
@@ -293,7 +351,6 @@ impl Clone for Erasure {
             block_size: self.block_size,
             uses_legacy: self.uses_legacy,
             _id: Uuid::new_v4(), // Generate new ID for clone
-            _buf: vec![0u8; self.block_size],
         }
     }
 }
@@ -338,7 +395,6 @@ impl Erasure {
             legacy_encoder,
             uses_legacy,
             _id: Uuid::new_v4(),
-            _buf: vec![0u8; block_size],
         }
     }
 
@@ -392,7 +448,7 @@ impl Erasure {
         Ok(shards)
     }
 
-    /// Decode and reconstruct missing shards in-place.
+    /// Decode and reconstruct missing data shards in-place.
     ///
     /// # Arguments
     /// * `shards` - Mutable slice of optional shard data. Missing shards should be `None`.
@@ -400,6 +456,25 @@ impl Erasure {
     /// # Returns
     /// Ok if reconstruction succeeds, error otherwise.
     pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        if self.parity_shards > 0 {
+            if self.uses_legacy {
+                if let Some(encoder) = self.legacy_encoder.as_ref() {
+                    encoder.reconstruct_data(shards)?;
+                } else {
+                    warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
+                }
+            } else if let Some(encoder) = self.encoder.as_ref() {
+                encoder.reconstruct_data(shards)?;
+            } else {
+                warn!("parity_shards > 0, but encoder is None");
+            }
+        }
+
+        Ok(())
+    }
+
+    /// Decode and reconstruct missing data shards, then regenerate parity shards.
+    pub fn decode_data_and_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         if self.parity_shards > 0 {
             if self.uses_legacy {
                 if let Some(encoder) = self.legacy_encoder.as_ref() {
@@ -499,18 +574,42 @@ impl Erasure {
         F: FnMut(std::io::Result<Vec<Bytes>>) -> Fut + Send,
         Fut: std::future::Future<Output = Result<(), E>> + Send,
     {
+        if self.block_size == 0 {
+            on_block(Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "erasure block_size must be non-zero",
+            )))
+            .await?;
+            return Ok(0);
+        }
+
         let block_size = self.block_size;
         let mut total = 0;
         let mut buf = vec![0u8; block_size];
         loop {
-            match rustfs_utils::read_full(&mut *reader, &mut buf).await {
-                Ok(n) if n > 0 => {
+            match rustfs_utils::read_full_or_eof(&mut *reader, &mut buf).await {
+                Ok(Some(n)) => {
+                    debug_assert!(n > 0, "non-zero block_size prevents zero-length reads");
                     warn!("encode_stream_callback_async read n={}", n);
                     total += n;
-                    let res = self.encode_data(&buf[..n]);
+                    let erasure = self.clone();
+                    let encode_buf = std::mem::take(&mut buf);
+                    let (res, returned_buf) = match tokio::task::spawn_blocking(move || {
+                        let res = erasure.encode_data(&encode_buf[..n]);
+                        (res, encode_buf)
+                    })
+                    .await
+                    {
+                        Ok(result) => result,
+                        Err(err) => {
+                            on_block(Err(std::io::Error::other(format!("EC encode task failed: {err}")))).await?;
+                            break;
+                        }
+                    };
+                    buf = returned_buf;
                     on_block(res).await?
                 }
-                Ok(_) => {
+                Ok(None) => {
                     warn!("encode_stream_callback_async read unexpected ok");
                     break;
                 }
@@ -534,6 +633,131 @@ mod tests {
 
     use super::*;
 
+    fn optional_shards(shards: &[Bytes]) -> Vec<Option<Vec<u8>>> {
+        shards.iter().map(|shard| Some(shard.to_vec())).collect()
+    }
+
+    #[test]
+    fn decode_data_keeps_missing_parity_shard_unreconstructed() {
+        let erasure = Erasure::new(2, 2, 64);
+        let data = b"read decode should not rebuild parity";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        let missing_parity = erasure.data_shards;
+        shards[missing_parity] = None;
+
+        erasure.decode_data(&mut shards).expect("decode should succeed");
+
+        assert!(shards[missing_parity].is_none(), "read decode should leave parity missing");
+        for index in 0..erasure.data_shards {
+            assert_eq!(
+                shards[index].as_deref(),
+                Some(encoded[index].as_ref()),
+                "data shard {index} should remain unchanged"
+            );
+        }
+    }
+
+    #[test]
+    fn legacy_decode_data_keeps_missing_parity_shard_unreconstructed() {
+        let erasure = Erasure::new_with_options(2, 2, 64, true);
+        let data = b"legacy read decode should not rebuild parity";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        let missing_parity = erasure.data_shards + 1;
+        shards[missing_parity] = None;
+
+        erasure.decode_data(&mut shards).expect("decode should succeed");
+
+        assert!(shards[missing_parity].is_none(), "legacy read decode should leave parity missing");
+        for index in 0..erasure.data_shards {
+            assert_eq!(
+                shards[index].as_deref(),
+                Some(encoded[index].as_ref()),
+                "legacy data shard {index} should remain unchanged"
+            );
+        }
+    }
+
+    #[test]
+    fn decode_data_and_parity_leaves_complete_shards_unchanged() {
+        let erasure = Erasure::new(4, 2, 128);
+        let data = b"complete shards should not be changed";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let original = optional_shards(&encoded);
+        let mut shards = original.clone();
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should succeed without missing shards");
+
+        assert_eq!(shards, original);
+    }
+
+    #[test]
+    fn decode_data_and_parity_reconstructs_missing_parity_shard() {
+        let erasure = Erasure::new(2, 2, 64);
+        let data = b"parity shard must be rebuilt";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        shards[2] = None;
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should rebuild parity");
+
+        for (index, shard) in shards.iter().enumerate() {
+            assert_eq!(
+                shard.as_deref(),
+                Some(encoded[index].as_ref()),
+                "shard {index} should match encoded source"
+            );
+        }
+    }
+
+    #[test]
+    fn decode_data_and_parity_reconstructs_missing_data_and_parity_shards() {
+        let erasure = Erasure::new(4, 2, 128);
+        let data = b"data and parity shards should both be reconstructed";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        shards[1] = None;
+        shards[4] = None;
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should rebuild all missing shards");
+
+        for (index, shard) in shards.iter().enumerate() {
+            assert_eq!(
+                shard.as_deref(),
+                Some(encoded[index].as_ref()),
+                "shard {index} should match encoded source"
+            );
+        }
+    }
+
+    #[test]
+    fn legacy_decode_data_and_parity_reconstructs_missing_parity_shard() {
+        let erasure = Erasure::new_with_options(2, 2, 64, true);
+        let data = b"legacy parity shard must be rebuilt";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        shards[3] = None;
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should rebuild parity");
+
+        for (index, shard) in shards.iter().enumerate() {
+            assert_eq!(
+                shard.as_deref(),
+                Some(encoded[index].as_ref()),
+                "shard {index} should match encoded source"
+            );
+        }
+    }
+
     #[test]
     fn test_shard_file_size_cases2() {
         let erasure = Erasure::new(12, 4, 1024 * 1024);
@@ -813,6 +1037,35 @@ mod tests {
         assert_eq!(&recovered, &data_clone);
     }
 
+    #[tokio::test]
+    async fn test_encode_stream_callback_async_reports_zero_block_size() {
+        use std::io::Cursor;
+        use std::sync::{Arc, Mutex};
+
+        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let mut reader = Cursor::new(b"payload".to_vec());
+        let observed = Arc::new(Mutex::new(None));
+        let observed_clone = observed.clone();
+
+        let total = erasure
+            .encode_stream_callback_async::<_, _, (), _>(&mut reader, move |res| {
+                let observed = observed_clone.clone();
+                async move {
+                    let err = res.expect_err("zero block size should report an error");
+                    *observed.lock().unwrap() = Some((err.kind(), err.to_string()));
+                    Ok(())
+                }
+            })
+            .await
+            .expect("callback should handle the zero block size error");
+
+        assert_eq!(total, 0);
+        let observed = observed.lock().unwrap();
+        let (kind, message) = observed.as_ref().expect("callback should be invoked once");
+        assert_eq!(*kind, std::io::ErrorKind::InvalidInput);
+        assert!(message.contains("block_size"));
+    }
+
     // SIMD mode specific tests
     mod simd_tests {
         use super::*;

crates/ecstore/src/erasure_coding/heal.rs

@@ -49,6 +49,10 @@ impl super::Erasure {
             end_block += 1;
         }
 
+        let available_writers = writers.iter().filter(|w| w.is_some()).count();
+        let write_quorum = available_writers.max(1);
+        let mut writers = MultiWriter::new(writers, write_quorum);
+
         for _ in start_block..end_block {
             let (mut shards, errs) = reader.read().await;
 
@@ -63,7 +67,7 @@ impl super::Erasure {
             }
 
             if self.parity_shards > 0 {
-                self.decode_data(&mut shards)?;
+                self.decode_data_and_parity(&mut shards)?;
             }
 
             let shards = shards
@@ -71,15 +75,122 @@ impl super::Erasure {
                 .map(|s| Bytes::from(s.unwrap_or_default()))
                 .collect::<Vec<_>>();
 
-            // Calculate proper write quorum for heal operation
-            // For heal, we only write to disks that need healing, so write quorum should be
-            // the number of available writers (disks that need healing)
-            let available_writers = writers.iter().filter(|w| w.is_some()).count();
-            let write_quorum = available_writers.max(1); // At least 1 writer must succeed
-            let mut writers = MultiWriter::new(writers, write_quorum);
             writers.write(shards).await?;
         }
 
+        writers.shutdown().await?;
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::erasure_coding::{CustomWriter, Erasure};
+    use rustfs_utils::HashAlgorithm;
+    use std::io::Cursor;
+
+    #[tokio::test]
+    async fn heal_reconstructs_missing_parity_shard() {
+        let erasure = Erasure::new(2, 2, 64);
+        let data = b"heal should write a rebuilt parity shard";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let missing_parity = erasure.data_shards;
+
+        let readers = encoded
+            .iter()
+            .enumerate()
+            .map(|(index, shard)| {
+                if index == missing_parity {
+                    None
+                } else {
+                    Some(BitrotReader::new(
+                        Cursor::new(shard.to_vec()),
+                        erasure.shard_size(),
+                        HashAlgorithm::None,
+                        false,
+                    ))
+                }
+            })
+            .collect::<Vec<_>>();
+
+        let mut writers = (0..erasure.total_shard_count())
+            .map(|index| {
+                if index == missing_parity {
+                    Some(BitrotWriterWrapper::new(
+                        CustomWriter::new_inline_buffer(),
+                        erasure.shard_size(),
+                        HashAlgorithm::None,
+                    ))
+                } else {
+                    None
+                }
+            })
+            .collect::<Vec<_>>();
+
+        erasure
+            .heal(&mut writers, readers, data.len(), &[])
+            .await
+            .expect("heal should rebuild parity");
+
+        let healed = writers[missing_parity]
+            .take()
+            .expect("parity writer should remain")
+            .into_inline_data()
+            .expect("inline writer should retain data");
+        assert_eq!(healed, encoded[missing_parity].to_vec());
+    }
+
+    #[tokio::test]
+    async fn heal_reconstructs_missing_data_shard_across_multiple_blocks() {
+        let erasure = Erasure::new(3, 2, 96);
+        let data = (0..erasure.block_size * 2 + 17)
+            .map(|index| (index % 251) as u8)
+            .collect::<Vec<_>>();
+        let encoded = erasure.encode_data(&data).expect("encode should succeed");
+        let missing_data = 1;
+
+        let readers = encoded
+            .iter()
+            .enumerate()
+            .map(|(index, shard)| {
+                if index == missing_data {
+                    None
+                } else {
+                    Some(BitrotReader::new(
+                        Cursor::new(shard.to_vec()),
+                        erasure.shard_size(),
+                        HashAlgorithm::None,
+                        false,
+                    ))
+                }
+            })
+            .collect::<Vec<_>>();
+
+        let mut writers = (0..erasure.total_shard_count())
+            .map(|index| {
+                if index == missing_data {
+                    Some(BitrotWriterWrapper::new(
+                        CustomWriter::new_inline_buffer(),
+                        erasure.shard_size(),
+                        HashAlgorithm::None,
+                    ))
+                } else {
+                    None
+                }
+            })
+            .collect::<Vec<_>>();
+
+        erasure
+            .heal(&mut writers, readers, data.len(), &[])
+            .await
+            .expect("heal should rebuild data");
+
+        let healed = writers[missing_data]
+            .take()
+            .expect("data writer should remain")
+            .into_inline_data()
+            .expect("inline writer should retain data");
+        assert_eq!(healed, encoded[missing_data].to_vec());
+    }
+}

crates/ecstore/src/error.rs

@@ -160,6 +160,14 @@ pub enum StorageError {
     NotModified,
     #[error("Invalid range specified: {0}")]
     InvalidRangeSpec(String),
+    #[error("Namespace lock quorum unavailable for {mode} lock on {bucket}/{object}: required {required}, achieved {achieved}")]
+    NamespaceLockQuorumUnavailable {
+        mode: &'static str,
+        bucket: String,
+        object: String,
+        required: usize,
+        achieved: usize,
+    },
 
     // ── Generic ──────────────────────────────────────────────────────
     #[error("Unexpected error")]
@@ -204,6 +212,7 @@ impl StorageError {
                 | StorageError::ErasureWriteQuorum
                 | StorageError::InsufficientReadQuorum(_, _)
                 | StorageError::InsufficientWriteQuorum(_, _)
+                | StorageError::NamespaceLockQuorumUnavailable { .. }
         )
     }
 }
@@ -433,6 +442,19 @@ impl Clone for StorageError {
             StorageError::NotModified => StorageError::NotModified,
             StorageError::InvalidPartNumber(a) => StorageError::InvalidPartNumber(*a),
             StorageError::InvalidRangeSpec(a) => StorageError::InvalidRangeSpec(a.clone()),
+            StorageError::NamespaceLockQuorumUnavailable {
+                mode,
+                bucket,
+                object,
+                required,
+                achieved,
+            } => StorageError::NamespaceLockQuorumUnavailable {
+                mode,
+                bucket: bucket.clone(),
+                object: object.clone(),
+                required: *required,
+                achieved: *achieved,
+            },
         }
     }
 }
@@ -505,6 +527,7 @@ impl StorageError {
             StorageError::InvalidRangeSpec(_) => 0x3D,
             StorageError::NotModified => 0x3E,
             StorageError::InvalidPartNumber(_) => 0x3F,
+            StorageError::NamespaceLockQuorumUnavailable { .. } => 0x42,
         }
     }
 
@@ -579,6 +602,13 @@ impl StorageError {
             0x3D => Some(StorageError::InvalidRangeSpec(Default::default())),
             0x3E => Some(StorageError::NotModified),
             0x3F => Some(StorageError::InvalidPartNumber(Default::default())),
+            0x42 => Some(StorageError::NamespaceLockQuorumUnavailable {
+                mode: "write",
+                bucket: Default::default(),
+                object: Default::default(),
+                required: Default::default(),
+                achieved: Default::default(),
+            }),
             _ => None,
         }
     }
@@ -984,6 +1014,17 @@ mod tests {
         assert_eq!(StorageError::DecommissionAlreadyRunning.to_u32(), 0x30);
         assert_eq!(StorageError::RebalanceAlreadyRunning.to_u32(), 0x40);
         assert_eq!(StorageError::OperationCanceled.to_u32(), 0x41);
+        assert_eq!(
+            StorageError::NamespaceLockQuorumUnavailable {
+                mode: "write",
+                bucket: "bucket".into(),
+                object: "object".into(),
+                required: 3,
+                achieved: 2,
+            }
+            .to_u32(),
+            0x42
+        );
     }
 
     #[test]
@@ -998,6 +1039,10 @@ mod tests {
         assert!(matches!(StorageError::from_u32(0x30), Some(StorageError::DecommissionAlreadyRunning)));
         assert!(matches!(StorageError::from_u32(0x40), Some(StorageError::RebalanceAlreadyRunning)));
         assert!(matches!(StorageError::from_u32(0x41), Some(StorageError::OperationCanceled)));
+        assert!(matches!(
+            StorageError::from_u32(0x42),
+            Some(StorageError::NamespaceLockQuorumUnavailable { .. })
+        ));
 
         // Test invalid code returns None
         assert!(StorageError::from_u32(0xFF).is_none());

crates/ecstore/src/notification_sys.rs

@@ -189,6 +189,65 @@ impl NotificationSys {
         join_all(futures).await
     }
 
+    pub async fn reload_dynamic_config(&self, sub_sys: &str) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter() {
+            let sub_sys = sub_sys.to_string();
+            futures.push(async move {
+                if let Some(client) = client {
+                    match client
+                        .signal_service(crate::rpc::SERVICE_SIGNAL_RELOAD_DYNAMIC, &sub_sys, false, SystemTime::UNIX_EPOCH)
+                        .await
+                    {
+                        Ok(_) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: None,
+                        },
+                        Err(e) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: Some(e),
+                        },
+                    }
+                } else {
+                    NotificationPeerErr {
+                        host: "".to_string(),
+                        err: Some(Error::other("peer is not reachable")),
+                    }
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
+    pub async fn refresh_config_snapshot(&self) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter() {
+            futures.push(async move {
+                if let Some(client) = client {
+                    match client
+                        .signal_service(crate::rpc::SERVICE_SIGNAL_REFRESH_CONFIG, "", false, SystemTime::UNIX_EPOCH)
+                        .await
+                    {
+                        Ok(_) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: None,
+                        },
+                        Err(e) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: Some(e),
+                        },
+                    }
+                } else {
+                    NotificationPeerErr {
+                        host: "".to_string(),
+                        err: Some(Error::other("peer is not reachable")),
+                    }
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
     pub async fn storage_info<S: StorageAPI>(&self, api: &S) -> rustfs_madmin::StorageInfo {
         let mut futures = Vec::with_capacity(self.peer_clients.len());
         let endpoints = get_global_endpoints();

crates/ecstore/src/rebalance.rs

@@ -1002,7 +1002,7 @@ impl ECStore {
                 return true;
             }
 
-            // Mark pool rebalance as done if within 5% of the PercentFreeGoal
+            // Mark pool rebalance as done only after it reaches the PercentFreeGoal.
             let pfi = if pool_stat.init_capacity == 0 {
                 0.0
             } else {
@@ -1032,7 +1032,7 @@ fn rebalance_goal_reached(init_free_space: u64, init_capacity: u64, bytes: u64,
     }
 
     let pfi = (init_free_space + bytes) as f64 / init_capacity as f64;
-    (pfi - percent_free_goal).abs() <= 0.05 + f64::EPSILON
+    pfi + f64::EPSILON >= percent_free_goal
 }
 
 fn percent_free_ratio(total_free: u64, total_cap: u64) -> f64 {
@@ -2664,23 +2664,20 @@ mod rebalance_unit_tests {
     }
 
     #[test]
-    fn test_rebalance_goal_reached_exact_tolerance_bound() {
+    fn test_rebalance_goal_reached_at_target() {
         let init_free_space = 200_u64;
         let init_capacity = 1_000_u64;
         let goal = 0.45_f64;
 
-        // goal - 0.05 => 200 + 200 = 400 free / 1000 => 0.4 exactly
-        assert!(rebalance_goal_reached(init_free_space, init_capacity, 200, goal));
-
-        // one byte above the tolerance boundary should be false
-        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 199, goal));
+        assert!(rebalance_goal_reached(init_free_space, init_capacity, 250, goal));
+        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 249, goal));
     }
 
     #[test]
-    fn test_rebalance_goal_reached_within_tolerance() {
+    fn test_rebalance_goal_reached_above_target() {
         let init_free_space = 200_u64;
         let init_capacity = 1_000_u64;
-        let bytes = 250_u64;
+        let bytes = 251_u64;
         let goal = 0.45_f64;
 
         assert!(rebalance_goal_reached(init_free_space, init_capacity, bytes, goal));
@@ -2702,12 +2699,12 @@ mod rebalance_unit_tests {
     }
 
     #[test]
-    fn test_rebalance_goal_above_one_is_true_when_within_tolerance() {
+    fn test_rebalance_goal_above_one_is_true_when_reached() {
         assert!(rebalance_goal_reached(950, 1_000, 100, 1.0));
     }
 
     #[test]
-    fn test_rebalance_goal_below_zero_is_true_when_within_tolerance() {
+    fn test_rebalance_goal_below_zero_is_true_when_reached() {
         assert!(rebalance_goal_reached(10, 1_000, 0, -0.01));
     }
 
@@ -3355,6 +3352,18 @@ mod rebalance_unit_tests {
         assert!(!should_pool_participate(300, 1_000, 0.3));
     }
 
+    #[test]
+    fn test_rebalance_goal_not_reached_for_issue_3137_initial_imbalance() {
+        let pool0_capacity = 10_000_u64;
+        let pool0_free = 9_135_u64;
+        let pool1_capacity = 10_000_u64;
+        let pool1_free = 9_800_u64;
+        let goal = percent_free_ratio(pool0_free + pool1_free, pool0_capacity + pool1_capacity);
+
+        assert!(should_pool_participate(pool0_free, pool0_capacity, goal));
+        assert!(!rebalance_goal_reached(pool0_free, pool0_capacity, 0, goal));
+    }
+
     #[test]
     fn test_complete_rebalance_pools_at_goal_marks_started_participants_completed() {
         let now = OffsetDateTime::from_unix_timestamp(1_000).unwrap();
@@ -3365,7 +3374,7 @@ mod rebalance_unit_tests {
                     participating: true,
                     init_free_space: 400,
                     init_capacity: 1_000,
-                    bytes: 50,
+                    bytes: 100,
                     info: RebalanceInfo {
                         status: RebalStatus::Started,
                         ..Default::default()
@@ -4129,13 +4138,13 @@ mod rebalance_unit_tests {
     }
 
     #[test]
-    fn test_rebalance_goal_reached_tolerance_and_regression() {
+    fn test_rebalance_goal_reached_requires_target_ratio() {
         let init_free_space = 150_u64;
         let init_capacity = 800_u64;
         let goal = 0.35_f64;
 
         assert!(!rebalance_goal_reached(init_free_space, init_capacity, 0, goal));
-        assert!(rebalance_goal_reached(init_free_space, init_capacity, 90, goal));
-        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 89, goal));
+        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 129, goal));
+        assert!(rebalance_goal_reached(init_free_space, init_capacity, 130, goal));
     }
 }

crates/ecstore/src/rpc/http_auth.rs

@@ -72,7 +72,7 @@ fn signature_payload(url: &str, method: &Method, timestamp: i64) -> String {
 /// Generate HMAC-SHA256 signature for the given data
 fn generate_signature(secret: &str, url: &str, method: &Method, timestamp: i64) -> String {
     let data = signature_payload(url, method, timestamp);
-    let mut mac = HmacSha256::new_from_slice(secret.as_bytes()).expect("HMAC can take key of any size");
+    let mut mac = <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).expect("HMAC can take key of any size");
     mac.update(data.as_bytes());
     let result = mac.finalize();
     general_purpose::STANDARD.encode(result.into_bytes())
@@ -84,7 +84,7 @@ fn verify_signature(secret: &str, url: &str, method: &Method, timestamp: i64, si
     };
 
     let data = signature_payload(url, method, timestamp);
-    let mut mac = HmacSha256::new_from_slice(secret.as_bytes()).expect("HMAC can take key of any size");
+    let mut mac = <HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).expect("HMAC can take key of any size");
     mac.update(data.as_bytes());
     mac.verify_slice(&signature).is_ok()
 }

crates/ecstore/src/rpc/internode_data_transport.rs

@@ -17,12 +17,14 @@ use crate::disk::{FileReader, FileWriter};
 use crate::rpc::build_auth_headers;
 use async_trait::async_trait;
 use http::{HeaderMap, HeaderValue, Method, header::CONTENT_TYPE};
-use rustfs_config::{DEFAULT_INTERNODE_DATA_TRANSPORT, ENV_RUSTFS_INTERNODE_DATA_TRANSPORT};
+use rustfs_config::{
+    DEFAULT_INTERNODE_DATA_TRANSPORT, ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP,
+    KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS,
+};
 use rustfs_rio::{HttpReader, HttpWriter};
 use std::sync::{Arc, OnceLock};
 use std::time::Duration;
 
-pub const INTERNODE_DATA_TRANSPORT_TCP: &str = "tcp";
 static INTERNODE_DATA_TRANSPORT: OnceLock<std::result::Result<Arc<dyn InternodeDataTransport>, String>> = OnceLock::new();
 
 const READ_FILE_STREAM_PATH: &str = "/rustfs/rpc/read_file_stream";
@@ -32,29 +34,36 @@ const CONTENT_TYPE_JSON: &str = "application/json";
 
 fn unsupported_transport_message(transport: &str) -> String {
     format!(
-        "invalid {ENV_RUSTFS_INTERNODE_DATA_TRANSPORT}={transport:?}; supported values: {DEFAULT_INTERNODE_DATA_TRANSPORT}, {INTERNODE_DATA_TRANSPORT_TCP}"
+        "invalid {ENV_RUSTFS_INTERNODE_DATA_TRANSPORT}={transport:?}; supported values: {}",
+        KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS.join(", ")
     )
 }
 
 #[derive(Debug, Clone, Copy, Eq, PartialEq)]
 pub struct InternodeDataTransportCapabilities {
-    pub stream_read: bool,
-    pub stream_write: bool,
-    pub walk_dir: bool,
-    pub registered_memory: bool,
-    pub scatter_gather: bool,
-    pub zero_copy_receive: bool,
+    /// Backend can open a streaming remote disk reader.
+    pub streaming_read: bool,
+    /// Backend can open a streaming remote disk writer.
+    pub streaming_write: bool,
+    /// Backend can stream walk-dir responses.
+    pub streaming_walk_dir: bool,
+    /// Backend preserves in-order delivery for each opened transfer.
+    pub ordered_delivery: bool,
+    /// Largest payload the backend accepts for one transfer, or no RustFS-level cap.
+    pub max_transfer_size: Option<usize>,
+    /// Backend can participate in the behavior-preserving TCP fallback path.
+    pub fallback_supported: bool,
 }
 
 impl InternodeDataTransportCapabilities {
     pub const fn tcp_http() -> Self {
         Self {
-            stream_read: true,
-            stream_write: true,
-            walk_dir: true,
-            registered_memory: false,
-            scatter_gather: false,
-            zero_copy_receive: false,
+            streaming_read: true,
+            streaming_write: true,
+            streaming_walk_dir: true,
+            ordered_delivery: true,
+            max_transfer_size: None,
+            fallback_supported: true,
         }
     }
 }
@@ -87,6 +96,14 @@ pub struct WalkDirStreamRequest {
     pub stall_timeout: Option<Duration>,
 }
 
+/// Data-plane stream opener used by `RemoteDisk`.
+///
+/// This boundary is limited to remote disk streams that can move large payloads.
+/// Internode metadata, lock, health, and administrative calls remain on the
+/// existing gRPC control plane.
+///
+/// Buffer ownership, backend selection, and fallback expectations are documented
+/// in `crates/ecstore/docs/internode-transport/`.
 #[async_trait]
 pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
     async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
@@ -215,16 +232,25 @@ mod tests {
         assert_eq!(
             transport.capabilities(),
             InternodeDataTransportCapabilities {
-                stream_read: true,
-                stream_write: true,
-                walk_dir: true,
-                registered_memory: false,
-                scatter_gather: false,
-                zero_copy_receive: false,
+                streaming_read: true,
+                streaming_write: true,
+                streaming_walk_dir: true,
+                ordered_delivery: true,
+                max_transfer_size: None,
+                fallback_supported: true,
             }
         );
     }
 
+    #[test]
+    fn tcp_http_capabilities_are_conservative() {
+        let capabilities = TcpHttpInternodeDataTransport.capabilities();
+
+        assert!(capabilities.ordered_delivery);
+        assert_eq!(capabilities.max_transfer_size, None);
+        assert!(capabilities.fallback_supported);
+    }
+
     #[test]
     fn read_file_stream_url_encodes_query_values() {
         let url = build_read_file_stream_url(&ReadStreamRequest {
@@ -302,20 +328,36 @@ mod tests {
         }
     }
 
+    #[test]
+    fn transport_config_known_backends_are_current_oss_values() {
+        assert_eq!(
+            KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS,
+            &[DEFAULT_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP]
+        );
+
+        for configured in KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS {
+            let transport = build_internode_data_transport(Some(configured)).unwrap();
+
+            assert_eq!(transport.name(), DEFAULT_INTERNODE_DATA_TRANSPORT);
+        }
+    }
+
     #[test]
     fn transport_config_rejects_unknown_backend() {
-        let err = build_internode_data_transport(Some("rdma")).expect_err("unknown backend should fail closed");
+        let err = build_internode_data_transport(Some("unsupported-backend")).expect_err("unknown backend should fail closed");
 
         assert!(err.to_string().contains(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT));
-        assert!(err.to_string().contains("rdma"));
+        assert!(err.to_string().contains("unsupported-backend"));
+        assert!(err.to_string().contains("supported values: tcp-http, tcp"));
     }
 
     #[test]
     fn cached_transport_config_error_uses_raw_message() {
-        let err = build_internode_data_transport_result(Some("rdma")).expect_err("unknown backend should fail closed");
+        let err =
+            build_internode_data_transport_result(Some("unsupported-backend")).expect_err("unknown backend should fail closed");
 
         assert!(!err.starts_with("io error "));
         assert!(err.contains(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT));
-        assert!(err.contains("rdma"));
+        assert!(err.contains("unsupported-backend"));
     }
 }

crates/ecstore/src/rpc/mod.rs

@@ -29,7 +29,10 @@ pub use internode_data_transport::{
     InternodeDataTransport, InternodeDataTransportCapabilities, ReadStreamRequest, TcpHttpInternodeDataTransport,
     WalkDirStreamRequest, WriteStreamRequest, build_internode_data_transport, build_internode_data_transport_from_env,
 };
-pub use peer_rest_client::PeerRestClient;
+pub use peer_rest_client::{
+    PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG,
+    SERVICE_SIGNAL_RELOAD_DYNAMIC,
+};
 pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, RemotePeerS3Client, S3PeerSys};
 pub use remote_disk::RemoteDisk;
 pub use remote_locker::RemoteClient;

crates/ecstore/src/rpc/peer_rest_client.rs

@@ -57,6 +57,8 @@ use tracing::warn;
 pub const PEER_RESTSIGNAL: &str = "signal";
 pub const PEER_RESTSUB_SYS: &str = "sub-sys";
 pub const PEER_RESTDRY_RUN: &str = "dry-run";
+pub const SERVICE_SIGNAL_REFRESH_CONFIG: u64 = 1;
+pub const SERVICE_SIGNAL_RELOAD_DYNAMIC: u64 = 2;
 const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
 const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
 

crates/ecstore/src/rpc/peer_s3_client.rs

@@ -843,12 +843,13 @@ impl PeerS3Client for RemotePeerS3Client {
                 });
                 let response = client.make_bucket(request).await?.into_inner();
 
-                // TODO: deal with error
                 if !response.success {
                     return if let Some(err) = response.error {
                         Err(err.into())
                     } else {
-                        Err(Error::other(""))
+                        Err(Error::other(format!(
+                            "make_bucket({bucket}): peer returned failure without error details"
+                        )))
                     };
                 }
 
@@ -1046,6 +1047,44 @@ async fn clone_drives() -> Vec<Option<DiskStore>> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::disk::WalkDirOptions;
+    use crate::disk::disk_store::LocalDiskWrapper;
+    use crate::disk::endpoint::Endpoint;
+    use crate::disk::local::LocalDisk;
+    use crate::endpoints::{Endpoints, PoolEndpoints};
+    use crate::global::{GLOBAL_LOCAL_DISK_ID_MAP, GLOBAL_LOCAL_DISK_MAP, GLOBAL_LOCAL_DISK_SET_DRIVES};
+    use crate::store::init_local_disks;
+    use rustfs_filemeta::FileInfo;
+    use serial_test::serial;
+    use std::{
+        io,
+        pin::Pin,
+        task::{Context, Poll},
+    };
+    use tempfile::TempDir;
+    use tokio::io::AsyncWrite;
+
+    struct PendingWriter;
+
+    impl AsyncWrite for PendingWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
+            Poll::Pending
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
+
+    async fn reset_local_disk_globals() {
+        GLOBAL_LOCAL_DISK_MAP.write().await.clear();
+        GLOBAL_LOCAL_DISK_ID_MAP.write().await.clear();
+        GLOBAL_LOCAL_DISK_SET_DRIVES.write().await.clear();
+    }
 
     #[derive(Debug)]
     struct TestPeerS3Client {
@@ -1152,6 +1191,77 @@ mod tests {
         client.cancel_token.cancel();
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn local_get_bucket_info_survives_prior_walk_timeout() {
+        reset_local_disk_globals().await;
+
+        let temp_dir = TempDir::new().expect("create temp dir for local peer listing regression");
+        let disk_path = temp_dir.path().join("disk1");
+        std::fs::create_dir_all(&disk_path).expect("create disk path");
+
+        let mut endpoint = Endpoint::try_from(disk_path.to_str().expect("disk path to str")).expect("endpoint");
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+
+        let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 1,
+            endpoints: Endpoints::from(vec![endpoint.clone()]),
+            cmd_line: "local-get-bucket-info-survives-prior-walk-timeout".to_string(),
+            platform: "test".to_string(),
+        }]);
+
+        init_local_disks(endpoint_pools).await.expect("init local disks");
+
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+        let wrapper = crate::disk::Disk::Local(Box::new(LocalDiskWrapper::new(disk, false)));
+        let disk_store: DiskStore = Arc::new(wrapper);
+        let bucket = "test-bucket";
+        let object = "test-object";
+
+        disk_store.make_volume(bucket).await.expect("bucket should be created");
+
+        let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+        file_info.volume = bucket.to_string();
+        file_info.name = object.to_string();
+        file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+        file_info.erasure.index = 1;
+
+        disk_store
+            .write_metadata("", bucket, object, file_info)
+            .await
+            .expect("object metadata should be written");
+
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let mut writer = PendingWriter;
+            let walk_err = disk_store
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await
+                .expect_err("walk_dir should time out against a non-draining writer");
+
+            assert_eq!(walk_err, DiskError::Timeout);
+
+            let info = LocalPeerS3Client::new(None, Some(vec![0]))
+                .get_bucket_info(bucket, &BucketOptions::default())
+                .await
+                .expect("bucket info should still succeed after prior walk timeout");
+            assert_eq!(info.name, bucket);
+        })
+        .await;
+
+        reset_local_disk_globals().await;
+    }
+
     #[test]
     fn test_reduce_pool_write_quorum_uses_only_pool_participants() {
         let clients = vec![

crates/ecstore/src/rpc/remote_disk.rs

@@ -35,7 +35,8 @@ use futures::lock::Mutex;
 use metrics::counter;
 use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
 use rustfs_io_metrics::internode_metrics::{
-    INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_WRITE_ALL, global_internode_metrics,
+    INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC,
+    global_internode_metrics,
 };
 use rustfs_protos::evict_failed_connection;
 use rustfs_protos::proto_gen::node_service::RenamePartRequest;
@@ -109,6 +110,15 @@ pub struct RemoteDisk {
 }
 
 impl RemoteDisk {
+    fn is_retryable_walk_dir_error(err: &DiskError) -> bool {
+        if is_network_like_disk_error(err) {
+            return true;
+        }
+
+        let err_text = err.to_string().to_ascii_lowercase();
+        err_text.contains("httpreader stream error") || err_text.contains("error decoding response body")
+    }
+
     pub(crate) async fn new(ep: &Endpoint, opt: &DiskOption, data_transport: Arc<dyn InternodeDataTransport>) -> Result<Self> {
         let addr = if let Some(port) = ep.url.port() {
             format!("{}://{}:{}", ep.url.scheme(), ep.url.host_str().unwrap(), port)
@@ -174,6 +184,10 @@ impl RemoteDisk {
         });
     }
 
+    fn mark_suspect_or_offline(&self, reason: &'static str) -> bool {
+        self.health.mark_failure(&self.endpoint, reason)
+    }
+
     /// Enable health monitoring after disk creation.
     /// Used to defer health checks until after startup format loading completes,
     /// so that remote peers have time to come online.
@@ -201,7 +215,10 @@ impl RemoteDisk {
         let mut interval = time::interval(get_drive_active_check_interval());
 
         // Perform basic connectivity check
-        if Self::perform_connectivity_check(&addr).await.is_err() && health.mark_offline(&endpoint, "connectivity_probe_failed") {
+        let initial_probe_ok = Self::perform_connectivity_check(&addr).await.is_ok();
+        if initial_probe_ok {
+            health.record_operation_success(&endpoint, "connectivity_probe_success");
+        } else if health.mark_failure(&endpoint, "connectivity_probe_failed") {
             warn!("Remote disk health check failed for {}: marking as faulty", addr);
 
             // Start recovery monitoring
@@ -244,7 +261,9 @@ impl RemoteDisk {
                     }
 
                     // Perform basic connectivity check
-                    if Self::perform_connectivity_check(&addr).await.is_err() && health.mark_offline(&endpoint, "connectivity_probe_failed") {
+                    if Self::perform_connectivity_check(&addr).await.is_ok() {
+                        health.record_operation_success(&endpoint, "connectivity_probe_success");
+                    } else if health.mark_failure(&endpoint, "connectivity_probe_failed") {
                         warn!("Remote disk health check failed for {}: marking as faulty", addr);
 
                         // Start recovery monitoring
@@ -285,7 +304,7 @@ impl RemoteDisk {
                             return;
                         }
                     } else {
-                        health.mark_offline(&endpoint, "connectivity_probe_failed");
+                        health.mark_failure(&endpoint, "connectivity_probe_failed");
                     }
                 }
             }
@@ -439,7 +458,11 @@ impl RemoteDisk {
     }
 
     async fn mark_faulty_and_evict(&self, reason: &'static str) {
-        if self.health.mark_offline(&self.endpoint, reason) {
+        let previous_state = self.runtime_state();
+        let transitioned_to_offline = self.mark_suspect_or_offline(reason);
+        let state = self.runtime_state();
+
+        if state != previous_state {
             self.spawn_recovery_monitor_if_needed();
             counter!(
                 "rustfs_drive_faulty_mark_total",
@@ -447,10 +470,17 @@ impl RemoteDisk {
                 "reason" => reason.to_string()
             )
             .increment(1);
-            warn!(
-                "Remote disk marked faulty after timeout: endpoint={}, addr={}, reason={}",
-                self.endpoint, self.addr, reason
-            );
+            if transitioned_to_offline {
+                warn!(
+                    "Remote disk marked faulty after timeout: endpoint={}, addr={}, reason={}",
+                    self.endpoint, self.addr, reason
+                );
+            } else {
+                warn!(
+                    "Remote disk marked suspect after timeout: endpoint={}, addr={}, reason={}, state={:?}",
+                    self.endpoint, self.addr, reason, state
+                );
+            }
             counter!(
                 "rustfs_drive_connection_evict_total",
                 "endpoint" => self.endpoint.to_string(),
@@ -1127,7 +1157,8 @@ impl DiskAPI for RemoteDisk {
     ) -> Result<RenameDataResp> {
         info!("rename_data {}/{}/{}/{}", self.addr, self.endpoint.to_string(), dst_volume, dst_path);
 
-        self.execute_with_timeout(
+        self.execute_with_timeout_for_op(
+            "rename_data",
             || async {
                 let file_info = serde_json::to_string(&fi)?;
                 let mut client = self
@@ -1194,24 +1225,58 @@ impl DiskAPI for RemoteDisk {
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
         info!("walk_dir {}", self.endpoint.to_string());
 
+        let disk = self.disk_ref().await;
+        let body = serde_json::to_vec(&opts)?;
+        let stall_timeout = get_drive_walkdir_stall_timeout();
+        let bucket = opts.bucket.clone();
+        let base_dir = opts.base_dir.clone();
+        let disk_for_log = disk.clone();
+
         self.execute_with_timeout_for_op_and_health_action(
             "walk_dir",
             || async {
-                let disk = self.disk_ref().await;
-                let opts = serde_json::to_vec(&opts)?;
-                let mut reader = self
-                    .data_transport
-                    .open_walk_dir(WalkDirStreamRequest {
-                        endpoint: self.endpoint.grid_host(),
-                        disk,
-                        body: opts,
-                        stall_timeout: Some(get_drive_walkdir_stall_timeout()),
-                    })
-                    .await?;
+                let mut last_err = None;
 
-                copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await?;
+                for attempt in 1..=2 {
+                    let mut reader = match self
+                        .data_transport
+                        .open_walk_dir(WalkDirStreamRequest {
+                            endpoint: self.endpoint.grid_host(),
+                            disk: disk.clone(),
+                            body: body.clone(),
+                            stall_timeout: Some(stall_timeout),
+                        })
+                        .await
+                    {
+                        Ok(reader) => reader,
+                        Err(err) => {
+                            if attempt == 1 && Self::is_retryable_walk_dir_error(&err) {
+                                warn!(
+                                    endpoint = %self.endpoint,
+                                    addr = %self.addr,
+                                    disk = %disk_for_log,
+                                    bucket = %bucket,
+                                    base_dir = %base_dir,
+                                    attempt,
+                                    stall_timeout_ms = stall_timeout.as_millis(),
+                                    error = %err,
+                                    "remote walk_dir returned retryable transport error; retrying"
+                                );
+                                last_err = Some(err);
+                                continue;
+                            }
 
-                Ok(())
+                            return Err(err);
+                        }
+                    };
+
+                    match copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await {
+                        Ok(_) => return Ok(()),
+                        Err(io_err) => return Err(DiskError::Io(io_err)),
+                    }
+                }
+
+                Err(last_err.unwrap_or_else(|| DiskError::other("walk_dir retry exhausted without captured error")))
             },
             get_drive_walkdir_timeout(),
             FailureHealthAction::IgnoreFailure,
@@ -1558,7 +1623,10 @@ impl DiskAPI for RemoteDisk {
                 let data_len = data.len();
                 let disk = self.disk_ref().await;
                 let mut client = self.get_client().await.map_err(|err| {
-                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+                    global_internode_metrics().record_error_for_operation_and_backend(
+                        INTERNODE_OPERATION_GRPC_WRITE_ALL,
+                        INTERNODE_TRANSPORT_BACKEND_GRPC,
+                    );
                     Error::other(format!("can not get client, err: {err}"))
                 })?;
                 let request = Request::new(WriteAllRequest {
@@ -1568,19 +1636,32 @@ impl DiskAPI for RemoteDisk {
                     data,
                 });
 
-                global_internode_metrics().record_outgoing_request_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+                global_internode_metrics().record_outgoing_request_for_operation_and_backend(
+                    INTERNODE_OPERATION_GRPC_WRITE_ALL,
+                    INTERNODE_TRANSPORT_BACKEND_GRPC,
+                );
                 let response = match client.write_all(request).await {
                     Ok(response) => response.into_inner(),
                     Err(err) => {
-                        global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+                        global_internode_metrics().record_error_for_operation_and_backend(
+                            INTERNODE_OPERATION_GRPC_WRITE_ALL,
+                            INTERNODE_TRANSPORT_BACKEND_GRPC,
+                        );
                         return Err(err.into());
                     }
                 };
 
-                global_internode_metrics().record_sent_bytes_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL, data_len);
+                global_internode_metrics().record_sent_bytes_for_operation_and_backend(
+                    INTERNODE_OPERATION_GRPC_WRITE_ALL,
+                    INTERNODE_TRANSPORT_BACKEND_GRPC,
+                    data_len,
+                );
 
                 if !response.success {
-                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+                    global_internode_metrics().record_error_for_operation_and_backend(
+                        INTERNODE_OPERATION_GRPC_WRITE_ALL,
+                        INTERNODE_TRANSPORT_BACKEND_GRPC,
+                    );
                     return Err(response.error.unwrap_or_default().into());
                 }
 
@@ -1599,7 +1680,10 @@ impl DiskAPI for RemoteDisk {
             || async {
                 let disk = self.disk_ref().await;
                 let mut client = self.get_client().await.map_err(|err| {
-                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+                    global_internode_metrics().record_error_for_operation_and_backend(
+                        INTERNODE_OPERATION_GRPC_READ_ALL,
+                        INTERNODE_TRANSPORT_BACKEND_GRPC,
+                    );
                     Error::other(format!("can not get client, err: {err}"))
                 })?;
                 let request = Request::new(ReadAllRequest {
@@ -1608,22 +1692,34 @@ impl DiskAPI for RemoteDisk {
                     path: path.to_string(),
                 });
 
-                global_internode_metrics().record_outgoing_request_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+                global_internode_metrics().record_outgoing_request_for_operation_and_backend(
+                    INTERNODE_OPERATION_GRPC_READ_ALL,
+                    INTERNODE_TRANSPORT_BACKEND_GRPC,
+                );
                 let response = match client.read_all(request).await {
                     Ok(response) => response.into_inner(),
                     Err(err) => {
-                        global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+                        global_internode_metrics().record_error_for_operation_and_backend(
+                            INTERNODE_OPERATION_GRPC_READ_ALL,
+                            INTERNODE_TRANSPORT_BACKEND_GRPC,
+                        );
                         return Err(err.into());
                     }
                 };
 
                 if !response.success {
-                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+                    global_internode_metrics().record_error_for_operation_and_backend(
+                        INTERNODE_OPERATION_GRPC_READ_ALL,
+                        INTERNODE_TRANSPORT_BACKEND_GRPC,
+                    );
                     return Err(response.error.unwrap_or_default().into());
                 }
 
-                global_internode_metrics()
-                    .record_recv_bytes_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL, response.data.len());
+                global_internode_metrics().record_recv_bytes_for_operation_and_backend(
+                    INTERNODE_OPERATION_GRPC_READ_ALL,
+                    INTERNODE_TRANSPORT_BACKEND_GRPC,
+                    response.data.len(),
+                );
                 Ok(response.data)
             },
             get_max_timeout_duration(),
@@ -1671,10 +1767,12 @@ impl DiskAPI for RemoteDisk {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::rpc::TcpHttpInternodeDataTransport;
+    use crate::rpc::{InternodeDataTransportCapabilities, TcpHttpInternodeDataTransport};
     use rustfs_common::GLOBAL_CONN_MAP;
-    use std::sync::Once;
-    use tokio::io::duplex;
+    use std::pin::Pin;
+    use std::sync::{Mutex as StdMutex, Once};
+    use std::task::{Context, Poll};
+    use tokio::io::{ReadBuf, duplex};
     use tokio::net::TcpListener;
     use tonic::transport::Endpoint as TonicEndpoint;
     use tracing::Level;
@@ -1682,6 +1780,184 @@ mod tests {
 
     static INIT: Once = Once::new();
 
+    #[derive(Debug, Clone)]
+    enum RecordedTransportCall {
+        Read(ReadStreamRequest),
+        Write(WriteStreamRequest),
+        WalkDir(WalkDirStreamRequest),
+    }
+
+    #[derive(Debug, Clone, Default)]
+    struct RecordingInternodeDataTransport {
+        calls: Arc<StdMutex<Vec<RecordedTransportCall>>>,
+    }
+
+    impl RecordingInternodeDataTransport {
+        fn calls(&self) -> Vec<RecordedTransportCall> {
+            self.calls.lock().expect("recorded transport calls lock poisoned").clone()
+        }
+
+        fn record(&self, call: RecordedTransportCall) {
+            self.calls.lock().expect("recorded transport calls lock poisoned").push(call);
+        }
+    }
+
+    #[derive(Debug)]
+    enum WalkDirTestStep {
+        Error(DiskError),
+        Data(Vec<u8>),
+        PartialDataThenError { data: Vec<u8>, error: io::Error },
+    }
+
+    #[derive(Debug, Clone, Default)]
+    struct RetryingWalkDirInternodeDataTransport {
+        calls: Arc<StdMutex<Vec<RecordedTransportCall>>>,
+        steps: Arc<StdMutex<Vec<WalkDirTestStep>>>,
+    }
+
+    impl RetryingWalkDirInternodeDataTransport {
+        fn with_steps(steps: Vec<WalkDirTestStep>) -> Self {
+            Self {
+                calls: Arc::new(StdMutex::new(Vec::new())),
+                steps: Arc::new(StdMutex::new(steps)),
+            }
+        }
+
+        fn calls(&self) -> Vec<RecordedTransportCall> {
+            self.calls.lock().expect("recorded transport calls lock poisoned").clone()
+        }
+
+        fn record(&self, call: RecordedTransportCall) {
+            self.calls.lock().expect("recorded transport calls lock poisoned").push(call);
+        }
+    }
+
+    #[derive(Debug, Default)]
+    struct EmptyTestReader;
+
+    impl AsyncRead for EmptyTestReader {
+        fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
+
+    #[derive(Debug, Default)]
+    struct SinkTestWriter;
+
+    impl AsyncWrite for SinkTestWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &[u8]) -> Poll<io::Result<usize>> {
+            Poll::Ready(Ok(buf.len()))
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl InternodeDataTransport for RecordingInternodeDataTransport {
+        async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader> {
+            self.record(RecordedTransportCall::Read(request));
+            Ok(Box::new(EmptyTestReader))
+        }
+
+        async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter> {
+            self.record(RecordedTransportCall::Write(request));
+            Ok(Box::new(SinkTestWriter))
+        }
+
+        async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader> {
+            self.record(RecordedTransportCall::WalkDir(request));
+            Ok(Box::new(EmptyTestReader))
+        }
+
+        fn name(&self) -> &'static str {
+            "recording"
+        }
+
+        fn capabilities(&self) -> InternodeDataTransportCapabilities {
+            InternodeDataTransportCapabilities::tcp_http()
+        }
+    }
+
+    #[async_trait::async_trait]
+    impl InternodeDataTransport for RetryingWalkDirInternodeDataTransport {
+        async fn open_read(&self, _request: ReadStreamRequest) -> Result<FileReader> {
+            panic!("open_read should not be used in walk_dir retry test");
+        }
+
+        async fn open_write(&self, _request: WriteStreamRequest) -> Result<FileWriter> {
+            panic!("open_write should not be used in walk_dir retry test");
+        }
+
+        async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader> {
+            self.record(RecordedTransportCall::WalkDir(request));
+            let step = self.steps.lock().expect("walk_dir retry steps lock poisoned").remove(0);
+            match step {
+                WalkDirTestStep::Error(err) => Err(err),
+                WalkDirTestStep::Data(data) => Ok(Box::new(Cursor::new(data))),
+                WalkDirTestStep::PartialDataThenError { data, error } => Ok(Box::new(PartialThenErrorReader {
+                    cursor: Cursor::new(data),
+                    error: Some(error),
+                })),
+            }
+        }
+
+        fn name(&self) -> &'static str {
+            "retrying-walk-dir"
+        }
+
+        fn capabilities(&self) -> InternodeDataTransportCapabilities {
+            InternodeDataTransportCapabilities::tcp_http()
+        }
+    }
+
+    async fn new_remote_disk_with_transport(data_transport: Arc<dyn InternodeDataTransport>) -> RemoteDisk {
+        let endpoint = Endpoint {
+            url: url::Url::parse("http://remote-node:9000/data/rustfs0").unwrap(),
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+        let disk_option = DiskOption {
+            cleanup: false,
+            health_check: false,
+        };
+
+        RemoteDisk::new(&endpoint, &disk_option, data_transport).await.unwrap()
+    }
+
+    #[derive(Debug)]
+    struct PartialThenErrorReader {
+        cursor: Cursor<Vec<u8>>,
+        error: Option<io::Error>,
+    }
+
+    impl AsyncRead for PartialThenErrorReader {
+        fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
+            let filled_before = buf.filled().len();
+            match Pin::new(&mut self.cursor).poll_read(cx, buf) {
+                Poll::Ready(Ok(())) => {
+                    if buf.filled().len() > filled_before {
+                        return Poll::Ready(Ok(()));
+                    }
+
+                    if let Some(err) = self.error.take() {
+                        return Poll::Ready(Err(err));
+                    }
+
+                    Poll::Ready(Ok(()))
+                }
+                other => other,
+            }
+        }
+    }
+
     fn init_tracing(filter_level: Level) {
         INIT.call_once(|| {
             let _ = tracing_subscriber::fmt()
@@ -1827,20 +2103,39 @@ mod tests {
             disk_idx: 0,
         };
 
-        let disk_option = DiskOption {
-            cleanup: false,
-            health_check: true,
-        };
+        temp_env::async_with_vars(
+            [
+                (rustfs_config::ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS, Some("1")),
+                (rustfs_config::ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS, Some("1")),
+            ],
+            async {
+                let disk_option = DiskOption {
+                    cleanup: false,
+                    health_check: true,
+                };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
-            .await
-            .unwrap();
-        remote_disk.enable_health_check();
+                let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+                    .await
+                    .unwrap();
+                remote_disk.enable_health_check();
 
-        // wait for health check connect timeout
-        tokio::time::sleep(Duration::from_secs(6)).await;
-
-        assert!(!remote_disk.is_online().await);
+                // Wait out the initial success-grace window so the active probe loop
+                // actually attempts a connectivity check. Under the new
+                // suspect-first semantics we only need to prove that the drive
+                // transitions away from a clean Online state at least once.
+                tokio::time::sleep(SKIP_IF_SUCCESS_BEFORE + Duration::from_secs(2)).await;
+                assert!(
+                    remote_disk.offline_duration_secs().is_some(),
+                    "missing listener should transition the drive through suspect/offline tracking"
+                );
+                assert_ne!(
+                    remote_disk.runtime_state(),
+                    RuntimeDriveHealthState::Online,
+                    "missing listener should not remain in a clean Online state after probing"
+                );
+            },
+        )
+        .await;
     }
 
     #[tokio::test]
@@ -1925,6 +2220,162 @@ mod tests {
         assert_eq!(remote_disk.disk_ref().await, disk_id.to_string());
     }
 
+    #[tokio::test]
+    async fn test_remote_disk_read_file_stream_uses_configured_data_transport() {
+        let transport = RecordingInternodeDataTransport::default();
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let expected_disk = remote_disk.disk_ref().await;
+
+        let _reader = remote_disk.read_file_stream("bucket", "object/part.1", 7, 11).await.unwrap();
+
+        let calls = transport.calls();
+        assert_eq!(calls.len(), 1);
+        match &calls[0] {
+            RecordedTransportCall::Read(request) => {
+                assert_eq!(request.endpoint, "http://remote-node:9000");
+                assert_eq!(request.disk, expected_disk);
+                assert_eq!(request.volume, "bucket");
+                assert_eq!(request.path, "object/part.1");
+                assert_eq!(request.offset, 7);
+                assert_eq!(request.length, 11);
+            }
+            other => panic!("expected read transport call, got {other:?}"),
+        }
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_create_and_append_file_use_configured_data_transport() {
+        let transport = RecordingInternodeDataTransport::default();
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let expected_disk = remote_disk.disk_ref().await;
+
+        let _created = remote_disk
+            .create_file("orig-bucket", "bucket", "object/part.1", 4096)
+            .await
+            .unwrap();
+        let _appended = remote_disk.append_file("bucket", "object/part.2").await.unwrap();
+
+        let calls = transport.calls();
+        assert_eq!(calls.len(), 2);
+
+        match &calls[0] {
+            RecordedTransportCall::Write(request) => {
+                assert_eq!(request.endpoint, "http://remote-node:9000");
+                assert_eq!(request.disk, expected_disk);
+                assert_eq!(request.volume, "bucket");
+                assert_eq!(request.path, "object/part.1");
+                assert!(!request.append);
+                assert_eq!(request.size, 4096);
+            }
+            other => panic!("expected create write transport call, got {other:?}"),
+        }
+
+        match &calls[1] {
+            RecordedTransportCall::Write(request) => {
+                assert_eq!(request.endpoint, "http://remote-node:9000");
+                assert_eq!(request.disk, expected_disk);
+                assert_eq!(request.volume, "bucket");
+                assert_eq!(request.path, "object/part.2");
+                assert!(request.append);
+                assert_eq!(request.size, 0);
+            }
+            other => panic!("expected append write transport call, got {other:?}"),
+        }
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_uses_configured_data_transport() {
+        let transport = RecordingInternodeDataTransport::default();
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let expected_disk = remote_disk.disk_ref().await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "prefix".to_string(),
+            recursive: true,
+            report_notfound: false,
+            filter_prefix: Some("part".to_string()),
+            forward_to: None,
+            limit: 10,
+            disk_id: String::new(),
+        };
+        let expected_body = serde_json::to_vec(&opts).unwrap();
+        let mut writer = Vec::new();
+
+        remote_disk.walk_dir(opts, &mut writer).await.unwrap();
+
+        let calls = transport.calls();
+        assert_eq!(calls.len(), 1);
+        match &calls[0] {
+            RecordedTransportCall::WalkDir(request) => {
+                assert_eq!(request.endpoint, "http://remote-node:9000");
+                assert_eq!(request.disk, expected_disk);
+                assert_eq!(request.body, expected_body);
+                assert_eq!(request.stall_timeout, Some(get_drive_walkdir_stall_timeout()));
+            }
+            other => panic!("expected walk-dir transport call, got {other:?}"),
+        }
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_retries_once_on_retryable_transport_error() {
+        let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![
+            WalkDirTestStep::Error(DiskError::other("HttpReader stream error: error decoding response body")),
+            WalkDirTestStep::Data(b"walk-dir-retry-ok".to_vec()),
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "config/iam".to_string(),
+            recursive: true,
+            report_notfound: false,
+            filter_prefix: None,
+            forward_to: None,
+            limit: 10,
+            disk_id: String::new(),
+        };
+        let mut writer = Vec::new();
+
+        remote_disk
+            .walk_dir(opts, &mut writer)
+            .await
+            .expect("retryable walk_dir error should recover");
+
+        assert_eq!(writer, b"walk-dir-retry-ok");
+        assert_eq!(transport.calls().len(), 2, "walk_dir should retry exactly once");
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_does_not_retry_after_partial_stream_failure() {
+        let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![
+            WalkDirTestStep::PartialDataThenError {
+                data: b"partial-walk-dir".to_vec(),
+                error: io::Error::new(io::ErrorKind::ConnectionReset, "connection reset"),
+            },
+            WalkDirTestStep::Data(b"walk-dir-retry-ok".to_vec()),
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "config/iam".to_string(),
+            recursive: true,
+            report_notfound: false,
+            filter_prefix: None,
+            forward_to: None,
+            limit: 10,
+            disk_id: String::new(),
+        };
+        let mut writer = Vec::new();
+
+        let err = remote_disk
+            .walk_dir(opts, &mut writer)
+            .await
+            .expect_err("partial stream failure should be returned without retry");
+
+        assert!(matches!(err, DiskError::Io(ref io_err) if io_err.kind() == io::ErrorKind::ConnectionReset));
+        assert_eq!(writer, b"partial-walk-dir");
+        assert_eq!(transport.calls().len(), 1, "walk_dir should not retry after writing partial bytes");
+    }
+
     #[tokio::test]
     async fn test_remote_disk_endpoints_with_different_schemes() {
         let test_cases = vec![
@@ -2064,7 +2515,12 @@ mod tests {
             .expect_err("timeout should fail");
 
         assert!(err.to_string().contains("timeout"));
-        assert!(!remote_disk.is_online().await, "remote disk should be marked faulty after timeout");
+        assert!(remote_disk.is_online().await, "first timeout should keep the remote disk online");
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first timeout should move the remote disk into suspect state"
+        );
     }
 
     #[tokio::test]
@@ -2230,7 +2686,15 @@ mod tests {
             },
             std::io::ErrorKind::TimedOut
         );
-        assert!(!remote_disk.is_online().await, "timeout-like errors should mark remote disk faulty");
+        assert!(
+            remote_disk.is_online().await,
+            "first timeout-like error should keep the remote disk online"
+        );
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first timeout-like error should move the remote disk into suspect state"
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "timeout-like errors should evict cached connection"
@@ -2283,7 +2747,15 @@ mod tests {
             },
             std::io::ErrorKind::ConnectionRefused
         );
-        assert!(!remote_disk.is_online().await, "network-like errors should mark remote disk faulty");
+        assert!(
+            remote_disk.is_online().await,
+            "first network-like error should keep the remote disk online"
+        );
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first network-like error should move the remote disk into suspect state"
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "network-like errors should evict cached connection"

crates/ecstore/src/rpc/remote_locker.rs

@@ -14,18 +14,21 @@
 
 use crate::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
 use async_trait::async_trait;
+use bytes::Bytes;
 use rustfs_lock::{
     LockClient, LockError, LockInfo, LockRequest, LockResponse, LockStats, LockStatus, LockType, Result,
     types::{LockId, LockMetadata, LockPriority},
 };
 use rustfs_protos::proto_gen::node_service::{BatchGenerallyLockRequest, GenerallyLockRequest, PingRequest};
-use rustfs_protos::{evict_failed_connection, proto_gen::node_service::node_service_client::NodeServiceClient};
+use rustfs_protos::{
+    evict_failed_connection, models::PingBodyBuilder, proto_gen::node_service::node_service_client::NodeServiceClient,
+};
 use std::time::Duration;
 use tokio::time::timeout;
 use tonic::Request;
 use tonic::service::interceptor::InterceptedService;
 use tonic::transport::Channel;
-use tracing::{info, warn};
+use tracing::{debug, info, warn};
 
 /// Remote lock client implementation
 #[derive(Debug, Clone)]
@@ -42,6 +45,20 @@ impl RemoteClient {
         Self { addr: url.to_string() }
     }
 
+    fn build_ping_request() -> PingRequest {
+        let mut fbb = flatbuffers::FlatBufferBuilder::new();
+        let payload = fbb.create_vector(b"health-check");
+        let mut builder = PingBodyBuilder::new(&mut fbb);
+        builder.add_payload(payload);
+        let root = builder.finish();
+        fbb.finish(root, None);
+
+        PingRequest {
+            version: 1,
+            body: Bytes::copy_from_slice(fbb.finished_data()),
+        }
+    }
+
     /// Create a minimal LockRequest for unlock operations using only lock_id
     fn create_unlock_request(lock_id: &LockId) -> LockRequest {
         LockRequest {
@@ -64,16 +81,44 @@ impl RemoteClient {
             .map_err(|err| LockError::internal(format!("can not get client, err: {err}")))
     }
 
-    async fn evict_connection(&self, op: &'static str, reason: &str) {
-        warn!(
-            addr = %self.addr,
-            op,
-            reason,
-            "Evicting cached remote lock connection after RPC failure"
-        );
+    fn is_scanner_leader_lock(resource_summary: &str) -> bool {
+        resource_summary == ".rustfs.sys/leader.lock@latest"
+    }
+
+    async fn evict_connection(&self, op: &'static str, reason: &str, resource_summary: &str) {
+        if Self::is_scanner_leader_lock(resource_summary) {
+            debug!(
+                addr = %self.addr,
+                op,
+                reason,
+                resource_summary,
+                "Evicting cached remote lock connection for scanner leader-lock RPC failure"
+            );
+        } else {
+            warn!(
+                addr = %self.addr,
+                op,
+                reason,
+                resource_summary,
+                "Evicting cached remote lock connection after RPC failure"
+            );
+        }
         evict_failed_connection(&self.addr).await;
     }
 
+    fn summarize_resources(requests: &[LockRequest]) -> String {
+        const LIMIT: usize = 3;
+        let mut resources = requests
+            .iter()
+            .take(LIMIT)
+            .map(|request| request.resource.to_string())
+            .collect::<Vec<_>>();
+        if requests.len() > LIMIT {
+            resources.push(format!("... (+{} more)", requests.len() - LIMIT));
+        }
+        resources.join(", ")
+    }
+
     fn rpc_timeout(timeout_duration: Duration) -> Duration {
         if timeout_duration.is_zero() {
             Duration::from_millis(1)
@@ -86,39 +131,74 @@ impl RemoteClient {
         &self,
         op: &'static str,
         timeout_duration: Duration,
+        resource_summary: &str,
         future: F,
     ) -> std::result::Result<T, LockError>
     where
         F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
     {
-        let timeout_duration = Self::rpc_timeout(timeout_duration);
-        match timeout(timeout_duration, future).await {
+        let lock_timeout = Self::rpc_timeout(timeout_duration);
+        match timeout(lock_timeout, future).await {
             Ok(Ok(response)) => Ok(response),
             Ok(Err(err)) => {
                 let reason = err.to_string();
-                self.evict_connection(op, &reason).await;
+                if Self::is_scanner_leader_lock(resource_summary) {
+                    debug!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        tonic_code = ?err.code(),
+                        tonic_message = err.message(),
+                        "Remote lock RPC returned tonic error for scanner leader lock"
+                    );
+                } else {
+                    warn!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        tonic_code = ?err.code(),
+                        tonic_message = err.message(),
+                        "Remote lock RPC returned tonic error"
+                    );
+                }
+                self.evict_connection(op, &reason, resource_summary).await;
                 Err(LockError::internal(format!("{op} RPC failed: {reason}")))
             }
             Err(_) => {
-                let reason = format!("RPC timed out after {:?}", timeout_duration);
-                self.evict_connection(op, &reason).await;
-                Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), timeout_duration))
+                let reason = format!("RPC timed out after {:?}", lock_timeout);
+                if Self::is_scanner_leader_lock(resource_summary) {
+                    debug!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        "Remote lock RPC timed out for scanner leader lock"
+                    );
+                } else {
+                    warn!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        "Remote lock RPC timed out"
+                    );
+                }
+                self.evict_connection(op, &reason, resource_summary).await;
+                Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), lock_timeout))
             }
         }
     }
 
-    fn timeout_failure_response(request: &LockRequest) -> LockResponse {
-        LockResponse::failure("Lock acquisition timeout", request.acquire_timeout)
+    fn rpc_timeout_failure_response(request: &LockRequest, err: &LockError) -> LockResponse {
+        LockResponse::failure(format!("Remote lock RPC timed out: {err}"), request.acquire_timeout)
     }
 
     fn rpc_failure_response(_request: &LockRequest, err: &LockError) -> LockResponse {
         LockResponse::failure(format!("Remote lock RPC failed: {err}"), Duration::ZERO)
     }
 
-    fn timeout_failure_batch(requests: &[LockRequest]) -> Vec<LockResponse> {
-        requests.iter().map(Self::timeout_failure_response).collect()
-    }
-
     fn rpc_failure_batch(requests: &[LockRequest], err: &LockError) -> Vec<LockResponse> {
         requests
             .iter()
@@ -126,6 +206,13 @@ impl RemoteClient {
             .collect()
     }
 
+    fn rpc_timeout_failure_batch(requests: &[LockRequest], err: &LockError) -> Vec<LockResponse> {
+        requests
+            .iter()
+            .map(|request| Self::rpc_timeout_failure_response(request, err))
+            .collect()
+    }
+
     fn batch_rpc_timeout(requests: &[LockRequest]) -> Duration {
         requests
             .iter()
@@ -179,14 +266,18 @@ impl LockClient for RemoteClient {
     async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
         info!("remote acquire_exclusive for {}", request.resource);
         let mut client = self.get_client().await?;
+        let resource_summary = request.resource.to_string();
         let req = Request::new(GenerallyLockRequest {
             args: serde_json::to_string(&request)
                 .map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
         });
 
-        let resp = match self.execute_rpc("lock", request.acquire_timeout, client.lock(req)).await {
+        let resp = match self
+            .execute_rpc("lock", request.acquire_timeout, &resource_summary, client.lock(req))
+            .await
+        {
             Ok(resp) => resp.into_inner(),
-            Err(LockError::Timeout { .. }) => return Ok(Self::timeout_failure_response(request)),
+            Err(err @ LockError::Timeout { .. }) => return Ok(Self::rpc_timeout_failure_response(request, &err)),
             Err(err) => return Ok(Self::rpc_failure_response(request, &err)),
         };
 
@@ -212,6 +303,7 @@ impl LockClient for RemoteClient {
         }
 
         let mut client = self.get_client().await?;
+        let resource_summary = Self::summarize_resources(requests);
         let req = Request::new(BatchGenerallyLockRequest {
             args: requests
                 .iter()
@@ -222,11 +314,11 @@ impl LockClient for RemoteClient {
         });
 
         let resp = match self
-            .execute_rpc("lock_batch", Self::batch_rpc_timeout(requests), client.lock_batch(req))
+            .execute_rpc("lock_batch", Self::batch_rpc_timeout(requests), &resource_summary, client.lock_batch(req))
             .await
         {
             Ok(resp) => resp.into_inner(),
-            Err(LockError::Timeout { .. }) => return Ok(Self::timeout_failure_batch(requests)),
+            Err(err @ LockError::Timeout { .. }) => return Ok(Self::rpc_timeout_failure_batch(requests, &err)),
             Err(err) => return Ok(Self::rpc_failure_batch(requests, &err)),
         };
 
@@ -435,10 +527,7 @@ impl LockClient for RemoteClient {
             }
         };
 
-        let ping_req = Request::new(PingRequest {
-            version: 1,
-            body: bytes::Bytes::new(),
-        });
+        let ping_req = Request::new(Self::build_ping_request());
 
         match client.ping(ping_req).await {
             Ok(_) => {
@@ -511,7 +600,14 @@ mod tests {
             "remote lock RPC should honor request timeout"
         );
         assert!(!response.success, "timed out lock acquisition should fail");
-        assert_eq!(response.error.as_deref(), Some("Lock acquisition timeout"));
+        assert!(
+            response
+                .error
+                .as_deref()
+                .is_some_and(|error| error.contains("Remote lock RPC timed out")),
+            "expected remote RPC timeout marker, got {:?}",
+            response.error
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "timeout should evict cached connection"
@@ -539,7 +635,14 @@ mod tests {
         );
         assert_eq!(responses.len(), 1);
         assert!(!responses[0].success, "timed out batch lock acquisition should fail");
-        assert_eq!(responses[0].error.as_deref(), Some("Lock acquisition timeout"));
+        assert!(
+            responses[0]
+                .error
+                .as_deref()
+                .is_some_and(|error| error.contains("Remote lock RPC timed out")),
+            "expected remote RPC timeout marker, got {:?}",
+            responses[0].error
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "batch timeout should evict cached connection"

crates/ecstore/src/set_disk/heal.rs

@@ -39,12 +39,12 @@ impl SetDisks {
 
         let write_lock_guard = if !opts.no_lock {
             let ns_lock = self.new_ns_lock(bucket, object).await?;
-            Some(ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| {
-                StorageError::other(format!(
-                    "Failed to acquire write lock: {}",
-                    self.format_lock_error_from_error(bucket, object, "write", &e)
-                ))
-            })?)
+            Some(
+                ns_lock
+                    .get_write_lock(get_lock_acquire_timeout())
+                    .await
+                    .map_err(|e| self.map_namespace_lock_error(bucket, object, "write", e))?,
+            )
         } else {
             None
         };
@@ -350,15 +350,6 @@ impl SetDisks {
 
                             for (part_index, part) in latest_meta.parts.iter().enumerate() {
                                 let till_offset = erasure.shard_file_offset(0, part.size, part.size);
-                                let checksum_info = erasure_info.get_checksum_info(part.number);
-                                let checksum_algo = if latest_meta.uses_legacy_checksum
-                                    && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S
-                                {
-                                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
-                                } else {
-                                    checksum_info.algorithm
-                                };
-
                                 // Read zero-copy configuration from environment variable
                                 // Default: enabled (true) for performance
                                 let use_zero_copy =
@@ -382,6 +373,15 @@ impl SetDisks {
                                     }
 
                                     if let (Some(disk), Some(metadata)) = (disk, &copy_parts_metadata[index]) {
+                                        let checksum_info = metadata.erasure.get_checksum_info(part.number);
+                                        let checksum_algo = if metadata.uses_legacy_checksum
+                                            && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S
+                                        {
+                                            rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
+                                        } else {
+                                            checksum_info.algorithm
+                                        };
+
                                         match create_bitrot_reader(
                                             metadata.data.as_deref(),
                                             Some(disk),
@@ -417,7 +417,7 @@ impl SetDisks {
                                 }
 
                                 let is_inline_buffer = {
-                                    if let Some(sc) = GLOBAL_STORAGE_CLASS.get() {
+                                    if let Some(sc) = get_global_storage_class() {
                                         sc.should_inline(erasure.shard_file_size(latest_meta.size), false)
                                     } else {
                                         false
@@ -713,13 +713,7 @@ impl SetDisks {
             .await?
             .get_write_lock(get_lock_acquire_timeout())
             .await
-            .map_err(|e| {
-                let message = format!(
-                    "Failed to acquire write lock: {}",
-                    self.format_lock_error_from_error(bucket, object, "write", &e)
-                );
-                DiskError::other(message)
-            })?;
+            .map_err(|e| DiskError::other(self.map_namespace_lock_error(bucket, object, "write", e).to_string()))?;
 
         self.heal_object_dir_locked(bucket, object, dry_run, remove).await
     }

crates/ecstore/src/set_disk/lock.rs

@@ -50,6 +50,30 @@ impl SetDisks {
         }
     }
 
+    pub(super) fn map_namespace_lock_error(
+        &self,
+        bucket: &str,
+        object: &str,
+        mode: &'static str,
+        err: rustfs_lock::error::LockError,
+    ) -> StorageError {
+        match err {
+            rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
+                StorageError::NamespaceLockQuorumUnavailable {
+                    mode,
+                    bucket: bucket.to_string(),
+                    object: object.to_string(),
+                    required,
+                    achieved,
+                }
+            }
+            other => StorageError::other(format!(
+                "Failed to acquire {mode} lock: {}",
+                self.format_lock_error_from_error(bucket, object, mode, &other)
+            )),
+        }
+    }
+
     pub(super) async fn get_disks_internal(&self) -> Vec<Option<DiskStore>> {
         let rl = self.disks.read().await;
 

crates/ecstore/src/set_disk/read.rs

@@ -144,18 +144,24 @@ impl SetDisks {
         for (part_idx, part_info) in part_meta_paths.iter().enumerate() {
             let mut part_meta_quorum = HashMap::new();
             let mut part_infos = Vec::new();
-            for (j, parts) in object_parts.iter().enumerate() {
+            let mut present_count = 0usize;
+            let mut missing_or_empty_count = 0usize;
+            let mut mismatched_response_count = 0usize;
+            for parts in object_parts.iter() {
                 if parts.len() != part_meta_paths.len() {
+                    mismatched_response_count += 1;
                     *part_meta_quorum.entry(part_info.clone()).or_insert(0) += 1;
                     continue;
                 }
 
                 if !parts[part_idx].etag.is_empty() {
+                    present_count += 1;
                     *part_meta_quorum.entry(parts[part_idx].etag.clone()).or_insert(0) += 1;
                     part_infos.push(parts[part_idx].clone());
                     continue;
                 }
 
+                missing_or_empty_count += 1;
                 *part_meta_quorum.entry(part_info.clone()).or_insert(0) += 1;
             }
 
@@ -194,6 +200,22 @@ impl SetDisks {
             {
                 ret[part_idx] = found.clone();
             } else {
+                if issue3031_diag_enabled() {
+                    warn!(
+                        target: "rustfs_ecstore::set_disk",
+                        bucket = %bucket,
+                        part_meta_path = %part_info,
+                        part_id = part_numbers[part_idx],
+                        read_quorum = read_quorum,
+                        max_quorum = max_quorum,
+                        disk_response_count = object_parts.len(),
+                        present_count = present_count,
+                        missing_or_empty_count = missing_or_empty_count,
+                        mismatched_response_count = mismatched_response_count,
+                        max_vote_is_missing_marker = max_etag.map(|etag| etag == part_info).unwrap_or(false),
+                        "issue3031_read_parts_part_quorum"
+                    );
+                }
                 ret[part_idx] = ObjectPartInfo {
                     number: part_numbers[part_idx],
                     error: Some(format!("part.{} not found", part_numbers[part_idx])),