feat(scanner): add runtime scanner controls and status (#3203)

* feat(scanner): add runtime scanner controls and status

* fix(scanner): validate persisted scanner config

* docs(scanner): clarify start delay behavior

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

.agents/skills/code-change-verification/SKILL.md

@@ -41,6 +41,19 @@ Use this skill to review code changes consistently before merge, before release,
   - hidden coupling to shared helpers/constants/features
 - If a point is uncertain, mark it as an open question instead of guessing.
 
+#### Rust-specific checks (apply to all Rust changes)
+
+- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
+- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
+- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
+- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
+- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
+- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
+- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
+- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
+- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
+- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
+
 ### 4) Findings-first output
 - Order findings by severity:
   - P0: critical failure, security breach, or data loss risk

.agents/skills/rust-code-quality/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: rust-code-quality
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+---
+
+# Rust Code Quality Gate
+
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
+
+## Quick Start
+
+1. Identify changed `.rs` files.
+2. Run automated checks on changed files.
+3. Run manual review checklist on the diff.
+4. Report findings; block merge if P0/P1 issues exist.
+
+## Automated Checks
+
+Run these on every changed `.rs` file (excluding test modules):
+
+```bash
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
+
+# 2. Silent type truncation via `as` cast
+rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
+
+# 3. String as error type
+rg -n 'Result<.*String>' <changed-files> | grep -v test
+
+# 4. Box<dyn Error> in public APIs
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
+
+# 5. println/eprintln in production
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
+
+# 6. Ordering::Relaxed usage (verify each is intentional)
+rg -n 'Ordering::Relaxed' <changed-files>
+```
+
+## Manual Review Checklist
+
+For every Rust code change, verify:
+
+### Error Handling
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
+- [ ] No `Result<_, String>` in public API signatures
+- [ ] No `Box<dyn Error>` in public trait/struct methods
+- [ ] `Error::source()` is overridden when inner error is stored
+- [ ] Error messages are actionable (what failed, with what input)
+
+### Type Safety
+- [ ] No silent `as` truncation (negative→unsigned, large→small)
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
+
+### Concurrency
+- [ ] Lock acquisition order is documented when multiple locks are used
+- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
+- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
+
+### Memory and Performance
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
+
+### Recursion Safety
+- [ ] Recursive functions have a depth limit or use iterative traversal
+- [ ] Tree/cache traversals handle corrupted/cyclic input safely
+
+### Testing
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
+
+### Serde
+- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
+- [ ] `#[serde(default)]` not used on security-critical fields without validation
+
+### Code Hygiene
+- [ ] No `#![allow(dead_code)]` at crate root
+- [ ] No camelCase statics or Hungarian notation
+- [ ] New string literals don't duplicate existing constants
+
+## Severity Classification
+
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
+
+## Output Template
+
+```
+## Rust Code Quality Report
+
+### Automated Scan
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
+
+### Findings
+- [P1] `path:line` — description
+  - Fix: ...
+  - Validation: ...
+
+### Verdict
+PASS / BLOCKED (list blocking findings)
+```

.agents/skills/rust-code-quality/references/checklist.md

@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```

.agents/skills/security-advisory-lessons/SKILL.md

@@ -70,6 +70,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
 - Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
 - Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
 
 ### Secrets, default credentials, and crypto
 - Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
@@ -86,6 +87,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
 - Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
 - RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
 
 ### Browser, CORS, and console surfaces
 - Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
@@ -115,5 +117,6 @@ Use these prompts while reviewing a diff:
 - Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
 - Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -38,6 +38,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
 - `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
 
 ### Secrets, defaults, and cryptographic misuse
 
@@ -55,6 +56,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
 - `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
 
 ### Browser, CORS, and console isolation
 
@@ -69,6 +71,13 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
+### Serde deserialization and input validation
+
+- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
+- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
+- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
+- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
+
 ## Useful Search Seeds
 
 Use these targeted searches when a diff touches security-sensitive code:
@@ -76,10 +85,12 @@ Use these targeted searches when a diff touches security-sensitive code:
 ```bash
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
 ## Minimum Regression Test Expectations
@@ -87,7 +98,8 @@ rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-
 - Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.docker/compose/docker-compose.cluster.local-build.yml

@@ -23,13 +23,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -53,13 +55,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -83,13 +87,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -113,13 +119,15 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
       - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
       - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"

.docker/compose/docker-compose.cluster.yaml

@@ -21,8 +21,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
-      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9003:9000" # Map port 9004 of the host to port 9000 of the container

.docker/observability/grafana/dashboards/rustfs.json

@@ -4551,7 +4551,7 @@
                   "index": 0,
                   "text": "INACTIVE"
                 },
-                "to": 1e-09
+                "to": 1e-9
               },
               "type": "range"
             }
@@ -5281,7 +5281,10 @@
       "id": 102,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5368,7 +5371,10 @@
       "id": 103,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5455,7 +5461,10 @@
       "id": 104,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5533,7 +5542,9 @@
       "id": 105,
       "options": {
         "legend": {
-          "calcs": ["mean"],
+          "calcs": [
+            "mean"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5629,7 +5640,10 @@
       "id": 106,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5725,7 +5739,10 @@
       "id": 107,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5803,7 +5820,10 @@
       "id": 108,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -8071,6 +8091,374 @@
         "x": 0,
         "y": 195
       },
+      "id": 150,
+      "panels": [],
+      "title": "Object Lock Diagnostics",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 196
+      },
+      "id": 151,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_acquire_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Acquires",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 196
+      },
+      "id": 152,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_hold_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Holds",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 204
+      },
+      "id": 153,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_acquire_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Acquire Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 204
+      },
+      "id": 154,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_hold_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Hold Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [
+            {
+              "options": {
+                "0": {
+                  "text": "Disabled"
+                },
+                "1": {
+                  "text": "Enabled"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": 0
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 212
+      },
+      "id": 155,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "percentChangeColorMode": "standard",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max(rustfs_object_lock_diag_enabled{job=~\"$job\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Diagnostics Enabled",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 213
+      },
       "id": 137,
       "panels": [],
       "title": "Debug / Raw Explorer",
@@ -8138,7 +8526,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 196
+        "y": 214
       },
       "id": 138,
       "options": {
@@ -8235,7 +8623,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 196
+        "y": 214
       },
       "id": 139,
       "options": {
@@ -8332,7 +8720,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 204
+        "y": 222
       },
       "id": 140,
       "options": {
@@ -8429,7 +8817,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 204
+        "y": 222
       },
       "id": 141,
       "options": {
@@ -8526,7 +8914,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 212
+        "y": 230
       },
       "id": 142,
       "options": {
@@ -8623,7 +9011,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 212
+        "y": 230
       },
       "id": 143,
       "options": {
@@ -8720,7 +9108,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 220
+        "y": 238
       },
       "id": 144,
       "options": {
@@ -8817,7 +9205,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 220
+        "y": 238
       },
       "id": 145,
       "options": {
@@ -8914,7 +9302,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 228
+        "y": 246
       },
       "id": 146,
       "options": {
@@ -9011,7 +9399,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 228
+        "y": 246
       },
       "id": 147,
       "options": {
@@ -9108,7 +9496,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 236
+        "y": 254
       },
       "id": 148,
       "options": {
@@ -9205,7 +9593,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 236
+        "y": 254
       },
       "id": 149,
       "options": {
@@ -9382,5 +9770,5 @@
   "timezone": "browser",
   "title": "RustFS",
   "uid": "rustfs-s3",
-  "version": 13
+  "version": 14
 }

.github/actions/setup/action.yml

@@ -54,6 +54,7 @@ runs:
           build-essential \
           pkg-config \
           libssl-dev \
+          unzip \
           protobuf-compiler
 
     - name: Install protoc
@@ -83,7 +84,7 @@ runs:
       uses: taiki-e/install-action@cargo-zigbuild
 
     - name: Install cargo-nextest
-      uses: taiki-e/install-action@cargo-nextest
+      uses: taiki-e/install-action@nextest
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@v2

.github/dependabot.yml

@@ -40,6 +40,8 @@ updates:
         versions: [ "1.x" ]
       - dependency-name: "ratelimit"
         versions: [ "2.x" ]
+      - dependency-name: "pyroscope"
+        versions: [ "2.x" ]
     groups:
       s3s:
         update-types:

.github/s3tests/s3tests.conf

@@ -11,8 +11,8 @@
 # host set for RustFS - will be substituted via envsubst
 host = ${S3_HOST}
 
-# port for RustFS
-port = 9000
+# port for RustFS - will be substituted via envsubst
+port = ${S3_PORT}
 
 ## say "False" to disable TLS
 is_secure = False

.github/workflows/audit.yml

@@ -40,7 +40,7 @@ env:
 jobs:
   security-audit:
     name: Security Audit
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 15
     steps:
       - name: Checkout repository
@@ -65,7 +65,7 @@ jobs:
 
   dependency-review:
     name: Dependency Review
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: github.event_name == 'pull_request'
     permissions:
       contents: read

.github/workflows/build.yml

@@ -73,7 +73,7 @@ jobs:
   # Build strategy check - determine build type based on trigger
   build-check:
     name: Build Strategy Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       build_type: ${{ steps.check.outputs.build_type }}
@@ -146,7 +146,7 @@ jobs:
   # Build RustFS binaries
   prepare-platform-matrix:
     name: Prepare Platform Matrix
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
       selected: ${{ steps.select.outputs.selected }}
@@ -169,7 +169,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":true,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -549,7 +549,7 @@ jobs:
     name: Build Summary
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Build completion summary
         shell: bash
@@ -601,7 +601,7 @@ jobs:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     outputs:
@@ -687,7 +687,7 @@ jobs:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
       actions: read
@@ -768,7 +768,7 @@ jobs:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/')
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Update latest.json
         env:
@@ -793,9 +793,10 @@ jobs:
 
           curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
           unzip "$OSSUTIL_ZIP"
-          mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+          OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
+          mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
           rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-          chmod +x /usr/local/bin/ossutil
+          chmod +x "$OSSUTIL_BIN"
 
           # Create latest.json
           cat > latest.json << EOF
@@ -809,7 +810,7 @@ jobs:
           EOF
 
           # Upload to OSS
-          ossutil cp latest.json oss://rustfs-version/latest.json --force
+          "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
 
           echo "✅ Updated latest.json for stable release $VERSION"
 
@@ -818,7 +819,7 @@ jobs:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     steps:

.github/workflows/ci.yml

@@ -78,7 +78,7 @@ jobs:
     permissions:
       actions: write
       contents: read
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_skip: ${{ steps.skip_check.outputs.should_skip }}
     steps:
@@ -95,7 +95,7 @@ jobs:
     name: Typos
     needs: skip-check
     if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
       - uses: dtolnay/rust-toolchain@stable
@@ -182,11 +182,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v6
 
-      - name: Clean up previous test run
-        run: |
-          rm -rf /tmp/rustfs
-          rm -f /tmp/rustfs.log
-
       - name: Download debug binary
         uses: actions/download-artifact@v7
         with:
@@ -209,14 +204,19 @@ jobs:
       - name: Run end-to-end tests
         run: |
           s3s-e2e --version
-          ./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          RUSTFS_TEST_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
+          RUSTFS_TEST_PORT="${RUSTFS_TEST_PORT}" \
+          RUSTFS_TEST_LOG="${RUN_ROOT}/rustfs.log" \
+          ./scripts/e2e-run.sh ./target/debug/rustfs "${RUN_ROOT}/data"
 
       - name: Upload test logs
         if: failure()
         uses: actions/upload-artifact@v6
         with:
           name: e2e-test-logs-${{ github.run_number }}
-          path: /tmp/rustfs.log
+          path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
   s3-implemented-tests:
@@ -240,10 +240,16 @@ jobs:
 
       - name: Run implemented s3-tests
         run: |
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
           DEPLOY_MODE=binary \
           RUSTFS_BINARY=./target/debug/rustfs \
           TEST_MODE=single \
           MAXFAIL=1 \
+          S3_PORT="${S3_PORT}" \
+          DATA_ROOT="${RUN_ROOT}" \
+          S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
           ./scripts/s3-tests/run.sh
 
       - name: Upload s3 test artifacts

.github/workflows/docker.yml

@@ -73,7 +73,7 @@ jobs:
   # Check if we should build Docker images
   build-check:
     name: Docker Build Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       should_push: ${{ steps.check.outputs.should_push }}
@@ -421,7 +421,7 @@ jobs:
     name: Docker Build Summary
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Docker build completion summary
         run: |

.github/workflows/e2e-s3tests.yml

@@ -40,8 +40,8 @@ on:
 
 env:
   # main user
-  S3_ACCESS_KEY: rustfs-ci-admin
-  S3_SECRET_KEY: rustfs-ci-secret
+  S3_ACCESS_KEY: rustfsadmin
+  S3_SECRET_KEY: rustfsadmin
   # alt user (must be different from main for many s3-tests)
   S3_ALT_ACCESS_KEY: rustfsalt
   S3_ALT_SECRET_KEY: rustfsalt

.github/workflows/helm-package.yml

@@ -31,7 +31,7 @@ permissions:
 
 jobs:
   build-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: |
       github.event_name == 'workflow_dispatch' ||
       (
@@ -99,7 +99,7 @@ jobs:
           retention-days: 1
 
   publish-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     needs: [ build-helm-package ]
     if: needs.build-helm-package.result == 'success'
 

.github/workflows/issue-translator.yml

@@ -25,7 +25,7 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubicloud-standard-4
+    runs-on: ubuntu-latest
     steps:
       - uses: usthe/issues-translate-action@v2.7
         with:

.github/workflows/nix.yml

@@ -38,41 +38,65 @@ concurrency:
   cancel-in-progress: true
 permissions:
   contents: read
+  id-token: write
 
 jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
     timeout-minutes: 60
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    permissions:
+      contents: read
+      id-token: write
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
 
       - name: Install Nix
-        uses: cachix/install-nix-action@v31
+        uses: DeterminateSystems/determinate-nix-action@v3.21.0
         with:
-          github_access_token: ${{ secrets.GITHUB_TOKEN }}
-          extra_nix_config: |
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          extra-conf: |
             experimental-features = nix-command flakes
+            max-jobs = 1
 
-      - name: Setup Magic Nix Cache
-        uses: DeterminateSystems/magic-nix-cache-action@v13
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@v3.21.0
 
-      - name: Setup Flake Checker
+      - name: Check Nix Flake Inputs
         uses: DeterminateSystems/flake-checker-action@v12
+        with:
+          fail-mode: true
+          ignore-missing-flake-lock: false
 
       - name: Verify Flake
         run: |
           echo "Checking flake structure and evaluation..."
           nix flake show
-          nix flake check --print-build-logs
+          for attempt in 1 2 3; do
+            if nix flake check --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix flake check failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Build RustFS
         run: |
           echo "Building the default package..."
-          nix build .#default --print-build-logs
+          for attempt in 1 2 3; do
+            if nix build .#default --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix build failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Test Binary
         run: |

AGENTS.md

@@ -103,6 +103,18 @@ Do not open a PR with code changes when the required checks fail.
 - Use environment variables or vault tooling for sensitive configuration.
 - For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
+## Serde Safety
+
+- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs).
+- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
+- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
+
+## Naming Conventions
+
+- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
+- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
+- If existing code violates naming conventions, do not widen the violation in new code. Fix opportunistically when touching the surrounding area.
+
 ## Scoped Guidance in This Repository
 
 - `.github/AGENTS.md`

Cargo.toml

@@ -60,7 +60,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.95.0"
-version = "1.0.0-beta.4"
+version = "1.0.0-beta.7"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -77,67 +77,67 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.4" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.4" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.4" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.4" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.4" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.4" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.4" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.4" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.4" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.4" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.4" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.4" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.4" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.4" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.4" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.4" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.4" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.4" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.4" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.4" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.4" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.4" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.4" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.4" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.4" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.4" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.4" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.4" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.4" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.4" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.4" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.4" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.4" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.4" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.4" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.4" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.4" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.7" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.7" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.7" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.7" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.7" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.7" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.7" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.7" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.7" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.7" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.7" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.7" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.7" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.7" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.7" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.7" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.7" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.7" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.7" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.7" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.7" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.7" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.7" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.7" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.7" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.7" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.7" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.7" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.7" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.7" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.7" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.7" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.7" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.7" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.7" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.7" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.7" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "2bad388283bc3ce48801fc2ffcd22445eb6f3d24" }
+mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-async-nats = "0.48.0"
+async-nats = "0.49.0"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
-pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
+pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime"] }
 lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
-hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
+hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
-http = "1.4.0"
+http = "1.4.1"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
-reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
-socket2 = { version = "0.6.3", features = ["all"] }
+socket2 = { version = "0.6.4", features = ["all"] }
 tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
@@ -164,7 +164,7 @@ serde_json = { version = "1.0.150", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
-aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
+aes-gcm = { version = "0.11.0-rc.4", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.8" }
 blake2 = "0.11.0-rc.6"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
@@ -185,7 +185,7 @@ zeroize = { version = "1.8.2", features = ["derive"] }
 # Time and Date
 chrono = { version = "0.4.44", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.24", features = ["serde"] }
+jiff = { version = "0.2.28", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Database
@@ -199,14 +199,14 @@ arc-swap = "1.9.1"
 astral-tokio-tar = "0.6.2"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.17" }
+aws-config = { version = "1.8.18" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.133.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.8" }
+aws-sdk-s3 = { version = "1.135.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.13", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.9" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
-brotli = "8.0.2"
+brotli = "8.0.3"
 clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
@@ -256,21 +256,21 @@ reed-solomon-erasure = { version = "6.0", default-features = false, features = [
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
 rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
-redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
+redis = { version = "1.2.2", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
 s3s = { git = "https://github.com/rustfs/s3s", rev = "507e1312b211c3ddc214b03875d6fabd15d22ed5", features = ["minio"] }
-serial_test = "3.4.0"
+serial_test = "3.5.0"
 shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
-snafu = "0.9.0"
+snafu = "0.9.1"
 snap = "1.1.1"
 starshard = { version = "2.2.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.39.2"
+sysinfo = "0.39.3"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -283,7 +283,7 @@ tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.23.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.23.2", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
@@ -298,17 +298,17 @@ dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.32.0" }
 opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
 opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.32.0", features = ["rt-tokio"] }
+opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
 opentelemetry-semantic-conventions = { version = "0.32.0", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.0.4", features = ["backend-pprof-rs"] }
+pyroscope = { version = "=2.0.5", features = ["backend-pprof-rs", "backend-jemalloc"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
 suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
 rcgen = "0.14.8"
-russh = { version = "0.60.3", git = "https://github.com/Eugeny/russh", rev = "fc6e3ab4cd4338e94ae64e17aeed2acee9335e6b" }
+russh = { version = "0.61.1",features = ["serde"] }
 russh-sftp = "2.3.0"
 
 # WebDAV
@@ -324,7 +324,9 @@ tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profilin
 jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
 # pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
-# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
+# Use pprof-pyroscope-fork to match pyroscope 2.0.5's internal pprof dependency (v0.1500.4).
+# Cargo unifies them into a single instance, avoiding duplicate perf_signal_handler symbols.
+# NOTE: Do NOT upgrade pyroscope to >=2.0.6 — it vendors pprof-rs internally, causing symbol collision.
 pprof = { package = "pprof-pyroscope-fork", version = "0.1500.4", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]

Dockerfile.glibc

@@ -105,6 +105,8 @@ RUN groupadd -g 10001 rustfs && \
 
 ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_CONSOLE_ADDRESS=":9001" \
+    RUSTFS_ACCESS_KEY="rustfsadmin" \
+    RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \

README.md

@@ -7,6 +7,7 @@
   <a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="Build and Push Docker Images" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
   <img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
   <img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
+  <a href="https://discord.gg/NcKBCEJp6P"><img alt="Discord" src="https://img.shields.io/badge/Discord-Join%20Chat-5865F2?logo=discord&logoColor=white" /></a>
   <a href="https://hellogithub.com/repository/rustfs/rustfs" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=b95bcb72bdc340b68f16fdf6790b7d5b&claim_uid=MsbvjYeLDKAH457&theme=small" alt="Featured｜HelloGitHub" /></a>
 </p>
 
@@ -115,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.7
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -295,6 +296,7 @@ If you have any questions or need assistance:
 - [Documentation](https://docs.rustfs.com) - The manual you should read
 - [Changelog](https://github.com/rustfs/rustfs/releases) - What we broke and fixed
 - [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Where the community lives
+- [Discord](https://discord.gg/NcKBCEJp6P) - Chat with the RustFS community
 
 ## Contact
 

README_ZH.md

@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.7
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

crates/AGENTS.md

@@ -14,6 +14,41 @@ Applies to all paths under `crates/`.
 - Keep integration tests under each crate's `tests/` directory.
 - Add regression tests for bug fixes and behavior changes.
 
+## Error Type Design
+
+- Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`.
+- Do not use `Box<dyn Error>` or `Box<dyn Error + Send + Sync>` in public trait methods or struct methods. Define a concrete error type with specific variants.
+- When implementing `std::error::Error`, always override `fn source()` if you store an inner error. Breaking the error chain makes debugging impossible.
+- Internal helpers that return `Result<_, String>` and are immediately wrapped via `.map_err(Error::other)` should return the actual error type directly.
+
+## Concurrency
+
+- Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths.
+- Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics).
+- When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff.
+- `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`.
+
+## Recursion Safety
+
+- Recursive tree/graph traversals must have a depth limit (e.g., `max_depth` counter) or use an iterative approach with an explicit `Vec` stack.
+- This applies to cache trees, directory walks, and any user-influenced hierarchy.
+- A corrupted or malicious input must not be able to overflow the thread stack.
+
+## Type Casting
+
+- Never use `as` for numeric conversions that may truncate or overflow. Use `try_into()` with explicit error handling, or clamp with `value.max(0) as usize` when the domain is bounded.
+- `f64 as usize` saturates but is fragile; clamp to `[0, usize::MAX as f64]` first.
+- Treat every `as` cast in a PR review as a potential bug; require justification.
+
+## Testing
+
+- Keep unit tests close to the module they test.
+- Keep integration tests under each crate's `tests/` directory.
+- Add regression tests for bug fixes and behavior changes.
+- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test.
+- In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input.
+
 ## Async and Performance
 
 - Keep async paths non-blocking.

crates/common/src/heal_channel.rs

@@ -110,6 +110,25 @@ pub enum HealScanMode {
     Deep = 2,
 }
 
+impl HealScanMode {
+    pub const fn as_str(self) -> &'static str {
+        match self {
+            Self::Unknown => "unknown",
+            Self::Normal => "normal",
+            Self::Deep => "deep",
+        }
+    }
+
+    pub const fn from_u8(value: u8) -> Option<Self> {
+        match value {
+            0 => Some(Self::Unknown),
+            1 => Some(Self::Normal),
+            2 => Some(Self::Deep),
+            _ => None,
+        }
+    }
+}
+
 impl Serialize for HealScanMode {
     fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
     where
@@ -137,12 +156,7 @@ impl<'de> Deserialize<'de> for HealScanMode {
             where
                 E: serde::de::Error,
             {
-                match value {
-                    0 => Ok(HealScanMode::Unknown),
-                    1 => Ok(HealScanMode::Normal),
-                    2 => Ok(HealScanMode::Deep),
-                    _ => Err(E::custom(format!("invalid HealScanMode value: {value}"))),
-                }
+                HealScanMode::from_u8(value).ok_or_else(|| E::custom(format!("invalid HealScanMode value: {value}")))
             }
 
             fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>

crates/config/README.md

@@ -70,6 +70,27 @@ Current guidance:
 - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` (deprecated alias for compatibility)
 - `RUSTFS_SCANNER_IDLE_MODE` (canonical)
 - `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical)
+
+## Health compatibility switches
+
+- `RUSTFS_HEALTH_ENDPOINT_ENABLE`
+  - controls canonical `/health`, `/health/live`, and `/health/ready` endpoint exposure.
+- `RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE`
+  - enables minimal payload mode for GET health responses (`status`, `ready` only).
+- `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
+  - TTL for readiness cache evaluation.
+- `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
+  - enables busy protection behavior for health probes.
+  - default is `false`.
+- `RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS`
+  - max active HTTP requests; health probes return `429` when active requests reach or exceed this value.
+  - `0` disables thresholding even if busy protection is enabled.
+- `RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE`
+  - enables KMS readiness enforcement for `/health/ready`.
+  - default is `false`.
 
 ## Drive timeout environment variables
 

crates/config/src/audit/kafka.rs

@@ -21,10 +21,14 @@ pub const ENV_AUDIT_KAFKA_TLS_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_TLS_ENABLE";
 pub const ENV_AUDIT_KAFKA_TLS_CA: &str = "RUSTFS_AUDIT_KAFKA_TLS_CA";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_KAFKA_SASL_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_SASL_ENABLE";
+pub const ENV_AUDIT_KAFKA_SASL_MECHANISM: &str = "RUSTFS_AUDIT_KAFKA_SASL_MECHANISM";
+pub const ENV_AUDIT_KAFKA_SASL_USERNAME: &str = "RUSTFS_AUDIT_KAFKA_SASL_USERNAME";
+pub const ENV_AUDIT_KAFKA_SASL_PASSWORD: &str = "RUSTFS_AUDIT_KAFKA_SASL_PASSWORD";
 pub const ENV_AUDIT_KAFKA_QUEUE_DIR: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_DIR";
 pub const ENV_AUDIT_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 14] = &[
     ENV_AUDIT_KAFKA_ENABLE,
     ENV_AUDIT_KAFKA_BROKERS,
     ENV_AUDIT_KAFKA_TOPIC,
@@ -33,6 +37,10 @@ pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
     ENV_AUDIT_KAFKA_TLS_CA,
     ENV_AUDIT_KAFKA_TLS_CLIENT_CERT,
     ENV_AUDIT_KAFKA_TLS_CLIENT_KEY,
+    ENV_AUDIT_KAFKA_SASL_ENABLE,
+    ENV_AUDIT_KAFKA_SASL_MECHANISM,
+    ENV_AUDIT_KAFKA_SASL_USERNAME,
+    ENV_AUDIT_KAFKA_SASL_PASSWORD,
     ENV_AUDIT_KAFKA_QUEUE_DIR,
     ENV_AUDIT_KAFKA_QUEUE_LIMIT,
 ];
@@ -47,6 +55,10 @@ pub const AUDIT_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,

crates/config/src/constants/app.rs

@@ -155,12 +155,6 @@ pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
-/// Environment variable to explicitly allow public default root credentials.
-///
-/// This is intended for local development only. Production startup paths should
-/// provide non-default `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` values.
-pub const ENV_RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS: &str = "RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS";
-
 /// Environment variable controlling startup behavior when unsupported filesystem types are detected.
 ///
 /// Accepted values:

crates/config/src/constants/health.rs

@@ -26,3 +26,20 @@ pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
 /// When enabled, only `status` and `ready` fields are returned.
 pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
 pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;
+
+/// Enable busy protection for health probes.
+/// When enabled with a positive request threshold, alias health probes may
+/// return 429 when active HTTP requests exceed the threshold.
+pub const ENV_HEALTH_COMPAT_BUSY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_CHECK_ENABLE: bool = false;
+
+/// Max active HTTP requests; alias health probes report busy (429) when active requests reach or exceed this value.
+/// Set to 0 to disable thresholding even when busy protection is enabled.
+pub const ENV_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: &str = "RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: usize = 0;
+
+/// Enable KMS readiness check for alias readiness probes.
+/// When enabled, `/health/ready` additionally requires KMS service to be
+/// in running state if a global KMS manager exists.
+pub const ENV_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: bool = false;

crates/config/src/constants/object.rs

@@ -281,6 +281,40 @@ pub const ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT: &str = "RUSTFS_OBJECT_LOCK_ACQUIRE_TI
 /// Default lock acquisition timeout: 5 seconds.
 pub const DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT: u64 = 5;
 
+/// Environment variable to enable object namespace lock diagnostics.
+///
+/// When enabled, RustFS emits slow lock acquisition and long lock hold
+/// warnings for object-level namespace locks. This is intended for
+/// production debugging of contention on hot object keys.
+///
+/// Default: false (disabled, can be overridden by `RUSTFS_OBJECT_LOCK_DIAG_ENABLE`).
+pub const ENV_OBJECT_LOCK_DIAG_ENABLE: &str = "RUSTFS_OBJECT_LOCK_DIAG_ENABLE";
+
+/// Default: object lock diagnostics are disabled.
+pub const DEFAULT_OBJECT_LOCK_DIAG_ENABLE: bool = false;
+
+/// Environment variable for the slow object lock acquisition threshold in milliseconds.
+///
+/// When a read or write namespace lock takes at least this long to acquire,
+/// RustFS emits a warning with the operation name and object key.
+///
+/// Default: 500 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS";
+
+/// Default slow object lock acquisition threshold: 500 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: u64 = 500;
+
+/// Environment variable for the long object lock hold threshold in milliseconds.
+///
+/// When a namespace lock guard is held for at least this long, RustFS emits
+/// a warning when the guard is dropped.
+///
+/// Default: 1000 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_HOLD_MS";
+
+/// Default long object lock hold threshold: 1000 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: u64 = 1000;
+
 // ============================================================================
 // I/O priority scheduling configuration
 // ============================================================================

crates/config/src/constants/oidc.rs

@@ -28,6 +28,7 @@ pub const OIDC_GROUPS_CLAIM: &str = "groups_claim";
 pub const OIDC_ROLES_CLAIM: &str = "roles_claim";
 pub const OIDC_EMAIL_CLAIM: &str = "email_claim";
 pub const OIDC_USERNAME_CLAIM: &str = "username_claim";
+pub const OIDC_HIDE_FROM_UI: &str = "hide_from_ui";
 
 // Environment variable names for OIDC
 pub const ENV_IDENTITY_OPENID_ENABLE: &str = "RUSTFS_IDENTITY_OPENID_ENABLE";
@@ -46,9 +47,10 @@ pub const ENV_IDENTITY_OPENID_GROUPS_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_GROUP
 pub const ENV_IDENTITY_OPENID_ROLES_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_ROLES_CLAIM";
 pub const ENV_IDENTITY_OPENID_EMAIL_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM";
 pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM";
+pub const ENV_IDENTITY_OPENID_HIDE_FROM_UI: &str = "RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI";
 
 /// List of all environment variable keys for an OIDC provider.
-pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
+pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
     ENV_IDENTITY_OPENID_ENABLE,
     ENV_IDENTITY_OPENID_CONFIG_URL,
     ENV_IDENTITY_OPENID_CLIENT_ID,
@@ -65,6 +67,7 @@ pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
     ENV_IDENTITY_OPENID_ROLES_CLAIM,
     ENV_IDENTITY_OPENID_EMAIL_CLAIM,
     ENV_IDENTITY_OPENID_USERNAME_CLAIM,
+    ENV_IDENTITY_OPENID_HIDE_FROM_UI,
 ];
 
 /// A list of all valid configuration keys for an OIDC provider.
@@ -85,6 +88,7 @@ pub const IDENTITY_OPENID_KEYS: &[&str] = &[
     OIDC_ROLES_CLAIM,
     OIDC_EMAIL_CLAIM,
     OIDC_USERNAME_CLAIM,
+    OIDC_HIDE_FROM_UI,
     crate::COMMENT_KEY,
 ];
 

crates/config/src/constants/runtime.rs

@@ -69,6 +69,10 @@ pub const DEFAULT_TRANSITION_QUEUE_CAPACITY: usize = 1000;
 pub const DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS: usize = 100;
 /// Test-only fault injection env var that forces the immediate transition enqueue timeout path.
 pub const ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT: &str = "RUSTFS_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT";
+/// Test-only fault injection env var that forces a number of IAM bootstrap failures.
+pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS";
+/// Test-only env var that overrides the deferred IAM retry interval in debug builds.
+pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
 /// Runtime env var controlling the transition worker count.
 pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
 /// Runtime env var controlling the absolute maximum transition workers.

crates/config/src/constants/scanner.rs

@@ -14,6 +14,76 @@
 
 use std::time::Duration;
 
+/// Scanner admin config subsystem name.
+pub const SCANNER_SUB_SYS: &str = "scanner";
+
+/// Scanner config key selecting the speed preset.
+pub const SCANNER_SPEED: &str = "speed";
+
+/// Scanner config key overriding the cycle interval in seconds.
+pub const SCANNER_CYCLE: &str = "cycle";
+
+/// Scanner config key setting the startup delay in seconds.
+///
+/// For compatibility, this also acts as the scanner cycle interval when
+/// `cycle` is unset.
+pub const SCANNER_START_DELAY: &str = "start_delay";
+
+/// Scanner config key capping one cycle's runtime in seconds.
+pub const SCANNER_CYCLE_MAX_DURATION: &str = "cycle_max_duration";
+
+/// Scanner config key capping objects processed by one cycle.
+pub const SCANNER_CYCLE_MAX_OBJECTS: &str = "cycle_max_objects";
+
+/// Scanner config key capping directories entered by one cycle.
+pub const SCANNER_CYCLE_MAX_DIRECTORIES: &str = "cycle_max_directories";
+
+/// Scanner config key setting the periodic bitrot scan cycle in seconds.
+pub const SCANNER_BITROT_CYCLE: &str = "bitrot_cycle";
+
+/// Scanner config key controlling whether scanner throttling is enabled.
+pub const SCANNER_IDLE_MODE: &str = "idle_mode";
+
+/// Scanner config key controlling scanner cache save timeout in seconds.
+pub const SCANNER_CACHE_SAVE_TIMEOUT: &str = "cache_save_timeout";
+
+/// Scanner config key capping concurrent scanner set tasks.
+pub const SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "max_concurrent_set_scans";
+
+/// Scanner config key capping concurrent scanner disk bucket walks per set.
+pub const SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "max_concurrent_disk_scans";
+
+/// Scanner config key controlling how often object loops yield.
+pub const SCANNER_YIELD_EVERY_N_OBJECTS: &str = "yield_every_n_objects";
+
+/// Scanner config key controlling object version count alerts.
+pub const SCANNER_ALERT_EXCESS_VERSIONS: &str = "alert_excess_versions";
+
+/// Scanner config key controlling retained version size alerts.
+pub const SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "alert_excess_version_size";
+
+/// Scanner config key controlling direct subfolder count alerts.
+pub const SCANNER_ALERT_EXCESS_FOLDERS: &str = "alert_excess_folders";
+
+/// Scanner config keys supported by the admin config subsystem.
+pub const SCANNER_KEYS: &[&str] = &[
+    SCANNER_SPEED,
+    SCANNER_CYCLE,
+    SCANNER_START_DELAY,
+    SCANNER_CYCLE_MAX_DURATION,
+    SCANNER_CYCLE_MAX_OBJECTS,
+    SCANNER_CYCLE_MAX_DIRECTORIES,
+    SCANNER_BITROT_CYCLE,
+    SCANNER_IDLE_MODE,
+    SCANNER_CACHE_SAVE_TIMEOUT,
+    SCANNER_MAX_CONCURRENT_SET_SCANS,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
+    SCANNER_ALERT_EXCESS_VERSIONS,
+    SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_FOLDERS,
+];
+
 /// Canonical environment variable name that specifies the scanner start delay in seconds.
 /// If set, this overrides the cycle interval derived from `RUSTFS_SCANNER_SPEED`.
 /// - Unit: seconds (u64).
@@ -31,6 +101,22 @@ pub const ENV_DATA_SCANNER_START_DELAY_SECS: &str = "RUSTFS_DATA_SCANNER_START_D
 /// - Example: `export RUSTFS_SCANNER_CYCLE=3600` (1 hour)
 pub const ENV_SCANNER_CYCLE: &str = "RUSTFS_SCANNER_CYCLE";
 
+/// Environment variable that caps one scanner cycle's runtime in seconds.
+/// A value of `0` disables the cycle runtime budget.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS=1800`
+pub const ENV_SCANNER_CYCLE_MAX_DURATION_SECS: &str = "RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS";
+
+/// Environment variable that caps objects processed by one scanner cycle.
+/// A value of `0` disables the per-cycle object budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_OBJECTS=1000000`
+pub const ENV_SCANNER_CYCLE_MAX_OBJECTS: &str = "RUSTFS_SCANNER_CYCLE_MAX_OBJECTS";
+
+/// Environment variable that caps directories entered by one scanner cycle.
+/// A value of `0` disables the per-cycle directory budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000`
+pub const ENV_SCANNER_CYCLE_MAX_DIRECTORIES: &str = "RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES";
+
 /// Environment variable that selects the scanner speed preset.
 /// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
 /// Controls the sleep factor, maximum sleep duration, and cycle interval.
@@ -40,6 +126,18 @@ pub const ENV_SCANNER_SPEED: &str = "RUSTFS_SCANNER_SPEED";
 /// Default scanner speed preset.
 pub const DEFAULT_SCANNER_SPEED: &str = "default";
 
+/// Default scanner cycle runtime budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS: u64 = 0;
+
+/// Default scanner per-cycle object budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_OBJECTS: u64 = 0;
+
+/// Default scanner per-cycle directory budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES: u64 = 0;
+
 /// Environment variable that specifies the periodic bitrot scan cycle in seconds.
 /// When set to `0`, `true`, `on`, or `yes`, every scanner cycle runs in deep mode.
 /// When set to `false`, `off`, `no`, or `disabled`, periodic deep scans are disabled.
@@ -65,11 +163,12 @@ pub const ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "RUSTFS_SCANNER_ALERT_EX
 pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE: u64 = 1024 * 1024 * 1024 * 1024;
 
 /// Environment variable that controls how many subfolders trigger scanner alerts.
-/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS=50000`
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS=65538`
 pub const ENV_SCANNER_ALERT_EXCESS_FOLDERS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS";
 
 /// Default subfolder count that triggers scanner alerts.
-pub const DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS: u64 = 50_000;
+/// Allows Proxmox Backup Server's chunk namespace layout.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS: u64 = 65_538;
 
 /// Environment variable that controls whether the scanner sleeps between operations.
 /// When `true` (default), the scanner throttles itself. When `false`, it runs at full speed.
@@ -82,9 +181,38 @@ pub const ENV_SCANNER_IDLE_MODE: &str = "RUSTFS_SCANNER_IDLE_MODE";
 /// - Example: `export RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS=30`
 pub const ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS: &str = "RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS";
 
+/// Default scanner cache save timeout in seconds.
+pub const DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS: u64 = 30;
+
+/// Environment variable that caps concurrent scanner set tasks.
+/// A value of `0` keeps the existing topology-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS=2`
+pub const ENV_SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS";
+
+/// Environment variable that caps concurrent scanner disk bucket walks per set.
+/// A value of `0` keeps the existing disk-count-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS=1`
+pub const ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS";
+
+/// Environment variable that controls how often scanner object loops yield to the async runtime.
+/// A value of `0` disables this extra object-count yield.
+/// - Example: `export RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS=32`
+pub const ENV_SCANNER_YIELD_EVERY_N_OBJECTS: &str = "RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS";
+
 /// Default scanner idle mode.
 pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
 
+/// Default set scan concurrency budget.
+/// `0` means no additional limit beyond deployment topology.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 0;
+
+/// Default disk scan concurrency budget.
+/// `0` means no additional limit beyond available disks in the set.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 0;
+
+/// Default object interval for cooperative scanner yields.
+pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
+
 /// Compatibility flag kept for Patch 3 rollback windows.
 ///
 /// Inline scanner heal execution has been removed in favor of heal-candidate enqueue.
@@ -148,15 +276,20 @@ impl ScannerSpeed {
         }
     }
 
-    pub fn from_env_str(s: &str) -> Self {
+    pub fn parse_str(s: &str) -> Option<Self> {
         match s.trim().to_ascii_lowercase().as_str() {
-            "fastest" => Self::Fastest,
-            "fast" => Self::Fast,
-            "slow" => Self::Slow,
-            "slowest" => Self::Slowest,
-            _ => Self::Default,
+            "fastest" => Some(Self::Fastest),
+            "fast" => Some(Self::Fast),
+            "default" => Some(Self::Default),
+            "slow" => Some(Self::Slow),
+            "slowest" => Some(Self::Slowest),
+            _ => None,
         }
     }
+
+    pub fn from_env_str(s: &str) -> Self {
+        Self::parse_str(s).unwrap_or_default()
+    }
 }
 
 impl std::fmt::Display for ScannerSpeed {

crates/config/src/constants/targets.rs

@@ -49,6 +49,10 @@ pub const KAFKA_TLS_ENABLE: &str = "tls_enable";
 pub const KAFKA_TLS_CA: &str = "tls_ca";
 pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const KAFKA_SASL_ENABLE: &str = "sasl_enable";
+pub const KAFKA_SASL_MECHANISM: &str = "sasl_mechanism";
+pub const KAFKA_SASL_USERNAME: &str = "sasl_username";
+pub const KAFKA_SASL_PASSWORD: &str = "sasl_password";
 
 pub const AMQP_URL: &str = "url";
 pub const AMQP_EXCHANGE: &str = "exchange";

crates/config/src/notify/kafka.rs

@@ -22,6 +22,10 @@ pub const NOTIFY_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,
@@ -36,10 +40,14 @@ pub const ENV_NOTIFY_KAFKA_TLS_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_TLS_ENABLE";
 pub const ENV_NOTIFY_KAFKA_TLS_CA: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CA";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_KAFKA_SASL_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_SASL_ENABLE";
+pub const ENV_NOTIFY_KAFKA_SASL_MECHANISM: &str = "RUSTFS_NOTIFY_KAFKA_SASL_MECHANISM";
+pub const ENV_NOTIFY_KAFKA_SASL_USERNAME: &str = "RUSTFS_NOTIFY_KAFKA_SASL_USERNAME";
+pub const ENV_NOTIFY_KAFKA_SASL_PASSWORD: &str = "RUSTFS_NOTIFY_KAFKA_SASL_PASSWORD";
 pub const ENV_NOTIFY_KAFKA_QUEUE_DIR: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_DIR";
 pub const ENV_NOTIFY_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 14] = &[
     ENV_NOTIFY_KAFKA_ENABLE,
     ENV_NOTIFY_KAFKA_BROKERS,
     ENV_NOTIFY_KAFKA_TOPIC,
@@ -48,6 +56,10 @@ pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
     ENV_NOTIFY_KAFKA_TLS_CA,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY,
+    ENV_NOTIFY_KAFKA_SASL_ENABLE,
+    ENV_NOTIFY_KAFKA_SASL_MECHANISM,
+    ENV_NOTIFY_KAFKA_SASL_USERNAME,
+    ENV_NOTIFY_KAFKA_SASL_PASSWORD,
     ENV_NOTIFY_KAFKA_QUEUE_DIR,
     ENV_NOTIFY_KAFKA_QUEUE_LIMIT,
 ];

crates/data-usage/src/data_usage.rs

@@ -688,15 +688,14 @@ impl DataUsageCache {
             return;
         }
 
-        let mut leaves = Vec::new();
+        let mut candidates = Vec::new();
         let mut remove = total - limit;
-        add(self, path, &mut leaves);
-        leaves.sort_by_key(|a| a.objects);
+        add(self, path, &mut candidates);
+        candidates.sort_by_key(|a| a.objects);
 
-        while remove > 0 && !leaves.is_empty() {
-            let Some(e) = leaves.first() else {
-                break;
-            };
+        let mut candidate_index = 0;
+        while remove > 0 && candidate_index < candidates.len() {
+            let e = &candidates[candidate_index];
             let candidate = e.path.clone();
             if candidate == *path && !compact_self {
                 break;
@@ -705,7 +704,7 @@ impl DataUsageCache {
             let mut flat = match self.size_recursive(&candidate.key()) {
                 Some(flat) => flat,
                 None => {
-                    leaves.remove(0);
+                    candidate_index += 1;
                     continue;
                 }
             };
@@ -714,8 +713,8 @@ impl DataUsageCache {
             self.delete_recursive(&candidate);
             self.replace_hashed(&candidate, &None, &flat);
 
-            remove -= removing;
-            leaves.remove(0);
+            remove = remove.saturating_sub(removing);
+            candidate_index += 1;
         }
     }
 
@@ -869,23 +868,24 @@ struct Inner {
     path: DataUsageHash,
 }
 
-fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, leaves: &mut Vec<Inner>) {
+fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, candidates: &mut Vec<Inner>) -> usize {
     let e = match data_usage_cache.cache.get(&path.key()) {
         Some(e) => e,
-        None => return,
+        None => return 0,
     };
-    if !e.children.is_empty() {
-        return;
-    }
-
-    let sz = data_usage_cache.size_recursive(&path.key()).unwrap_or_default();
-    leaves.push(Inner {
-        objects: sz.objects,
-        path: path.clone(),
-    });
+    let mut objects = e.objects;
     for ch in e.children.iter() {
-        add(data_usage_cache, &DataUsageHash(ch.clone()), leaves);
+        objects += add(data_usage_cache, &DataUsageHash(ch.clone()), candidates);
     }
+    // Collect internal nodes (with children) as compaction candidates.
+    // Leaf nodes have no children to remove, so compacting them is a no-op.
+    if !e.children.is_empty() {
+        candidates.push(Inner {
+            objects,
+            path: path.clone(),
+        });
+    }
+    objects
 }
 
 fn mark(duc: &DataUsageCache, entry: &DataUsageEntry, found: &mut HashSet<String>) {
@@ -1361,4 +1361,161 @@ mod tests {
         assert_eq!(child_entry.size, 30);
         assert_eq!(child_entry.objects, 3);
     }
+
+    // --- Tests for `add` and `reduce_children_of` (bug fixes) ---
+
+    /// Build a small tree: root -> child1 (leaf), child2 -> grandchild (leaf).
+    fn build_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let c1 = hash_path("bucket/a");
+        let c2 = hash_path("bucket/b");
+        let gc = hash_path("bucket/b/c");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(&root, &None, &DataUsageEntry::default());
+        cache.replace_hashed(
+            &c1,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                size: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &c2,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 2,
+                size: 20,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &gc,
+            &Some(c2.clone()),
+            &DataUsageEntry {
+                objects: 3,
+                size: 30,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    fn build_underflow_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let small = hash_path("bucket/small");
+        let small_a = hash_path("bucket/small/a");
+        let small_b = hash_path("bucket/small/b");
+        let large = hash_path("bucket/large");
+        let large_a = hash_path("bucket/large/a");
+        let large_b = hash_path("bucket/large/b");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(
+            &root,
+            &None,
+            &DataUsageEntry {
+                objects: 100,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&small, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &small_a,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &small_b,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&large, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &large_a,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &large_b,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    #[test]
+    fn test_add_collects_internal_nodes_as_compaction_candidates() {
+        let (cache, root) = build_test_tree();
+        let mut candidates = Vec::new();
+        add(&cache, &root, &mut candidates);
+
+        let mut paths: Vec<String> = candidates.iter().map(|l| l.path.key()).collect();
+        paths.sort();
+        assert_eq!(paths.len(), 2, "add() should find internal nodes with children");
+        assert!(paths.contains(&hash_path("bucket").key()));
+        assert!(paths.contains(&hash_path("bucket/b").key()));
+    }
+
+    #[test]
+    fn test_add_skips_leaf_node() {
+        let mut cache = DataUsageCache::default();
+        let h = hash_path("single-leaf");
+        cache.replace_hashed(
+            &h,
+            &None,
+            &DataUsageEntry {
+                objects: 5,
+                size: 50,
+                ..Default::default()
+            },
+        );
+
+        let mut candidates = Vec::new();
+        add(&cache, &h, &mut candidates);
+        assert!(candidates.is_empty(), "leaf node should not be a compaction candidate");
+    }
+
+    #[test]
+    fn test_reduce_children_of_compacts_internal_node() {
+        let (mut cache, root) = build_test_tree();
+        cache.reduce_children_of(&root, 2, false);
+
+        let entry_c2 = cache.find("bucket/b").unwrap();
+        assert!(entry_c2.compacted, "internal node 'bucket/b' should be compacted");
+        let entry_c1 = cache.find("bucket/a").unwrap();
+        assert!(!entry_c1.compacted, "leaf 'bucket/a' should not be compacted");
+        assert!(cache.find("bucket/b/c").is_none(), "grandchild should be removed");
+    }
+
+    #[test]
+    fn test_reduce_children_of_usize_underflow_saturates() {
+        let (mut cache, root) = build_underflow_test_tree();
+
+        // total children=6, limit=5, remove=1. The smallest candidate removes
+        // two descendants, so plain subtraction would underflow and compact the
+        // next candidate too.
+        cache.reduce_children_of(&root, 5, false);
+
+        assert!(cache.find("bucket/small").is_some_and(|entry| entry.compacted));
+        assert!(cache.find("bucket/small/a").is_none());
+        assert!(cache.find("bucket/small/b").is_none());
+        assert!(cache.find("bucket/large").is_some_and(|entry| !entry.compacted));
+        assert!(cache.find("bucket/large/a").is_some());
+        assert!(cache.find("bucket/large/b").is_some());
+    }
 }

crates/e2e_test/src/kms/README.md

@@ -191,7 +191,7 @@ RUST_LOG=debug cargo test test_local_kms_end_to_end -- --nocapture
 4. **Monitor ports**
    ```bash
    netstat -an | grep 9050
-   curl http://127.0.0.1:9050/minio/health/ready
+   curl http://127.0.0.1:9050/health/ready
    ```
 
 ## 📊 Coverage

crates/e2e_test/src/lib.rs

@@ -63,6 +63,10 @@ mod archive_download_integrity_test;
 #[cfg(test)]
 mod list_objects_v2_pagination_test;
 
+// Regression test for Issue #3107: mc mirror small-bucket listing must not time out.
+#[cfg(test)]
+mod mc_mirror_small_bucket_test;
+
 // Policy variables tests
 #[cfg(test)]
 mod policy;
@@ -142,4 +146,7 @@ mod replication_extension_test;
 #[cfg(test)]
 mod snowball_auto_extract_test;
 
+#[cfg(test)]
+mod namespace_lock_quorum_test;
+
 pub mod tls_gen;

crates/e2e_test/src/mc_mirror_small_bucket_test.rs

@@ -0,0 +1,124 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, RustFSTestEnvironment};
+use serial_test::serial;
+use std::path::Path;
+use std::process::Command;
+use std::time::Duration;
+use tokio::fs;
+use tokio::time::timeout;
+use tracing::info;
+
+const BUCKET: &str = "ddd";
+const OBJECT_COUNT: usize = 484;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn create_issue_3107_fixture(root: &Path) -> TestResult {
+    for idx in 0..OBJECT_COUNT {
+        let object_path = root.join("ddd").join("requirements").join(format!("file-{idx:04}.txt"));
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, format!("issue-3107 payload {idx}\n").repeat(128)).await?;
+    }
+
+    for dir in ["aaa", "ccc", "ccc-package"] {
+        let object_path = root.join(dir).join("marker.txt");
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, dir.as_bytes()).await?;
+    }
+
+    Ok(())
+}
+
+fn mc_available() -> bool {
+    Command::new("mc")
+        .arg("--version")
+        .output()
+        .is_ok_and(|output| output.status.success())
+}
+
+fn run_mc(args: &[&str]) -> TestResult {
+    let output = Command::new("mc").args(args).output()?;
+    if !output.status.success() {
+        return Err(format!(
+            "mc {:?} failed: status={:?}, stdout={}, stderr={}",
+            args,
+            output.status.code(),
+            String::from_utf8_lossy(&output.stdout),
+            String::from_utf8_lossy(&output.stderr)
+        )
+        .into());
+    }
+    Ok(())
+}
+
+fn count_files(root: &Path) -> usize {
+    walkdir::WalkDir::new(root)
+        .into_iter()
+        .filter_map(Result::ok)
+        .filter(|entry| entry.file_type().is_file())
+        .count()
+}
+
+#[tokio::test]
+#[serial]
+async fn test_mc_mirror_small_bucket_completes_without_list_timeout() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting issue #3107 mc mirror regression test");
+    if !mc_available() {
+        info!("Skipping issue #3107 mc mirror regression test because mc is not installed");
+        return Ok(());
+    }
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    env.create_test_bucket(BUCKET).await?;
+
+    let alias = format!("rustfs-issue-3107-{}", uuid::Uuid::new_v4());
+    run_mc(&["alias", "set", &alias, &env.url, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY])?;
+
+    let fixture = Path::new(&env.temp_dir).join("issue-3107-fixture");
+    let backup = Path::new(&env.temp_dir).join("issue-3107-backup");
+    fs::create_dir_all(&fixture).await?;
+    fs::create_dir_all(&backup).await?;
+    create_issue_3107_fixture(&fixture).await?;
+
+    let bucket_target = format!("{alias}/{BUCKET}");
+    let source_path = fixture.to_string_lossy().to_string();
+    run_mc(&["mirror", "--overwrite", &source_path, &bucket_target])?;
+
+    let backup_path = backup.join("ddd-backup");
+    let backup_path_string = backup_path.to_string_lossy().to_string();
+    timeout(
+        Duration::from_secs(20),
+        tokio::task::spawn_blocking({
+            let bucket_target = bucket_target.clone();
+            let backup_path_string = backup_path_string.clone();
+            move || run_mc(&["mirror", "--overwrite", &bucket_target, &backup_path_string])
+        }),
+    )
+    .await
+    .map_err(|_| "mc mirror timed out while listing/comparing a small bucket")???;
+
+    let mirrored_count = count_files(&backup_path);
+    assert_eq!(
+        mirrored_count,
+        OBJECT_COUNT + 3,
+        "mc mirror should copy every object from the issue #3107 small bucket fixture"
+    );
+
+    run_mc(&["alias", "remove", &alias])?;
+    Ok(())
+}

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -0,0 +1,122 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::RustFSTestClusterEnvironment;
+use aws_sdk_s3::Client;
+use aws_sdk_s3::error::SdkError;
+use bytes::Bytes;
+use serial_test::serial;
+use std::sync::Arc;
+use tokio::sync::Barrier;
+use tracing::{info, warn};
+
+const BUCKET: &str = "namespace-lock-quorum-bucket";
+const KEY: &str = "thumb/79/concurrent-overwrite.jpg";
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .body(Bytes::from(payload).into())
+        .send()
+        .await
+        .map(|_| ())
+        .map_err(|err| format_s3_error(err, writer_id))
+}
+
+fn format_s3_error(err: SdkError<aws_sdk_s3::operation::put_object::PutObjectError>, writer_id: usize) -> String {
+    match err {
+        SdkError::ServiceError(service_err) => {
+            let err = service_err.err();
+            let code = err.meta().code().unwrap_or("<unknown>");
+            let message = err.meta().message().unwrap_or("<empty>");
+            format!("writer {writer_id} returned service error {code}: {message}")
+        }
+        other => format!("writer {writer_id} returned SDK error: {other:?}"),
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting namespace lock quorum regression test with auto cluster");
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    // Keep the regression focused on false quorum-loss errors, not ordinary lock
+    // wait exhaustion under a heavily contended same-key overwrite workload.
+    cluster.set_env("RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT", "20");
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let clients = cluster.create_all_clients()?;
+    let first_payload = b"initial object contents".to_vec();
+    put_object(clients[0].clone(), first_payload, 0).await?;
+
+    let writer_count = clients.len() * 2;
+    let barrier = Arc::new(Barrier::new(writer_count));
+    let mut handles = Vec::with_capacity(writer_count);
+
+    for writer_id in 0..writer_count {
+        let client = clients[writer_id % clients.len()].clone();
+        let barrier = barrier.clone();
+        let payload = format!("replacement payload from writer {writer_id:02}").into_bytes();
+        handles.push(tokio::spawn(async move {
+            barrier.wait().await;
+            put_object(client, payload, writer_id).await
+        }));
+    }
+
+    let mut failures = Vec::new();
+    for handle in handles {
+        match handle.await {
+            Ok(Ok(())) => {}
+            Ok(Err(err)) => failures.push(err),
+            Err(err) => failures.push(format!("writer task join failed: {err}")),
+        }
+    }
+
+    if !failures.is_empty() {
+        for failure in &failures {
+            warn!("concurrent overwrite failure: {}", failure);
+        }
+    }
+
+    assert!(
+        failures.is_empty(),
+        "concurrent overwrites must not surface namespace lock quorum failures: {failures:#?}"
+    );
+
+    let body = clients[0]
+        .get_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .send()
+        .await?
+        .body
+        .collect()
+        .await?
+        .into_bytes();
+    let body = std::str::from_utf8(&body)?;
+    assert!(
+        body.starts_with("replacement payload from writer "),
+        "final object body should be one of the successful overwrite payloads, got {body:?}"
+    );
+
+    clients[0].delete_object().bucket(BUCKET).key(KEY).send().await?;
+    Ok(())
+}

crates/e2e_test/src/snowball_auto_extract_test.rs

@@ -70,6 +70,36 @@ mod tests {
         Ok(builder.into_inner().await?.into_inner())
     }
 
+    fn build_archive_with_parent_dir_entry(victim_bucket: &str) -> Vec<u8> {
+        let path = format!("../{victim_bucket}/evil-injected.txt");
+        let data = b"injected-body";
+        let mut header = [0u8; 512];
+
+        header[..path.len()].copy_from_slice(path.as_bytes());
+        header[100..108].copy_from_slice(b"0000644\0");
+        header[108..116].copy_from_slice(b"0000000\0");
+        header[116..124].copy_from_slice(b"0000000\0");
+        let size = format!("{:011o}\0", data.len());
+        header[124..136].copy_from_slice(size.as_bytes());
+        header[136..148].copy_from_slice(b"00000000000\0");
+        header[148..156].fill(b' ');
+        header[156] = b'0';
+        header[257..263].copy_from_slice(b"ustar\0");
+        header[263..265].copy_from_slice(b"00");
+
+        let checksum: u32 = header.iter().map(|byte| *byte as u32).sum();
+        let checksum = format!("{:06o}\0 ", checksum);
+        header[148..156].copy_from_slice(checksum.as_bytes());
+
+        let mut archive = Vec::new();
+        archive.extend_from_slice(&header);
+        archive.extend_from_slice(data);
+        let padding = (512 - (data.len() % 512)) % 512;
+        archive.extend(std::iter::repeat_n(0, padding));
+        archive.extend_from_slice(&[0u8; 1024]);
+        archive
+    }
+
     #[tokio::test]
     #[serial]
     async fn snowball_auto_extract_supports_minio_prefix_and_directory_markers() -> Result<(), Box<dyn Error + Send + Sync>> {
@@ -273,6 +303,49 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn snowball_auto_extract_rejects_parent_dir_entry_without_cross_bucket_write()
+    -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(vec![]).await?;
+
+        let client = env.create_s3_client();
+        let attacker_bucket = "snowball-traversal-source";
+        let victim_bucket = "snowball-traversal-victim";
+        let archive = build_archive_with_parent_dir_entry(victim_bucket);
+
+        client.create_bucket().bucket(attacker_bucket).send().await?;
+        client.create_bucket().bucket(victim_bucket).send().await?;
+
+        let err = client
+            .put_object()
+            .bucket(attacker_bucket)
+            .key("fixture.tar")
+            .metadata("Snowball-Auto-Extract", "true")
+            .body(ByteStream::from(archive))
+            .send()
+            .await
+            .expect_err("parent directory archive entry should be rejected");
+        let service_err = err.into_service_error();
+        assert_eq!(service_err.code(), Some("InvalidArgument"));
+
+        let victim_err = client
+            .head_object()
+            .bucket(victim_bucket)
+            .key("evil-injected.txt")
+            .send()
+            .await
+            .expect_err("rejected archive entry must not write into the victim bucket");
+        let victim_service_err = victim_err.into_service_error();
+        assert_eq!(victim_service_err.code(), Some("NotFound"));
+
+        env.stop_server();
+        Ok(())
+    }
+
     #[tokio::test]
     #[serial]
     async fn snowball_auto_extract_prefers_exact_minio_prefix_over_suffix_fallback() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/ecstore/AGENTS.md

@@ -14,6 +14,30 @@ Applies to `crates/ecstore/`.
 - Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
 - Benchmark-sensitive changes should include measurable rationale.
 
+## Allocation Discipline in Hot Paths
+
+- Do not implement `Clone` on structs with >5 heap-allocated fields without considering `Arc` for heavy fields.
+- Before cloning a struct in a loop or per-request path, check if a reference or `Cow` would suffice.
+- When a struct contains a large buffer (e.g., erasure coding block), wrap it in `Arc` and pass by reference rather than cloning.
+- Use `&str` and `Cow<str>` instead of `String` for temporary computations (header parsing, signature building, path manipulation).
+- Use `HashMap::with_capacity()` / `Vec::with_capacity()` when the size is known or estimable.
+
+## Lock Ordering
+
+- When a function acquires multiple locks, document the acquisition order in a comment.
+- Never acquire the same set of locks in different orders across code paths — this is a deadlock.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters.
+
+## Recursion Safety
+
+- Recursive tree traversals (cache trees, directory walks) must have a depth limit or use iterative traversal with an explicit `Vec` stack.
+- `flatten()`, `delete_recursive()`, `copy_with_children()`, `total_children_rec()`, `mark()` — all must handle deep or corrupted trees safely.
+
+## Dead Code
+
+- Do not add `#![allow(dead_code)]` at the crate root. If code is unused, remove it or gate it behind a feature flag.
+- Each `#[allow(dead_code)]` annotation must have a comment explaining why the code is kept.
+
 ## Cross-Module Coordination
 
 - Validate behavior impacts on:

crates/ecstore/Cargo.toml

@@ -131,7 +131,7 @@ aws-smithy-http-client.workspace = true
 metrics = { workspace = true }
 
 [dev-dependencies]
-tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
+tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true, features = ["async_closure"] }
 tracing-subscriber = { workspace = true }

crates/ecstore/docs/s3-table/s3-table-concept-model.md

@@ -0,0 +1,130 @@
+# RustFS S3 Table Core Concepts
+
+This document keeps only the core concepts for adding table catalog capability
+to RustFS. The direction is to build an Iceberg REST Catalog-compatible layer
+on top of existing RustFS object storage.
+
+## Concept Model
+
+```text
+TableBucket
+  Warehouse
+    Namespace
+      TableIdentifier
+        TableMetadataLocation
+        TableMetadata
+          Schema
+          Snapshot
+          Manifest
+          DataFile
+```
+
+## TableBucket
+
+`TableBucket` is a normal RustFS bucket enabled for table catalog use. It is not
+a new physical bucket type.
+
+RustFS can mark it with internal bucket metadata such as `table-bucket.json`.
+That marker is an implementation detail, not a public compatibility API.
+
+## Warehouse
+
+`Warehouse` is the catalog storage root exposed to Iceberg clients. In the first
+RustFS model, a warehouse maps to a table-enabled bucket.
+
+Clients should interact with the warehouse through catalog APIs, not by knowing
+RustFS internal metadata paths.
+
+## Namespace
+
+`Namespace` groups tables inside a warehouse. It is a first-class catalog
+resource, not a raw object prefix.
+
+Namespace validation must be conservative: no path traversal, encoded
+separators, empty segments, control characters, or ambiguous normalization.
+
+## TableIdentifier
+
+`TableIdentifier` names one table inside a warehouse and namespace.
+
+```text
+warehouse + namespace + table name
+```
+
+It is resolved by the catalog layer to internal metadata objects. It should not
+be treated as an S3 object key.
+
+## TableMetadataLocation
+
+`TableMetadataLocation` points to the current Iceberg table metadata file.
+
+It is the main mutable table pointer. Only catalog commit code may update it.
+Updates must use conflict detection so stale writers cannot overwrite newer
+table state.
+
+## TableMetadata
+
+`TableMetadata` is the Iceberg metadata JSON referenced by the current metadata
+location.
+
+RustFS should preserve Iceberg metadata exactly where possible. Full parsing can
+be added incrementally for fields RustFS must enforce, such as schemas,
+snapshots, manifest lists, table location, and properties.
+
+## Schema
+
+`Schema` describes table columns.
+
+RustFS does not need to interpret every schema detail at the beginning, but it
+must preserve schema metadata without corrupting unknown Iceberg fields.
+
+## Snapshot
+
+`Snapshot` represents an immutable table state at a point in time.
+
+Catalog commits advance the current metadata to a new snapshot. Maintenance
+features such as rollback and snapshot expiration can be added later.
+
+## Manifest
+
+`Manifest` and manifest lists describe table data-file additions, deletions, and
+references.
+
+RustFS should store and protect these as catalog-owned metadata. Full manifest
+parsing can be deferred until validation or maintenance requires it.
+
+## DataFile
+
+`DataFile` is a table data object, commonly Parquet, Avro, or ORC.
+
+Data files live in object storage. The catalog metadata decides which data files
+belong to a table snapshot.
+
+## Reserved Metadata Boundary
+
+Catalog-owned metadata should live under a reserved internal prefix, for
+example:
+
+```text
+.rustfs-table/
+  warehouses/
+    <warehouse-id>/
+      namespaces/
+        <namespace-id>/
+          tables/
+            <table-id>/
+              current.json
+              metadata/
+              manifests/
+```
+
+Ordinary S3 mutating APIs must not create, overwrite, copy into, delete, or
+mutate objects under this reserved prefix for table-enabled buckets.
+
+## Catalog Boundary
+
+S3 object APIs remain object APIs.
+
+Namespace, table, metadata pointer, and commit operations belong to a separate
+catalog layer. Public compatibility should target Iceberg REST Catalog semantics
+rather than exposing RustFS internal metadata objects directly.

crates/ecstore/src/batch_processor.rs

@@ -274,10 +274,10 @@ mod tests {
         assert!(successes.len() >= 2);
     }
 
-    #[tokio::test]
+    #[tokio::test(start_paused = true)]
     async fn test_batch_processor_quorum_returns_before_slow_tail() {
         let processor = AsyncBatchProcessor::new(4);
-        let started = std::time::Instant::now();
+        let started = tokio::time::Instant::now();
 
         let tasks: Vec<_> = [(10_u64, Ok(1_i32)), (15, Ok(2)), (250, Ok(3))]
             .into_iter()
@@ -295,10 +295,10 @@ mod tests {
         assert!(started.elapsed() < Duration::from_millis(100));
     }
 
-    #[tokio::test]
+    #[tokio::test(start_paused = true)]
     async fn test_batch_processor_quorum_fails_once_quorum_becomes_impossible() {
         let processor = AsyncBatchProcessor::new(4);
-        let started = std::time::Instant::now();
+        let started = tokio::time::Instant::now();
 
         let tasks: Vec<_> = vec![
             (10_u64, Ok(1_i32)),

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -48,8 +48,8 @@ use rustfs_common::heal_channel::rep_has_active_rules;
 use rustfs_common::metrics::{IlmAction, Metrics};
 use rustfs_config::{
     DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
-    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, ENV_TRANSITION_QUEUE_CAPACITY,
-    ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
+    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TRANSITION_QUEUE_CAPACITY, ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS,
+    ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
 };
 use rustfs_data_usage::TierStats;
 use rustfs_filemeta::{
@@ -75,6 +75,8 @@ use time::OffsetDateTime;
 use tokio::select;
 use tokio::sync::mpsc::{Receiver, Sender};
 use tokio::sync::{RwLock, mpsc};
+use tokio::task::JoinHandle;
+use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, info, warn};
 use uuid::Uuid;
 use xxhash_rust::xxh64;
@@ -145,7 +147,7 @@ fn is_immediate_transition_source(src: &LcEventSrc) -> bool {
 
 #[cfg(any(test, debug_assertions))]
 fn should_force_immediate_transition_enqueue_timeout() -> bool {
-    env::var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
+    env::var(rustfs_config::ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
         .ok()
         .is_some_and(|value| value == "1")
 }
@@ -587,12 +589,16 @@ impl ExpiryOp for TransitionTask {
     }
 }
 
+struct TransitionWorker {
+    cancel: CancellationToken,
+    handle: JoinHandle<()>,
+}
+
 pub struct TransitionState {
     transition_tx: A_Sender<Option<TransitionTask>>,
     transition_rx: A_Receiver<Option<TransitionTask>>,
     pub num_workers: AtomicI64,
-    kill_tx: A_Sender<()>,
-    kill_rx: A_Receiver<()>,
+    workers: Mutex<Vec<TransitionWorker>>,
     transition_queue_capacity: usize,
     transition_queue_send_timeout: StdDuration,
     active_tasks: AtomicI64,
@@ -621,13 +627,11 @@ impl TransitionState {
         let capacity = capacity.max(1);
         let queue_send_timeout = resolve_transition_queue_send_timeout();
         let (tx1, rx1) = bounded(capacity);
-        let (tx2, rx2) = bounded(1);
         Arc::new(Self {
             transition_tx: tx1,
             transition_rx: rx1,
             num_workers: AtomicI64::new(0),
-            kill_tx: tx2,
-            kill_rx: rx2,
+            workers: Mutex::new(Vec::new()),
             transition_queue_capacity: capacity,
             transition_queue_send_timeout: queue_send_timeout,
             active_tasks: AtomicI64::new(0),
@@ -849,10 +853,12 @@ impl TransitionState {
         Self::counter_value(&self.compensation_running_tasks)
     }
 
-    pub async fn worker(api: Arc<ECStore>) {
+    async fn worker_with_cancel(api: Arc<ECStore>, cancel_token: CancellationToken) {
         loop {
             select! {
-                _ = GLOBAL_TransitionState.kill_rx.recv() => {
+                biased;
+
+                _ = cancel_token.cancelled() => {
                     return;
                 }
                 task = GLOBAL_TransitionState.transition_rx.recv() => {
@@ -952,33 +958,45 @@ impl TransitionState {
         }
         // Allow environment override of maximum workers
         let absolute_max = resolve_transition_workers_absolute_max();
-        n = std::cmp::min(n, absolute_max);
+        n = n.clamp(0, absolute_max);
 
-        let previous_num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        let mut num_workers = previous_num_workers;
-        while num_workers < n {
+        Self::resize_workers_to(api, n, requested, absolute_max);
+    }
+
+    fn resize_workers_to(api: Arc<ECStore>, n: i64, requested: i64, absolute_max: i64) {
+        let target = n as usize;
+        let mut workers = GLOBAL_TransitionState.workers.lock().unwrap();
+        let tracked_workers = workers.len();
+        workers.retain(|worker| !worker.handle.is_finished());
+        let pruned_finished_workers = tracked_workers.saturating_sub(workers.len());
+        let previous_num_workers = workers.len() as i64;
+
+        while workers.len() < target {
             let clone_api = api.clone();
-            tokio::spawn(async move {
-                TransitionState::worker(clone_api).await;
+            let cancel = CancellationToken::new();
+            let worker_cancel = cancel.clone();
+            let handle = tokio::spawn(async move {
+                TransitionState::worker_with_cancel(clone_api, worker_cancel).await;
             });
-            num_workers += 1;
-            GLOBAL_TransitionState.num_workers.fetch_add(1, Ordering::SeqCst);
+            workers.push(TransitionWorker { cancel, handle });
         }
 
-        let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        while num_workers > n {
-            let worker = GLOBAL_TransitionState.kill_tx.clone();
-            let _ = worker.send(()).await;
-            num_workers -= 1;
-            GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
+        while workers.len() > target {
+            if let Some(worker) = workers.pop() {
+                worker.cancel.cancel();
+            }
         }
 
+        let current_workers = workers.len() as i64;
+        GLOBAL_TransitionState.num_workers.store(current_workers, Ordering::SeqCst);
+
         info!(
             requested_transition_workers = requested,
             effective_transition_workers = n,
             absolute_max_workers = absolute_max,
             previous_transition_workers = previous_num_workers,
-            current_transition_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst),
+            current_transition_workers = current_workers,
+            pruned_finished_transition_workers = pruned_finished_workers,
             "transition workers updated"
         );
     }
@@ -1850,11 +1868,11 @@ pub async fn put_restore_opts(
             ..Default::default()
         });
     }
-    for (k, v) in &oi.user_defined {
+    for (k, v) in oi.user_defined.iter() {
         meta.insert(k.to_string(), v.clone());
     }
     if !oi.user_tags.is_empty() {
-        meta.insert(AMZ_OBJECT_TAGGING.to_string(), oi.user_tags.clone());
+        meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
     }
     let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
     meta.insert(
@@ -1885,7 +1903,7 @@ impl LifecycleOps for ObjectInfo {
     fn to_lifecycle_opts(&self) -> lifecycle::ObjectOpts {
         lifecycle::ObjectOpts {
             name: self.name.clone(),
-            user_tags: self.user_tags.clone(),
+            user_tags: (*self.user_tags).clone(),
             version_id: self.version_id,
             mod_time: self.mod_time,
             size: self.size as usize,
@@ -2264,7 +2282,8 @@ mod tests {
         lifecycle_rule_has_date_expiration, lifecycle_version_purge_state_from_completed_targets,
         mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate, replication_state_for_delete,
         resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
-        should_defer_date_expiry_for_recent_config_update, should_reuse_lifecycle_delete_replication_state,
+        resolve_transition_workers_absolute_max, should_defer_date_expiry_for_recent_config_update,
+        should_reuse_lifecycle_delete_replication_state,
     };
     use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
     use crate::bucket::metadata_sys;
@@ -2655,15 +2674,45 @@ mod tests {
         })
         .await;
 
-        let current_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
-        if original_workers > 0 {
-            TransitionState::update_workers(ecstore, original_workers).await;
-        } else {
-            for _ in 0..current_workers {
-                let _ = GLOBAL_TransitionState.kill_tx.send(()).await;
-                GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
-            }
-        }
+        let absolute_max = resolve_transition_workers_absolute_max();
+        TransitionState::resize_workers_to(ecstore, original_workers, original_workers, absolute_max);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn transition_worker_resize_cancels_removed_workers_directly() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let original_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        let absolute_max = resolve_transition_workers_absolute_max();
+
+        TransitionState::resize_workers_to(ecstore.clone(), 0, 0, absolute_max);
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 0);
+
+        TransitionState::resize_workers_to(ecstore.clone(), 2, 2, absolute_max);
+        let worker_tokens = {
+            let workers = GLOBAL_TransitionState.workers.lock().unwrap();
+            assert_eq!(workers.len(), 2);
+            workers.iter().map(|worker| worker.cancel.clone()).collect::<Vec<_>>()
+        };
+
+        TransitionState::resize_workers_to(ecstore.clone(), 1, 1, absolute_max);
+
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 1);
+        assert_eq!(worker_tokens.iter().filter(|token| token.is_cancelled()).count(), 1);
+
+        let remaining_token = {
+            let workers = GLOBAL_TransitionState.workers.lock().unwrap();
+            assert_eq!(workers.len(), 1);
+            let token = workers[0].cancel.clone();
+            assert!(!token.is_cancelled());
+            token
+        };
+
+        TransitionState::resize_workers_to(ecstore.clone(), 0, 0, absolute_max);
+        assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 0);
+        assert!(remaining_token.is_cancelled());
+
+        TransitionState::resize_workers_to(ecstore, original_workers, original_workers, absolute_max);
     }
 
     #[test]
@@ -2994,6 +3043,52 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    #[ignore = "requires isolated global object layer state"]
+    #[serial]
+    async fn ecstore_new_succeeds_on_fresh_local_volumes() {
+        let test_base_dir = format!("/tmp/rustfs_ecstore_empty_boot_{}", Uuid::new_v4());
+        let temp_dir = PathBuf::from(&test_base_dir);
+        if temp_dir.exists() {
+            fs::remove_dir_all(&temp_dir).await.ok();
+        }
+        fs::create_dir_all(&temp_dir).await.unwrap();
+
+        let disk_paths = vec![
+            temp_dir.join("disk1"),
+            temp_dir.join("disk2"),
+            temp_dir.join("disk3"),
+            temp_dir.join("disk4"),
+        ];
+
+        for disk_path in &disk_paths {
+            fs::create_dir_all(disk_path).await.unwrap();
+        }
+
+        let mut endpoints = Vec::new();
+        for (i, disk_path) in disk_paths.iter().enumerate() {
+            let mut endpoint = Endpoint::try_from(disk_path.to_str().unwrap()).unwrap();
+            endpoint.set_pool_index(0);
+            endpoint.set_set_index(0);
+            endpoint.set_disk_index(i);
+            endpoints.push(endpoint);
+        }
+
+        let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 4,
+            endpoints: Endpoints::from(endpoints),
+            cmd_line: "fresh-boot-test".to_string(),
+            platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
+        }]);
+
+        crate::store::init_local_disks(endpoint_pools.clone()).await.unwrap();
+
+        let ecstore = ECStore::new("127.0.0.1:0".parse().unwrap(), endpoint_pools, CancellationToken::new()).await;
+        assert!(ecstore.is_ok(), "fresh local ECStore boot should succeed, got {ecstore:?}");
+    }
+
     #[tokio::test]
     #[serial]
     async fn stale_multipart_cleanup_uses_default_expiry_without_lifecycle() {
@@ -3138,6 +3233,76 @@ mod tests {
         assert!(!parts.user_defined.contains_key(RUSTFS_MULTIPART_OBJECT_KEY));
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn repeated_upload_part_overwrites_previous_part_state() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let bucket = format!("multipart-overwrite-{}", Uuid::new_v4().simple());
+        let object = "overwrite/object.txt";
+        create_test_bucket(&ecstore, &bucket).await;
+
+        let upload = ecstore
+            .new_multipart_upload(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect("multipart upload should be created");
+
+        let mut first = PutObjReader::from_vec(vec![1, 2, 3]);
+        let first_part = ecstore
+            .put_object_part(&bucket, object, &upload.upload_id, 1, &mut first, &ObjectOptions::default())
+            .await
+            .expect("first multipart part should be uploaded");
+
+        let mut second = PutObjReader::from_vec(vec![4, 5, 6, 7]);
+        let second_part = ecstore
+            .put_object_part(&bucket, object, &upload.upload_id, 1, &mut second, &ObjectOptions::default())
+            .await
+            .expect("second multipart part should overwrite the previous part");
+
+        assert_ne!(
+            first_part.etag, second_part.etag,
+            "the overwrite path should persist the latest part metadata rather than reusing stale state"
+        );
+
+        let parts = ecstore
+            .list_object_parts(
+                &bucket,
+                object,
+                &upload.upload_id,
+                None,
+                crate::set_disk::MAX_PARTS_COUNT,
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("multipart parts should be readable after overwrite");
+
+        assert_eq!(parts.parts.len(), 1, "only the latest version of part 1 should remain visible");
+        assert_eq!(parts.parts[0].part_num, 1);
+        assert_eq!(parts.parts[0].etag, second_part.etag);
+        assert_eq!(parts.parts[0].size, second_part.size);
+        assert_eq!(parts.parts[0].actual_size, second_part.actual_size);
+
+        let completed = ecstore
+            .complete_multipart_upload(
+                &bucket,
+                object,
+                &upload.upload_id,
+                vec![crate::store_api::CompletePart {
+                    part_num: 1,
+                    etag: second_part.etag.clone(),
+                    checksum_crc32: None,
+                    checksum_crc32c: None,
+                    checksum_sha1: None,
+                    checksum_sha256: None,
+                    checksum_crc64nvme: None,
+                }],
+                &ObjectOptions::default(),
+            )
+            .await
+            .expect("complete multipart upload should succeed with the latest overwritten part");
+
+        assert_eq!(completed.size, second_part.size as i64);
+    }
+
     #[tokio::test]
     #[serial]
     async fn cleanup_removes_empty_multipart_sha_dirs() {

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -881,7 +881,7 @@ impl ObjectOpts {
     pub fn from_object_info(oi: &ObjectInfo) -> Self {
         Self {
             name: oi.name.clone(),
-            user_tags: oi.user_tags.clone(),
+            user_tags: (*oi.user_tags).clone(),
             mod_time: oi.mod_time,
             size: oi.size as usize,
             version_id: oi.version_id,
@@ -894,7 +894,7 @@ impl ObjectOpts {
             restore_expires: oi.restore_expires,
             versioned: false,
             version_suspended: false,
-            user_defined: oi.user_defined.clone(),
+            user_defined: (*oi.user_defined).clone(),
             version_purge_status: oi.version_purge_status.clone(),
             replication_status: oi.replication_status.clone(),
         }

crates/ecstore/src/bucket/metadata.rs

@@ -244,6 +244,8 @@ pub const BUCKET_ACCELERATE_CONFIG: &str = "accelerate.xml";
 pub const BUCKET_REQUEST_PAYMENT_CONFIG: &str = "request-payment.xml";
 pub const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = "public-access-block.xml";
 pub const BUCKET_ACL_CONFIG: &str = "bucket-acl.json";
+pub const BUCKET_TABLE_CONFIG: &str = "table-bucket.json";
+pub const BUCKET_TABLE_RESERVED_PREFIX: &str = ".rustfs-table";
 
 #[derive(Debug, Clone)]
 pub struct BucketMetadata {
@@ -268,6 +270,7 @@ pub struct BucketMetadata {
     pub request_payment_config_xml: Vec<u8>,
     pub public_access_block_config_xml: Vec<u8>,
     pub bucket_acl_config_json: Vec<u8>,
+    pub table_bucket_config_json: Vec<u8>,
 
     pub policy_config_updated_at: OffsetDateTime,
     pub object_lock_config_updated_at: OffsetDateTime,
@@ -287,6 +290,7 @@ pub struct BucketMetadata {
     pub request_payment_config_updated_at: OffsetDateTime,
     pub public_access_block_config_updated_at: OffsetDateTime,
     pub bucket_acl_config_updated_at: OffsetDateTime,
+    pub table_bucket_config_updated_at: OffsetDateTime,
 
     pub new_field_updated_at: OffsetDateTime,
 
@@ -334,6 +338,7 @@ impl Default for BucketMetadata {
             request_payment_config_xml: Default::default(),
             public_access_block_config_xml: Default::default(),
             bucket_acl_config_json: Default::default(),
+            table_bucket_config_json: Default::default(),
             policy_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             object_lock_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             encryption_config_updated_at: OffsetDateTime::UNIX_EPOCH,
@@ -352,6 +357,7 @@ impl Default for BucketMetadata {
             request_payment_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             public_access_block_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             bucket_acl_config_updated_at: OffsetDateTime::UNIX_EPOCH,
+            table_bucket_config_updated_at: OffsetDateTime::UNIX_EPOCH,
             new_field_updated_at: OffsetDateTime::UNIX_EPOCH,
             policy_config: Default::default(),
             notification_config: Default::default(),
@@ -397,6 +403,10 @@ impl BucketMetadata {
         self.lock_enabled || (self.versioning_config.as_ref().is_some_and(|v| v.enabled()))
     }
 
+    pub fn table_bucket_enabled(&self) -> bool {
+        !self.table_bucket_config_json.is_empty()
+    }
+
     /// Decode from msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn decode_from<R: Read>(&mut self, rd: &mut R) -> Result<()> {
         let mut fields = rmp::decode::read_map_len(rd)?;
@@ -447,6 +457,7 @@ impl BucketMetadata {
                     self.public_access_block_config_xml = read_msgp_bin(rd)?
                 }
                 "BucketAclConfigJSON" | "BucketAclConfigJson" => self.bucket_acl_config_json = read_msgp_bin(rd)?,
+                "TableBucketConfigJSON" | "TableBucketConfigJson" => self.table_bucket_config_json = read_msgp_bin(rd)?,
                 "CorsConfigUpdatedAt" => self.cors_config_updated_at = read_msgp_time_value(rd)?,
                 "LoggingConfigUpdatedAt" => self.logging_config_updated_at = read_msgp_time_value(rd)?,
                 "WebsiteConfigUpdatedAt" => self.website_config_updated_at = read_msgp_time_value(rd)?,
@@ -454,6 +465,7 @@ impl BucketMetadata {
                 "RequestPaymentConfigUpdatedAt" => self.request_payment_config_updated_at = read_msgp_time_value(rd)?,
                 "PublicAccessBlockConfigUpdatedAt" => self.public_access_block_config_updated_at = read_msgp_time_value(rd)?,
                 "BucketAclConfigUpdatedAt" => self.bucket_acl_config_updated_at = read_msgp_time_value(rd)?,
+                "TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
                 other => {
                     tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
                     skip_msgp_value(rd)?;
@@ -466,8 +478,8 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Map size: MinIO fields (25) + RustFS extensions (14)
-        let map_len: u32 = 39;
+        // Map size: MinIO fields (25) + RustFS extensions (16)
+        let map_len: u32 = 41;
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -523,6 +535,7 @@ impl BucketMetadata {
         write_bin_field(wr, "RequestPaymentConfigXML", &self.request_payment_config_xml)?;
         write_bin_field(wr, "PublicAccessBlockConfigXML", &self.public_access_block_config_xml)?;
         write_bin_field(wr, "BucketAclConfigJSON", &self.bucket_acl_config_json)?;
+        write_bin_field(wr, "TableBucketConfigJSON", &self.table_bucket_config_json)?;
         rmp::encode::write_str(wr, "CorsConfigUpdatedAt")?;
         write_msgp_time(wr, self.cors_config_updated_at)?;
         rmp::encode::write_str(wr, "LoggingConfigUpdatedAt")?;
@@ -537,6 +550,8 @@ impl BucketMetadata {
         write_msgp_time(wr, self.public_access_block_config_updated_at)?;
         rmp::encode::write_str(wr, "BucketAclConfigUpdatedAt")?;
         write_msgp_time(wr, self.bucket_acl_config_updated_at)?;
+        rmp::encode::write_str(wr, "TableBucketConfigUpdatedAt")?;
+        write_msgp_time(wr, self.table_bucket_config_updated_at)?;
 
         Ok(())
     }
@@ -632,6 +647,9 @@ impl BucketMetadata {
         if self.bucket_acl_config_updated_at == OffsetDateTime::UNIX_EPOCH {
             self.bucket_acl_config_updated_at = self.created
         }
+        if self.table_bucket_config_updated_at == OffsetDateTime::UNIX_EPOCH {
+            self.table_bucket_config_updated_at = self.created
+        }
     }
 
     pub fn update_config(&mut self, config_file: &str, data: Vec<u8>) -> Result<OffsetDateTime> {
@@ -709,6 +727,10 @@ impl BucketMetadata {
                 self.bucket_acl_config_json = data;
                 self.bucket_acl_config_updated_at = updated;
             }
+            BUCKET_TABLE_CONFIG => {
+                self.table_bucket_config_json = data;
+                self.table_bucket_config_updated_at = updated;
+            }
             _ => return Err(Error::other(format!("config file not found : {config_file}"))),
         }
 
@@ -1028,6 +1050,10 @@ mod test {
         bm.bucket_acl_config_json = bucket_acl.as_bytes().to_vec();
         bm.bucket_acl_config_updated_at = OffsetDateTime::now_utc();
 
+        let table_bucket_marker = r#"{"enabled":true}"#;
+        bm.table_bucket_config_json = table_bucket_marker.as_bytes().to_vec();
+        bm.table_bucket_config_updated_at = OffsetDateTime::now_utc();
+
         // Test serialization
         let buf = bm.marshal_msg().unwrap();
         assert!(!buf.is_empty(), "Serialized buffer should not be empty");
@@ -1049,6 +1075,7 @@ mod test {
         assert_eq!(bm.quota_config_json, deserialized_bm.quota_config_json);
         assert_eq!(bm.public_access_block_config_xml, deserialized_bm.public_access_block_config_xml);
         assert_eq!(bm.bucket_acl_config_json, deserialized_bm.bucket_acl_config_json);
+        assert_eq!(bm.table_bucket_config_json, deserialized_bm.table_bucket_config_json);
         assert_eq!(bm.object_lock_config_xml, deserialized_bm.object_lock_config_xml);
         assert_eq!(bm.notification_config_xml, deserialized_bm.notification_config_xml);
         assert_eq!(bm.replication_config_xml, deserialized_bm.replication_config_xml);
@@ -1100,6 +1127,11 @@ mod test {
             bm.bucket_targets_config_meta_updated_at.unix_timestamp(),
             deserialized_bm.bucket_targets_config_meta_updated_at.unix_timestamp()
         );
+        assert_eq!(
+            bm.table_bucket_config_updated_at.unix_timestamp(),
+            deserialized_bm.table_bucket_config_updated_at.unix_timestamp()
+        );
+        assert!(deserialized_bm.table_bucket_enabled());
 
         // Test that the serialized data contains expected content
         let buf_str = String::from_utf8_lossy(&buf);
@@ -1116,6 +1148,20 @@ mod test {
         println!("   - Serialized buffer size: {} bytes", buf.len());
     }
 
+    #[test]
+    fn table_bucket_marker_tracks_config_presence() {
+        let mut bm = BucketMetadata::new("table-bucket");
+        assert!(!bm.table_bucket_enabled());
+
+        bm.update_config(BUCKET_TABLE_CONFIG, br#"{"enabled":true}"#.to_vec())
+            .unwrap();
+        assert!(bm.table_bucket_enabled());
+        assert!(!bm.table_bucket_config_json.is_empty());
+
+        bm.update_config(BUCKET_TABLE_CONFIG, Vec::new()).unwrap();
+        assert!(!bm.table_bucket_enabled());
+    }
+
     /// After policy deletion (policy_config_json cleared), parse_policy_config sets policy_config to None.
     #[test]
     fn test_parse_policy_config_clears_cache_when_json_empty() {

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -1079,7 +1079,7 @@ pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc:
         target_statuses: tgt_statuses,
         target_purge_statuses: purge_statuses,
         replication_timestamp: tm,
-        user_tags: oi.user_tags,
+        user_tags: (*oi.user_tags).clone(),
         checksum: None,
         retry_count: 0,
         event_type: "".to_string(),

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -934,7 +934,7 @@ fn heal_should_use_check_replicate_delete(oi: &ObjectInfo) -> bool {
 
 pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationConfig) -> ReplicateObjectInfo {
     let mut oi = oi.clone();
-    let mut user_defined = oi.user_defined.clone();
+    let mut user_defined = (*oi.user_defined).clone();
 
     if let Some(rc) = rcfg.config.as_ref()
         && !rc.role.is_empty()
@@ -982,7 +982,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
             &oi.name,
             MustReplicateOptions::new(
                 &user_defined,
-                oi.user_tags.clone(),
+                (*oi.user_tags).clone(),
                 ReplicationStatusType::Empty,
                 ReplicationType::Heal,
                 ObjectOptions::default(),
@@ -1020,7 +1020,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
         target_purge_statuses,
         replication_timestamp: None,
         ssec: false, // TODO: add ssec support
-        user_tags: oi.user_tags.clone(),
+        user_tags: (*oi.user_tags).clone(),
         checksum: oi.checksum.clone(),
         retry_count: 0,
     }
@@ -1158,7 +1158,7 @@ impl ReplicationConfig {
             return self.resync_internal(oi, dsc, status);
         }
 
-        let mut user_defined = oi.user_defined.clone();
+        let mut user_defined = (*oi.user_defined).clone();
         user_defined.remove(AMZ_BUCKET_REPLICATION_STATUS);
 
         let dsc = must_replicate(
@@ -1166,7 +1166,7 @@ impl ReplicationConfig {
             &oi.name,
             MustReplicateOptions::new(
                 &user_defined,
-                oi.user_tags.clone(),
+                (*oi.user_tags).clone(),
                 ReplicationStatusType::Empty,
                 ReplicationType::ExistingObject,
                 ObjectOptions::default(),
@@ -1296,7 +1296,7 @@ impl MustReplicateOptions {
     }
 
     pub fn from_object_info(oi: &ObjectInfo, op_type: ReplicationType, opts: ObjectOptions) -> Self {
-        Self::new(&oi.user_defined, oi.user_tags.clone(), oi.replication_status.clone(), op_type, opts)
+        Self::new(&oi.user_defined, (*oi.user_tags).clone(), oi.replication_status.clone(), op_type, opts)
     }
 
     pub fn replication_status(&self) -> ReplicationStatusType {
@@ -1411,7 +1411,7 @@ fn delete_replication_object_opts(dobj: &ObjectToDelete, oi: &ObjectInfo) -> Obj
     ObjectOpts {
         name: dobj.object_name.clone(),
         ssec: is_ssec_encrypted(&oi.user_defined),
-        user_tags: oi.user_tags.clone(),
+        user_tags: (*oi.user_tags).clone(),
         delete_marker: oi.delete_marker,
         version_id: dobj.version_id,
         op_type: ReplicationType::Delete,
@@ -2995,7 +2995,7 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
             mod_time: self.mod_time,
             version_id: self.version_id,
             size: self.size,
-            user_tags: self.user_tags.clone(),
+            user_tags: Arc::new(self.user_tags.clone()),
             actual_size: self.actual_size,
             replication_status_internal: self.replication_status_internal.clone(),
             replication_status: self.replication_status.clone(),
@@ -3221,7 +3221,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
     }
 
     // Use case-insensitive lookup for headers
-    let lk_map = object_info.user_defined.clone();
+    let lk_map = &*object_info.user_defined;
 
     if let Some(lang) = lk_map.lookup(CONTENT_LANGUAGE) {
         put_op.content_language = lang.to_string();
@@ -3500,7 +3500,7 @@ fn get_replication_action(oi1: &ObjectInfo, oi2: &HeadObjectOutput, op_type: Rep
 
     // compare metadata on both maps to see if meta is identical
     let mut compare_meta1 = HashMap::new();
-    for (k, v) in &oi1.user_defined {
+    for (k, v) in oi1.user_defined.iter() {
         let mut found = false;
         for prefix in &compare_keys {
             if strings_has_prefix_fold(k, prefix) {

crates/ecstore/src/cache_value/metacache_set.rs

@@ -442,10 +442,13 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 {
                     finished_fn(&errs).await;
                 }
-                if errs.iter().flatten().any(|err| *err == DiskError::Timeout) {
-                    return Err(DiskError::Timeout);
-                }
-
+                // All remaining readers are either at EOF or failed. Earlier logic
+                // returned Timeout here for even a single stalled drive, despite
+                // `has_err` being within the tolerated drive-failure budget. That
+                // makes small distributed listings fail once healthy quorum readers
+                // reach EOF but one remote walk stream is slow/stalled. Only the
+                // `has_err > opts.disks.len() - opts.min_disks` branch above should
+                // turn tolerated reader failures into request failures.
                 // error!("list_path_raw: at_eof + has_err == readers.len() break {:?}", &errs);
                 break;
             }
@@ -486,13 +489,23 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
         return Err(err);
     }
 
+    // The merge consumer can finish successfully before every producer finishes
+    // (for example after reaching EOF quorum while a tolerated drive is stalled,
+    // or after the requested listing limit is satisfied). Cancel remaining walk
+    // jobs before joining them so list calls do not wait for slow remote streams.
+    cancel_rx.cancel();
+
     let results = join_all(jobs).await;
     let mut job_errs = Vec::new();
     for result in results {
         match result {
             Ok(Ok(())) => {}
             Ok(Err(err)) => {
-                error!("list_path_raw producer err {:?}", err);
+                if matches!(err, DiskError::FileNotFound | DiskError::VolumeNotFound) {
+                    warn!("list_path_raw producer missing path {:?}", err);
+                } else {
+                    error!("list_path_raw producer err {:?}", err);
+                }
                 job_errs.push(err);
             }
             Err(err) => {
@@ -541,18 +554,34 @@ mod tests {
             CancellationToken::new(),
             ListPathRawOptions {
                 disks: vec![None, None],
-                min_disks: 1,
+                min_disks: 2,
                 test_reader_behaviors: vec![TestReaderBehavior::Stall, TestReaderBehavior::Eof],
                 peek_timeout: Some(Duration::from_millis(20)),
                 ..Default::default()
             },
         )
         .await
-        .expect_err("stalled reader should make listing fail explicitly");
+        .expect_err("stalled reader should fail when read quorum cannot be met");
 
         assert_eq!(err, DiskError::Timeout);
     }
 
+    #[tokio::test]
+    async fn list_path_raw_tolerates_stalled_reader_after_quorum_eof() {
+        list_path_raw(
+            CancellationToken::new(),
+            ListPathRawOptions {
+                disks: vec![None, None, None],
+                min_disks: 2,
+                test_reader_behaviors: vec![TestReaderBehavior::Eof, TestReaderBehavior::Eof, TestReaderBehavior::Stall],
+                peek_timeout: Some(Duration::from_millis(20)),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("listing should complete when healthy quorum reached EOF and only a tolerated drive stalled");
+    }
+
     #[tokio::test]
     async fn list_path_raw_returns_timeout_when_producer_fails_after_partial_entry() {
         let seen = Arc::new(Mutex::new(Vec::new()));

crates/ecstore/src/config/com.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::config::{Config, GLOBAL_STORAGE_CLASS, KVS, audit, notify, oidc, storageclass};
+use crate::config::{Config, KVS, audit, notify, oidc, set_global_storage_class, storageclass};
 use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result};
 use crate::global::is_first_cluster_node_local;
@@ -1164,11 +1164,8 @@ async fn apply_dynamic_config_for_sub_sys<S: StorageAPI>(cfg: &mut Config, api:
         for (i, count) in set_drive_counts.iter().enumerate() {
             match storageclass::lookup_config(&kvs, *count) {
                 Ok(res) => {
-                    if i == 0
-                        && GLOBAL_STORAGE_CLASS.get().is_none()
-                        && let Err(r) = GLOBAL_STORAGE_CLASS.set(res)
-                    {
-                        error!("GLOBAL_STORAGE_CLASS.set failed {:?}", r);
+                    if i == 0 {
+                        set_global_storage_class(res);
                     }
                 }
                 Err(err) => {
@@ -1358,7 +1355,7 @@ mod tests {
                 size: self.data.len() as i64,
                 actual_size: self.data.len() as i64,
                 is_dir: false,
-                user_defined: HashMap::new(),
+                user_defined: Arc::new(HashMap::new()),
                 parity_blocks: 0,
                 data_blocks: 0,
                 version_id: None,
@@ -1366,8 +1363,8 @@ mod tests {
                 transitioned_object: Default::default(),
                 restore_ongoing: false,
                 restore_expires: None,
-                user_tags: String::new(),
-                parts: Vec::new(),
+                user_tags: Arc::new(String::new()),
+                parts: Arc::new(Vec::new()),
                 is_latest: true,
                 content_type: Some("application/json".to_string()),
                 content_encoding: None,
@@ -2638,4 +2635,87 @@ mod tests {
         assert_eq!(object_info.bucket, crate::disk::RUSTFS_META_BUCKET);
         assert_eq!(object_info.name, "config/test.json");
     }
+
+    #[test]
+    fn test_encode_decode_roundtrip_preserves_storage_class() {
+        use crate::config::STORAGE_CLASS_SUB_SYS;
+
+        // Create a config with custom storage class values
+        let mut cfg = Config::new();
+        let kvs = storage_class_kvs_mut(&mut cfg);
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        kvs.insert("rrs".to_string(), "EC:2".to_string());
+        kvs.insert("optimize".to_string(), "availability".to_string());
+
+        // Encode to external format (what save_server_config produces)
+        let encoded = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+
+        // Verify the encoded data uses "storageclass" (no underscore)
+        let v: serde_json::Value = serde_json::from_slice(&encoded).expect("should be valid json");
+        assert!(v.get("storageclass").is_some(), "encoded should have 'storageclass' key");
+        assert!(v.get("storage_class").is_none(), "encoded should NOT have 'storage_class' key");
+
+        // Decode back (what load_server_config_from_store produces)
+        let decoded = decode_server_config_blob(&encoded).expect("decode should succeed");
+
+        // Verify the decoded config has "storage_class" (with underscore) subsystem
+        let kvs = decoded
+            .get_value(STORAGE_CLASS_SUB_SYS, crate::config::DEFAULT_DELIMITER)
+            .expect("decoded config should have storage_class subsystem");
+        assert_eq!(kvs.get("standard"), "EC:4", "standard should be EC:4");
+        assert_eq!(kvs.get("rrs"), "EC:2", "rrs should be EC:2");
+        assert_eq!(kvs.get("optimize"), "availability", "optimize should be availability");
+    }
+
+    #[test]
+    fn test_set_then_load_preserves_storage_class_values() {
+        use crate::config::STORAGE_CLASS_SUB_SYS;
+
+        // Step 1: Start with a fresh config (simulates initial server state)
+        let mut cfg = Config::new();
+
+        // Step 2: Apply set directives (simulates "mc admin config set storage_class standard=EC:4 rrs=EC:2")
+        let kvs = cfg
+            .0
+            .entry(STORAGE_CLASS_SUB_SYS.to_string())
+            .or_default()
+            .entry(DEFAULT_DELIMITER.to_string())
+            .or_default();
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        kvs.insert("rrs".to_string(), "EC:2".to_string());
+        cfg.set_defaults();
+
+        // Step 3: Save (encode to external format)
+        let encoded = encode_server_config_blob(&cfg, None).expect("encode should succeed");
+
+        // Step 4: Load (decode from external format)
+        let loaded = decode_server_config_blob(&encoded).expect("decode should succeed");
+
+        // Step 5: Verify the values are preserved
+        let loaded_kvs = loaded
+            .get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("should have storage_class");
+        assert_eq!(loaded_kvs.get("standard"), "EC:4");
+        assert_eq!(loaded_kvs.get("rrs"), "EC:2");
+
+        // Step 6: Verify the subsystem is accessible by name (what render_selected_config does)
+        let targets = loaded
+            .0
+            .get(STORAGE_CLASS_SUB_SYS)
+            .expect("storage_class subsystem should exist");
+        let target_kvs = targets.get(DEFAULT_DELIMITER).expect("default target should exist");
+        assert_eq!(target_kvs.get("standard"), "EC:4");
+        assert_eq!(target_kvs.get("rrs"), "EC:2");
+    }
+
+    #[test]
+    fn test_config_unmarshal_fails_on_external_format() {
+        // This verifies that the external "storageclass" format cannot be
+        // mistakenly parsed by Config::unmarshal (which expects internal format).
+        // If unmarshal succeeds, the config would have "storageclass" instead of
+        // "storage_class", causing render_selected_config to fail.
+        let external_json = r#"{"version":"33","storageclass":{"standard":"EC:4","rrs":"EC:2"}}"#;
+        let result = Config::unmarshal(external_json.as_bytes());
+        assert!(result.is_err(), "Config::unmarshal should fail on external format, but got: {:?}", result);
+    }
 }

crates/ecstore/src/config/mod.rs

@@ -18,6 +18,7 @@ pub mod com;
 pub mod heal;
 mod notify;
 mod oidc;
+mod scanner;
 pub mod storageclass;
 
 use crate::error::Result;
@@ -37,11 +38,12 @@ use rustfs_config::oidc::IDENTITY_OPENID_SUB_SYS;
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::sync::LazyLock;
-use std::sync::{Arc, OnceLock};
+use std::sync::{Arc, OnceLock, RwLock};
 
-pub static GLOBAL_STORAGE_CLASS: LazyLock<OnceLock<storageclass::Config>> = LazyLock::new(OnceLock::new);
+pub static GLOBAL_STORAGE_CLASS: LazyLock<RwLock<storageclass::Config>> =
+    LazyLock::new(|| RwLock::new(storageclass::Config::default()));
 pub static DEFAULT_KVS: LazyLock<OnceLock<HashMap<String, KVS>>> = LazyLock::new(OnceLock::new);
-pub static GLOBAL_SERVER_CONFIG: LazyLock<OnceLock<Config>> = LazyLock::new(OnceLock::new);
+pub static GLOBAL_SERVER_CONFIG: LazyLock<RwLock<Option<Config>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_CONFIG_SYS: LazyLock<ConfigSys> = LazyLock::new(ConfigSys::new);
 
 pub static RUSTFS_CONFIG_PREFIX: &str = "config";
@@ -63,14 +65,30 @@ impl ConfigSys {
 
         lookup_configs(&mut cfg, api).await;
 
-        let _ = GLOBAL_SERVER_CONFIG.set(cfg);
+        set_global_server_config(cfg);
 
         Ok(())
     }
 }
 
 pub fn get_global_server_config() -> Option<Config> {
-    GLOBAL_SERVER_CONFIG.get().cloned()
+    GLOBAL_SERVER_CONFIG.read().ok().and_then(|guard| (*guard).clone())
+}
+
+pub fn set_global_server_config(cfg: Config) {
+    if let Ok(mut guard) = GLOBAL_SERVER_CONFIG.write() {
+        *guard = Some(cfg);
+    }
+}
+
+pub fn get_global_storage_class() -> Option<storageclass::Config> {
+    GLOBAL_STORAGE_CLASS.read().ok().map(|guard| (*guard).clone())
+}
+
+pub fn set_global_storage_class(cfg: storageclass::Config) {
+    if let Ok(mut guard) = GLOBAL_STORAGE_CLASS.write() {
+        *guard = cfg;
+    }
 }
 
 pub async fn init_global_config_sys(api: Arc<ECStore>) -> Result<()> {
@@ -237,6 +255,7 @@ pub fn init() {
     let mut kvs = HashMap::new();
     // Load storageclass default configuration
     kvs.insert(STORAGE_CLASS_SUB_SYS.to_owned(), storageclass::DEFAULT_KVS.clone());
+    kvs.insert(rustfs_config::SCANNER_SUB_SYS.to_owned(), scanner::DEFAULT_KVS.clone());
     // New: Loading default configurations for notify_webhook and notify_mqtt
     // Referring subsystem names through constants to improve the readability and maintainability of the code
     kvs.insert(NOTIFY_WEBHOOK_SUB_SYS.to_owned(), notify::DEFAULT_NOTIFY_WEBHOOK_KVS.clone());
@@ -262,3 +281,38 @@ pub fn init() {
     // Register all default configurations
     register_default_kvs(kvs)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use rustfs_config::{DEFAULT_DELIMITER, DEFAULT_SCANNER_SPEED, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_SPEED, SCANNER_SUB_SYS};
+
+    #[test]
+    fn global_server_config_set_and_get_roundtrip() {
+        init();
+        let mut cfg = Config::new();
+        let mut kvs = KVS::new();
+        kvs.insert("standard".to_string(), "EC:4".to_string());
+        cfg.0
+            .insert(STORAGE_CLASS_SUB_SYS.to_string(), HashMap::from([("_".to_string(), kvs)]));
+
+        set_global_server_config(cfg.clone());
+        let loaded = get_global_server_config().expect("global config should be set");
+        let sc_kvs = loaded
+            .get_value(STORAGE_CLASS_SUB_SYS, "_")
+            .expect("storage_class should exist");
+        assert_eq!(sc_kvs.get("standard"), "EC:4");
+    }
+
+    #[test]
+    fn scanner_defaults_are_registered_for_admin_config() {
+        init();
+        let cfg = Config::new();
+        let scanner_kvs = cfg
+            .get_value(SCANNER_SUB_SYS, DEFAULT_DELIMITER)
+            .expect("scanner defaults should exist");
+
+        assert_eq!(scanner_kvs.get(SCANNER_SPEED), DEFAULT_SCANNER_SPEED);
+        assert_eq!(scanner_kvs.get(SCANNER_CYCLE_MAX_OBJECTS), "0");
+    }
+}

crates/ecstore/src/config/scanner.rs

@@ -0,0 +1,106 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::config::{KV, KVS};
+use rustfs_config::{
+    DEFAULT_SCANNER_BITROT_CYCLE_SECS, DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS, DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES,
+    DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS, DEFAULT_SCANNER_CYCLE_MAX_OBJECTS, DEFAULT_SCANNER_IDLE_MODE,
+    DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS, DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS, DEFAULT_SCANNER_SPEED,
+    DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS, SCANNER_ALERT_EXCESS_FOLDERS, SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_VERSIONS, SCANNER_BITROT_CYCLE, SCANNER_CACHE_SAVE_TIMEOUT, SCANNER_CYCLE,
+    SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_IDLE_MODE,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS, SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_SPEED, SCANNER_START_DELAY,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
+};
+use std::sync::LazyLock;
+
+pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
+    KVS(vec![
+        KV {
+            key: SCANNER_SPEED.to_owned(),
+            value: DEFAULT_SCANNER_SPEED.to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_START_DELAY.to_owned(),
+            value: String::new(),
+            hidden_if_empty: true,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_DURATION.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_OBJECTS.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_OBJECTS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CYCLE_MAX_DIRECTORIES.to_owned(),
+            value: DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_BITROT_CYCLE.to_owned(),
+            value: DEFAULT_SCANNER_BITROT_CYCLE_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_IDLE_MODE.to_owned(),
+            value: DEFAULT_SCANNER_IDLE_MODE.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_CACHE_SAVE_TIMEOUT.to_owned(),
+            value: DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_MAX_CONCURRENT_SET_SCANS.to_owned(),
+            value: DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_MAX_CONCURRENT_DISK_SCANS.to_owned(),
+            value: DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_YIELD_EVERY_N_OBJECTS.to_owned(),
+            value: DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_VERSIONS.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_VERSION_SIZE.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: SCANNER_ALERT_EXCESS_FOLDERS.to_owned(),
+            value: rustfs_config::DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS.to_string(),
+            hidden_if_empty: false,
+        },
+    ])
+});

crates/ecstore/src/config/storageclass.rs

@@ -101,13 +101,13 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
 });
 
 // StorageClass - holds storage class information
-#[derive(Serialize, Deserialize, Debug, Default)]
+#[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct StorageClass {
     parity: usize,
 }
 
 // Config storage class configuration
-#[derive(Serialize, Deserialize, Debug, Default)]
+#[derive(Serialize, Deserialize, Debug, Default, Clone)]
 pub struct Config {
     standard: StorageClass,
     rrs: StorageClass,
@@ -205,7 +205,7 @@ pub fn lookup_config(kvs: &KVS, set_drive_count: usize) -> Result<Config> {
             if let Ok(ssc_str) = env::var(RRS_ENV) {
                 ssc_str
             } else {
-                kvs.get(RRS)
+                kvs.get(CLASS_RRS)
             }
         };
 
@@ -336,3 +336,36 @@ pub fn validate_parity_inner(ss_parity: usize, rrs_parity: usize, set_drive_coun
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn lookup_config_reads_rrs_from_class_rrs_key() {
+        // Regression: kvs.get(RRS) used RRS="REDUCED_REDUNDANCY" instead of
+        // CLASS_RRS="rrs", so admin-written RRS values were never read back.
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
+        kvs.insert(CLASS_RRS.to_string(), "EC:2".to_string());
+
+        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        assert_eq!(cfg.standard.parity, 4, "standard parity should be 4");
+        assert_eq!(cfg.rrs.parity, 2, "rrs parity should be 2");
+    }
+
+    #[test]
+    fn lookup_config_ignores_redundancy_key_name() {
+        // Ensure the old key name "REDUCED_REDUNDANCY" is NOT read.
+        let mut kvs = KVS::new();
+        kvs.insert(CLASS_STANDARD.to_string(), "EC:4".to_string());
+        kvs.insert(RRS.to_string(), "EC:2".to_string());
+
+        let cfg = lookup_config(&kvs, 8).expect("lookup should succeed");
+        assert_eq!(cfg.standard.parity, 4);
+        assert_eq!(
+            cfg.rrs.parity, DEFAULT_RRS_PARITY,
+            "rrs should fall back to default when CLASS_RRS key is absent"
+        );
+    }
+}

crates/ecstore/src/data_movement.rs

@@ -94,7 +94,7 @@ fn data_movement_new_multipart_opts(object_info: &ObjectInfo, src_pool_idx: usiz
     ObjectOptions {
         versioned: object_info.version_id.is_some(),
         version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
-        user_defined: object_info.user_defined.clone(),
+        user_defined: (*object_info.user_defined).clone(),
         preserve_etag: object_info.etag.clone(),
         src_pool_idx,
         data_movement: true,
@@ -120,7 +120,7 @@ fn data_movement_put_object_opts(object_info: &ObjectInfo, src_pool_idx: usize)
         data_movement: true,
         version_id: object_info.version_id.as_ref().map(|v| v.to_string()),
         mod_time: object_info.mod_time,
-        user_defined: object_info.user_defined.clone(),
+        user_defined: (*object_info.user_defined).clone(),
         preserve_etag: object_info.etag.clone(),
         ..Default::default()
     }
@@ -341,7 +341,7 @@ mod tests {
         let object_info = ObjectInfo {
             version_id: Some(version_id),
             etag: Some("etag-value".to_string()),
-            user_defined: std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())]),
+            user_defined: Arc::new(std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())])),
             ..Default::default()
         };
 
@@ -382,7 +382,7 @@ mod tests {
             version_id: Some(version_id),
             mod_time: Some(OffsetDateTime::UNIX_EPOCH),
             etag: Some("etag-value".to_string()),
-            user_defined: std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())]),
+            user_defined: Arc::new(std::collections::HashMap::from([("x-amz-meta-key".to_string(), "value".to_string())])),
             ..Default::default()
         };
 

crates/ecstore/src/disk/disk_store.rs

@@ -1140,7 +1140,8 @@ impl DiskAPI for LocalDiskWrapper {
             "walk_dir",
             || async { self.disk.walk_dir(opts, wr).await },
             get_drive_walkdir_timeout(),
-            self.scanner_timeout_health_action(),
+            // Listing/scanner backpressure should fail only the current walk, not poison drive health.
+            TimeoutHealthAction::IgnoreFailure,
         )
         .await
     }
@@ -1241,7 +1242,8 @@ impl DiskAPI for LocalDiskWrapper {
         dst_volume: &str,
         dst_path: &str,
     ) -> Result<RenameDataResp> {
-        self.track_disk_health(
+        self.track_disk_health_with_op(
+            "rename_data",
             || async { self.disk.rename_data(src_volume, src_path, fi, dst_volume, dst_path).await },
             get_max_timeout_duration(),
         )
@@ -1603,7 +1605,7 @@ mod tests {
     }
 
     #[tokio::test]
-    async fn walk_dir_writer_backpressure_timeout_marks_drive_failure_by_default() {
+    async fn walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure_by_default() {
         temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
             let dir = tempfile::tempdir().expect("temp dir should be created");
             let endpoint =
@@ -1639,7 +1641,60 @@ mod tests {
                 .await;
 
             assert_eq!(result.expect_err("walk_dir should time out"), DiskError::Timeout);
-            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Suspect);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    async fn walk_dir_timeout_does_not_break_followup_stat_volume() {
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+            let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+            let wrapper = LocalDiskWrapper::new(disk, false);
+            let bucket = "test-bucket";
+            let object = "test-object";
+
+            wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+            let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+            file_info.volume = bucket.to_string();
+            file_info.name = object.to_string();
+            file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+            file_info.erasure.index = 1;
+
+            wrapper
+                .write_metadata("", bucket, object, file_info)
+                .await
+                .expect("object metadata should be written");
+
+            let mut writer = PendingWriter;
+            let walk_err = wrapper
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await
+                .expect_err("walk_dir should time out");
+
+            assert_eq!(walk_err, DiskError::Timeout);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+
+            let info = wrapper
+                .stat_volume(bucket)
+                .await
+                .expect("follow-up bucket stat should still succeed after walk timeout");
+            assert_eq!(info.name, bucket);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
         })
         .await;
     }

crates/ecstore/src/disk/endpoint.rs

@@ -19,6 +19,23 @@ use std::{fmt::Display, path::Path};
 use tracing::debug;
 use url::{ParseError, Url};
 
+#[cfg(windows)]
+pub(crate) fn windows_fallback_local_path(
+    path: &str,
+    canonicalize_error: &std::io::Error,
+    context: &'static str,
+) -> std::io::Result<std::path::PathBuf> {
+    let absolute = Path::new(path).absolutize()?.to_path_buf();
+    tracing::warn!(
+        path = %path,
+        canonicalize_error = %canonicalize_error,
+        resolved = ?absolute,
+        context = context,
+        "using windows fallback path resolution for local endpoint"
+    );
+    Ok(absolute)
+}
+
 /// enum for endpoint type.
 #[derive(PartialEq, Eq, Debug)]
 pub enum EndpointType {

crates/ecstore/src/disk/fs.rs

@@ -161,33 +161,39 @@ pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
     fs::create_dir_all(path.as_ref()).await
 }
 
+fn is_dir_error(e: &io::Error) -> bool {
+    e.raw_os_error() == Some(libc::EISDIR)
+        || e.kind() == io::ErrorKind::IsADirectory
+        // macOS: remove_file on a directory returns EPERM
+        || (cfg!(target_os = "macos") && e.raw_os_error() == Some(libc::EPERM))
+}
+
 #[tracing::instrument(level = "debug", skip_all)]
 pub async fn remove(path: impl AsRef<Path>) -> io::Result<()> {
-    let meta = fs::metadata(path.as_ref()).await?;
-    if meta.is_dir() {
-        fs::remove_dir(path.as_ref()).await
-    } else {
-        fs::remove_file(path.as_ref()).await
+    // Try remove_file first; fall back to remove_dir if it's a directory
+    match fs::remove_file(path.as_ref()).await {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => fs::remove_dir(path.as_ref()).await,
+        Err(e) => Err(e),
     }
 }
 
 pub async fn remove_all(path: impl AsRef<Path>) -> io::Result<()> {
-    let meta = fs::metadata(path.as_ref()).await?;
-    if meta.is_dir() {
-        fs::remove_dir_all(path.as_ref()).await
-    } else {
-        fs::remove_file(path.as_ref()).await
+    // Try remove_file first; fall back to remove_dir_all if it's a directory
+    match fs::remove_file(path.as_ref()).await {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => fs::remove_dir_all(path.as_ref()).await,
+        Err(e) => Err(e),
     }
 }
 
 #[tracing::instrument(level = "debug", skip_all)]
 pub fn remove_std(path: impl AsRef<Path>) -> io::Result<()> {
-    let path = path.as_ref();
-    let meta = std::fs::metadata(path)?;
-    if meta.is_dir() {
-        std::fs::remove_dir(path)
-    } else {
-        std::fs::remove_file(path)
+    // Try remove_file first; fall back to remove_dir if it's a directory
+    match std::fs::remove_file(path.as_ref()) {
+        Ok(()) => Ok(()),
+        Err(e) if is_dir_error(&e) => std::fs::remove_dir(path.as_ref()),
+        Err(e) => Err(e),
     }
 }
 

crates/ecstore/src/disk/local.rs

@@ -26,7 +26,7 @@ use crate::disk::{
     format::FormatV3,
     fs::{O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, lstat, lstat_std, remove, remove_all_std, remove_std, rename},
     os,
-    os::{check_path_length, is_empty_dir, is_root_disk, rename_all},
+    os::{check_path_length, is_empty_dir, is_root_disk, rename_all, rename_all_ignore_missing_source},
 };
 use crate::erasure_coding::bitrot_verify;
 use crate::global::{GLOBAL_IsErasureSD, GLOBAL_RootDiskThreshold};
@@ -57,14 +57,15 @@ use std::{
 use time::OffsetDateTime;
 use tokio::fs::{self, File};
 use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWrite, AsyncWriteExt, ErrorKind};
-use tokio::sync::RwLock;
-use tokio::time::interval;
+use tokio::sync::{Notify, RwLock};
+use tokio::time::{Instant, interval_at, timeout};
 use tracing::{debug, error, info, warn};
 use uuid::Uuid;
 
 const DELETED_OBJECTS_CLEANUP_INTERVAL: Duration = Duration::from_secs(60 * 5);
 const STALE_TMP_OBJECT_EXPIRY: Duration = Duration::from_secs(24 * 60 * 60);
 const RUSTFS_META_TMP_OLD_BUCKET: &str = ".rustfs.sys/tmp-old";
+const STARTUP_CLEANUP_WAIT_TIMEOUT: Duration = Duration::from_secs(2);
 
 #[derive(Debug, Clone)]
 pub struct FormatInfo {
@@ -331,6 +332,8 @@ pub struct LocalDisk {
     // pub format_data: Mutex<Vec<u8>>,
     // pub format_file_info: Mutex<Option<Metadata>>,
     // pub format_last_check: Mutex<Option<OffsetDateTime>>,
+    startup_cleanup_ready: Arc<AtomicU32>,
+    startup_cleanup_notify: Arc<Notify>,
     exit_signal: Option<tokio::sync::broadcast::Sender<()>>,
 }
 
@@ -353,24 +356,72 @@ impl Debug for LocalDisk {
     }
 }
 
+/// Resolve the local disk root path from an endpoint path.
+///
+/// Tries `canonicalize` first (fast path). On Windows, if canonicalization reports
+/// `NotFound` for paths that may still be valid mount roots, falls back to
+/// `absolutize` + metadata check to accept valid local directory roots that
+/// don't support full canonicalization.
+fn resolve_local_disk_root(ep_path: &str) -> Result<PathBuf> {
+    match rustfs_utils::canonicalize(ep_path) {
+        Ok(path) => Ok(path),
+        Err(err) => {
+            if err.kind() != ErrorKind::NotFound {
+                return Err(to_file_error(err).into());
+            }
+
+            #[cfg(windows)]
+            {
+                // On Windows, canonicalize can fail for ZFS volumes, junction points,
+                // subst drives, and other non-standard filesystem mounts. Try a fallback
+                // path resolution using absolutize + metadata check.
+                let absolute = match crate::disk::endpoint::windows_fallback_local_path(ep_path, &err, "local disk root") {
+                    Ok(path) => path,
+                    Err(_) => {
+                        return Err(DiskError::VolumeNotFound);
+                    }
+                };
+
+                match std::fs::metadata(&absolute) {
+                    Ok(metadata) => {
+                        if !metadata.is_dir() {
+                            return Err(DiskError::DiskNotDir);
+                        }
+                        return Ok(absolute);
+                    }
+                    Err(meta_err) => {
+                        if meta_err.kind() == ErrorKind::NotFound {
+                            return Err(DiskError::VolumeNotFound);
+                        }
+                        return Err(to_file_error(meta_err).into());
+                    }
+                }
+            }
+
+            #[cfg(not(windows))]
+            {
+                Err(DiskError::VolumeNotFound)
+            }
+        }
+    }
+}
+
 impl LocalDisk {
     pub async fn new(ep: &Endpoint, cleanup: bool) -> Result<Self> {
         debug!("Creating local disk");
-        // Use optimized path resolution instead of absolutize() for better performance
-        // Use dunce::canonicalize instead of std::fs::canonicalize to avoid UNC paths on Windows
-        let root = match rustfs_utils::canonicalize(ep.get_file_path()) {
-            Ok(path) => path,
-            Err(e) => {
-                if e.kind() == ErrorKind::NotFound {
-                    return Err(DiskError::VolumeNotFound);
-                }
-                return Err(to_file_error(e).into());
-            }
-        };
+        let root = resolve_local_disk_root(&ep.get_file_path())?;
 
         ensure_data_usage_layout(&root).await.map_err(DiskError::from)?;
 
-        if cleanup && let Err(err) = Self::cleanup_tmp_on_startup(&root).await {
+        let startup_cleanup_ready = Arc::new(AtomicU32::new(u32::from(!cleanup)));
+        let startup_cleanup_notify = Arc::new(Notify::new());
+
+        if cleanup
+            && let Err(err) =
+                Self::cleanup_tmp_on_startup(&root, startup_cleanup_ready.clone(), startup_cleanup_notify.clone()).await
+        {
+            startup_cleanup_ready.store(1, Ordering::Release);
+            startup_cleanup_notify.notify_waiters();
             warn!(root = ?root, error = ?err, "failed to cleanup temporary data during disk startup");
         }
 
@@ -466,6 +517,8 @@ impl LocalDisk {
             // format_last_check: Mutex::new(format_last_check),
             path_cache: Arc::new(ParkingLotRwLock::new(HashMap::with_capacity(2048))),
             current_dir: Arc::new(OnceLock::new()),
+            startup_cleanup_ready,
+            startup_cleanup_notify,
             exit_signal: None,
         };
         let (info, _root) = get_disk_info(root).await?;
@@ -497,7 +550,8 @@ impl LocalDisk {
     }
 
     async fn cleanup_deleted_objects_loop(root: PathBuf, mut exit_rx: tokio::sync::broadcast::Receiver<()>) {
-        let mut interval = interval(DELETED_OBJECTS_CLEANUP_INTERVAL);
+        let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);
         loop {
             tokio::select! {
                 _ = interval.tick() => {
@@ -525,12 +579,18 @@ impl LocalDisk {
         root.join(meta_path)
     }
 
-    async fn cleanup_tmp_on_startup(root: &Path) -> Result<()> {
+    async fn cleanup_tmp_on_startup(
+        root: &Path,
+        startup_cleanup_ready: Arc<AtomicU32>,
+        startup_cleanup_notify: Arc<Notify>,
+    ) -> Result<()> {
         let tmp_path = Self::meta_path(root, RUSTFS_META_TMP_BUCKET);
         let tmp_old_path = Self::meta_path(root, RUSTFS_META_TMP_OLD_BUCKET).join(Uuid::new_v4().to_string());
 
         rename_all(&tmp_path, &tmp_old_path, root).await?;
 
+        tokio::fs::create_dir_all(Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET)).await?;
+
         let tmp_old_root = Self::meta_path(root, RUSTFS_META_TMP_OLD_BUCKET);
         tokio::spawn(async move {
             if let Err(err) = tokio::fs::remove_dir_all(&tmp_old_root).await
@@ -538,12 +598,35 @@ impl LocalDisk {
             {
                 warn!(path = ?tmp_old_root, error = ?err, "failed to remove old temporary data");
             }
+            startup_cleanup_ready.store(1, Ordering::Release);
+            startup_cleanup_notify.notify_waiters();
         });
 
-        tokio::fs::create_dir_all(Self::meta_path(root, RUSTFS_META_TMP_DELETED_BUCKET)).await?;
         Ok(())
     }
 
+    async fn wait_for_startup_cleanup(&self) {
+        if self.startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+            return;
+        }
+
+        if wait_for_startup_cleanup_signal(
+            self.startup_cleanup_ready.as_ref(),
+            self.startup_cleanup_notify.as_ref(),
+            STARTUP_CLEANUP_WAIT_TIMEOUT,
+        )
+        .await
+        {
+            debug!(disk = %self.endpoint, "startup cleanup barrier released before walk_dir");
+        } else {
+            warn!(
+                disk = %self.endpoint,
+                timeout_ms = STARTUP_CLEANUP_WAIT_TIMEOUT.as_millis(),
+                "startup cleanup barrier timed out; continuing walk_dir"
+            );
+        }
+    }
+
     async fn cleanup_stale_tmp_objects(root: PathBuf) -> Result<()> {
         Self::cleanup_stale_tmp_objects_with_expiry(root, STALE_TMP_OBJECT_EXPIRY).await
     }
@@ -846,19 +929,20 @@ impl LocalDisk {
         // }
 
         let err = if recursive {
-            rename_all(delete_path, trash_path, self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?)
+            rename_all_ignore_missing_source(delete_path, trash_path, self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?)
                 .await
                 .err()
         } else {
-            rename(&delete_path, &trash_path)
-                .await
-                .map_err(|e| to_file_error(e).into())
-                .err()
+            match rename(&delete_path, &trash_path).await {
+                Ok(()) => None,
+                Err(err) if err.kind() == ErrorKind::NotFound => None,
+                Err(err) => Some(to_file_error(err).into()),
+            }
         };
 
         if immediate_purge || delete_path.to_string_lossy().ends_with(SLASH_SEPARATOR) {
             let trash_path2 = self.get_object_path(RUSTFS_META_TMP_DELETED_BUCKET, Uuid::new_v4().to_string().as_str())?;
-            let _ = rename_all(
+            let _ = rename_all_ignore_missing_source(
                 encode_dir_object(delete_path.to_string_lossy().as_ref()),
                 trash_path2,
                 self.get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET)?,
@@ -2375,6 +2459,8 @@ impl DiskAPI for LocalDisk {
     // FIXME: TODO: io.writer TODO cancel
     #[tracing::instrument(level = "debug", skip(self, wr))]
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
+        self.wait_for_startup_cleanup().await;
+
         let volume_dir = self.get_bucket_path(&opts.bucket)?;
 
         if !skip_access_checks(&opts.bucket)
@@ -2512,75 +2598,57 @@ impl DiskAPI for LocalDisk {
         check_path_length(src_file_path.to_string_lossy().to_string().as_str())?;
         check_path_length(dst_file_path.to_string_lossy().to_string().as_str())?;
 
-        // Read the previous xl.meta
-
-        let has_dst_buf = match super::fs::read_file(&dst_file_path).await {
-            Ok(res) => Some(res),
-            Err(e) => {
-                let e: DiskError = to_file_error(e).into();
-
-                if e != DiskError::FileNotFound {
-                    return Err(e);
-                }
-
-                None
-            }
-        };
-
-        let mut xlmeta = FileMeta::new();
-
-        if let Some(dst_buf) = has_dst_buf.as_ref()
-            && FileMeta::is_xl2_v1_format(dst_buf)
-            && let Ok(nmeta) = FileMeta::load(dst_buf)
-        {
-            xlmeta = nmeta
-        }
-
-        let mut skip_parent = dst_volume_dir.clone();
-        if has_dst_buf.as_ref().is_some()
-            && let Some(parent) = dst_file_path.parent()
-        {
-            skip_parent = parent.to_path_buf();
-        }
-
-        // TODO: Healing
-
-        let version_id = fi.version_id.unwrap_or_default();
-        let search_version_id = Some(version_id);
         let no_inline = fi.data.is_none() && fi.size > 0;
 
-        // Check if there's an existing version with the same version_id that has a data_dir to clean up
-        // Reuse one metadata scan to find the version data_dir and determine whether it is shared.
-        let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(search_version_id);
-        if let Some(old_data_dir) = has_old_data_dir.as_ref() {
-            let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
-        }
-
-        xlmeta.add_version(fi)?;
-
-        if xlmeta.versions.len() <= 10 {
-            // TODO: Sign
-        }
-
-        if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
-            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
-            let meta_skip_parent = if no_inline {
-                src_file_parent
-            } else {
-                src_volume_dir.as_path()
+        if no_inline {
+            // Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
+            let has_dst_buf = match super::fs::read_file(&dst_file_path).await {
+                Ok(res) => Some(res),
+                Err(e) => {
+                    let e: DiskError = to_file_error(e).into();
+                    if e != DiskError::FileNotFound {
+                        return Err(e);
+                    }
+                    None
+                }
             };
+
+            let mut xlmeta = FileMeta::new();
+            if let Some(dst_buf) = has_dst_buf.as_ref()
+                && FileMeta::is_xl2_v1_format(dst_buf)
+                && let Ok(nmeta) = FileMeta::load(dst_buf)
+            {
+                xlmeta = nmeta
+            }
+
+            let mut skip_parent = dst_volume_dir.clone();
+            if has_dst_buf.as_ref().is_some()
+                && let Some(parent) = dst_file_path.parent()
+            {
+                skip_parent = parent.to_path_buf();
+            }
+
+            let version_id = fi.version_id.unwrap_or_default();
+            let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
+            if let Some(old_data_dir) = has_old_data_dir.as_ref() {
+                let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
+            }
+            xlmeta.add_version(fi)?;
             let new_dst_buf = xlmeta.marshal_msg()?;
 
+            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
             self.write_all_private(
                 src_volume,
-                format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(),
+                &format!("{}/{}", &src_path, STORAGE_FORMAT_FILE),
                 new_dst_buf.into(),
                 true,
-                meta_skip_parent,
+                src_file_parent,
             )
             .await?;
 
-            if no_inline && let Err(err) = rename_all(&src_data_path, &dst_data_path, &skip_parent).await {
+            if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref()
+                && let Err(err) = rename_all(src_data_path, dst_data_path, &skip_parent).await
+            {
                 let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
                 info!(
                     "rename all failed src_data_path: {:?}, dst_data_path: {:?}, err: {:?}",
@@ -2588,19 +2656,21 @@ impl DiskAPI for LocalDisk {
                 );
                 return Err(err);
             }
-        } else {
-            let new_dst_buf = xlmeta.marshal_msg()?;
-            self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
-                .await?;
-        }
 
-        if let Some(old_data_dir) = has_old_data_dir {
-            // preserve current xl.meta inside the oldDataDir.
-            if let Some(dst_buf) = has_dst_buf
+            if let Err(err) = rename_all(&src_file_path, &dst_file_path, &skip_parent).await {
+                if let Some((_, dst_data_path)) = has_data_dir_path.as_ref() {
+                    let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
+                }
+                info!("rename all failed err: {:?}", err);
+                return Err(err);
+            }
+
+            if let Some(old_data_dir) = has_old_data_dir
+                && let Some(dst_buf) = has_dst_buf
                 && let Err(err) = self
                     .write_all_private(
                         dst_volume,
-                        format!("{}/{}/{}", &dst_path, &old_data_dir.to_string(), STORAGE_FORMAT_FILE).as_str(),
+                        &format!("{}/{}/{}", &dst_path, &old_data_dir.to_string(), STORAGE_FORMAT_FILE),
                         dst_buf.into(),
                         true,
                         &skip_parent,
@@ -2610,30 +2680,106 @@ impl DiskAPI for LocalDisk {
                 info!("write_all_private failed err: {:?}", err);
                 return Err(err);
             }
-        }
 
-        if let Err(err) = rename_all(&src_file_path, &dst_file_path, &skip_parent).await {
-            if let Some((_, dst_data_path)) = has_data_dir_path.as_ref() {
-                let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
+            if let Some(src_file_path_parent) = src_file_path.parent() {
+                if src_volume != super::RUSTFS_META_MULTIPART_BUCKET {
+                    let _ = remove_std(src_file_path_parent);
+                } else {
+                    let _ = self
+                        .delete_file(&dst_volume_dir, &src_file_path_parent.to_path_buf(), true, false)
+                        .await;
+                }
             }
-            info!("rename all failed err: {:?}", err);
-            return Err(err);
-        }
 
-        if let Some(src_file_path_parent) = src_file_path.parent() {
-            if src_volume != super::RUSTFS_META_MULTIPART_BUCKET {
-                let _ = remove_std(src_file_path_parent);
+            Ok(RenameDataResp {
+                old_data_dir: has_old_data_dir,
+                sign: None,
+            })
+        } else {
+            // Inline: merge read + parse + write + rename into single spawn_blocking
+            let src = src_file_path.clone();
+            let dst = dst_file_path.clone();
+            let cleanup_path = if src_volume == super::RUSTFS_META_MULTIPART_BUCKET {
+                src_file_path.parent().map(|p| p.to_path_buf())
             } else {
-                let _ = self
-                    .delete_file(&dst_volume_dir, &src_file_path_parent.to_path_buf(), true, false)
-                    .await;
-            }
-        }
+                None
+            };
 
-        Ok(RenameDataResp {
-            old_data_dir: has_old_data_dir,
-            sign: None, // TODO:
-        })
+            let (old_data_dir, _dst_buf) = tokio::task::spawn_blocking(move || {
+                // Read existing xl.meta
+                let has_dst_buf = match std::fs::read(&dst) {
+                    Ok(buf) => Some(Bytes::from(buf)),
+                    Err(e) if e.kind() == std::io::ErrorKind::NotFound => None,
+                    Err(e) => return Err(to_file_error(e)),
+                };
+
+                let mut xlmeta = FileMeta::new();
+                if let Some(ref buf) = has_dst_buf
+                    && FileMeta::is_xl2_v1_format(buf)
+                    && let Ok(nmeta) = FileMeta::load(buf)
+                {
+                    xlmeta = nmeta
+                }
+
+                let version_id = fi.version_id.unwrap_or_default();
+                let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
+                if let Some(d) = old_data_dir.as_ref() {
+                    let _ = xlmeta.data.remove_two(version_id, *d);
+                }
+                xlmeta.add_version(fi)?;
+                let new_buf = xlmeta.marshal_msg()?;
+
+                // Write new xl.meta + rename
+                if let Some(parent) = src.parent() {
+                    std::fs::create_dir_all(parent)?;
+                }
+                let mut f = std::fs::OpenOptions::new()
+                    .create(true)
+                    .write(true)
+                    .truncate(true)
+                    .open(&src)?;
+                std::io::Write::write_all(&mut f, &new_buf)?;
+                match std::fs::rename(&src, &dst) {
+                    Ok(()) => Ok(()),
+                    Err(err) if err.kind() == std::io::ErrorKind::NotFound && !src.exists() => Ok(()),
+                    Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+                        if let Some(parent) = dst.parent() {
+                            std::fs::create_dir_all(parent)?;
+                        }
+                        std::fs::rename(&src, &dst).map_err(to_file_error)?;
+                        Ok(())
+                    }
+                    Err(err) => Err(to_file_error(err)),
+                }?;
+
+                if let Some(old_dir) = old_data_dir.as_ref()
+                    && let Some(ref buf) = has_dst_buf
+                    && let Some(dst_parent) = dst.parent()
+                {
+                    let old_path = dst_parent.join(old_dir.to_string()).join(STORAGE_FORMAT_FILE);
+                    if let Some(old_parent) = old_path.parent() {
+                        std::fs::create_dir_all(old_parent)?;
+                    }
+                    std::fs::write(&old_path, buf).map_err(to_file_error)?;
+                }
+
+                Ok::<(Option<uuid::Uuid>, Option<Bytes>), std::io::Error>((old_data_dir, has_dst_buf))
+            })
+            .await
+            .map_err(DiskError::from)??;
+
+            // Cleanup
+            if let Some(ref cleanup) = cleanup_path {
+                let _ = self.delete_file(&dst_volume_dir, cleanup, true, false).await;
+            } else if let Some(parent) = src_file_path.parent() {
+                let _ = remove_std(parent);
+            }
+
+            Ok(RenameDataResp {
+                old_data_dir,
+                sign: None,
+            })
+        }
     }
 
     #[tracing::instrument(skip(self))]
@@ -3105,6 +3251,31 @@ impl DiskAPI for LocalDisk {
     }
 }
 
+async fn wait_for_startup_cleanup_signal(
+    startup_cleanup_ready: &AtomicU32,
+    startup_cleanup_notify: &Notify,
+    wait_timeout: Duration,
+) -> bool {
+    if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+        return true;
+    }
+
+    timeout(wait_timeout, async {
+        loop {
+            if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+                return;
+            }
+            let notified = startup_cleanup_notify.notified();
+            if startup_cleanup_ready.load(Ordering::Acquire) != 0 {
+                return;
+            }
+            notified.await;
+        }
+    })
+    .await
+    .is_ok()
+}
+
 #[tracing::instrument]
 async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInfo, bool)> {
     let drive_path = drive_path.to_string_lossy().to_string();
@@ -3157,7 +3328,9 @@ mod test {
         fs::create_dir_all(leftover.parent().unwrap()).await.unwrap();
         fs::write(&leftover, b"temporary").await.unwrap();
 
-        LocalDisk::cleanup_tmp_on_startup(dir.path()).await.unwrap();
+        LocalDisk::cleanup_tmp_on_startup(dir.path(), Arc::new(AtomicU32::new(0)), Arc::new(Notify::new()))
+            .await
+            .unwrap();
 
         assert!(!tmp.join("leftover").exists());
         assert!(LocalDisk::meta_path(dir.path(), RUSTFS_META_TMP_DELETED_BUCKET).exists());
@@ -3213,6 +3386,54 @@ mod test {
         assert!(entries.next_entry().await.unwrap().is_none());
     }
 
+    #[tokio::test(start_paused = true)]
+    async fn cleanup_loop_interval_does_not_tick_immediately() {
+        let start_at = tokio::time::Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);
+
+        assert!(tokio::time::timeout(Duration::from_secs(1), interval.tick()).await.is_err());
+
+        tokio::time::advance(DELETED_OBJECTS_CLEANUP_INTERVAL).await;
+        interval.tick().await;
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn startup_cleanup_barrier_waits_for_notification() {
+        let ready = Arc::new(AtomicU32::new(0));
+        let notify = Arc::new(Notify::new());
+
+        let wait = tokio::spawn({
+            let ready = ready.clone();
+            let notify = notify.clone();
+            async move { wait_for_startup_cleanup_signal(ready.as_ref(), notify.as_ref(), Duration::from_secs(2)).await }
+        });
+
+        tokio::task::yield_now().await;
+        assert!(!wait.is_finished());
+
+        ready.store(1, Ordering::Release);
+        notify.notify_waiters();
+
+        assert!(wait.await.unwrap());
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn startup_cleanup_barrier_times_out() {
+        let ready = Arc::new(AtomicU32::new(0));
+        let notify = Arc::new(Notify::new());
+
+        let wait = tokio::spawn({
+            let ready = ready.clone();
+            let notify = notify.clone();
+            async move { wait_for_startup_cleanup_signal(ready.as_ref(), notify.as_ref(), Duration::from_secs(2)).await }
+        });
+
+        tokio::task::yield_now().await;
+        tokio::time::advance(Duration::from_secs(2)).await;
+
+        assert!(!wait.await.unwrap());
+    }
+
     #[tokio::test]
     async fn test_scan_dir_includes_nested_object_dirs() {
         use rustfs_filemeta::MetacacheReader;

crates/ecstore/src/disk/os.rs

@@ -140,36 +140,55 @@ pub async fn rename_all(
     Ok(())
 }
 
+#[tracing::instrument(level = "debug", skip_all)]
+pub async fn rename_all_ignore_missing_source(
+    src_file_path: impl AsRef<Path>,
+    dst_file_path: impl AsRef<Path>,
+    base_dir: impl AsRef<Path>,
+) -> Result<()> {
+    match reliable_rename_inner(src_file_path, dst_file_path.as_ref(), base_dir, false).await {
+        Ok(()) => Ok(()),
+        Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(()),
+        Err(err) => Err(to_file_error(err).into()),
+    }
+}
+
 async fn reliable_rename(
     src_file_path: impl AsRef<Path>,
     dst_file_path: impl AsRef<Path>,
     base_dir: impl AsRef<Path>,
+) -> io::Result<()> {
+    reliable_rename_inner(src_file_path, dst_file_path, base_dir, true).await
+}
+
+async fn reliable_rename_inner(
+    src_file_path: impl AsRef<Path>,
+    dst_file_path: impl AsRef<Path>,
+    base_dir: impl AsRef<Path>,
+    warn_on_failure: bool,
 ) -> io::Result<()> {
     if let Some(parent) = dst_file_path.as_ref().parent()
         && !file_exists(parent)
     {
-        // info!("reliable_rename reliable_mkdir_all parent: {:?}", parent);
         reliable_mkdir_all(parent, base_dir.as_ref()).await?;
     }
 
     let mut i = 0;
     loop {
         if let Err(e) = super::fs::rename_std(src_file_path.as_ref(), dst_file_path.as_ref()) {
-            if e.kind() == io::ErrorKind::NotFound {
-                break;
-            }
-
             if i == 0 {
                 i += 1;
                 continue;
             }
-            warn!(
-                "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
-                src_file_path.as_ref(),
-                dst_file_path.as_ref(),
-                base_dir.as_ref(),
-                e
-            );
+            if warn_on_failure {
+                warn!(
+                    "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
+                    src_file_path.as_ref(),
+                    dst_file_path.as_ref(),
+                    base_dir.as_ref(),
+                    e
+                );
+            }
             return Err(e);
         }
 
@@ -249,3 +268,36 @@ pub async fn os_mkdir_all(dir_path: impl AsRef<Path>, base_dir: impl AsRef<Path>
 pub fn file_exists(path: impl AsRef<Path>) -> bool {
     std::fs::metadata(path.as_ref()).map(|_| true).unwrap_or(false)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tempfile::tempdir;
+
+    #[tokio::test]
+    async fn rename_all_missing_source_returns_file_not_found() {
+        let temp_dir = tempdir().expect("create temp dir");
+        let src = temp_dir.path().join("missing");
+        let dst = temp_dir.path().join("dst");
+
+        let err = rename_all(&src, &dst, temp_dir.path())
+            .await
+            .expect_err("missing source must fail");
+
+        assert!(matches!(err, DiskError::FileNotFound));
+        assert!(!dst.exists());
+    }
+
+    #[tokio::test]
+    async fn rename_all_ignore_missing_source_returns_ok() {
+        let temp_dir = tempdir().expect("create temp dir");
+        let src = temp_dir.path().join("missing");
+        let dst = temp_dir.path().join("dst");
+
+        rename_all_ignore_missing_source(&src, &dst, temp_dir.path())
+            .await
+            .expect("missing cleanup source must be ignored");
+
+        assert!(!dst.exists());
+    }
+}

crates/ecstore/src/endpoints.rs

@@ -683,14 +683,46 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
     }
 
     let mut device_paths = BTreeMap::<String, BTreeSet<String>>::new();
+    #[cfg(not(windows))]
     let mut missing_paths = Vec::new();
 
     for path in &local_paths {
         let canonical = match rustfs_utils::canonicalize(path) {
             Ok(path) => path,
             Err(err) if err.kind() == ErrorKind::NotFound => {
-                missing_paths.push(path.clone());
-                continue;
+                // On Windows, canonicalize can fail for ZFS volumes, junction points,
+                // subst drives, and other non-standard mounts. Try absolutize as fallback.
+                #[cfg(windows)]
+                {
+                    match crate::disk::endpoint::windows_fallback_local_path(path, &err, "disk independence validation") {
+                        Ok(absolute) => {
+                            let abs_path = absolute.to_string_lossy().into_owned();
+                            match rustfs_utils::os::get_physical_device_ids(&abs_path) {
+                                Ok(ids) => {
+                                    for device_id in ids {
+                                        device_paths.entry(device_id).or_default().insert(abs_path.clone());
+                                    }
+                                }
+                                Err(device_err) => {
+                                    return Err(Error::other(format!(
+                                        "failed to inspect physical disk for local endpoint '{abs_path}' after fallback path resolution: {device_err}"
+                                    )));
+                                }
+                            }
+                            continue;
+                        }
+                        Err(fallback_err) => {
+                            return Err(Error::other(format!(
+                                "failed to resolve local endpoint path '{path}' for disk validation: {err}; fallback resolution failed: {fallback_err}"
+                            )));
+                        }
+                    }
+                }
+                #[cfg(not(windows))]
+                {
+                    missing_paths.push(path.clone());
+                    continue;
+                }
             }
             Err(err) => {
                 return Err(Error::other(format!(
@@ -708,6 +740,7 @@ fn validate_local_physical_disk_independence(pools: &[Endpoints]) -> Result<()>
         }
     }
 
+    #[cfg(not(windows))]
     if !missing_paths.is_empty() {
         warn!(
             missing_paths = ?missing_paths,

crates/ecstore/src/erasure_coding/encode.rs

@@ -209,6 +209,13 @@ impl Erasure {
     where
         R: AsyncRead + Send + Sync + Unpin + 'static,
     {
+        if self.block_size == 0 {
+            return Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "erasure block_size must be non-zero",
+            ));
+        }
+
         // Bound queued encoded blocks by memory budget to avoid per-request spikes.
         let expanded_block_bytes = self.shard_size().saturating_mul(self.total_shard_count());
         let max_inflight_bytes = rustfs_utils::get_env_usize(
@@ -223,10 +230,20 @@ impl Erasure {
             let mut total = 0;
             let mut buf = vec![0u8; block_size];
             loop {
-                match rustfs_utils::read_full(&mut reader, &mut buf).await {
-                    Ok(n) if n > 0 => {
+                match rustfs_utils::read_full_or_eof(&mut reader, &mut buf).await {
+                    Ok(Some(n)) => {
+                        debug_assert!(n > 0, "non-zero block_size prevents zero-length reads");
                         total += n;
-                        let res = self.encode_data(&buf[..n])?;
+                        let erasure = self.clone();
+                        let encode_buf = std::mem::take(&mut buf);
+                        let (res, returned_buf) = tokio::task::spawn_blocking(move || {
+                            let res = erasure.encode_data(&encode_buf[..n]);
+                            (res, encode_buf)
+                        })
+                        .await
+                        .map_err(|err| std::io::Error::other(format!("EC encode task failed: {err}")))?;
+                        buf = returned_buf;
+                        let res = res?;
                         let queued_bytes = queued_block_bytes(&res);
                         rustfs_io_metrics::add_ec_encode_inflight_bytes(queued_bytes);
                         if let Err(err) = tx.send(res).await {
@@ -234,7 +251,7 @@ impl Erasure {
                             return Err(std::io::Error::other(format!("Failed to send encoded data : {err}")));
                         }
                     }
-                    Ok(_) => {
+                    Ok(None) => {
                         break;
                     }
                     Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
@@ -244,7 +261,7 @@ impl Erasure {
                         {
                             return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()));
                         }
-                        break;
+                        return Err(e);
                     }
                     Err(e) => {
                         return Err(e);
@@ -285,13 +302,42 @@ impl Erasure {
         writers.shutdown().await?;
         Ok((reader, total))
     }
+
+    /// Fast path for small inline objects: skip tokio::spawn + mpsc channel.
+    /// Reads all data, encodes directly, writes shards sequentially.
+    pub async fn encode_inline_small<R>(
+        self: Arc<Self>,
+        mut reader: R,
+        writers: &mut [Option<BitrotWriterWrapper>],
+        quorum: usize,
+    ) -> std::io::Result<(R, usize)>
+    where
+        R: AsyncRead + Send + Sync + Unpin,
+    {
+        use tokio::io::AsyncReadExt;
+
+        let mut buf = Vec::with_capacity(self.block_size);
+        let total = reader.read_to_end(&mut buf).await?;
+
+        if total == 0 {
+            return Ok((reader, 0));
+        }
+
+        let shards = self.encode_data(&buf)?;
+        let mut mw = MultiWriter::new(writers, quorum);
+        mw.write(shards).await?;
+        mw.shutdown().await?;
+        Ok((reader, total))
+    }
 }
 
 #[cfg(test)]
 mod tests {
     use super::*;
     use crate::erasure_coding::{BitrotWriterWrapper, CustomWriter};
+    use rustfs_rio::HardLimitReader;
     use rustfs_utils::HashAlgorithm;
+    use std::io::Cursor;
     use std::pin::Pin;
     use std::sync::{Arc, Mutex};
     use std::task::{Context, Poll};
@@ -348,6 +394,103 @@ mod tests {
         assert!(!committed.lock().unwrap().is_empty());
     }
 
+    #[tokio::test]
+    async fn encode_returns_unexpected_eof_for_truncated_limited_reader() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed);
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 16));
+        let truncated = HardLimitReader::new(Cursor::new(b"short".to_vec()), 10);
+
+        let err = match erasure.encode(truncated, &mut writers, 1).await {
+            Ok(_) => panic!("truncated input must fail"),
+            Err(err) => err,
+        };
+
+        assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
+    }
+
+    #[tokio::test]
+    async fn encode_rejects_zero_block_size() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed);
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(b"payload".to_vec()));
+        let err = erasure
+            .encode(reader, &mut writers, 1)
+            .await
+            .expect_err("zero block size must be rejected");
+
+        assert_eq!(err.kind(), std::io::ErrorKind::InvalidInput);
+        assert!(err.to_string().contains("block_size"));
+    }
+
+    /// encode_inline_small: empty reader returns (reader, 0) without writing to any shard.
+    #[tokio::test]
+    async fn encode_inline_small_empty_stream_returns_zero() {
+        let committed = Arc::new(Mutex::new(Vec::new()));
+        let writer = DeferredCommitWriter::new(committed.clone());
+        // 1 data shard, 0 parity shards, block_size = 16
+        let mut writers = vec![Some(BitrotWriterWrapper::new(
+            CustomWriter::new_tokio_writer(writer),
+            16,
+            HashAlgorithm::HighwayHash256S,
+        ))];
+
+        let erasure = Arc::new(Erasure::new(1, 0, 16));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(Vec::<u8>::new()));
+        let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, 1).await.unwrap();
+
+        assert_eq!(total, 0);
+        // No shutdown was called, so nothing should be committed
+        assert!(committed.lock().unwrap().is_empty());
+    }
+
+    /// encode_inline_small: small payload is encoded into the correct number of shards
+    /// and each writer receives data after shutdown.
+    #[tokio::test]
+    async fn encode_inline_small_payload_writes_all_shards() {
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const TOTAL_SHARDS: usize = DATA_SHARDS + PARITY_SHARDS;
+        const BLOCK_SIZE: usize = 64;
+
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..TOTAL_SHARDS).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+
+        let mut writers: Vec<Option<BitrotWriterWrapper>> = committed
+            .iter()
+            .map(|c| {
+                Some(BitrotWriterWrapper::new(
+                    CustomWriter::new_tokio_writer(DeferredCommitWriter::new(c.clone())),
+                    BLOCK_SIZE / DATA_SHARDS,
+                    HashAlgorithm::HighwayHash256S,
+                ))
+            })
+            .collect();
+
+        let payload = b"hello inline small";
+        let erasure = Arc::new(Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE));
+        let reader = tokio::io::BufReader::new(std::io::Cursor::new(payload.to_vec()));
+        let (_reader, total) = erasure.encode_inline_small(reader, &mut writers, DATA_SHARDS).await.unwrap();
+
+        assert_eq!(total, payload.len());
+        // All shards must have received data (shutdown flushed the bitrot header + shard bytes)
+        for (i, c) in committed.iter().enumerate() {
+            assert!(!c.lock().unwrap().is_empty(), "shard {i} should have received data");
+        }
+    }
+
     #[test]
     fn encode_channel_capacity_never_returns_zero() {
         assert_eq!(encode_channel_capacity(0, 1024), 1);

crates/ecstore/src/erasure_coding/erasure.rs

@@ -308,7 +308,6 @@ where
 /// - `encoder`: Optional ReedSolomon encoder instance.
 /// - `block_size`: Block size for each shard.
 /// - `_id`: Unique identifier for the erasure instance.
-/// - `_buf`: Internal buffer for block operations.
 ///
 /// # Example
 /// ```ignore
@@ -326,7 +325,6 @@ pub struct Erasure {
     pub block_size: usize,
     uses_legacy: bool,
     _id: Uuid,
-    _buf: Vec<u8>,
 }
 
 impl Default for Erasure {
@@ -339,7 +337,6 @@ impl Default for Erasure {
             block_size: 0,
             uses_legacy: false,
             _id: Uuid::nil(),
-            _buf: vec![],
         }
     }
 }
@@ -354,7 +351,6 @@ impl Clone for Erasure {
             block_size: self.block_size,
             uses_legacy: self.uses_legacy,
             _id: Uuid::new_v4(), // Generate new ID for clone
-            _buf: vec![0u8; self.block_size],
         }
     }
 }
@@ -399,7 +395,6 @@ impl Erasure {
             legacy_encoder,
             uses_legacy,
             _id: Uuid::new_v4(),
-            _buf: vec![0u8; block_size],
         }
     }
 
@@ -579,18 +574,42 @@ impl Erasure {
         F: FnMut(std::io::Result<Vec<Bytes>>) -> Fut + Send,
         Fut: std::future::Future<Output = Result<(), E>> + Send,
     {
+        if self.block_size == 0 {
+            on_block(Err(std::io::Error::new(
+                std::io::ErrorKind::InvalidInput,
+                "erasure block_size must be non-zero",
+            )))
+            .await?;
+            return Ok(0);
+        }
+
         let block_size = self.block_size;
         let mut total = 0;
         let mut buf = vec![0u8; block_size];
         loop {
-            match rustfs_utils::read_full(&mut *reader, &mut buf).await {
-                Ok(n) if n > 0 => {
+            match rustfs_utils::read_full_or_eof(&mut *reader, &mut buf).await {
+                Ok(Some(n)) => {
+                    debug_assert!(n > 0, "non-zero block_size prevents zero-length reads");
                     warn!("encode_stream_callback_async read n={}", n);
                     total += n;
-                    let res = self.encode_data(&buf[..n]);
+                    let erasure = self.clone();
+                    let encode_buf = std::mem::take(&mut buf);
+                    let (res, returned_buf) = match tokio::task::spawn_blocking(move || {
+                        let res = erasure.encode_data(&encode_buf[..n]);
+                        (res, encode_buf)
+                    })
+                    .await
+                    {
+                        Ok(result) => result,
+                        Err(err) => {
+                            on_block(Err(std::io::Error::other(format!("EC encode task failed: {err}")))).await?;
+                            break;
+                        }
+                    };
+                    buf = returned_buf;
                     on_block(res).await?
                 }
-                Ok(_) => {
+                Ok(None) => {
                     warn!("encode_stream_callback_async read unexpected ok");
                     break;
                 }
@@ -1018,6 +1037,35 @@ mod tests {
         assert_eq!(&recovered, &data_clone);
     }
 
+    #[tokio::test]
+    async fn test_encode_stream_callback_async_reports_zero_block_size() {
+        use std::io::Cursor;
+        use std::sync::{Arc, Mutex};
+
+        let erasure = Arc::new(Erasure::new(1, 0, 0));
+        let mut reader = Cursor::new(b"payload".to_vec());
+        let observed = Arc::new(Mutex::new(None));
+        let observed_clone = observed.clone();
+
+        let total = erasure
+            .encode_stream_callback_async::<_, _, (), _>(&mut reader, move |res| {
+                let observed = observed_clone.clone();
+                async move {
+                    let err = res.expect_err("zero block size should report an error");
+                    *observed.lock().unwrap() = Some((err.kind(), err.to_string()));
+                    Ok(())
+                }
+            })
+            .await
+            .expect("callback should handle the zero block size error");
+
+        assert_eq!(total, 0);
+        let observed = observed.lock().unwrap();
+        let (kind, message) = observed.as_ref().expect("callback should be invoked once");
+        assert_eq!(*kind, std::io::ErrorKind::InvalidInput);
+        assert!(message.contains("block_size"));
+    }
+
     // SIMD mode specific tests
     mod simd_tests {
         use super::*;

crates/ecstore/src/error.rs

@@ -160,6 +160,14 @@ pub enum StorageError {
     NotModified,
     #[error("Invalid range specified: {0}")]
     InvalidRangeSpec(String),
+    #[error("Namespace lock quorum unavailable for {mode} lock on {bucket}/{object}: required {required}, achieved {achieved}")]
+    NamespaceLockQuorumUnavailable {
+        mode: &'static str,
+        bucket: String,
+        object: String,
+        required: usize,
+        achieved: usize,
+    },
 
     // ── Generic ──────────────────────────────────────────────────────
     #[error("Unexpected error")]
@@ -204,6 +212,7 @@ impl StorageError {
                 | StorageError::ErasureWriteQuorum
                 | StorageError::InsufficientReadQuorum(_, _)
                 | StorageError::InsufficientWriteQuorum(_, _)
+                | StorageError::NamespaceLockQuorumUnavailable { .. }
         )
     }
 }
@@ -433,6 +442,19 @@ impl Clone for StorageError {
             StorageError::NotModified => StorageError::NotModified,
             StorageError::InvalidPartNumber(a) => StorageError::InvalidPartNumber(*a),
             StorageError::InvalidRangeSpec(a) => StorageError::InvalidRangeSpec(a.clone()),
+            StorageError::NamespaceLockQuorumUnavailable {
+                mode,
+                bucket,
+                object,
+                required,
+                achieved,
+            } => StorageError::NamespaceLockQuorumUnavailable {
+                mode,
+                bucket: bucket.clone(),
+                object: object.clone(),
+                required: *required,
+                achieved: *achieved,
+            },
         }
     }
 }
@@ -505,6 +527,7 @@ impl StorageError {
             StorageError::InvalidRangeSpec(_) => 0x3D,
             StorageError::NotModified => 0x3E,
             StorageError::InvalidPartNumber(_) => 0x3F,
+            StorageError::NamespaceLockQuorumUnavailable { .. } => 0x42,
         }
     }
 
@@ -579,6 +602,13 @@ impl StorageError {
             0x3D => Some(StorageError::InvalidRangeSpec(Default::default())),
             0x3E => Some(StorageError::NotModified),
             0x3F => Some(StorageError::InvalidPartNumber(Default::default())),
+            0x42 => Some(StorageError::NamespaceLockQuorumUnavailable {
+                mode: "write",
+                bucket: Default::default(),
+                object: Default::default(),
+                required: Default::default(),
+                achieved: Default::default(),
+            }),
             _ => None,
         }
     }
@@ -984,6 +1014,17 @@ mod tests {
         assert_eq!(StorageError::DecommissionAlreadyRunning.to_u32(), 0x30);
         assert_eq!(StorageError::RebalanceAlreadyRunning.to_u32(), 0x40);
         assert_eq!(StorageError::OperationCanceled.to_u32(), 0x41);
+        assert_eq!(
+            StorageError::NamespaceLockQuorumUnavailable {
+                mode: "write",
+                bucket: "bucket".into(),
+                object: "object".into(),
+                required: 3,
+                achieved: 2,
+            }
+            .to_u32(),
+            0x42
+        );
     }
 
     #[test]
@@ -998,6 +1039,10 @@ mod tests {
         assert!(matches!(StorageError::from_u32(0x30), Some(StorageError::DecommissionAlreadyRunning)));
         assert!(matches!(StorageError::from_u32(0x40), Some(StorageError::RebalanceAlreadyRunning)));
         assert!(matches!(StorageError::from_u32(0x41), Some(StorageError::OperationCanceled)));
+        assert!(matches!(
+            StorageError::from_u32(0x42),
+            Some(StorageError::NamespaceLockQuorumUnavailable { .. })
+        ));
 
         // Test invalid code returns None
         assert!(StorageError::from_u32(0xFF).is_none());

crates/ecstore/src/notification_sys.rs

@@ -189,6 +189,65 @@ impl NotificationSys {
         join_all(futures).await
     }
 
+    pub async fn reload_dynamic_config(&self, sub_sys: &str) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter() {
+            let sub_sys = sub_sys.to_string();
+            futures.push(async move {
+                if let Some(client) = client {
+                    match client
+                        .signal_service(crate::rpc::SERVICE_SIGNAL_RELOAD_DYNAMIC, &sub_sys, false, SystemTime::UNIX_EPOCH)
+                        .await
+                    {
+                        Ok(_) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: None,
+                        },
+                        Err(e) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: Some(e),
+                        },
+                    }
+                } else {
+                    NotificationPeerErr {
+                        host: "".to_string(),
+                        err: Some(Error::other("peer is not reachable")),
+                    }
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
+    pub async fn refresh_config_snapshot(&self) -> Vec<NotificationPeerErr> {
+        let mut futures = Vec::with_capacity(self.peer_clients.len());
+        for client in self.peer_clients.iter() {
+            futures.push(async move {
+                if let Some(client) = client {
+                    match client
+                        .signal_service(crate::rpc::SERVICE_SIGNAL_REFRESH_CONFIG, "", false, SystemTime::UNIX_EPOCH)
+                        .await
+                    {
+                        Ok(_) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: None,
+                        },
+                        Err(e) => NotificationPeerErr {
+                            host: client.host.to_string(),
+                            err: Some(e),
+                        },
+                    }
+                } else {
+                    NotificationPeerErr {
+                        host: "".to_string(),
+                        err: Some(Error::other("peer is not reachable")),
+                    }
+                }
+            });
+        }
+        join_all(futures).await
+    }
+
     pub async fn storage_info<S: StorageAPI>(&self, api: &S) -> rustfs_madmin::StorageInfo {
         let mut futures = Vec::with_capacity(self.peer_clients.len());
         let endpoints = get_global_endpoints();

crates/ecstore/src/rebalance.rs

@@ -1002,7 +1002,7 @@ impl ECStore {
                 return true;
             }
 
-            // Mark pool rebalance as done if within 5% of the PercentFreeGoal
+            // Mark pool rebalance as done only after it reaches the PercentFreeGoal.
             let pfi = if pool_stat.init_capacity == 0 {
                 0.0
             } else {
@@ -1032,7 +1032,7 @@ fn rebalance_goal_reached(init_free_space: u64, init_capacity: u64, bytes: u64,
     }
 
     let pfi = (init_free_space + bytes) as f64 / init_capacity as f64;
-    (pfi - percent_free_goal).abs() <= 0.05 + f64::EPSILON
+    pfi + f64::EPSILON >= percent_free_goal
 }
 
 fn percent_free_ratio(total_free: u64, total_cap: u64) -> f64 {
@@ -2664,23 +2664,20 @@ mod rebalance_unit_tests {
     }
 
     #[test]
-    fn test_rebalance_goal_reached_exact_tolerance_bound() {
+    fn test_rebalance_goal_reached_at_target() {
         let init_free_space = 200_u64;
         let init_capacity = 1_000_u64;
         let goal = 0.45_f64;
 
-        // goal - 0.05 => 200 + 200 = 400 free / 1000 => 0.4 exactly
-        assert!(rebalance_goal_reached(init_free_space, init_capacity, 200, goal));
-
-        // one byte above the tolerance boundary should be false
-        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 199, goal));
+        assert!(rebalance_goal_reached(init_free_space, init_capacity, 250, goal));
+        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 249, goal));
     }
 
     #[test]
-    fn test_rebalance_goal_reached_within_tolerance() {
+    fn test_rebalance_goal_reached_above_target() {
         let init_free_space = 200_u64;
         let init_capacity = 1_000_u64;
-        let bytes = 250_u64;
+        let bytes = 251_u64;
         let goal = 0.45_f64;
 
         assert!(rebalance_goal_reached(init_free_space, init_capacity, bytes, goal));
@@ -2702,12 +2699,12 @@ mod rebalance_unit_tests {
     }
 
     #[test]
-    fn test_rebalance_goal_above_one_is_true_when_within_tolerance() {
+    fn test_rebalance_goal_above_one_is_true_when_reached() {
         assert!(rebalance_goal_reached(950, 1_000, 100, 1.0));
     }
 
     #[test]
-    fn test_rebalance_goal_below_zero_is_true_when_within_tolerance() {
+    fn test_rebalance_goal_below_zero_is_true_when_reached() {
         assert!(rebalance_goal_reached(10, 1_000, 0, -0.01));
     }
 
@@ -3355,6 +3352,18 @@ mod rebalance_unit_tests {
         assert!(!should_pool_participate(300, 1_000, 0.3));
     }
 
+    #[test]
+    fn test_rebalance_goal_not_reached_for_issue_3137_initial_imbalance() {
+        let pool0_capacity = 10_000_u64;
+        let pool0_free = 9_135_u64;
+        let pool1_capacity = 10_000_u64;
+        let pool1_free = 9_800_u64;
+        let goal = percent_free_ratio(pool0_free + pool1_free, pool0_capacity + pool1_capacity);
+
+        assert!(should_pool_participate(pool0_free, pool0_capacity, goal));
+        assert!(!rebalance_goal_reached(pool0_free, pool0_capacity, 0, goal));
+    }
+
     #[test]
     fn test_complete_rebalance_pools_at_goal_marks_started_participants_completed() {
         let now = OffsetDateTime::from_unix_timestamp(1_000).unwrap();
@@ -3365,7 +3374,7 @@ mod rebalance_unit_tests {
                     participating: true,
                     init_free_space: 400,
                     init_capacity: 1_000,
-                    bytes: 50,
+                    bytes: 100,
                     info: RebalanceInfo {
                         status: RebalStatus::Started,
                         ..Default::default()
@@ -4129,13 +4138,13 @@ mod rebalance_unit_tests {
     }
 
     #[test]
-    fn test_rebalance_goal_reached_tolerance_and_regression() {
+    fn test_rebalance_goal_reached_requires_target_ratio() {
         let init_free_space = 150_u64;
         let init_capacity = 800_u64;
         let goal = 0.35_f64;
 
         assert!(!rebalance_goal_reached(init_free_space, init_capacity, 0, goal));
-        assert!(rebalance_goal_reached(init_free_space, init_capacity, 90, goal));
-        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 89, goal));
+        assert!(!rebalance_goal_reached(init_free_space, init_capacity, 129, goal));
+        assert!(rebalance_goal_reached(init_free_space, init_capacity, 130, goal));
     }
 }

crates/ecstore/src/rpc/mod.rs

@@ -29,7 +29,10 @@ pub use internode_data_transport::{
     InternodeDataTransport, InternodeDataTransportCapabilities, ReadStreamRequest, TcpHttpInternodeDataTransport,
     WalkDirStreamRequest, WriteStreamRequest, build_internode_data_transport, build_internode_data_transport_from_env,
 };
-pub use peer_rest_client::PeerRestClient;
+pub use peer_rest_client::{
+    PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG,
+    SERVICE_SIGNAL_RELOAD_DYNAMIC,
+};
 pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, RemotePeerS3Client, S3PeerSys};
 pub use remote_disk::RemoteDisk;
 pub use remote_locker::RemoteClient;

crates/ecstore/src/rpc/peer_rest_client.rs

@@ -57,6 +57,8 @@ use tracing::warn;
 pub const PEER_RESTSIGNAL: &str = "signal";
 pub const PEER_RESTSUB_SYS: &str = "sub-sys";
 pub const PEER_RESTDRY_RUN: &str = "dry-run";
+pub const SERVICE_SIGNAL_REFRESH_CONFIG: u64 = 1;
+pub const SERVICE_SIGNAL_RELOAD_DYNAMIC: u64 = 2;
 const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
 const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
 

crates/ecstore/src/rpc/peer_s3_client.rs

@@ -843,12 +843,13 @@ impl PeerS3Client for RemotePeerS3Client {
                 });
                 let response = client.make_bucket(request).await?.into_inner();
 
-                // TODO: deal with error
                 if !response.success {
                     return if let Some(err) = response.error {
                         Err(err.into())
                     } else {
-                        Err(Error::other(""))
+                        Err(Error::other(format!(
+                            "make_bucket({bucket}): peer returned failure without error details"
+                        )))
                     };
                 }
 
@@ -1046,6 +1047,44 @@ async fn clone_drives() -> Vec<Option<DiskStore>> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::disk::WalkDirOptions;
+    use crate::disk::disk_store::LocalDiskWrapper;
+    use crate::disk::endpoint::Endpoint;
+    use crate::disk::local::LocalDisk;
+    use crate::endpoints::{Endpoints, PoolEndpoints};
+    use crate::global::{GLOBAL_LOCAL_DISK_ID_MAP, GLOBAL_LOCAL_DISK_MAP, GLOBAL_LOCAL_DISK_SET_DRIVES};
+    use crate::store::init_local_disks;
+    use rustfs_filemeta::FileInfo;
+    use serial_test::serial;
+    use std::{
+        io,
+        pin::Pin,
+        task::{Context, Poll},
+    };
+    use tempfile::TempDir;
+    use tokio::io::AsyncWrite;
+
+    struct PendingWriter;
+
+    impl AsyncWrite for PendingWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
+            Poll::Pending
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
+
+    async fn reset_local_disk_globals() {
+        GLOBAL_LOCAL_DISK_MAP.write().await.clear();
+        GLOBAL_LOCAL_DISK_ID_MAP.write().await.clear();
+        GLOBAL_LOCAL_DISK_SET_DRIVES.write().await.clear();
+    }
 
     #[derive(Debug)]
     struct TestPeerS3Client {
@@ -1152,6 +1191,77 @@ mod tests {
         client.cancel_token.cancel();
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn local_get_bucket_info_survives_prior_walk_timeout() {
+        reset_local_disk_globals().await;
+
+        let temp_dir = TempDir::new().expect("create temp dir for local peer listing regression");
+        let disk_path = temp_dir.path().join("disk1");
+        std::fs::create_dir_all(&disk_path).expect("create disk path");
+
+        let mut endpoint = Endpoint::try_from(disk_path.to_str().expect("disk path to str")).expect("endpoint");
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+
+        let endpoint_pools = EndpointServerPools(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 1,
+            endpoints: Endpoints::from(vec![endpoint.clone()]),
+            cmd_line: "local-get-bucket-info-survives-prior-walk-timeout".to_string(),
+            platform: "test".to_string(),
+        }]);
+
+        init_local_disks(endpoint_pools).await.expect("init local disks");
+
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+        let wrapper = crate::disk::Disk::Local(Box::new(LocalDiskWrapper::new(disk, false)));
+        let disk_store: DiskStore = Arc::new(wrapper);
+        let bucket = "test-bucket";
+        let object = "test-object";
+
+        disk_store.make_volume(bucket).await.expect("bucket should be created");
+
+        let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+        file_info.volume = bucket.to_string();
+        file_info.name = object.to_string();
+        file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+        file_info.erasure.index = 1;
+
+        disk_store
+            .write_metadata("", bucket, object, file_info)
+            .await
+            .expect("object metadata should be written");
+
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let mut writer = PendingWriter;
+            let walk_err = disk_store
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await
+                .expect_err("walk_dir should time out against a non-draining writer");
+
+            assert_eq!(walk_err, DiskError::Timeout);
+
+            let info = LocalPeerS3Client::new(None, Some(vec![0]))
+                .get_bucket_info(bucket, &BucketOptions::default())
+                .await
+                .expect("bucket info should still succeed after prior walk timeout");
+            assert_eq!(info.name, bucket);
+        })
+        .await;
+
+        reset_local_disk_globals().await;
+    }
+
     #[test]
     fn test_reduce_pool_write_quorum_uses_only_pool_participants() {
         let clients = vec![

crates/ecstore/src/rpc/remote_disk.rs

@@ -110,6 +110,15 @@ pub struct RemoteDisk {
 }
 
 impl RemoteDisk {
+    fn is_retryable_walk_dir_error(err: &DiskError) -> bool {
+        if is_network_like_disk_error(err) {
+            return true;
+        }
+
+        let err_text = err.to_string().to_ascii_lowercase();
+        err_text.contains("httpreader stream error") || err_text.contains("error decoding response body")
+    }
+
     pub(crate) async fn new(ep: &Endpoint, opt: &DiskOption, data_transport: Arc<dyn InternodeDataTransport>) -> Result<Self> {
         let addr = if let Some(port) = ep.url.port() {
             format!("{}://{}:{}", ep.url.scheme(), ep.url.host_str().unwrap(), port)
@@ -175,6 +184,10 @@ impl RemoteDisk {
         });
     }
 
+    fn mark_suspect_or_offline(&self, reason: &'static str) -> bool {
+        self.health.mark_failure(&self.endpoint, reason)
+    }
+
     /// Enable health monitoring after disk creation.
     /// Used to defer health checks until after startup format loading completes,
     /// so that remote peers have time to come online.
@@ -202,7 +215,10 @@ impl RemoteDisk {
         let mut interval = time::interval(get_drive_active_check_interval());
 
         // Perform basic connectivity check
-        if Self::perform_connectivity_check(&addr).await.is_err() && health.mark_offline(&endpoint, "connectivity_probe_failed") {
+        let initial_probe_ok = Self::perform_connectivity_check(&addr).await.is_ok();
+        if initial_probe_ok {
+            health.record_operation_success(&endpoint, "connectivity_probe_success");
+        } else if health.mark_failure(&endpoint, "connectivity_probe_failed") {
             warn!("Remote disk health check failed for {}: marking as faulty", addr);
 
             // Start recovery monitoring
@@ -245,7 +261,9 @@ impl RemoteDisk {
                     }
 
                     // Perform basic connectivity check
-                    if Self::perform_connectivity_check(&addr).await.is_err() && health.mark_offline(&endpoint, "connectivity_probe_failed") {
+                    if Self::perform_connectivity_check(&addr).await.is_ok() {
+                        health.record_operation_success(&endpoint, "connectivity_probe_success");
+                    } else if health.mark_failure(&endpoint, "connectivity_probe_failed") {
                         warn!("Remote disk health check failed for {}: marking as faulty", addr);
 
                         // Start recovery monitoring
@@ -286,7 +304,7 @@ impl RemoteDisk {
                             return;
                         }
                     } else {
-                        health.mark_offline(&endpoint, "connectivity_probe_failed");
+                        health.mark_failure(&endpoint, "connectivity_probe_failed");
                     }
                 }
             }
@@ -440,7 +458,11 @@ impl RemoteDisk {
     }
 
     async fn mark_faulty_and_evict(&self, reason: &'static str) {
-        if self.health.mark_offline(&self.endpoint, reason) {
+        let previous_state = self.runtime_state();
+        let transitioned_to_offline = self.mark_suspect_or_offline(reason);
+        let state = self.runtime_state();
+
+        if state != previous_state {
             self.spawn_recovery_monitor_if_needed();
             counter!(
                 "rustfs_drive_faulty_mark_total",
@@ -448,10 +470,17 @@ impl RemoteDisk {
                 "reason" => reason.to_string()
             )
             .increment(1);
-            warn!(
-                "Remote disk marked faulty after timeout: endpoint={}, addr={}, reason={}",
-                self.endpoint, self.addr, reason
-            );
+            if transitioned_to_offline {
+                warn!(
+                    "Remote disk marked faulty after timeout: endpoint={}, addr={}, reason={}",
+                    self.endpoint, self.addr, reason
+                );
+            } else {
+                warn!(
+                    "Remote disk marked suspect after timeout: endpoint={}, addr={}, reason={}, state={:?}",
+                    self.endpoint, self.addr, reason, state
+                );
+            }
             counter!(
                 "rustfs_drive_connection_evict_total",
                 "endpoint" => self.endpoint.to_string(),
@@ -1128,7 +1157,8 @@ impl DiskAPI for RemoteDisk {
     ) -> Result<RenameDataResp> {
         info!("rename_data {}/{}/{}/{}", self.addr, self.endpoint.to_string(), dst_volume, dst_path);
 
-        self.execute_with_timeout(
+        self.execute_with_timeout_for_op(
+            "rename_data",
             || async {
                 let file_info = serde_json::to_string(&fi)?;
                 let mut client = self
@@ -1195,24 +1225,58 @@ impl DiskAPI for RemoteDisk {
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
         info!("walk_dir {}", self.endpoint.to_string());
 
+        let disk = self.disk_ref().await;
+        let body = serde_json::to_vec(&opts)?;
+        let stall_timeout = get_drive_walkdir_stall_timeout();
+        let bucket = opts.bucket.clone();
+        let base_dir = opts.base_dir.clone();
+        let disk_for_log = disk.clone();
+
         self.execute_with_timeout_for_op_and_health_action(
             "walk_dir",
             || async {
-                let disk = self.disk_ref().await;
-                let opts = serde_json::to_vec(&opts)?;
-                let mut reader = self
-                    .data_transport
-                    .open_walk_dir(WalkDirStreamRequest {
-                        endpoint: self.endpoint.grid_host(),
-                        disk,
-                        body: opts,
-                        stall_timeout: Some(get_drive_walkdir_stall_timeout()),
-                    })
-                    .await?;
+                let mut last_err = None;
 
-                copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await?;
+                for attempt in 1..=2 {
+                    let mut reader = match self
+                        .data_transport
+                        .open_walk_dir(WalkDirStreamRequest {
+                            endpoint: self.endpoint.grid_host(),
+                            disk: disk.clone(),
+                            body: body.clone(),
+                            stall_timeout: Some(stall_timeout),
+                        })
+                        .await
+                    {
+                        Ok(reader) => reader,
+                        Err(err) => {
+                            if attempt == 1 && Self::is_retryable_walk_dir_error(&err) {
+                                warn!(
+                                    endpoint = %self.endpoint,
+                                    addr = %self.addr,
+                                    disk = %disk_for_log,
+                                    bucket = %bucket,
+                                    base_dir = %base_dir,
+                                    attempt,
+                                    stall_timeout_ms = stall_timeout.as_millis(),
+                                    error = %err,
+                                    "remote walk_dir returned retryable transport error; retrying"
+                                );
+                                last_err = Some(err);
+                                continue;
+                            }
 
-                Ok(())
+                            return Err(err);
+                        }
+                    };
+
+                    match copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await {
+                        Ok(_) => return Ok(()),
+                        Err(io_err) => return Err(DiskError::Io(io_err)),
+                    }
+                }
+
+                Err(last_err.unwrap_or_else(|| DiskError::other("walk_dir retry exhausted without captured error")))
             },
             get_drive_walkdir_timeout(),
             FailureHealthAction::IgnoreFailure,
@@ -1738,6 +1802,36 @@ mod tests {
         }
     }
 
+    #[derive(Debug)]
+    enum WalkDirTestStep {
+        Error(DiskError),
+        Data(Vec<u8>),
+        PartialDataThenError { data: Vec<u8>, error: io::Error },
+    }
+
+    #[derive(Debug, Clone, Default)]
+    struct RetryingWalkDirInternodeDataTransport {
+        calls: Arc<StdMutex<Vec<RecordedTransportCall>>>,
+        steps: Arc<StdMutex<Vec<WalkDirTestStep>>>,
+    }
+
+    impl RetryingWalkDirInternodeDataTransport {
+        fn with_steps(steps: Vec<WalkDirTestStep>) -> Self {
+            Self {
+                calls: Arc::new(StdMutex::new(Vec::new())),
+                steps: Arc::new(StdMutex::new(steps)),
+            }
+        }
+
+        fn calls(&self) -> Vec<RecordedTransportCall> {
+            self.calls.lock().expect("recorded transport calls lock poisoned").clone()
+        }
+
+        fn record(&self, call: RecordedTransportCall) {
+            self.calls.lock().expect("recorded transport calls lock poisoned").push(call);
+        }
+    }
+
     #[derive(Debug, Default)]
     struct EmptyTestReader;
 
@@ -1790,6 +1884,38 @@ mod tests {
         }
     }
 
+    #[async_trait::async_trait]
+    impl InternodeDataTransport for RetryingWalkDirInternodeDataTransport {
+        async fn open_read(&self, _request: ReadStreamRequest) -> Result<FileReader> {
+            panic!("open_read should not be used in walk_dir retry test");
+        }
+
+        async fn open_write(&self, _request: WriteStreamRequest) -> Result<FileWriter> {
+            panic!("open_write should not be used in walk_dir retry test");
+        }
+
+        async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader> {
+            self.record(RecordedTransportCall::WalkDir(request));
+            let step = self.steps.lock().expect("walk_dir retry steps lock poisoned").remove(0);
+            match step {
+                WalkDirTestStep::Error(err) => Err(err),
+                WalkDirTestStep::Data(data) => Ok(Box::new(Cursor::new(data))),
+                WalkDirTestStep::PartialDataThenError { data, error } => Ok(Box::new(PartialThenErrorReader {
+                    cursor: Cursor::new(data),
+                    error: Some(error),
+                })),
+            }
+        }
+
+        fn name(&self) -> &'static str {
+            "retrying-walk-dir"
+        }
+
+        fn capabilities(&self) -> InternodeDataTransportCapabilities {
+            InternodeDataTransportCapabilities::tcp_http()
+        }
+    }
+
     async fn new_remote_disk_with_transport(data_transport: Arc<dyn InternodeDataTransport>) -> RemoteDisk {
         let endpoint = Endpoint {
             url: url::Url::parse("http://remote-node:9000/data/rustfs0").unwrap(),
@@ -1806,6 +1932,32 @@ mod tests {
         RemoteDisk::new(&endpoint, &disk_option, data_transport).await.unwrap()
     }
 
+    #[derive(Debug)]
+    struct PartialThenErrorReader {
+        cursor: Cursor<Vec<u8>>,
+        error: Option<io::Error>,
+    }
+
+    impl AsyncRead for PartialThenErrorReader {
+        fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
+            let filled_before = buf.filled().len();
+            match Pin::new(&mut self.cursor).poll_read(cx, buf) {
+                Poll::Ready(Ok(())) => {
+                    if buf.filled().len() > filled_before {
+                        return Poll::Ready(Ok(()));
+                    }
+
+                    if let Some(err) = self.error.take() {
+                        return Poll::Ready(Err(err));
+                    }
+
+                    Poll::Ready(Ok(()))
+                }
+                other => other,
+            }
+        }
+    }
+
     fn init_tracing(filter_level: Level) {
         INIT.call_once(|| {
             let _ = tracing_subscriber::fmt()
@@ -1951,20 +2103,39 @@ mod tests {
             disk_idx: 0,
         };
 
-        let disk_option = DiskOption {
-            cleanup: false,
-            health_check: true,
-        };
+        temp_env::async_with_vars(
+            [
+                (rustfs_config::ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS, Some("1")),
+                (rustfs_config::ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS, Some("1")),
+            ],
+            async {
+                let disk_option = DiskOption {
+                    cleanup: false,
+                    health_check: true,
+                };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
-            .await
-            .unwrap();
-        remote_disk.enable_health_check();
+                let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+                    .await
+                    .unwrap();
+                remote_disk.enable_health_check();
 
-        // wait for health check connect timeout
-        tokio::time::sleep(Duration::from_secs(6)).await;
-
-        assert!(!remote_disk.is_online().await);
+                // Wait out the initial success-grace window so the active probe loop
+                // actually attempts a connectivity check. Under the new
+                // suspect-first semantics we only need to prove that the drive
+                // transitions away from a clean Online state at least once.
+                tokio::time::sleep(SKIP_IF_SUCCESS_BEFORE + Duration::from_secs(2)).await;
+                assert!(
+                    remote_disk.offline_duration_secs().is_some(),
+                    "missing listener should transition the drive through suspect/offline tracking"
+                );
+                assert_ne!(
+                    remote_disk.runtime_state(),
+                    RuntimeDriveHealthState::Online,
+                    "missing listener should not remain in a clean Online state after probing"
+                );
+            },
+        )
+        .await;
     }
 
     #[tokio::test]
@@ -2145,6 +2316,66 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_retries_once_on_retryable_transport_error() {
+        let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![
+            WalkDirTestStep::Error(DiskError::other("HttpReader stream error: error decoding response body")),
+            WalkDirTestStep::Data(b"walk-dir-retry-ok".to_vec()),
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "config/iam".to_string(),
+            recursive: true,
+            report_notfound: false,
+            filter_prefix: None,
+            forward_to: None,
+            limit: 10,
+            disk_id: String::new(),
+        };
+        let mut writer = Vec::new();
+
+        remote_disk
+            .walk_dir(opts, &mut writer)
+            .await
+            .expect("retryable walk_dir error should recover");
+
+        assert_eq!(writer, b"walk-dir-retry-ok");
+        assert_eq!(transport.calls().len(), 2, "walk_dir should retry exactly once");
+    }
+
+    #[tokio::test]
+    async fn test_remote_disk_walk_dir_does_not_retry_after_partial_stream_failure() {
+        let transport = RetryingWalkDirInternodeDataTransport::with_steps(vec![
+            WalkDirTestStep::PartialDataThenError {
+                data: b"partial-walk-dir".to_vec(),
+                error: io::Error::new(io::ErrorKind::ConnectionReset, "connection reset"),
+            },
+            WalkDirTestStep::Data(b"walk-dir-retry-ok".to_vec()),
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+        let opts = WalkDirOptions {
+            bucket: "bucket".to_string(),
+            base_dir: "config/iam".to_string(),
+            recursive: true,
+            report_notfound: false,
+            filter_prefix: None,
+            forward_to: None,
+            limit: 10,
+            disk_id: String::new(),
+        };
+        let mut writer = Vec::new();
+
+        let err = remote_disk
+            .walk_dir(opts, &mut writer)
+            .await
+            .expect_err("partial stream failure should be returned without retry");
+
+        assert!(matches!(err, DiskError::Io(ref io_err) if io_err.kind() == io::ErrorKind::ConnectionReset));
+        assert_eq!(writer, b"partial-walk-dir");
+        assert_eq!(transport.calls().len(), 1, "walk_dir should not retry after writing partial bytes");
+    }
+
     #[tokio::test]
     async fn test_remote_disk_endpoints_with_different_schemes() {
         let test_cases = vec![
@@ -2284,7 +2515,12 @@ mod tests {
             .expect_err("timeout should fail");
 
         assert!(err.to_string().contains("timeout"));
-        assert!(!remote_disk.is_online().await, "remote disk should be marked faulty after timeout");
+        assert!(remote_disk.is_online().await, "first timeout should keep the remote disk online");
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first timeout should move the remote disk into suspect state"
+        );
     }
 
     #[tokio::test]
@@ -2450,7 +2686,15 @@ mod tests {
             },
             std::io::ErrorKind::TimedOut
         );
-        assert!(!remote_disk.is_online().await, "timeout-like errors should mark remote disk faulty");
+        assert!(
+            remote_disk.is_online().await,
+            "first timeout-like error should keep the remote disk online"
+        );
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first timeout-like error should move the remote disk into suspect state"
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "timeout-like errors should evict cached connection"
@@ -2503,7 +2747,15 @@ mod tests {
             },
             std::io::ErrorKind::ConnectionRefused
         );
-        assert!(!remote_disk.is_online().await, "network-like errors should mark remote disk faulty");
+        assert!(
+            remote_disk.is_online().await,
+            "first network-like error should keep the remote disk online"
+        );
+        assert_eq!(
+            remote_disk.runtime_state(),
+            RuntimeDriveHealthState::Suspect,
+            "first network-like error should move the remote disk into suspect state"
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "network-like errors should evict cached connection"

crates/ecstore/src/rpc/remote_locker.rs

@@ -14,18 +14,21 @@
 
 use crate::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
 use async_trait::async_trait;
+use bytes::Bytes;
 use rustfs_lock::{
     LockClient, LockError, LockInfo, LockRequest, LockResponse, LockStats, LockStatus, LockType, Result,
     types::{LockId, LockMetadata, LockPriority},
 };
 use rustfs_protos::proto_gen::node_service::{BatchGenerallyLockRequest, GenerallyLockRequest, PingRequest};
-use rustfs_protos::{evict_failed_connection, proto_gen::node_service::node_service_client::NodeServiceClient};
+use rustfs_protos::{
+    evict_failed_connection, models::PingBodyBuilder, proto_gen::node_service::node_service_client::NodeServiceClient,
+};
 use std::time::Duration;
 use tokio::time::timeout;
 use tonic::Request;
 use tonic::service::interceptor::InterceptedService;
 use tonic::transport::Channel;
-use tracing::{info, warn};
+use tracing::{debug, info, warn};
 
 /// Remote lock client implementation
 #[derive(Debug, Clone)]
@@ -42,6 +45,20 @@ impl RemoteClient {
         Self { addr: url.to_string() }
     }
 
+    fn build_ping_request() -> PingRequest {
+        let mut fbb = flatbuffers::FlatBufferBuilder::new();
+        let payload = fbb.create_vector(b"health-check");
+        let mut builder = PingBodyBuilder::new(&mut fbb);
+        builder.add_payload(payload);
+        let root = builder.finish();
+        fbb.finish(root, None);
+
+        PingRequest {
+            version: 1,
+            body: Bytes::copy_from_slice(fbb.finished_data()),
+        }
+    }
+
     /// Create a minimal LockRequest for unlock operations using only lock_id
     fn create_unlock_request(lock_id: &LockId) -> LockRequest {
         LockRequest {
@@ -64,16 +81,44 @@ impl RemoteClient {
             .map_err(|err| LockError::internal(format!("can not get client, err: {err}")))
     }
 
-    async fn evict_connection(&self, op: &'static str, reason: &str) {
-        warn!(
-            addr = %self.addr,
-            op,
-            reason,
-            "Evicting cached remote lock connection after RPC failure"
-        );
+    fn is_scanner_leader_lock(resource_summary: &str) -> bool {
+        resource_summary == ".rustfs.sys/leader.lock@latest"
+    }
+
+    async fn evict_connection(&self, op: &'static str, reason: &str, resource_summary: &str) {
+        if Self::is_scanner_leader_lock(resource_summary) {
+            debug!(
+                addr = %self.addr,
+                op,
+                reason,
+                resource_summary,
+                "Evicting cached remote lock connection for scanner leader-lock RPC failure"
+            );
+        } else {
+            warn!(
+                addr = %self.addr,
+                op,
+                reason,
+                resource_summary,
+                "Evicting cached remote lock connection after RPC failure"
+            );
+        }
         evict_failed_connection(&self.addr).await;
     }
 
+    fn summarize_resources(requests: &[LockRequest]) -> String {
+        const LIMIT: usize = 3;
+        let mut resources = requests
+            .iter()
+            .take(LIMIT)
+            .map(|request| request.resource.to_string())
+            .collect::<Vec<_>>();
+        if requests.len() > LIMIT {
+            resources.push(format!("... (+{} more)", requests.len() - LIMIT));
+        }
+        resources.join(", ")
+    }
+
     fn rpc_timeout(timeout_duration: Duration) -> Duration {
         if timeout_duration.is_zero() {
             Duration::from_millis(1)
@@ -86,39 +131,74 @@ impl RemoteClient {
         &self,
         op: &'static str,
         timeout_duration: Duration,
+        resource_summary: &str,
         future: F,
     ) -> std::result::Result<T, LockError>
     where
         F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
     {
-        let timeout_duration = Self::rpc_timeout(timeout_duration);
-        match timeout(timeout_duration, future).await {
+        let lock_timeout = Self::rpc_timeout(timeout_duration);
+        match timeout(lock_timeout, future).await {
             Ok(Ok(response)) => Ok(response),
             Ok(Err(err)) => {
                 let reason = err.to_string();
-                self.evict_connection(op, &reason).await;
+                if Self::is_scanner_leader_lock(resource_summary) {
+                    debug!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        tonic_code = ?err.code(),
+                        tonic_message = err.message(),
+                        "Remote lock RPC returned tonic error for scanner leader lock"
+                    );
+                } else {
+                    warn!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        tonic_code = ?err.code(),
+                        tonic_message = err.message(),
+                        "Remote lock RPC returned tonic error"
+                    );
+                }
+                self.evict_connection(op, &reason, resource_summary).await;
                 Err(LockError::internal(format!("{op} RPC failed: {reason}")))
             }
             Err(_) => {
-                let reason = format!("RPC timed out after {:?}", timeout_duration);
-                self.evict_connection(op, &reason).await;
-                Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), timeout_duration))
+                let reason = format!("RPC timed out after {:?}", lock_timeout);
+                if Self::is_scanner_leader_lock(resource_summary) {
+                    debug!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        "Remote lock RPC timed out for scanner leader lock"
+                    );
+                } else {
+                    warn!(
+                        addr = %self.addr,
+                        op,
+                        timeout_ms = lock_timeout.as_millis(),
+                        resource_summary,
+                        "Remote lock RPC timed out"
+                    );
+                }
+                self.evict_connection(op, &reason, resource_summary).await;
+                Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), lock_timeout))
             }
         }
     }
 
-    fn timeout_failure_response(request: &LockRequest) -> LockResponse {
-        LockResponse::failure("Lock acquisition timeout", request.acquire_timeout)
+    fn rpc_timeout_failure_response(request: &LockRequest, err: &LockError) -> LockResponse {
+        LockResponse::failure(format!("Remote lock RPC timed out: {err}"), request.acquire_timeout)
     }
 
     fn rpc_failure_response(_request: &LockRequest, err: &LockError) -> LockResponse {
         LockResponse::failure(format!("Remote lock RPC failed: {err}"), Duration::ZERO)
     }
 
-    fn timeout_failure_batch(requests: &[LockRequest]) -> Vec<LockResponse> {
-        requests.iter().map(Self::timeout_failure_response).collect()
-    }
-
     fn rpc_failure_batch(requests: &[LockRequest], err: &LockError) -> Vec<LockResponse> {
         requests
             .iter()
@@ -126,6 +206,13 @@ impl RemoteClient {
             .collect()
     }
 
+    fn rpc_timeout_failure_batch(requests: &[LockRequest], err: &LockError) -> Vec<LockResponse> {
+        requests
+            .iter()
+            .map(|request| Self::rpc_timeout_failure_response(request, err))
+            .collect()
+    }
+
     fn batch_rpc_timeout(requests: &[LockRequest]) -> Duration {
         requests
             .iter()
@@ -179,14 +266,18 @@ impl LockClient for RemoteClient {
     async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
         info!("remote acquire_exclusive for {}", request.resource);
         let mut client = self.get_client().await?;
+        let resource_summary = request.resource.to_string();
         let req = Request::new(GenerallyLockRequest {
             args: serde_json::to_string(&request)
                 .map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
         });
 
-        let resp = match self.execute_rpc("lock", request.acquire_timeout, client.lock(req)).await {
+        let resp = match self
+            .execute_rpc("lock", request.acquire_timeout, &resource_summary, client.lock(req))
+            .await
+        {
             Ok(resp) => resp.into_inner(),
-            Err(LockError::Timeout { .. }) => return Ok(Self::timeout_failure_response(request)),
+            Err(err @ LockError::Timeout { .. }) => return Ok(Self::rpc_timeout_failure_response(request, &err)),
             Err(err) => return Ok(Self::rpc_failure_response(request, &err)),
         };
 
@@ -212,6 +303,7 @@ impl LockClient for RemoteClient {
         }
 
         let mut client = self.get_client().await?;
+        let resource_summary = Self::summarize_resources(requests);
         let req = Request::new(BatchGenerallyLockRequest {
             args: requests
                 .iter()
@@ -222,11 +314,11 @@ impl LockClient for RemoteClient {
         });
 
         let resp = match self
-            .execute_rpc("lock_batch", Self::batch_rpc_timeout(requests), client.lock_batch(req))
+            .execute_rpc("lock_batch", Self::batch_rpc_timeout(requests), &resource_summary, client.lock_batch(req))
             .await
         {
             Ok(resp) => resp.into_inner(),
-            Err(LockError::Timeout { .. }) => return Ok(Self::timeout_failure_batch(requests)),
+            Err(err @ LockError::Timeout { .. }) => return Ok(Self::rpc_timeout_failure_batch(requests, &err)),
             Err(err) => return Ok(Self::rpc_failure_batch(requests, &err)),
         };
 
@@ -435,10 +527,7 @@ impl LockClient for RemoteClient {
             }
         };
 
-        let ping_req = Request::new(PingRequest {
-            version: 1,
-            body: bytes::Bytes::new(),
-        });
+        let ping_req = Request::new(Self::build_ping_request());
 
         match client.ping(ping_req).await {
             Ok(_) => {
@@ -511,7 +600,14 @@ mod tests {
             "remote lock RPC should honor request timeout"
         );
         assert!(!response.success, "timed out lock acquisition should fail");
-        assert_eq!(response.error.as_deref(), Some("Lock acquisition timeout"));
+        assert!(
+            response
+                .error
+                .as_deref()
+                .is_some_and(|error| error.contains("Remote lock RPC timed out")),
+            "expected remote RPC timeout marker, got {:?}",
+            response.error
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "timeout should evict cached connection"
@@ -539,7 +635,14 @@ mod tests {
         );
         assert_eq!(responses.len(), 1);
         assert!(!responses[0].success, "timed out batch lock acquisition should fail");
-        assert_eq!(responses[0].error.as_deref(), Some("Lock acquisition timeout"));
+        assert!(
+            responses[0]
+                .error
+                .as_deref()
+                .is_some_and(|error| error.contains("Remote lock RPC timed out")),
+            "expected remote RPC timeout marker, got {:?}",
+            responses[0].error
+        );
         assert!(
             !GLOBAL_CONN_MAP.read().await.contains_key(&addr),
             "batch timeout should evict cached connection"

crates/ecstore/src/set_disk/heal.rs

@@ -39,12 +39,12 @@ impl SetDisks {
 
         let write_lock_guard = if !opts.no_lock {
             let ns_lock = self.new_ns_lock(bucket, object).await?;
-            Some(ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| {
-                StorageError::other(format!(
-                    "Failed to acquire write lock: {}",
-                    self.format_lock_error_from_error(bucket, object, "write", &e)
-                ))
-            })?)
+            Some(
+                ns_lock
+                    .get_write_lock(get_lock_acquire_timeout())
+                    .await
+                    .map_err(|e| self.map_namespace_lock_error(bucket, object, "write", e))?,
+            )
         } else {
             None
         };
@@ -417,7 +417,7 @@ impl SetDisks {
                                 }
 
                                 let is_inline_buffer = {
-                                    if let Some(sc) = GLOBAL_STORAGE_CLASS.get() {
+                                    if let Some(sc) = get_global_storage_class() {
                                         sc.should_inline(erasure.shard_file_size(latest_meta.size), false)
                                     } else {
                                         false
@@ -713,13 +713,7 @@ impl SetDisks {
             .await?
             .get_write_lock(get_lock_acquire_timeout())
             .await
-            .map_err(|e| {
-                let message = format!(
-                    "Failed to acquire write lock: {}",
-                    self.format_lock_error_from_error(bucket, object, "write", &e)
-                );
-                DiskError::other(message)
-            })?;
+            .map_err(|e| DiskError::other(self.map_namespace_lock_error(bucket, object, "write", e).to_string()))?;
 
         self.heal_object_dir_locked(bucket, object, dry_run, remove).await
     }

crates/ecstore/src/set_disk/lock.rs

@@ -50,6 +50,30 @@ impl SetDisks {
         }
     }
 
+    pub(super) fn map_namespace_lock_error(
+        &self,
+        bucket: &str,
+        object: &str,
+        mode: &'static str,
+        err: rustfs_lock::error::LockError,
+    ) -> StorageError {
+        match err {
+            rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
+                StorageError::NamespaceLockQuorumUnavailable {
+                    mode,
+                    bucket: bucket.to_string(),
+                    object: object.to_string(),
+                    required,
+                    achieved,
+                }
+            }
+            other => StorageError::other(format!(
+                "Failed to acquire {mode} lock: {}",
+                self.format_lock_error_from_error(bucket, object, mode, &other)
+            )),
+        }
+    }
+
     pub(super) async fn get_disks_internal(&self) -> Vec<Option<DiskStore>> {
         let rl = self.disks.read().await;
 

crates/ecstore/src/set_disk/replication.rs

@@ -25,7 +25,7 @@ impl SetDisks {
         let mut oi = obj_info.clone();
         oi.metadata_only = true;
 
-        oi.user_defined.remove(X_AMZ_RESTORE.as_str());
+        Arc::make_mut(&mut oi.user_defined).remove(X_AMZ_RESTORE.as_str());
 
         let version_id = oi.version_id.map(|v| v.to_string());
         let _obj = self

crates/ecstore/src/set_disk/write.rs

@@ -262,6 +262,11 @@ impl SetDisks {
         let dst_bucket = Arc::new(dst_bucket.to_string());
         let dst_object = Arc::new(dst_object.to_string());
 
+        // Match MinIO's multipart overwrite semantics: clear any stale destination
+        // part payload and metadata before the new per-disk rename fan-out begins.
+        self.cleanup_multipart_path(&[dst_object.to_string(), format!("{dst_object}.meta")])
+            .await;
+
         let mut errs = Vec::with_capacity(disks.len());
 
         let futures = disks.iter().map(|disk| {
@@ -378,30 +383,35 @@ impl SetDisks {
         }
 
         if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum) {
-            // TODO: add concurrency
+            let mut revert_futures = Vec::with_capacity(disks.len());
             for (i, err) in errs.iter().enumerate() {
                 if err.is_some() {
                     continue;
                 }
 
                 if let Some(disk) = disks[i].as_ref() {
-                    let _ = disk
-                        .delete(
-                            bucket,
-                            &path_join_buf(&[prefix, STORAGE_FORMAT_FILE]),
-                            DeleteOptions {
-                                recursive: true,
-                                ..Default::default()
-                            },
-                        )
-                        .await
-                        .map_err(|e| {
-                            warn!("write meta revert err {:?}", e);
-                            e
-                        });
+                    let disk = disk.clone();
+                    let bucket = bucket.to_string();
+                    let path = path_join_buf(&[prefix, STORAGE_FORMAT_FILE]);
+                    revert_futures.push(async move {
+                        if let Err(err) = disk
+                            .delete(
+                                &bucket,
+                                &path,
+                                DeleteOptions {
+                                    recursive: true,
+                                    ..Default::default()
+                                },
+                            )
+                            .await
+                        {
+                            warn!("write meta revert err {:?}", err);
+                        }
+                    });
                 }
             }
 
+            join_all(revert_futures).await;
             return Err(err);
         }
         Ok(())

crates/ecstore/src/sets.rs

@@ -306,12 +306,10 @@ impl Sets {
     //     unimplemented!()
     // }
 
-    async fn delete_prefix(&self, bucket: &str, object: &str) -> Result<()> {
+    async fn delete_prefix(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<()> {
         let mut futures = Vec::new();
-        let opt = ObjectOptions {
-            delete_prefix: true,
-            ..Default::default()
-        };
+        let mut opt = opts.clone();
+        opt.delete_prefix = true;
 
         for set in self.disk_set.iter() {
             futures.push(set.delete_object(bucket, object, opt.clone()));
@@ -463,6 +461,7 @@ impl ObjectOperations for Sets {
             versioned: dst_opts.versioned,
             version_id: dst_opts.version_id.clone(),
             mod_time: dst_opts.mod_time,
+            http_preconditions: dst_opts.http_preconditions.clone(),
             ..Default::default()
         };
 
@@ -485,7 +484,7 @@ impl ObjectOperations for Sets {
     #[tracing::instrument(skip(self))]
     async fn delete_object(&self, bucket: &str, object: &str, opts: ObjectOptions) -> Result<ObjectInfo> {
         if opts.delete_prefix && !opts.delete_prefix_object {
-            self.delete_prefix(bucket, object).await?;
+            self.delete_prefix(bucket, object, &opts).await?;
             return Ok(ObjectInfo::default());
         }
 

crates/ecstore/src/store.rs

@@ -31,7 +31,6 @@ use crate::bucket::utils::check_object_args;
 use crate::bucket::utils::check_put_object_args;
 use crate::bucket::utils::check_put_object_part_args;
 use crate::bucket::utils::{check_valid_bucket_name, check_valid_bucket_name_strict, is_meta_bucketname};
-use crate::config::GLOBAL_STORAGE_CLASS;
 use crate::config::storageclass;
 use crate::disk::endpoint::{Endpoint, EndpointType};
 use crate::disk::{DiskAPI, DiskInfo, DiskInfoOptions};

crates/ecstore/src/store/bucket.rs

@@ -13,12 +13,48 @@
 // limitations under the License.
 
 use super::*;
-use crate::bucket::utils::is_meta_bucketname;
+use crate::bucket::{metadata::BUCKET_TABLE_RESERVED_PREFIX, utils::is_meta_bucketname};
+use crate::set_disk::get_lock_acquire_timeout;
 
 fn should_override_created_from_metadata(created: OffsetDateTime) -> bool {
     created != OffsetDateTime::UNIX_EPOCH
 }
 
+fn validate_table_bucket_delete_allowed(
+    bucket: &str,
+    table_bucket_enabled: bool,
+    table_catalog_metadata_exists: bool,
+) -> Result<()> {
+    if table_bucket_enabled && table_catalog_metadata_exists {
+        return Err(StorageError::BucketNotEmpty(bucket.to_string()));
+    }
+
+    Ok(())
+}
+
+async fn table_catalog_metadata_exists(bucket: &str) -> bool {
+    let local_disks = all_local_disk().await;
+    for disk in local_disks.iter() {
+        let catalog_path = disk.path().join(bucket).join(BUCKET_TABLE_RESERVED_PREFIX);
+        if has_xlmeta_files(&catalog_path).await {
+            return true;
+        }
+    }
+
+    false
+}
+
+async fn validate_table_bucket_delete_guard(bucket: &str) -> Result<()> {
+    let table_bucket_enabled = metadata_sys::get(bucket)
+        .await
+        .is_ok_and(|metadata| metadata.table_bucket_enabled());
+    if table_bucket_enabled {
+        validate_table_bucket_delete_allowed(bucket, true, table_catalog_metadata_exists(bucket).await)?;
+    }
+
+    Ok(())
+}
+
 impl ECStore {
     #[instrument(skip(self))]
     pub(super) async fn handle_make_bucket(&self, bucket: &str, opts: &MakeBucketOptions) -> Result<()> {
@@ -28,7 +64,28 @@ impl ECStore {
             return Err(StorageError::BucketNameInvalid(err.to_string()));
         }
 
-        // TODO: nslock
+        let _ns_guard = if !opts.no_lock {
+            let ns_lock = self.new_ns_lock(bucket, bucket).await?;
+            Some(
+                ns_lock
+                    .get_write_lock(get_lock_acquire_timeout())
+                    .await
+                    .map_err(|e| match e {
+                        rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
+                            StorageError::NamespaceLockQuorumUnavailable {
+                                mode: "write",
+                                bucket: bucket.to_string(),
+                                object: bucket.to_string(),
+                                required,
+                                achieved,
+                            }
+                        }
+                        other => StorageError::other(format!("make_bucket: failed to acquire write lock on {bucket}: {other}")),
+                    })?,
+            )
+        } else {
+            None
+        };
 
         if let Err(err) = self.peer_sys.make_bucket(bucket, opts).await {
             let err = to_object_err(err.into(), vec![bucket]);
@@ -111,7 +168,28 @@ impl ECStore {
             return Err(StorageError::BucketNameInvalid(err.to_string()));
         }
 
-        // TODO: nslock
+        let _ns_guard = if !opts.no_lock {
+            let ns_lock = self.new_ns_lock(bucket, bucket).await?;
+            Some(
+                ns_lock
+                    .get_write_lock(get_lock_acquire_timeout())
+                    .await
+                    .map_err(|e| match e {
+                        rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
+                            StorageError::NamespaceLockQuorumUnavailable {
+                                mode: "write",
+                                bucket: bucket.to_string(),
+                                object: bucket.to_string(),
+                                required,
+                                achieved,
+                            }
+                        }
+                        other => StorageError::other(format!("delete_bucket: failed to acquire write lock on {bucket}: {other}")),
+                    })?,
+            )
+        } else {
+            None
+        };
 
         // Check bucket exists before deletion (per S3 API spec)
         // If bucket doesn't exist, return NoSuchBucket error
@@ -124,6 +202,8 @@ impl ECStore {
             return Err(to_object_err(storage_err, vec![bucket]));
         }
 
+        validate_table_bucket_delete_guard(bucket).await?;
+
         // Check bucket is empty before deletion (per S3 API spec)
         // If bucket is not empty (contains actual objects with xl.meta files) and force
         // is not set, return BucketNotEmpty error.
@@ -160,7 +240,8 @@ impl ECStore {
 
 #[cfg(test)]
 mod tests {
-    use super::should_override_created_from_metadata;
+    use super::{should_override_created_from_metadata, validate_table_bucket_delete_allowed};
+    use crate::error::StorageError;
     use time::OffsetDateTime;
 
     #[test]
@@ -173,4 +254,13 @@ mod tests {
         let created = OffsetDateTime::from_unix_timestamp(1704067200).expect("valid timestamp");
         assert!(should_override_created_from_metadata(created));
     }
+
+    #[test]
+    fn table_bucket_delete_guard_rejects_remaining_catalog_metadata() {
+        let err = validate_table_bucket_delete_allowed("table-bucket", true, true).unwrap_err();
+
+        assert!(matches!(err, StorageError::BucketNotEmpty(bucket) if bucket == "table-bucket"));
+        assert!(validate_table_bucket_delete_allowed("table-bucket", true, false).is_ok());
+        assert!(validate_table_bucket_delete_allowed("regular-bucket", false, true).is_ok());
+    }
 }

crates/ecstore/src/store/object.rs

@@ -13,6 +13,151 @@
 // limitations under the License.
 
 use super::*;
+use crate::set_disk::{
+    get_lock_acquire_timeout, get_object_lock_diag_slow_acquire_threshold, get_object_lock_diag_slow_hold_threshold,
+    is_lock_optimization_enabled, is_object_lock_diag_enabled,
+};
+use rustfs_io_metrics::{
+    record_object_lock_diag_acquire_duration, record_object_lock_diag_hold_duration, record_object_lock_diag_slow_acquire,
+    record_object_lock_diag_slow_hold,
+};
+use std::{
+    fmt,
+    pin::Pin,
+    task::{Context, Poll},
+    time::{Duration, Instant},
+};
+use tokio::io::{AsyncRead, ReadBuf};
+
+struct LockGuardedReader {
+    inner: Box<dyn AsyncRead + Unpin + Send + Sync>,
+    guard: Option<ObjectLockDiagGuard>,
+}
+
+impl AsyncRead for LockGuardedReader {
+    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
+        let had_capacity = buf.remaining() > 0;
+        let filled_before = buf.filled().len();
+        let poll = Pin::new(&mut self.inner).poll_read(cx, buf);
+        if had_capacity && matches!(poll, Poll::Ready(Ok(()))) && buf.filled().len() == filled_before {
+            self.guard.take();
+        }
+        poll
+    }
+}
+
+#[derive(Clone, Copy, Debug)]
+enum ObjectLockDiagMode {
+    Read,
+    Write,
+}
+
+impl ObjectLockDiagMode {
+    fn as_str(self) -> &'static str {
+        match self {
+            Self::Read => "read",
+            Self::Write => "write",
+        }
+    }
+}
+
+impl fmt::Display for ObjectLockDiagMode {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.write_str(self.as_str())
+    }
+}
+
+struct ObjectLockDiagGuard {
+    guard: rustfs_lock::NamespaceLockGuard,
+    enabled: bool,
+    op: &'static str,
+    bucket: Option<String>,
+    object: Option<String>,
+    owner: Option<String>,
+    mode: ObjectLockDiagMode,
+    acquired_at: Instant,
+}
+
+impl ObjectLockDiagGuard {
+    fn new(
+        guard: rustfs_lock::NamespaceLockGuard,
+        enabled: bool,
+        op: &'static str,
+        bucket: Option<String>,
+        object: Option<String>,
+        owner: Option<String>,
+        mode: ObjectLockDiagMode,
+    ) -> Self {
+        Self {
+            guard,
+            enabled,
+            op,
+            bucket,
+            object,
+            owner,
+            mode,
+            acquired_at: Instant::now(),
+        }
+    }
+}
+
+impl Drop for ObjectLockDiagGuard {
+    fn drop(&mut self) {
+        if !self.enabled || self.guard.is_released() {
+            return;
+        }
+
+        let hold = self.acquired_at.elapsed();
+        record_object_lock_diag_hold_duration(self.op, self.mode.as_str(), hold);
+        let threshold = get_object_lock_diag_slow_hold_threshold();
+        if hold >= threshold {
+            record_object_lock_diag_slow_hold(self.op, self.mode.as_str());
+            warn!(
+                target: "rustfs_ecstore::object_lock_diag",
+                op = self.op,
+                bucket = %self.bucket.as_deref().unwrap_or_default(),
+                object = %self.object.as_deref().unwrap_or_default(),
+                mode = %self.mode,
+                owner = %self.owner.as_deref().unwrap_or_default(),
+                hold_ms = hold.as_millis(),
+                threshold_ms = threshold.as_millis(),
+                "object namespace lock held longer than threshold"
+            );
+        }
+    }
+}
+
+fn log_object_lock_acquire_if_slow(
+    op: &'static str,
+    bucket: &str,
+    object: &str,
+    owner: Option<&str>,
+    mode: ObjectLockDiagMode,
+    elapsed: Duration,
+    diag_enabled: bool,
+) {
+    if !diag_enabled {
+        return;
+    }
+
+    let threshold = get_object_lock_diag_slow_acquire_threshold();
+    record_object_lock_diag_acquire_duration(op, mode.as_str(), elapsed);
+    if elapsed >= threshold {
+        record_object_lock_diag_slow_acquire(op, mode.as_str());
+        warn!(
+            target: "rustfs_ecstore::object_lock_diag",
+            op,
+            bucket,
+            object,
+            mode = %mode,
+            owner = owner.unwrap_or_default(),
+            acquire_ms = elapsed.as_millis(),
+            threshold_ms = threshold.as_millis(),
+            "object namespace lock acquisition exceeded threshold"
+        );
+    }
+}
+
 fn select_data_movement_target_pool(
     existing_pool_idx: Result<usize>,
     src_pool_idx: usize,
@@ -89,6 +234,116 @@ fn data_movement_pool_lookup_opts(opts: &ObjectOptions, no_lock: bool) -> Object
 }
 
 impl ECStore {
+    fn map_namespace_lock_error(bucket: &str, object: &str, mode: &'static str, err: rustfs_lock::LockError) -> StorageError {
+        match err {
+            rustfs_lock::LockError::QuorumNotReached { required, achieved } => StorageError::NamespaceLockQuorumUnavailable {
+                mode,
+                bucket: bucket.to_string(),
+                object: object.to_string(),
+                required,
+                achieved,
+            },
+            other => StorageError::other(format!("Failed to acquire {mode} lock on {bucket}/{object}: {other}")),
+        }
+    }
+
+    async fn acquire_object_write_lock_if_needed(
+        &self,
+        op: &'static str,
+        bucket: &str,
+        object: &str,
+        opts: &mut ObjectOptions,
+    ) -> Result<Option<ObjectLockDiagGuard>> {
+        if opts.no_lock {
+            return Ok(None);
+        }
+
+        let diag_enabled = is_object_lock_diag_enabled();
+        let ns_lock = self.handle_new_ns_lock(bucket, object).await?;
+        let acquire_start = Instant::now();
+        let guard = ns_lock
+            .get_write_lock(get_lock_acquire_timeout())
+            .await
+            .map_err(|err| Self::map_namespace_lock_error(bucket, object, "write", err))?;
+        let owner = diag_enabled.then(|| ns_lock.owner().to_string());
+        log_object_lock_acquire_if_slow(
+            op,
+            bucket,
+            object,
+            owner.as_deref(),
+            ObjectLockDiagMode::Write,
+            acquire_start.elapsed(),
+            diag_enabled,
+        );
+        opts.no_lock = true;
+
+        Ok(Some(ObjectLockDiagGuard::new(
+            guard,
+            diag_enabled,
+            op,
+            diag_enabled.then(|| bucket.to_string()),
+            diag_enabled.then(|| object.to_string()),
+            owner,
+            ObjectLockDiagMode::Write,
+        )))
+    }
+
+    async fn acquire_object_read_lock_if_needed(
+        &self,
+        op: &'static str,
+        bucket: &str,
+        object: &str,
+        opts: &mut ObjectOptions,
+    ) -> Result<Option<ObjectLockDiagGuard>> {
+        if opts.no_lock {
+            return Ok(None);
+        }
+
+        let diag_enabled = is_object_lock_diag_enabled();
+        let ns_lock = self.handle_new_ns_lock(bucket, object).await?;
+        let acquire_start = Instant::now();
+        let guard = ns_lock
+            .get_read_lock(get_lock_acquire_timeout())
+            .await
+            .map_err(|err| Self::map_namespace_lock_error(bucket, object, "read", err))?;
+        let owner = diag_enabled.then(|| ns_lock.owner().to_string());
+        log_object_lock_acquire_if_slow(
+            op,
+            bucket,
+            object,
+            owner.as_deref(),
+            ObjectLockDiagMode::Read,
+            acquire_start.elapsed(),
+            diag_enabled,
+        );
+        opts.no_lock = true;
+
+        Ok(Some(ObjectLockDiagGuard::new(
+            guard,
+            diag_enabled,
+            op,
+            diag_enabled.then(|| bucket.to_string()),
+            diag_enabled.then(|| object.to_string()),
+            owner,
+            ObjectLockDiagMode::Read,
+        )))
+    }
+
+    fn attach_read_lock_guard(mut reader: GetObjectReader, guard: Option<ObjectLockDiagGuard>) -> GetObjectReader {
+        if is_lock_optimization_enabled() {
+            return reader;
+        }
+
+        if let Some(guard) = guard {
+            reader.stream = Box::new(LockGuardedReader {
+                inner: reader.stream,
+                guard: Some(guard),
+            });
+        }
+
+        reader
+    }
+
     async fn get_latest_accessible_object_info_with_idx(
         &self,
         bucket: &str,
@@ -193,23 +448,25 @@ impl ECStore {
         check_get_obj_args(bucket, object)?;
 
         let object = encode_dir_object(object);
-
-        if self.single_pool() {
-            return self.pools[0].get_object_reader(bucket, object.as_str(), range, h, opts).await;
-        }
-
-        // TODO: nslock
-
         let mut opts = opts.clone();
-
-        opts.no_lock = true;
-
-        let (_, idx) = self
-            .get_latest_accessible_object_info_with_idx(bucket, &object, &opts)
+        let read_lock_guard = self
+            .acquire_object_read_lock_if_needed("get_object", bucket, &object, &mut opts)
             .await?;
-        self.pools[idx]
-            .get_object_reader(bucket, object.as_str(), range, h, &opts)
-            .await
+
+        let reader = if self.single_pool() {
+            self.pools[0]
+                .get_object_reader(bucket, object.as_str(), range, h, &opts)
+                .await?
+        } else {
+            let (_, idx) = self
+                .get_latest_accessible_object_info_with_idx(bucket, &object, &opts)
+                .await?;
+            self.pools[idx]
+                .get_object_reader(bucket, object.as_str(), range, h, &opts)
+                .await?
+        };
+
+        Ok(Self::attach_read_lock_guard(reader, read_lock_guard))
     }
 
     #[instrument(level = "debug", skip(self, data))]
@@ -224,6 +481,8 @@ impl ECStore {
 
         let object = encode_dir_object(object);
 
+        // Keep PUT atomic-read friendly: SetDisks takes the object write lock only
+        // around precondition checks and the final rename/commit.
         if self.single_pool() {
             return self.pools[0].put_object(bucket, object.as_str(), data, opts).await;
         }
@@ -251,16 +510,18 @@ impl ECStore {
         check_object_args(bucket, object)?;
 
         let object = encode_dir_object(object);
-
-        if self.single_pool() {
-            return self.pools[0].get_object_info(bucket, object.as_str(), opts).await;
-        }
-
-        // TODO: nslock
-
-        let (info, _) = self
-            .get_latest_accessible_object_info_with_idx(bucket, object.as_str(), opts)
+        let mut opts = opts.clone();
+        let _object_lock_guard = self
+            .acquire_object_read_lock_if_needed("get_object_info", bucket, &object, &mut opts)
             .await?;
+
+        let info = if self.single_pool() {
+            self.pools[0].get_object_info(bucket, object.as_str(), &opts).await?
+        } else {
+            self.get_latest_accessible_object_info_with_idx(bucket, object.as_str(), &opts)
+                .await?
+                .0
+        };
         opts.precondition_check(&info)?;
         Ok(info)
     }
@@ -285,43 +546,56 @@ impl ECStore {
 
         let cp_src_dst_same = path_join_buf(&[src_bucket, &src_object]) == path_join_buf(&[dst_bucket, &dst_object]);
 
-        // TODO: nslock
-
-        let pool_idx = self
-            .get_pool_info_existing_with_opts(src_bucket, &src_object, &version_aware_lookup_opts(src_opts, true))
-            .await?
-            .0
-            .index;
+        let mut dst_opts = dst_opts.clone();
+        let _dst_lock_guard = if cp_src_dst_same {
+            self.acquire_object_write_lock_if_needed("copy_object", dst_bucket, &dst_object, &mut dst_opts)
+                .await?
+        } else {
+            None
+        };
 
         if cp_src_dst_same {
+            let pool_idx = self
+                .get_pool_info_existing_with_opts(src_bucket, &src_object, &version_aware_lookup_opts(src_opts, true))
+                .await?
+                .0
+                .index;
+
             if let (Some(src_vid), Some(dst_vid)) = (&src_opts.version_id, &dst_opts.version_id)
                 && src_vid == dst_vid
             {
                 return self.pools[pool_idx]
-                    .copy_object(src_bucket, &src_object, dst_bucket, &dst_object, src_info, src_opts, dst_opts)
+                    .copy_object(src_bucket, &src_object, dst_bucket, &dst_object, src_info, src_opts, &dst_opts)
                     .await;
             }
 
             if !dst_opts.versioned && src_opts.version_id.is_none() {
                 return self.pools[pool_idx]
-                    .copy_object(src_bucket, &src_object, dst_bucket, &dst_object, src_info, src_opts, dst_opts)
+                    .copy_object(src_bucket, &src_object, dst_bucket, &dst_object, src_info, src_opts, &dst_opts)
                     .await;
             }
 
             if dst_opts.versioned && src_opts.version_id != dst_opts.version_id {
                 src_info.version_only = true;
                 return self.pools[pool_idx]
-                    .copy_object(src_bucket, &src_object, dst_bucket, &dst_object, src_info, src_opts, dst_opts)
+                    .copy_object(src_bucket, &src_object, dst_bucket, &dst_object, src_info, src_opts, &dst_opts)
                     .await;
             }
         }
 
+        let pool_idx = if dst_opts.no_lock {
+            self.get_pool_idx_no_lock(dst_bucket, &dst_object, src_info.size).await?
+        } else {
+            self.get_pool_idx(dst_bucket, &dst_object, src_info.size).await?
+        };
+
         let put_opts = ObjectOptions {
-            user_defined: src_info.user_defined.clone(),
+            user_defined: (*src_info.user_defined).clone(),
             versioned: dst_opts.versioned,
             version_id: dst_opts.version_id.clone(),
-            no_lock: true,
+            no_lock: dst_opts.no_lock,
             mod_time: dst_opts.mod_time,
+            http_preconditions: dst_opts.http_preconditions.clone(),
             ..Default::default()
         };
 
@@ -342,15 +616,29 @@ impl ECStore {
     pub(super) async fn handle_delete_object(&self, bucket: &str, object: &str, opts: ObjectOptions) -> Result<ObjectInfo> {
         check_del_obj_args(bucket, object)?;
 
-        if opts.delete_prefix {
-            self.delete_prefix(bucket, object).await?;
+        let object = if opts.delete_prefix && !opts.delete_prefix_object {
+            object.to_owned()
+        } else {
+            encode_dir_object(object)
+        };
+        let object = object.as_str();
+        let mut opts = opts;
+
+        if opts.delete_prefix && !opts.delete_prefix_object {
+            // Prefix deletes cover multiple object keys; an exact lock on the prefix string
+            // would not protect child objects.
+            self.delete_prefix(bucket, object, &opts).await?;
             return Ok(ObjectInfo::default());
         }
 
-        // TODO: nslock
+        let _object_lock_guard = self
+            .acquire_object_write_lock_if_needed("delete_object", bucket, object, &mut opts)
+            .await?;
 
-        let object = encode_dir_object(object);
-        let object = object.as_str();
+        if opts.delete_prefix {
+            self.delete_prefix(bucket, object, &opts).await?;
+            return Ok(ObjectInfo::default());
+        }
 
         let gopts = version_aware_lookup_opts(&opts, true);
 
@@ -704,7 +992,7 @@ impl ECStore {
         }
 
         let (oi, _) = self.get_latest_accessible_object_info_with_idx(bucket, &object, opts).await?;
-        Ok(oi.user_tags)
+        Ok((*oi.user_tags).clone())
     }
 
     #[instrument(level = "debug", skip(self))]
@@ -775,6 +1063,8 @@ impl ECStore {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use std::io::Cursor;
+    use tokio::io::AsyncReadExt;
 
     #[test]
     fn delete_marker_data_movement_falls_back_when_only_source_pool_has_object() {
@@ -983,4 +1273,80 @@ mod tests {
         assert!(lookup_opts.skip_decommissioned);
         assert!(lookup_opts.skip_rebalancing);
     }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn reader_lock_is_held_when_optimization_is_disabled() {
+        temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE, Some("false"))], async {
+            let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
+            let lock = rustfs_lock::NamespaceLock::with_local_manager("test".to_string(), manager);
+            let key = rustfs_lock::ObjectKey::new("bucket", "object");
+            let read_guard = lock
+                .get_read_lock(key.clone(), "reader", Duration::from_secs(1))
+                .await
+                .expect("read lock should be acquired");
+            let read_guard = ObjectLockDiagGuard::new(
+                read_guard,
+                true,
+                "test_get_object",
+                Some("bucket".to_string()),
+                Some("object".to_string()),
+                Some("reader".to_string()),
+                ObjectLockDiagMode::Read,
+            );
+            let reader = GetObjectReader {
+                stream: Box::new(Cursor::new(Vec::<u8>::new())),
+                object_info: ObjectInfo::default(),
+            };
+
+            let reader = ECStore::attach_read_lock_guard(reader, Some(read_guard));
+
+            lock.get_write_lock(key.clone(), "writer", Duration::from_millis(20))
+                .await
+                .expect_err("reader should hold the read lock");
+            drop(reader);
+            lock.get_write_lock(key, "writer", Duration::from_secs(1))
+                .await
+                .expect("dropping the reader should release the read lock");
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn reader_lock_is_released_after_stream_eof() {
+        temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_OPTIMIZATION_ENABLE, Some("false"))], async {
+            let manager = Arc::new(rustfs_lock::GlobalLockManager::new());
+            let lock = rustfs_lock::NamespaceLock::with_local_manager("test".to_string(), manager);
+            let key = rustfs_lock::ObjectKey::new("bucket", "object");
+            let read_guard = lock
+                .get_read_lock(key.clone(), "reader", Duration::from_secs(1))
+                .await
+                .expect("read lock should be acquired");
+            let read_guard = ObjectLockDiagGuard::new(
+                read_guard,
+                true,
+                "test_get_object",
+                Some("bucket".to_string()),
+                Some("object".to_string()),
+                Some("reader".to_string()),
+                ObjectLockDiagMode::Read,
+            );
+            let reader = GetObjectReader {
+                stream: Box::new(Cursor::new(vec![1, 2, 3])),
+                object_info: ObjectInfo::default(),
+            };
+
+            let mut reader = ECStore::attach_read_lock_guard(reader, Some(read_guard));
+            let mut output = Vec::new();
+            reader.stream.read_to_end(&mut output).await.expect("reader should reach EOF");
+            assert_eq!(output, vec![1, 2, 3]);
+
+            lock.get_write_lock(key, "writer", Duration::from_secs(1))
+                .await
+                .expect("EOF should release the read lock before the reader is dropped");
+            drop(reader);
+        })
+        .await;
+    }
 }

crates/ecstore/src/store/rebalance.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use super::*;
+use crate::config::get_global_storage_class;
 
 struct LatestObjectInfoCandidate {
     info: Option<ObjectInfo>,
@@ -182,17 +183,11 @@ impl ECStore {
         Ok(())
     }
 
-    pub(super) async fn delete_prefix(&self, bucket: &str, object: &str) -> Result<()> {
+    pub(super) async fn delete_prefix(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<()> {
         for pool in self.pools.iter() {
-            pool.delete_object(
-                bucket,
-                object,
-                ObjectOptions {
-                    delete_prefix: true,
-                    ..Default::default()
-                },
-            )
-            .await?;
+            let mut opts = opts.clone();
+            opts.delete_prefix = true;
+            pool.delete_object(bucket, object, opts).await?;
         }
 
         Ok(())
@@ -605,7 +600,7 @@ impl ECStore {
     #[instrument(skip(self))]
     pub(super) async fn handle_backend_info(&self) -> rustfs_madmin::BackendInfo {
         let (standard_sc_parity, rr_sc_parity) = {
-            if let Some(sc) = GLOBAL_STORAGE_CLASS.get() {
+            if let Some(sc) = get_global_storage_class() {
                 let sc_parity = sc
                     .get_parity_for_sc(storageclass::CLASS_STANDARD)
                     .or(Some(self.pools[0].default_parity_count));

crates/ecstore/src/store_api/readers.rs

@@ -652,12 +652,20 @@ fn multipart_part_numbers(parts: &[rustfs_filemeta::ObjectPartInfo]) -> Vec<usiz
     parts.iter().map(|part| part.number).collect()
 }
 
+fn metadata_get<'a>(metadata: &'a HashMap<String, String>, key: &str) -> Option<&'a str> {
+    metadata.get(key).map(String::as_str).or_else(|| {
+        metadata
+            .iter()
+            .find_map(|(candidate, value)| candidate.eq_ignore_ascii_case(key).then_some(value.as_str()))
+    })
+}
+
 async fn resolve_encryption_material(oi: &ObjectInfo, headers: &HeaderMap<HeaderValue>) -> Result<EncryptionMaterial> {
-    if oi.user_defined.contains_key(SSEC_ALGORITHM_HEADER) {
+    if metadata_get(&oi.user_defined, SSEC_ALGORITHM_HEADER).is_some() {
         return resolve_ssec_material(oi, headers);
     }
 
-    if oi.user_defined.contains_key(INTERNAL_ENCRYPTION_KEY_HEADER) {
+    if metadata_get(&oi.user_defined, INTERNAL_ENCRYPTION_KEY_HEADER).is_some() {
         return resolve_managed_material(&oi.user_defined).await;
     }
 
@@ -697,11 +705,9 @@ fn resolve_ssec_material(oi: &ObjectInfo, headers: &HeaderMap<HeaderValue>) -> R
         return Err(Error::other("SSE-C key MD5 mismatch"));
     }
 
-    let stored_md5 = oi
-        .user_defined
-        .get(SSEC_KEY_MD5_HEADER)
-        .ok_or_else(|| Error::other("missing stored SSE-C key md5"))?;
-    if stored_md5 != &expected_md5 {
+    let stored_md5 =
+        metadata_get(&oi.user_defined, SSEC_KEY_MD5_HEADER).ok_or_else(|| Error::other("missing stored SSE-C key md5"))?;
+    if stored_md5 != expected_md5 {
         return Err(Error::other("SSE-C key does not match object metadata"));
     }
 
@@ -712,16 +718,14 @@ fn resolve_ssec_material(oi: &ObjectInfo, headers: &HeaderMap<HeaderValue>) -> R
 }
 
 async fn resolve_managed_material(metadata: &HashMap<String, String>) -> Result<EncryptionMaterial> {
-    let encrypted_dek = metadata
-        .get(INTERNAL_ENCRYPTION_KEY_HEADER)
-        .ok_or_else(|| Error::other("missing managed encrypted DEK"))?;
+    let encrypted_dek =
+        metadata_get(metadata, INTERNAL_ENCRYPTION_KEY_HEADER).ok_or_else(|| Error::other("missing managed encrypted DEK"))?;
     let encrypted_dek = BASE64_STANDARD
         .decode(encrypted_dek)
         .map_err(|e| Error::other(format!("failed to decode managed encrypted DEK: {e}")))?;
 
-    let iv_b64 = metadata
-        .get(INTERNAL_ENCRYPTION_IV_HEADER)
-        .ok_or_else(|| Error::other("missing managed encryption IV"))?;
+    let iv_b64 =
+        metadata_get(metadata, INTERNAL_ENCRYPTION_IV_HEADER).ok_or_else(|| Error::other("missing managed encryption IV"))?;
     let iv = BASE64_STANDARD
         .decode(iv_b64)
         .map_err(|e| Error::other(format!("failed to decode managed encryption IV: {e}")))?;
@@ -730,10 +734,7 @@ async fn resolve_managed_material(metadata: &HashMap<String, String>) -> Result<
         .try_into()
         .map_err(|_| Error::other("managed encryption IV must be 12 bytes"))?;
 
-    let kms_key_id = metadata
-        .get(INTERNAL_ENCRYPTION_KEY_ID_HEADER)
-        .map(String::as_str)
-        .unwrap_or("default");
+    let kms_key_id = metadata_get(metadata, INTERNAL_ENCRYPTION_KEY_ID_HEADER).unwrap_or("default");
 
     let key_bytes = if let Some(service) = get_global_encryption_service().await {
         service
@@ -960,7 +961,7 @@ mod tests {
     fn test_http_range_spec_from_object_info_valid_and_invalid_parts() {
         let object_info = ObjectInfo {
             size: 300,
-            parts: vec![
+            parts: Arc::new(vec![
                 ObjectPartInfo {
                     etag: String::new(),
                     number: 1,
@@ -982,7 +983,7 @@ mod tests {
                     actual_size: 100,
                     ..Default::default()
                 },
-            ],
+            ]),
             ..Default::default()
         };
 
@@ -998,7 +999,7 @@ mod tests {
     fn test_http_range_spec_from_object_info_uses_actual_size() {
         let object_info = ObjectInfo {
             size: 90,
-            parts: vec![
+            parts: Arc::new(vec![
                 ObjectPartInfo {
                     etag: String::new(),
                     number: 1,
@@ -1020,7 +1021,7 @@ mod tests {
                     actual_size: 50,
                     ..Default::default()
                 },
-            ],
+            ]),
             ..Default::default()
         };
 
@@ -1033,7 +1034,7 @@ mod tests {
     fn test_http_range_spec_from_object_info_falls_back_to_part_size_when_actual_size_missing() {
         let object_info = ObjectInfo {
             size: 90,
-            parts: vec![
+            parts: Arc::new(vec![
                 ObjectPartInfo {
                     etag: String::new(),
                     number: 1,
@@ -1055,7 +1056,7 @@ mod tests {
                     actual_size: 0,
                     ..Default::default()
                 },
-            ],
+            ]),
             ..Default::default()
         };
 
@@ -1122,14 +1123,59 @@ mod tests {
         format!("{}:{}", BASE64_STANDARD.encode(nonce), BASE64_STANDARD.encode(ciphertext))
     }
 
+    #[tokio::test]
+    async fn resolve_managed_material_accepts_case_insensitive_metadata_keys() {
+        async_with_vars([("__RUSTFS_SSE_SIMPLE_CMK", Some(BASE64_STANDARD.encode([0u8; 32])))], async {
+            let data_key = [0x24; 32];
+            let base_nonce = [0x14; 12];
+            let encrypted_dek = encrypt_managed_dek_for_test(data_key, [0u8; 32]);
+            let metadata = HashMap::from([
+                ("X-Rustfs-Encryption-Key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
+                ("X-Rustfs-Encryption-IV".to_string(), BASE64_STANDARD.encode(base_nonce)),
+            ]);
+
+            let material = resolve_managed_material(&metadata)
+                .await
+                .expect("managed material should resolve mixed-case metadata keys");
+
+            assert_eq!(material.key_bytes, data_key);
+            assert_eq!(material.base_nonce, base_nonce);
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    async fn resolve_encryption_material_accepts_case_insensitive_metadata_keys() {
+        async_with_vars([("__RUSTFS_SSE_SIMPLE_CMK", Some(BASE64_STANDARD.encode([0u8; 32])))], async {
+            let data_key = [0x24; 32];
+            let base_nonce = [0x14; 12];
+            let encrypted_dek = encrypt_managed_dek_for_test(data_key, [0u8; 32]);
+            let metadata = HashMap::from([
+                ("X-Rustfs-Encryption-Key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
+                ("X-Rustfs-Encryption-IV".to_string(), BASE64_STANDARD.encode(base_nonce)),
+            ]);
+            let object_info = ObjectInfo {
+                user_defined: Arc::new(metadata),
+                ..Default::default()
+            };
+            let material = resolve_encryption_material(&object_info, &HeaderMap::new())
+                .await
+                .expect("resolve_encryption_material should accept mixed-case managed metadata");
+
+            assert_eq!(material.key_bytes, data_key);
+            assert_eq!(material.base_nonce, base_nonce);
+        })
+        .await;
+    }
+
     #[tokio::test]
     async fn test_get_object_reader_rejects_ssec_read_without_headers() {
         let object_info = ObjectInfo {
             size: 10,
-            user_defined: HashMap::from([
+            user_defined: Arc::new(HashMap::from([
                 ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
                 ("x-amz-server-side-encryption-customer-original-size".to_string(), "20".to_string()),
-            ]),
+            ])),
             ..Default::default()
         };
 
@@ -1155,10 +1201,10 @@ mod tests {
     async fn test_get_object_reader_restore_request_bypasses_encryption_range_rewrite() {
         let object_info = ObjectInfo {
             size: 10,
-            user_defined: HashMap::from([
+            user_defined: Arc::new(HashMap::from([
                 ("x-rustfs-encryption-key".to_string(), "encrypted-key".to_string()),
                 ("x-rustfs-encryption-original-size".to_string(), "20".to_string()),
-            ]),
+            ])),
             ..Default::default()
         };
 
@@ -1201,12 +1247,12 @@ mod tests {
 
             let object_info = ObjectInfo {
                 size: encrypted.len() as i64,
-                user_defined: HashMap::from([
+                user_defined: Arc::new(HashMap::from([
                     ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
                     ("x-rustfs-encryption-key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
                     ("x-rustfs-encryption-iv".to_string(), BASE64_STANDARD.encode(base_nonce)),
                     ("x-rustfs-encryption-original-size".to_string(), plaintext.len().to_string()),
-                ]),
+                ])),
                 ..Default::default()
             };
 
@@ -1247,12 +1293,12 @@ mod tests {
 
             let object_info = ObjectInfo {
                 size: encrypted.len() as i64,
-                user_defined: HashMap::from([
+                user_defined: Arc::new(HashMap::from([
                     ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
                     ("x-rustfs-encryption-key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
                     ("x-rustfs-encryption-iv".to_string(), BASE64_STANDARD.encode(base_nonce)),
                     ("x-rustfs-encryption-original-size".to_string(), plaintext.len().to_string()),
-                ]),
+                ])),
                 ..Default::default()
             };
             let range = HTTPRangeSpec {
@@ -1303,12 +1349,12 @@ mod tests {
 
                 let object_info = ObjectInfo {
                     size: encrypted.len() as i64,
-                    user_defined: HashMap::from([
+                    user_defined: Arc::new(HashMap::from([
                         ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
                         ("x-rustfs-encryption-key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
                         ("x-rustfs-encryption-iv".to_string(), BASE64_STANDARD.encode(base_nonce)),
                         ("x-rustfs-encryption-original-size".to_string(), plaintext.len().to_string()),
-                    ]),
+                    ])),
                     ..Default::default()
                 };
 
@@ -1340,18 +1386,18 @@ mod tests {
 
         let object_info = ObjectInfo {
             size: 3_000_000,
-            parts: vec![ObjectPartInfo {
+            parts: Arc::new(vec![ObjectPartInfo {
                 etag: String::new(),
                 number: 1,
                 size: 3_000_000,
                 actual_size: 4_194_304,
                 index: Some(index.into_vec()),
                 ..Default::default()
-            }],
-            user_defined: HashMap::from([
+            }]),
+            user_defined: Arc::new(HashMap::from([
                 ("x-minio-internal-compression".to_string(), "gzip".to_string()),
                 ("x-minio-internal-actual-size".to_string(), "4194304".to_string()),
-            ]),
+            ])),
             ..Default::default()
         };
 
@@ -1397,7 +1443,7 @@ mod tests {
             bucket: bucket.to_string(),
             name: object.to_string(),
             size: encrypted.len() as i64,
-            user_defined: HashMap::from([
+            user_defined: Arc::new(HashMap::from([
                 ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
                 (
                     "x-amz-server-side-encryption-customer-key-md5".to_string(),
@@ -1407,7 +1453,7 @@ mod tests {
                     "x-amz-server-side-encryption-customer-original-size".to_string(),
                     plaintext.len().to_string(),
                 ),
-            ]),
+            ])),
             ..Default::default()
         };
 
@@ -1450,7 +1496,7 @@ mod tests {
             bucket: bucket.to_string(),
             name: object.to_string(),
             size: encrypted.len() as i64,
-            user_defined: HashMap::from([
+            user_defined: Arc::new(HashMap::from([
                 ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
                 (
                     "x-amz-server-side-encryption-customer-key-md5".to_string(),
@@ -1460,7 +1506,7 @@ mod tests {
                     "x-amz-server-side-encryption-customer-original-size".to_string(),
                     plaintext.len().to_string(),
                 ),
-            ]),
+            ])),
             ..Default::default()
         };
         let range = HTTPRangeSpec {
@@ -1514,7 +1560,7 @@ mod tests {
             bucket: bucket.to_string(),
             name: object.to_string(),
             size: encrypted.len() as i64,
-            user_defined: HashMap::from([
+            user_defined: Arc::new(HashMap::from([
                 ("x-amz-server-side-encryption-customer-algorithm".to_string(), "AES256".to_string()),
                 (
                     "x-amz-server-side-encryption-customer-key-md5".to_string(),
@@ -1526,7 +1572,7 @@ mod tests {
                 ),
                 ("x-minio-internal-compression".to_string(), CompressionAlgorithm::default().to_string()),
                 ("x-minio-internal-actual-size".to_string(), plaintext.len().to_string()),
-            ]),
+            ])),
             ..Default::default()
         };
         let range = HTTPRangeSpec {

crates/ecstore/src/store_api/types.rs

@@ -293,7 +293,7 @@ pub struct ObjectInfo {
     // Actual size is the real size of the object uploaded by client.
     pub actual_size: i64,
     pub is_dir: bool,
-    pub user_defined: HashMap<String, String>,
+    pub user_defined: Arc<HashMap<String, String>>,
     pub parity_blocks: usize,
     pub data_blocks: usize,
     pub version_id: Option<Uuid>,
@@ -301,8 +301,8 @@ pub struct ObjectInfo {
     pub transitioned_object: TransitionedObject,
     pub restore_ongoing: bool,
     pub restore_expires: Option<OffsetDateTime>,
-    pub user_tags: String,
-    pub parts: Vec<ObjectPartInfo>,
+    pub user_tags: Arc<String>,
+    pub parts: Arc<Vec<ObjectPartInfo>>,
     pub is_latest: bool,
     pub content_type: Option<String>,
     pub content_encoding: Option<String>,
@@ -384,14 +384,24 @@ impl ObjectInfo {
     }
 
     pub fn is_encrypted(&self) -> bool {
+        // Corresponding to the logic in rustfs/src/sse.rs/encryption_material_to_metadata function
         use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
 
-        self.user_defined
-            .keys()
-            .any(|key| rustfs_utils::http::is_encryption_metadata_key(key))
-            || self.user_defined.contains_key(SSEC_ALGORITHM_HEADER)
-            || self.user_defined.contains_key(SSEC_KEY_HEADER)
-            || self.user_defined.contains_key(SSEC_KEY_MD5_HEADER)
+        self.user_defined.keys().any(|key| {
+            let key = key.to_lowercase();
+            key.starts_with("x-minio-encryption-")
+                || matches!(
+                    key.as_str(),
+                    "x-rustfs-encryption-key"
+                        | "x-rustfs-encryption-algorithm"
+                        | "x-rustfs-encryption-iv"
+                        | "x-amz-server-side-encryption-aws-kms-key-id"
+                        | SSEC_ALGORITHM_HEADER
+                        | SSEC_KEY_HEADER
+                        | SSEC_KEY_MD5_HEADER
+                        | "x-amz-server-side-encryption"
+                )
+        })
     }
 
     pub fn encryption_original_size(&self) -> std::io::Result<Option<i64>> {
@@ -467,7 +477,11 @@ impl ObjectInfo {
         };
 
         // tags
-        let user_tags = fi.metadata.get(AMZ_OBJECT_TAGGING).cloned().unwrap_or_default();
+        let user_tags: Arc<String> = fi
+            .metadata
+            .get(AMZ_OBJECT_TAGGING)
+            .map(|s| Arc::new(s.clone()))
+            .unwrap_or_default();
 
         let inlined = fi.inline_data();
 
@@ -561,7 +575,7 @@ impl ObjectInfo {
                 number: part.number,
                 error: part.error.clone(),
             })
-            .collect();
+            .collect::<Vec<_>>();
 
         // TODO: part checksums
 
@@ -575,7 +589,7 @@ impl ObjectInfo {
             delete_marker: fi.deleted,
             mod_time: fi.mod_time,
             size: fi.size,
-            parts,
+            parts: Arc::new(parts),
             is_latest: fi.is_latest,
             user_tags,
             content_type,
@@ -585,7 +599,7 @@ impl ObjectInfo {
             successor_mod_time: fi.successor_mod_time,
             etag,
             inlined,
-            user_defined: metadata,
+            user_defined: Arc::new(metadata),
             transitioned_object,
             checksum: fi.checksum.clone(),
             storage_class,
@@ -605,11 +619,12 @@ impl ObjectInfo {
         bucket: &str,
         prefix: &str,
         delimiter: Option<String>,
-        after_version_id: Option<Uuid>,
+        after_version_marker: Option<VersionMarker>,
     ) -> Vec<ObjectInfo> {
         let vcfg = get_versioning_config(bucket).await.ok();
         let mut objects = Vec::with_capacity(entries.entries().len());
         let mut prev_prefix = "";
+        let mut after_version_marker = after_version_marker;
         for entry in entries.entries() {
             if entry.is_object() {
                 if let Some(delimiter) = &delimiter {
@@ -646,12 +661,8 @@ impl ObjectInfo {
                     }
                 };
 
-                let versions = if let Some(vid) = after_version_id {
-                    if let Some(idx) = file_infos.find_version_index(vid) {
-                        &file_infos.versions[idx + 1..]
-                    } else {
-                        &file_infos.versions
-                    }
+                let versions = if let Some(marker) = after_version_marker.take() {
+                    versions_after_marker(&file_infos, marker)
                 } else {
                     &file_infos.versions
                 };
@@ -823,6 +834,23 @@ impl ObjectInfo {
     }
 }
 
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum VersionMarker {
+    Null,
+    Version(Uuid),
+}
+
+fn versions_after_marker(file_infos: &rustfs_filemeta::FileInfoVersions, marker: VersionMarker) -> &[FileInfo] {
+    let marker_idx = match marker {
+        VersionMarker::Null => file_infos.versions.iter().position(|version| version.version_id.is_none()),
+        VersionMarker::Version(vid) => file_infos.find_version_index(vid),
+    };
+
+    marker_idx
+        .map(|idx| &file_infos.versions[idx + 1..])
+        .unwrap_or(&file_infos.versions)
+}
+
 #[derive(Debug, Default)]
 pub struct ListObjectsInfo {
     // Indicates whether the returned list objects response is truncated. A
@@ -1066,6 +1094,100 @@ mod tests {
     use super::*;
     use rustfs_filemeta::ReplicationState;
 
+    #[test]
+    fn versions_after_marker_handles_null_version_marker() {
+        let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();
+        let last_version = Uuid::parse_str("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee").unwrap();
+        let file_infos = rustfs_filemeta::FileInfoVersions {
+            versions: vec![
+                FileInfo {
+                    version_id: Some(first_version),
+                    ..Default::default()
+                },
+                FileInfo {
+                    version_id: None,
+                    ..Default::default()
+                },
+                FileInfo {
+                    version_id: Some(last_version),
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        let versions = versions_after_marker(&file_infos, VersionMarker::Null);
+
+        assert_eq!(versions.len(), 1);
+        assert_eq!(versions[0].version_id, Some(last_version));
+    }
+
+    #[test]
+    fn versions_after_marker_handles_uuid_version_marker() {
+        let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();
+        let last_version = Uuid::parse_str("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee").unwrap();
+        let file_infos = rustfs_filemeta::FileInfoVersions {
+            versions: vec![
+                FileInfo {
+                    version_id: Some(first_version),
+                    ..Default::default()
+                },
+                FileInfo {
+                    version_id: None,
+                    ..Default::default()
+                },
+                FileInfo {
+                    version_id: Some(last_version),
+                    ..Default::default()
+                },
+            ],
+            ..Default::default()
+        };
+
+        let versions = versions_after_marker(&file_infos, VersionMarker::Version(first_version));
+
+        assert_eq!(versions.len(), 2);
+        assert_eq!(versions[0].version_id, None);
+        assert_eq!(versions[1].version_id, Some(last_version));
+    }
+
+    #[tokio::test]
+    async fn versions_listing_applies_version_marker_only_to_first_entry() {
+        let metadata = rustfs_filemeta::test_data::create_real_xlmeta().expect("test metadata should be valid");
+        let entries = rustfs_filemeta::MetaCacheEntriesSorted {
+            o: rustfs_filemeta::MetaCacheEntries(vec![
+                Some(rustfs_filemeta::MetaCacheEntry {
+                    name: "obj-a".to_owned(),
+                    metadata: metadata.clone(),
+                    ..Default::default()
+                }),
+                Some(rustfs_filemeta::MetaCacheEntry {
+                    name: "obj-b".to_owned(),
+                    metadata,
+                    ..Default::default()
+                }),
+            ]),
+            ..Default::default()
+        };
+        let marker_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();
+
+        let objects = ObjectInfo::from_meta_cache_entries_sorted_versions(
+            &entries,
+            "bucket",
+            "",
+            None,
+            Some(VersionMarker::Version(marker_version)),
+        )
+        .await;
+
+        let obj_a_count = objects.iter().filter(|object| object.name == "obj-a").count();
+        let obj_b_count = objects.iter().filter(|object| object.name == "obj-b").count();
+
+        assert_eq!(obj_a_count, 2);
+        assert_eq!(obj_b_count, 3);
+        assert_eq!(objects.len(), 5);
+    }
+
     #[test]
     fn get_actual_size_prefers_actual_size_field() {
         let info = ObjectInfo {
@@ -1089,7 +1211,7 @@ mod tests {
         let info = ObjectInfo {
             size: 100,
             actual_size: 0,
-            user_defined,
+            user_defined: Arc::new(user_defined),
             ..Default::default()
         };
 
@@ -1107,7 +1229,7 @@ mod tests {
         let info = ObjectInfo {
             size: 100,
             actual_size: 0,
-            user_defined,
+            user_defined: Arc::new(user_defined),
             ..Default::default()
         };
 
@@ -1159,8 +1281,8 @@ mod tests {
         let info = ObjectInfo {
             size: 12,
             actual_size: 0,
-            user_defined,
-            parts: vec![
+            user_defined: Arc::new(user_defined),
+            parts: Arc::new(vec![
                 rustfs_filemeta::ObjectPartInfo {
                     actual_size: 4,
                     ..Default::default()
@@ -1169,7 +1291,7 @@ mod tests {
                     actual_size: 5,
                     ..Default::default()
                 },
-            ],
+            ]),
             ..Default::default()
         };
 
@@ -1187,10 +1309,139 @@ mod tests {
         let info = ObjectInfo {
             size: 12,
             actual_size: 0,
-            user_defined,
+            user_defined: Arc::new(user_defined),
             ..Default::default()
         };
 
         assert!(info.get_actual_size().is_err());
     }
+
+    #[test]
+    fn is_encrypted_correct_for_old_version_fileinfo() {
+        let mut user_defined: HashMap<String, String> = HashMap::new();
+
+        let metadata = vec![
+            ("content-type", "text/plain"),
+            ("etag", "e4336b5de4e2180a53fe2e17d03abe4f-4"),
+            ("x-minio-internal-actual-size", "67108864"),
+            ("x-rustfs-encryption-original-size", "67108864"),
+            ("x-rustfs-internal-actual-size", "67108864"),
+        ];
+
+        metadata.into_iter().for_each(|(key, value)| {
+            user_defined.insert(key.to_string(), value.to_string());
+        });
+
+        let info = ObjectInfo {
+            user_defined: Arc::new(user_defined),
+            ..Default::default()
+        };
+
+        assert!(!info.is_encrypted());
+    }
+
+    #[test]
+    fn is_encrypted_returns_true_when_encryption_metadata_present() {
+        let mut user_defined: HashMap<String, String> = HashMap::new();
+
+        let metadata = vec![
+            ("content-type", "text/plain"),
+            ("etag", "f1c9645dbc14efddc7d8a322685f26eb"),
+            ("x-amz-server-side-encryption", "AES256"),
+            ("x-rustfs-encryption-algorithm", "AES256"),
+            ("x-rustfs-encryption-iv", "Fb9moBlEBRE0D14F"),
+            (
+                "x-rustfs-encryption-key",
+                "QUFBQUFBQUFBQUFBQUFBQTpZQk5sNnNJdmJHWWl3QmxZbCtsMTJlVlZCeXVoVml4UlV4b3JPbTNoRk5odUlYVnBPdlpXNWVyT0FTcklXMWJr",
+            ),
+            ("x-rustfs-encryption-key-id", "default"),
+            ("x-rustfs-encryption-original-size", "10485760"),
+        ];
+
+        metadata.into_iter().for_each(|(key, value)| {
+            user_defined.insert(key.to_string(), value.to_string());
+        });
+
+        let info = ObjectInfo {
+            user_defined: Arc::new(user_defined),
+            ..Default::default()
+        };
+
+        assert!(info.is_encrypted());
+    }
+
+    #[test]
+    fn is_encrypted_handles_case_insensitive_rustfs_metadata_keys() {
+        let mut user_defined: HashMap<String, String> = HashMap::new();
+        user_defined.insert("X-Rustfs-Encryption-Key".to_string(), "encrypted-key".to_string());
+
+        let info = ObjectInfo {
+            user_defined: Arc::new(user_defined),
+            ..Default::default()
+        };
+
+        assert!(info.is_encrypted());
+    }
+
+    #[test]
+    fn objectinfo_clone_shares_arc_data_and_is_correct() {
+        let mut ud = HashMap::new();
+        ud.insert("content-type".to_string(), "application/octet-stream".to_string());
+        ud.insert("x-custom-header".to_string(), "custom-value".to_string());
+
+        let original = ObjectInfo {
+            bucket: "test-bucket".to_string(),
+            name: "test-object".to_string(),
+            user_defined: Arc::new(ud),
+            user_tags: Arc::new("env=prod&team=storage".to_string()),
+            parts: Arc::new(vec![
+                rustfs_filemeta::ObjectPartInfo {
+                    number: 1,
+                    size: 1024,
+                    actual_size: 1024,
+                    ..Default::default()
+                },
+                rustfs_filemeta::ObjectPartInfo {
+                    number: 2,
+                    size: 512,
+                    actual_size: 512,
+                    ..Default::default()
+                },
+            ]),
+            size: 1536,
+            etag: Some("abc123".to_string()),
+            ..Default::default()
+        };
+
+        let cloned = original.clone();
+
+        // Verify cloned values are correct
+        assert_eq!(cloned.bucket, "test-bucket");
+        assert_eq!(cloned.name, "test-object");
+        assert_eq!(cloned.size, 1536);
+        assert_eq!(cloned.etag, Some("abc123".to_string()));
+
+        // Verify Arc fields share the same allocation
+        assert!(Arc::ptr_eq(&original.user_defined, &cloned.user_defined));
+        assert!(Arc::ptr_eq(&original.user_tags, &cloned.user_tags));
+        assert!(Arc::ptr_eq(&original.parts, &cloned.parts));
+
+        // Verify Arc-wrapped data is accessible through the clone
+        assert_eq!(
+            cloned.user_defined.get("content-type").map(String::as_str),
+            Some("application/octet-stream")
+        );
+        assert_eq!(cloned.user_tags.as_str(), "env=prod&team=storage");
+        assert_eq!(cloned.parts.len(), 2);
+        assert_eq!(cloned.parts[0].number, 1);
+        assert_eq!(cloned.parts[1].size, 512);
+
+        // Verify default ObjectInfo clone also works
+        let default_obj = ObjectInfo::default();
+        let default_cloned = default_obj.clone();
+        assert!(default_obj.user_defined.is_empty());
+        assert!(default_cloned.user_defined.is_empty());
+        assert!(default_cloned.user_tags.is_empty());
+        assert!(default_cloned.parts.is_empty());
+    }
 }

crates/ecstore/src/store_list_objects.rs

@@ -23,8 +23,8 @@ use crate::error::{
 };
 use crate::set_disk::SetDisks;
 use crate::store_api::{
-    ListObjectVersionsInfo, ListObjectsInfo, ObjectInfo, ObjectInfoOrErr, ObjectOperations, ObjectOptions, WalkOptions,
-    WalkVersionsSortOrder,
+    ListObjectVersionsInfo, ListObjectsInfo, ObjectInfo, ObjectInfoOrErr, ObjectOperations, ObjectOptions, VersionMarker,
+    WalkOptions, WalkVersionsSortOrder,
 };
 use crate::store_utils::is_reserved_or_invalid_bucket;
 use crate::{store::ECStore, store_api::ListObjectsV2Info};
@@ -118,9 +118,12 @@ pub struct ListPathOptions {
     pub filter_prefix: Option<String>,
 
     // Marker to resume listing.
-    // The response will be the first entry >= this object name.
     pub marker: Option<String>,
 
+    // Include marker itself in the returned entries. Version listings need this
+    // when a version marker selects a later version from the marker object.
+    pub include_marker: bool,
+
     // Limit the number of results.
     pub limit: i32,
 
@@ -159,6 +162,67 @@ pub struct ListPathOptions {
 }
 
 const MARKER_TAG_VERSION: &str = "v1";
+const ENV_API_LIST_QUORUM: &str = "RUSTFS_API_LIST_QUORUM";
+const DEFAULT_API_LIST_QUORUM: &str = "strict";
+
+fn normalize_list_quorum(value: &str) -> &'static str {
+    let value = value.trim();
+    if value.eq_ignore_ascii_case("disk") {
+        "disk"
+    } else if value.eq_ignore_ascii_case("reduced") {
+        "reduced"
+    } else if value.eq_ignore_ascii_case("optimal") {
+        "optimal"
+    } else if value.eq_ignore_ascii_case("auto") {
+        "auto"
+    } else {
+        DEFAULT_API_LIST_QUORUM
+    }
+}
+
+fn list_quorum_from_env() -> String {
+    let value = rustfs_utils::get_env_str(ENV_API_LIST_QUORUM, DEFAULT_API_LIST_QUORUM);
+    normalize_list_quorum(&value).to_owned()
+}
+
+fn list_metadata_resolution_params(bucket: String, listing_quorum: usize, versioned: bool) -> MetadataResolutionParams {
+    let mut resolver = MetadataResolutionParams {
+        dir_quorum: listing_quorum,
+        obj_quorum: listing_quorum,
+        bucket,
+        ..Default::default()
+    };
+
+    if !versioned {
+        resolver.requested_versions = 1;
+    }
+
+    resolver
+}
+
+fn parse_version_marker(marker: String) -> Result<VersionMarker> {
+    if marker == "null" {
+        Ok(VersionMarker::Null)
+    } else {
+        Ok(VersionMarker::Version(Uuid::parse_str(&marker)?))
+    }
+}
+
+fn version_marker_for_entries(
+    entries: Option<&MetaCacheEntriesSorted>,
+    key_marker: Option<&str>,
+    version_marker: Option<VersionMarker>,
+) -> Option<VersionMarker> {
+    let marker = version_marker?;
+    let Some(key_marker) = key_marker else {
+        return Some(marker);
+    };
+
+    entries
+        .and_then(|entries| entries.entries().first().map(|entry| entry.name.as_str() == key_marker))
+        .unwrap_or_default()
+        .then_some(marker)
+}
 
 impl ListPathOptions {
     pub fn set_filter(&mut self) {
@@ -182,50 +246,64 @@ impl ListPathOptions {
     }
 
     pub fn parse_marker(&mut self) {
-        if let Some(marker) = &self.marker {
-            let s = marker.clone();
-            if !s.contains(format!("[rustfs_cache:{MARKER_TAG_VERSION}").as_str()) {
-                return;
-            }
+        let Some(marker) = self.marker.clone() else {
+            return;
+        };
+        let Some(start_idx) = marker.rfind("[rustfs_cache:") else {
+            return;
+        };
+        let Some(end_offset) = marker[start_idx..].rfind(']') else {
+            return;
+        };
 
-            if let (Some(start_idx), Some(end_idx)) = (s.find("["), s.find("]")) {
-                self.marker = Some(s[0..start_idx].to_owned());
-                let tags: Vec<_> = s[start_idx..end_idx].trim_matches(['[', ']']).split(",").collect();
+        let end_idx = start_idx + end_offset;
+        let tag_body = marker[start_idx + 1..end_idx].to_owned();
+        let mut supported_marker = false;
 
-                for &tag in tags.iter() {
-                    let kv: Vec<_> = tag.split(":").collect();
-                    if kv.len() != 2 {
-                        continue;
-                    }
-
-                    match kv[0] {
-                        "rustfs_cache" if kv[1] != MARKER_TAG_VERSION => {
-                            continue;
-                        }
-                        "id" => self.id = Some(kv[1].to_owned()),
-                        "return" => {
-                            self.id = Some(Uuid::new_v4().to_string());
-                            self.create = true;
-                        }
-                        "p" => match kv[1].parse::<usize>() {
-                            Ok(res) => self.pool_idx = Some(res),
-                            Err(_) => {
-                                self.id = Some(Uuid::new_v4().to_string());
-                                self.create = true;
-                                continue;
-                            }
-                        },
-                        "s" => match kv[1].parse::<usize>() {
-                            Ok(res) => self.set_idx = Some(res),
-                            Err(_) => {
-                                self.id = Some(Uuid::new_v4().to_string());
-                                self.create = true;
-                                continue;
-                            }
-                        },
-                        _ => (),
-                    }
+        for tag in tag_body.split(',') {
+            let Some((key, value)) = tag.split_once(':') else {
+                continue;
+            };
+            if key == "rustfs_cache" {
+                if value != MARKER_TAG_VERSION {
+                    return;
                 }
+                supported_marker = true;
+            }
+        }
+
+        if !supported_marker {
+            return;
+        }
+
+        self.marker = Some(marker[..start_idx].to_owned());
+        for tag in tag_body.split(',') {
+            let Some((key, value)) = tag.split_once(':') else {
+                continue;
+            };
+
+            match key {
+                "rustfs_cache" => {}
+                "id" => self.id = Some(value.to_owned()),
+                "return" => {
+                    self.id = Some(Uuid::new_v4().to_string());
+                    self.create = true;
+                }
+                "p" => match value.parse::<usize>() {
+                    Ok(res) => self.pool_idx = Some(res),
+                    Err(_) => {
+                        self.id = Some(Uuid::new_v4().to_string());
+                        self.create = true;
+                    }
+                },
+                "s" => match value.parse::<usize>() {
+                    Ok(res) => self.set_idx = Some(res),
+                    Err(_) => {
+                        self.id = Some(Uuid::new_v4().to_string());
+                        self.create = true;
+                    }
+                },
+                _ => (),
             }
         }
     }
@@ -301,7 +379,7 @@ impl ECStore {
             limit: effective_max_keys,
             marker,
             incl_deleted,
-            ask_disks: "strict".to_owned(), //TODO: from config
+            ask_disks: list_quorum_from_env(),
             ..Default::default()
         };
 
@@ -444,13 +522,9 @@ impl ECStore {
             return Err(StorageError::NotImplemented);
         }
 
+        let has_version_marker = version_marker.is_some();
         let version_marker = if let Some(marker) = version_marker {
-            // "null" is used for non-versioned objects in AWS S3 API
-            if marker == "null" {
-                None
-            } else {
-                Some(Uuid::parse_str(&marker)?)
-            }
+            Some(parse_version_marker(marker)?)
         } else {
             None
         };
@@ -464,8 +538,9 @@ impl ECStore {
             limit: effective_max_keys,
             marker,
             incl_deleted: true,
-            ask_disks: "strict".to_owned(),
+            ask_disks: list_quorum_from_env(),
             versioned: true,
+            include_marker: has_version_marker,
             ..Default::default()
         };
 
@@ -486,10 +561,14 @@ impl ECStore {
             return Err(to_object_err(err.into(), vec![bucket, prefix]));
         }
 
-        if let Some(result) = list_result.entries.as_mut() {
-            result.forward_past(opts.marker);
+        if let Some(result) = list_result.entries.as_mut()
+            && !has_version_marker
+        {
+            result.forward_past(opts.marker.clone());
         }
 
+        let version_marker = version_marker_for_entries(list_result.entries.as_ref(), opts.marker.as_deref(), version_marker);
+
         let mut get_objects = ObjectInfo::from_meta_cache_entries_sorted_versions(
             &list_result.entries.unwrap_or_default(),
             bucket,
@@ -762,11 +841,13 @@ impl ECStore {
             }
         }
 
+        // All sets returned not-found — the listing is simply empty.
+        // We intentionally do NOT distinguish VolumeNotFound here: during
+        // listing, a missing volume on a set is equivalent to "no entries"
+        // and must not surface as an error to the caller. The original
+        // VolumeNotFound check caused spurious errors when a pool had not
+        // yet created the bucket prefix on every set.
         if is_all_not_found(&errs) {
-            if is_all_volume_not_found(&errs) {
-                return Err(StorageError::VolumeNotFound);
-            }
-
             return Ok(Vec::new());
         }
 
@@ -1080,7 +1161,7 @@ async fn gather_results(
         }
 
         if let Some(marker) = &opts.marker
-            && &entry.name <= marker
+            && ((!opts.include_marker && &entry.name <= marker) || (opts.include_marker && &entry.name < marker))
         {
             continue;
         }
@@ -1137,12 +1218,18 @@ async fn gather_results(
 }
 
 async fn select_from(
+    rx: &CancellationToken,
     in_channels: &mut [Receiver<MetaCacheEntry>],
     idx: usize,
     top: &mut [Option<MetaCacheEntry>],
     n_done: &mut usize,
-) -> Result<()> {
-    match in_channels[idx].recv().await {
+) -> Result<bool> {
+    let entry = tokio::select! {
+        entry = in_channels[idx].recv() => entry,
+        _ = rx.cancelled() => return Ok(false),
+    };
+
+    match entry {
         Some(entry) => {
             top[idx] = Some(entry);
         }
@@ -1151,10 +1238,19 @@ async fn select_from(
             *n_done += 1;
         }
     }
-    Ok(())
+    Ok(true)
+}
+
+async fn send_or_cancel(rx: &CancellationToken, out_channel: &Sender<MetaCacheEntry>, entry: MetaCacheEntry) -> Result<bool> {
+    tokio::select! {
+        result = out_channel.send(entry) => {
+            result.map_err(Error::other)?;
+            Ok(true)
+        }
+        _ = rx.cancelled() => Ok(false),
+    }
 }
 
-// TODO: exit when cancel
 async fn merge_entry_channels(
     rx: CancellationToken,
     in_channels: Vec<Receiver<MetaCacheEntry>>,
@@ -1172,7 +1268,9 @@ async fn merge_entry_channels(
                 has_entry = in_channels[0].recv()=>{
                     if let Some(entry) = has_entry{
                         // warn!("merge_entry_channels entry {}", &entry.name);
-                        out_channel.send(entry).await.map_err(Error::other)?;
+                        if !send_or_cancel(&rx, &out_channel, entry).await? {
+                            return Ok(());
+                        }
                     } else {
                         return Ok(())
                     }
@@ -1191,7 +1289,9 @@ async fn merge_entry_channels(
     let in_channels_len = in_channels.len();
 
     for idx in 0..in_channels_len {
-        select_from(&mut in_channels, idx, &mut top, &mut n_done).await?;
+        if !select_from(&rx, &mut in_channels, idx, &mut top, &mut n_done).await? {
+            return Ok(());
+        }
     }
 
     let mut last = String::new();
@@ -1205,16 +1305,13 @@ async fn merge_entry_channels(
         let mut best_idx = 0;
         to_merge.clear();
 
-        // FIXME: top move when select_from call
-        // let vtop = top.clone();
-
-        // let vtop = top.as_slice();
+        // Note: `select_from` mutates `top[idx]` during the inner loop, but this is safe
+        // because each borrow from `top[other_idx]` is only used before any later
+        // `select_from` call that can mutate that slot.
 
         for other_idx in 1..top.len() {
             if let Some(other_entry) = &top[other_idx] {
                 if let Some(best_entry) = &best {
-                    // println!("get other_entry {:?}", other_entry.name);
-
                     if path::clean(&best_entry.name) == path::clean(&other_entry.name) {
                         let dir_matches = best_entry.is_dir() && other_entry.is_dir();
                         let suffix_matches =
@@ -1229,7 +1326,9 @@ async fn merge_entry_channels(
                             // dir and object has the save name
                             if other_entry.is_dir() {
                                 // TODO: read next entry to top
-                                select_from(&mut in_channels, other_idx, &mut top, &mut n_done).await?;
+                                if !select_from(&rx, &mut in_channels, other_idx, &mut top, &mut n_done).await? {
+                                    return Ok(());
+                                }
                                 continue;
                             }
 
@@ -1270,7 +1369,9 @@ async fn merge_entry_channels(
                         let xl2 = match entry.clone().xl_meta() {
                             Ok(res) => res,
                             Err(_) => {
-                                select_from(&mut in_channels, idx, &mut top, &mut n_done).await?;
+                                if !select_from(&rx, &mut in_channels, idx, &mut top, &mut n_done).await? {
+                                    return Ok(());
+                                }
 
                                 continue;
                             }
@@ -1279,13 +1380,17 @@ async fn merge_entry_channels(
                         versions.push(xl2.versions.clone());
 
                         if has_xl.is_none() {
-                            select_from(&mut in_channels, best_idx, &mut top, &mut n_done).await?;
+                            if !select_from(&rx, &mut in_channels, best_idx, &mut top, &mut n_done).await? {
+                                return Ok(());
+                            }
 
                             best_idx = idx;
                             best = Some(entry.clone());
                             has_xl = Some(xl2);
                         } else {
-                            select_from(&mut in_channels, best_idx, &mut top, &mut n_done).await?;
+                            if !select_from(&rx, &mut in_channels, best_idx, &mut top, &mut n_done).await? {
+                                return Ok(());
+                            }
                         }
                     }
                 }
@@ -1310,11 +1415,15 @@ async fn merge_entry_channels(
         if let Some(best_entry) = &best
             && best_entry.name > last
         {
-            out_channel.send(best_entry.clone()).await.map_err(Error::other)?;
+            if !send_or_cancel(&rx, &out_channel, best_entry.clone()).await? {
+                return Ok(());
+            }
             last = best_entry.name.clone();
         }
 
-        select_from(&mut in_channels, best_idx, &mut top, &mut n_done).await?;
+        if !select_from(&rx, &mut in_channels, best_idx, &mut top, &mut n_done).await? {
+            return Ok(());
+        }
     }
 }
 
@@ -1349,16 +1458,7 @@ impl SetDisks {
             fallback_disks = disks.split_off(ask_disks as usize);
         }
 
-        let mut resolver = MetadataResolutionParams {
-            dir_quorum: listing_quorum,
-            obj_quorum: listing_quorum,
-            bucket: opts.bucket.clone(),
-            ..Default::default()
-        };
-
-        if opts.versioned {
-            resolver.requested_versions = 1;
-        }
+        let resolver = list_metadata_resolution_params(opts.bucket.clone(), listing_quorum, opts.versioned);
 
         let limit = {
             if opts.limit > 0 && opts.stop_disk_at_limit {
@@ -1483,9 +1583,13 @@ fn calc_common_counter(infos: &[DiskInfo], read_quorum: usize) -> u64 {
 
 #[cfg(test)]
 mod test {
-    use super::{ListPathOptions, MAX_OBJECT_LIST, gather_results, max_keys_plus_one, walk_result_from_set_errors};
+    use super::{
+        ENV_API_LIST_QUORUM, ListPathOptions, MAX_OBJECT_LIST, VersionMarker, gather_results, list_metadata_resolution_params,
+        list_quorum_from_env, max_keys_plus_one, merge_entry_channels, normalize_list_quorum, parse_version_marker,
+        version_marker_for_entries, walk_result_from_set_errors,
+    };
     use crate::error::StorageError;
-    use rustfs_filemeta::MetaCacheEntry;
+    use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntriesSorted, MetaCacheEntry};
     use std::time::Duration;
     use tokio::sync::mpsc;
     use tokio::time::timeout;
@@ -1499,6 +1603,13 @@ mod test {
         }
     }
 
+    fn sorted_entries(names: &[&str]) -> MetaCacheEntriesSorted {
+        MetaCacheEntriesSorted {
+            o: MetaCacheEntries(names.iter().map(|name| Some(test_meta_entry(name))).collect()),
+            ..Default::default()
+        }
+    }
+
     #[tokio::test]
     async fn gather_results_returns_after_limit_without_waiting_for_input_close() {
         let (entry_tx, entry_rx) = mpsc::channel(4);
@@ -1531,6 +1642,109 @@ mod test {
             .expect("gather_results should succeed");
     }
 
+    #[tokio::test]
+    async fn gather_results_keeps_marker_entry_for_version_marker_listing() {
+        let (entry_tx, entry_rx) = mpsc::channel(4);
+        let (result_tx, mut result_rx) = mpsc::channel(1);
+
+        entry_tx.send(test_meta_entry("obj-a")).await.unwrap();
+        entry_tx.send(test_meta_entry("obj-b")).await.unwrap();
+
+        let handle = tokio::spawn(gather_results(
+            CancellationToken::new(),
+            ListPathOptions {
+                bucket: "bucket".to_owned(),
+                marker: Some("obj-a".to_owned()),
+                include_marker: true,
+                limit: 2,
+                incl_deleted: true,
+                versioned: true,
+                ..Default::default()
+            },
+            entry_rx,
+            result_tx,
+        ));
+
+        let result = timeout(Duration::from_secs(1), result_rx.recv())
+            .await
+            .expect("limited result should be sent promptly")
+            .expect("limited result should be present");
+        let entries = result.entries.unwrap();
+        let names = entries
+            .entries()
+            .into_iter()
+            .map(|entry| entry.name.as_str())
+            .collect::<Vec<_>>();
+
+        assert_eq!(names, ["obj-a", "obj-b"]);
+
+        timeout(Duration::from_secs(1), handle)
+            .await
+            .expect("gather_results should finish after sending a limited result")
+            .expect("gather_results task should not panic")
+            .expect("gather_results should succeed");
+    }
+
+    #[test]
+    fn version_marker_is_applied_only_when_key_marker_entry_is_present() {
+        let version_marker = Some(VersionMarker::Null);
+
+        let listed_after_deleted_marker = sorted_entries(&["obj-b", "obj-c"]);
+        assert_eq!(
+            version_marker_for_entries(Some(&listed_after_deleted_marker), Some("obj-a"), version_marker),
+            None
+        );
+
+        let listed_with_marker = sorted_entries(&["obj-a", "obj-b"]);
+        assert_eq!(
+            version_marker_for_entries(Some(&listed_with_marker), Some("obj-a"), version_marker),
+            version_marker
+        );
+    }
+
+    #[tokio::test]
+    async fn gather_results_skips_marker_entry_by_default() {
+        let (entry_tx, entry_rx) = mpsc::channel(4);
+        let (result_tx, mut result_rx) = mpsc::channel(1);
+
+        entry_tx.send(test_meta_entry("obj-a")).await.unwrap();
+        entry_tx.send(test_meta_entry("obj-b")).await.unwrap();
+        drop(entry_tx);
+
+        let handle = tokio::spawn(gather_results(
+            CancellationToken::new(),
+            ListPathOptions {
+                bucket: "bucket".to_owned(),
+                marker: Some("obj-a".to_owned()),
+                limit: 2,
+                incl_deleted: true,
+                ..Default::default()
+            },
+            entry_rx,
+            result_tx,
+        ));
+
+        let result = timeout(Duration::from_secs(1), result_rx.recv())
+            .await
+            .expect("eof result should be sent promptly")
+            .expect("eof result should be present");
+        let entries = result.entries.unwrap();
+        let names = entries
+            .entries()
+            .into_iter()
+            .map(|entry| entry.name.as_str())
+            .collect::<Vec<_>>();
+
+        assert_eq!(names, ["obj-b"]);
+        assert!(result.err.is_some());
+
+        timeout(Duration::from_secs(1), handle)
+            .await
+            .expect("gather_results should finish after input closes")
+            .expect("gather_results task should not panic")
+            .expect("gather_results should succeed");
+    }
+
     #[test]
     fn test_max_keys_plus_one_caps_before_lookahead() {
         assert_eq!(max_keys_plus_one(999, true), 1000);
@@ -1540,28 +1754,74 @@ mod test {
         assert_eq!(max_keys_plus_one(-1, true), MAX_OBJECT_LIST + 1);
     }
 
-    /// Test that "null" version marker is handled correctly
-    /// AWS S3 API uses "null" string to represent non-versioned objects
+    #[test]
+    fn normalize_list_quorum_accepts_supported_values() {
+        assert_eq!(normalize_list_quorum("strict"), "strict");
+        assert_eq!(normalize_list_quorum("disk"), "disk");
+        assert_eq!(normalize_list_quorum("reduced"), "reduced");
+        assert_eq!(normalize_list_quorum("optimal"), "optimal");
+        assert_eq!(normalize_list_quorum("auto"), "auto");
+        assert_eq!(normalize_list_quorum(" OPTIMAL "), "optimal");
+    }
+
+    #[test]
+    fn normalize_list_quorum_falls_back_to_strict() {
+        assert_eq!(normalize_list_quorum(""), "strict");
+        assert_eq!(normalize_list_quorum("unknown"), "strict");
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn list_quorum_from_env_defaults_to_strict() {
+        temp_env::with_var_unset(ENV_API_LIST_QUORUM, || {
+            assert_eq!(list_quorum_from_env(), "strict");
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn list_quorum_from_env_honors_supported_value() {
+        temp_env::with_var(ENV_API_LIST_QUORUM, Some("auto"), || {
+            assert_eq!(list_quorum_from_env(), "auto");
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn list_quorum_from_env_rejects_unknown_value() {
+        temp_env::with_var(ENV_API_LIST_QUORUM, Some("unsafe"), || {
+            assert_eq!(list_quorum_from_env(), "strict");
+        });
+    }
+
+    #[test]
+    fn list_metadata_resolution_params_limits_plain_listing_to_latest_version() {
+        let resolver = list_metadata_resolution_params("bucket".to_string(), 2, false);
+
+        assert_eq!(resolver.dir_quorum, 2);
+        assert_eq!(resolver.obj_quorum, 2);
+        assert_eq!(resolver.bucket, "bucket");
+        assert_eq!(resolver.requested_versions, 1);
+    }
+
+    #[test]
+    fn list_metadata_resolution_params_keeps_all_versions_for_version_listing() {
+        let resolver = list_metadata_resolution_params("bucket".to_string(), 3, true);
+
+        assert_eq!(resolver.dir_quorum, 3);
+        assert_eq!(resolver.obj_quorum, 3);
+        assert_eq!(resolver.bucket, "bucket");
+        assert_eq!(resolver.requested_versions, 0);
+    }
+
     #[test]
     fn test_null_version_marker_handling() {
-        // "null" should be treated as None (non-versioned)
-        let version_marker = "null";
-        let parsed: Option<Uuid> = if version_marker == "null" {
-            None
-        } else {
-            Uuid::parse_str(version_marker).ok()
-        };
-        assert!(parsed.is_none(), "\"null\" should be parsed as None");
+        let parsed = parse_version_marker("null".to_string()).expect("null marker should parse");
+        assert_eq!(parsed, VersionMarker::Null);
 
-        // Valid UUID should be parsed correctly
         let valid_uuid = "550e8400-e29b-41d4-a716-446655440000";
-        let parsed: Option<Uuid> = if valid_uuid == "null" {
-            None
-        } else {
-            Uuid::parse_str(valid_uuid).ok()
-        };
-        assert!(parsed.is_some(), "Valid UUID should be parsed correctly");
-        assert_eq!(parsed.unwrap().to_string(), "550e8400-e29b-41d4-a716-446655440000");
+        let parsed = parse_version_marker(valid_uuid.to_string()).expect("uuid marker should parse");
+        assert_eq!(parsed, VersionMarker::Version(Uuid::parse_str(valid_uuid).unwrap()));
     }
 
     /// Test that next_version_idmarker returns "null" for non-versioned objects
@@ -1584,15 +1844,11 @@ mod test {
         // Scenario 1: Non-versioned object
         // Server returns "null" as NextVersionIdMarker
         // Client sends "null" as VersionIdMarker
-        // Server parses "null" as None
+        // Server parses "null" as an explicit null-version marker
         let server_response = "null";
         let client_request = server_response;
-        let parsed: Option<Uuid> = if client_request == "null" {
-            None
-        } else {
-            Uuid::parse_str(client_request).ok()
-        };
-        assert!(parsed.is_none());
+        let parsed = parse_version_marker(client_request.to_string()).expect("null marker should parse");
+        assert_eq!(parsed, VersionMarker::Null);
 
         // Scenario 2: Versioned object
         // Server returns UUID as NextVersionIdMarker
@@ -1601,13 +1857,8 @@ mod test {
         let uuid_str = "550e8400-e29b-41d4-a716-446655440000";
         let server_response = uuid_str;
         let client_request = server_response;
-        let parsed: Option<Uuid> = if client_request == "null" {
-            None
-        } else {
-            Uuid::parse_str(client_request).ok()
-        };
-        assert!(parsed.is_some());
-        assert_eq!(parsed.unwrap().to_string(), uuid_str);
+        let parsed = parse_version_marker(client_request.to_string()).expect("uuid marker should parse");
+        assert_eq!(parsed, VersionMarker::Version(Uuid::parse_str(uuid_str).unwrap()));
     }
 
     #[test]
@@ -1639,6 +1890,41 @@ mod test {
         assert!(!parsed.create);
     }
 
+    #[test]
+    fn list_path_marker_parser_uses_trailing_cache_tag() {
+        let mut parsed = ListPathOptions {
+            marker: Some(format!(
+                "photos/[archive]/image.jpg[rustfs_cache:{},id:list-cache-id,p:3,s:7]",
+                super::MARKER_TAG_VERSION
+            )),
+            ..Default::default()
+        };
+
+        parsed.parse_marker();
+
+        assert_eq!(parsed.marker.as_deref(), Some("photos/[archive]/image.jpg"));
+        assert_eq!(parsed.id.as_deref(), Some("list-cache-id"));
+        assert_eq!(parsed.pool_idx, Some(3));
+        assert_eq!(parsed.set_idx, Some(7));
+    }
+
+    #[test]
+    fn list_path_marker_parser_ignores_unsupported_cache_tag_version() {
+        let marker = "photos/image.jpg[rustfs_cache:v0,id:list-cache-id,p:3,s:7]".to_string();
+        let mut parsed = ListPathOptions {
+            marker: Some(marker.clone()),
+            ..Default::default()
+        };
+
+        parsed.parse_marker();
+
+        assert_eq!(parsed.marker.as_deref(), Some(marker.as_str()));
+        assert!(parsed.id.is_none());
+        assert!(parsed.pool_idx.is_none());
+        assert!(parsed.set_idx.is_none());
+        assert!(!parsed.create);
+    }
+
     #[test]
     fn walk_result_from_set_errors_returns_non_eof_error() {
         let err = walk_result_from_set_errors(&[Some(StorageError::Unexpected), Some(StorageError::FileAccessDenied)])
@@ -1966,4 +2252,172 @@ mod test {
     //         println!("get entry {:?}", entry)
     //     }
     // }
+
+    #[tokio::test]
+    async fn merge_entry_channels_produces_sorted_unique_output_from_two_channels() {
+        let (tx_a, rx_a) = mpsc::channel(4);
+        let (tx_b, rx_b) = mpsc::channel(4);
+        let (out_tx, mut out_rx) = mpsc::channel(8);
+
+        // Send sorted entries from two channels with some overlap
+        tx_a.send(test_meta_entry("obj-a")).await.unwrap();
+        tx_a.send(test_meta_entry("obj-c")).await.unwrap();
+        tx_a.send(test_meta_entry("obj-e")).await.unwrap();
+        drop(tx_a);
+
+        tx_b.send(test_meta_entry("obj-b")).await.unwrap();
+        tx_b.send(test_meta_entry("obj-d")).await.unwrap();
+        drop(tx_b);
+
+        let rx = CancellationToken::new();
+        let handle = tokio::spawn(merge_entry_channels(rx, vec![rx_a, rx_b], out_tx, 1));
+
+        let mut results = Vec::new();
+        while let Some(entry) = out_rx.recv().await {
+            results.push(entry.name.clone());
+        }
+
+        handle.await.unwrap().unwrap();
+
+        // Results should be sorted and deduplicated
+        assert_eq!(results, vec!["obj-a", "obj-b", "obj-c", "obj-d", "obj-e"]);
+    }
+
+    #[tokio::test]
+    async fn merge_entry_channels_deduplicates_entries_across_channels() {
+        let (tx_a, rx_a) = mpsc::channel(4);
+        let (tx_b, rx_b) = mpsc::channel(4);
+        let (out_tx, mut out_rx) = mpsc::channel(8);
+
+        // Both channels have the same entry
+        tx_a.send(test_meta_entry("obj-a")).await.unwrap();
+        tx_a.send(test_meta_entry("obj-c")).await.unwrap();
+        drop(tx_a);
+
+        tx_b.send(test_meta_entry("obj-a")).await.unwrap();
+        tx_b.send(test_meta_entry("obj-b")).await.unwrap();
+        drop(tx_b);
+
+        let rx = CancellationToken::new();
+        let handle = tokio::spawn(merge_entry_channels(rx, vec![rx_a, rx_b], out_tx, 1));
+
+        let mut results = Vec::new();
+        while let Some(entry) = out_rx.recv().await {
+            results.push(entry.name.clone());
+        }
+
+        handle.await.unwrap().unwrap();
+
+        // "obj-a" should appear only once despite being in both channels
+        assert_eq!(results, vec!["obj-a", "obj-b", "obj-c"]);
+    }
+
+    #[tokio::test]
+    async fn merge_entry_channels_handles_single_channel() {
+        let (tx, rx) = mpsc::channel(4);
+        let (out_tx, mut out_rx) = mpsc::channel(8);
+
+        tx.send(test_meta_entry("obj-a")).await.unwrap();
+        tx.send(test_meta_entry("obj-b")).await.unwrap();
+        drop(tx);
+
+        let cancel = CancellationToken::new();
+        let handle = tokio::spawn(merge_entry_channels(cancel, vec![rx], out_tx, 1));
+
+        let mut results = Vec::new();
+        while let Some(entry) = out_rx.recv().await {
+            results.push(entry.name.clone());
+        }
+
+        handle.await.unwrap().unwrap();
+
+        assert_eq!(results, vec!["obj-a", "obj-b"]);
+    }
+
+    #[tokio::test]
+    async fn merge_entry_channels_respects_cancellation() {
+        let (tx, rx) = mpsc::channel::<MetaCacheEntry>(4);
+        let (out_tx, mut out_rx) = mpsc::channel(8);
+
+        let cancel = CancellationToken::new();
+        let cancel_clone = cancel.clone();
+
+        // Use a single channel so the cancellation check in tokio::select! is exercised.
+        let handle = tokio::spawn(merge_entry_channels(cancel_clone, vec![rx], out_tx, 1));
+
+        // Send an entry to prove the function is running and processing data.
+        tx.send(test_meta_entry("a")).await.unwrap();
+        let received = timeout(Duration::from_millis(500), out_rx.recv())
+            .await
+            .expect("should receive entry before cancellation")
+            .map(|e| e.name);
+        assert_eq!(received, Some("a".to_string()));
+
+        // Cancel while the sender is still alive (channel is not closed).
+        cancel.cancel();
+
+        let result = timeout(Duration::from_secs(2), handle)
+            .await
+            .expect("merge should not hang after cancellation")
+            .expect("task should not panic");
+        assert!(result.is_ok(), "merge should return Ok on cancellation");
+
+        // Keep tx alive until after the assertion so the channel doesn't close prematurely.
+        drop(tx);
+    }
+
+    #[tokio::test]
+    async fn merge_entry_channels_respects_cancellation_with_multiple_live_channels() {
+        let (tx_a, rx_a) = mpsc::channel::<MetaCacheEntry>(4);
+        let (tx_b, rx_b) = mpsc::channel::<MetaCacheEntry>(4);
+        let (out_tx, _out_rx) = mpsc::channel(8);
+
+        let cancel = CancellationToken::new();
+        let cancel_clone = cancel.clone();
+
+        let handle = tokio::spawn(merge_entry_channels(cancel_clone, vec![rx_a, rx_b], out_tx, 1));
+        cancel.cancel();
+
+        let result = timeout(Duration::from_secs(2), handle)
+            .await
+            .expect("multi-channel merge should not hang after cancellation")
+            .expect("task should not panic");
+        assert!(result.is_ok(), "multi-channel merge should return Ok on cancellation");
+
+        drop(tx_a);
+        drop(tx_b);
+    }
+
+    #[tokio::test]
+    async fn merge_entry_channels_respects_cancellation_when_output_is_full() {
+        let (tx_a, rx_a) = mpsc::channel::<MetaCacheEntry>(4);
+        let (tx_b, rx_b) = mpsc::channel::<MetaCacheEntry>(4);
+        let (out_tx, _out_rx) = mpsc::channel(1);
+
+        out_tx
+            .send(test_meta_entry("already-buffered"))
+            .await
+            .expect("output channel should accept initial entry");
+        tx_a.send(test_meta_entry("a"))
+            .await
+            .expect("input channel a should accept entry");
+        tx_b.send(test_meta_entry("b"))
+            .await
+            .expect("input channel b should accept entry");
+
+        let cancel = CancellationToken::new();
+        let cancel_clone = cancel.clone();
+
+        let handle = tokio::spawn(merge_entry_channels(cancel_clone, vec![rx_a, rx_b], out_tx, 1));
+        cancel.cancel();
+
+        let result = timeout(Duration::from_secs(2), handle)
+            .await
+            .expect("merge should not hang when output is full and cancellation fires")
+            .expect("task should not panic");
+        assert!(result.is_ok(), "merge should return Ok when cancelled during output send");
+
+        drop(tx_a);
+        drop(tx_b);
+    }
 }

crates/heal/src/heal/channel.rs

@@ -281,6 +281,12 @@ impl HealChannelProcessor {
                 data: Some("stopped".as_bytes().to_vec()),
                 error: None,
             },
+            Err(Error::TaskNotFound { .. }) if client_token.is_empty() => HealChannelResponse {
+                request_id,
+                success: true,
+                data: Some("stopped".as_bytes().to_vec()),
+                error: None,
+            },
             Err(err) => HealChannelResponse {
                 request_id,
                 success: false,
@@ -965,6 +971,27 @@ mod tests {
         ));
     }
 
+    #[tokio::test]
+    async fn test_process_cancel_request_treats_unknown_path_as_stopped() {
+        let heal_manager = create_test_heal_manager();
+        let processor = HealChannelProcessor::new(heal_manager);
+        let (tx, rx) = oneshot::channel();
+
+        processor
+            .process_cancel_request(".".to_string(), String::new(), tx)
+            .await
+            .expect("cancel should process");
+
+        let response = rx
+            .await
+            .expect("oneshot should resolve")
+            .expect("cancel response should be returned");
+        assert!(response.success);
+        assert_eq!(response.request_id, ".");
+        assert_eq!(response.data.as_deref(), Some("stopped".as_bytes()));
+        assert!(response.error.is_none());
+    }
+
     #[tokio::test]
     async fn test_process_cancel_request_reports_unknown_task() {
         let heal_manager = create_test_heal_manager();

crates/heal/src/heal/manager.rs

@@ -149,6 +149,50 @@ impl PriorityHealQueue {
         QueuePushOutcome::Accepted
     }
 
+    fn can_displace_lower_priority(&self, priority: HealPriority) -> bool {
+        self.heap.iter().any(|item| item.priority < priority)
+    }
+
+    fn push_displacing_lower_priority(&mut self, request: HealRequest) -> Option<HealRequest> {
+        let mut retained = BinaryHeap::new();
+        let mut displaced: Option<PriorityQueueItem> = None;
+
+        while let Some(item) = self.heap.pop() {
+            if item.priority < request.priority {
+                let should_displace = displaced
+                    .as_ref()
+                    .map(|current| {
+                        item.priority < current.priority
+                            || (item.priority == current.priority && item.sequence > current.sequence)
+                    })
+                    .unwrap_or(true);
+                if should_displace {
+                    if let Some(current) = displaced.replace(item) {
+                        retained.push(current);
+                    }
+                } else {
+                    retained.push(item);
+                }
+            } else {
+                retained.push(item);
+            }
+        }
+
+        self.heap = retained;
+
+        let displaced = displaced.map(|item| {
+            let key = Self::make_dedup_key(&item.request);
+            Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &key);
+            item.request
+        });
+
+        if displaced.is_some() {
+            debug_assert_eq!(self.push(request), QueuePushOutcome::Accepted);
+        }
+
+        displaced
+    }
+
     /// Get statistics about queue contents by priority
     fn get_priority_stats(&self) -> HashMap<HealPriority, usize> {
         let mut stats = HashMap::new();
@@ -578,6 +622,37 @@ impl HealManager {
         }
 
         if queue_len >= queue_capacity && !request.force_start {
+            if queue.can_displace_lower_priority(request.priority) {
+                let request_id = request.id.clone();
+                let priority = request.priority;
+                if let Some(displaced) = queue.push_displacing_lower_priority(request) {
+                    publish_heal_queue_length(&queue);
+                    warn!(
+                        request_id = %request_id,
+                        priority = ?priority,
+                        displaced_request_id = %displaced.id,
+                        displaced_priority = ?displaced.priority,
+                        queue_len,
+                        queue_capacity,
+                        "Admitted heal request by displacing lower-priority queued work"
+                    );
+                    drop(queue);
+                    if config.event_driven_scheduler_enable {
+                        self.notify.notify_one();
+                    }
+                    return Ok(HealAdmissionResult::Accepted);
+                }
+
+                warn!(
+                    request_id = %request_id,
+                    priority = ?priority,
+                    queue_len,
+                    queue_capacity,
+                    "Heal queue was full and lower-priority work was unavailable for displacement"
+                );
+                return Ok(HealAdmissionResult::Full);
+            }
+
             let admission = Self::classify_full_admission(&request, &config);
             match admission {
                 HealAdmissionResult::Dropped(reason) => {
@@ -2135,6 +2210,59 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn test_high_priority_request_displaces_lower_priority_when_queue_full() {
+        let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
+        let manager = HealManager::new(
+            storage,
+            Some(HealConfig {
+                queue_size: 1,
+                ..HealConfig::default()
+            }),
+        );
+
+        let low = HealRequest::new(
+            HealType::Bucket {
+                bucket: "background-bucket".to_string(),
+            },
+            HealOptions::default(),
+            HealPriority::Low,
+        );
+        let low_id = low.id.clone();
+        let high = HealRequest::new(
+            HealType::Bucket {
+                bucket: "manual-bucket".to_string(),
+            },
+            HealOptions::default(),
+            HealPriority::High,
+        );
+        let high_id = high.id.clone();
+
+        assert_eq!(
+            manager
+                .submit_heal_request(low)
+                .await
+                .expect("low priority request should be accepted first"),
+            HealAdmissionResult::Accepted
+        );
+        assert_eq!(
+            manager
+                .submit_heal_request(high)
+                .await
+                .expect("high priority request should be admitted by displacing lower priority work"),
+            HealAdmissionResult::Accepted
+        );
+        assert_eq!(manager.get_queue_length().await, 1);
+        assert!(matches!(manager.get_task_status(&low_id).await, Err(Error::TaskNotFound { .. })));
+        assert_eq!(
+            manager
+                .get_task_status(&high_id)
+                .await
+                .expect("high priority request should remain queued"),
+            HealTaskStatus::Pending
+        );
+    }
+
     #[tokio::test]
     async fn test_force_start_bypasses_duplicate_and_full_admission() {
         let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);

crates/iam/src/cache.rs

@@ -16,10 +16,10 @@ use std::{
     collections::{HashMap, HashSet},
     ops::{Deref, DerefMut},
     ptr,
-    sync::Arc,
+    sync::{Arc, Mutex},
 };
 
-use arc_swap::{ArcSwap, AsRaw, Guard};
+use arc_swap::{ArcSwap, Guard};
 use rustfs_policy::{
     auth::UserIdentity,
     policy::{Args, PolicyDoc},
@@ -29,80 +29,302 @@ use tracing::warn;
 
 use crate::store::{GroupInfo, MappedPolicy};
 
+/// Immutable IAM cache snapshot published atomically by [`Cache`].
+///
+/// Readers should load one `CacheState` and read all related maps from it. Writers
+/// must go through `Cache`/`LockedCache` so multi-map updates publish as one state.
+#[derive(Clone)]
+pub(crate) struct CacheState {
+    pub(crate) policy_docs: Arc<CacheEntity<PolicyDoc>>,
+    pub(crate) users: Arc<CacheEntity<UserIdentity>>,
+    pub(crate) user_policies: Arc<CacheEntity<MappedPolicy>>,
+    pub(crate) sts_accounts: Arc<CacheEntity<UserIdentity>>,
+    pub(crate) sts_policies: Arc<CacheEntity<MappedPolicy>>,
+    pub(crate) groups: Arc<CacheEntity<GroupInfo>>,
+    pub(crate) user_group_memberships: Arc<CacheEntity<HashSet<String>>>,
+    pub(crate) group_policies: Arc<CacheEntity<MappedPolicy>>,
+}
+
+impl Default for CacheState {
+    fn default() -> Self {
+        Self {
+            policy_docs: Arc::new(CacheEntity::default()),
+            users: Arc::new(CacheEntity::default()),
+            user_policies: Arc::new(CacheEntity::default()),
+            sts_accounts: Arc::new(CacheEntity::default()),
+            sts_policies: Arc::new(CacheEntity::default()),
+            groups: Arc::new(CacheEntity::default()),
+            user_group_memberships: Arc::new(CacheEntity::default()),
+            group_policies: Arc::new(CacheEntity::default()),
+        }
+    }
+}
+
 pub struct Cache {
-    pub policy_docs: ArcSwap<CacheEntity<PolicyDoc>>,
-    pub users: ArcSwap<CacheEntity<UserIdentity>>,
-    pub user_policies: ArcSwap<CacheEntity<MappedPolicy>>,
-    pub sts_accounts: ArcSwap<CacheEntity<UserIdentity>>,
-    pub sts_policies: ArcSwap<CacheEntity<MappedPolicy>>,
-    pub groups: ArcSwap<CacheEntity<GroupInfo>>,
-    pub user_group_memberships: ArcSwap<CacheEntity<HashSet<String>>>,
-    pub group_policies: ArcSwap<CacheEntity<MappedPolicy>>,
+    state: ArcSwap<CacheState>,
+    write_lock: Mutex<()>,
 }
 
 impl Default for Cache {
     fn default() -> Self {
         Self {
-            policy_docs: ArcSwap::new(Arc::new(CacheEntity::default())),
-            users: ArcSwap::new(Arc::new(CacheEntity::default())),
-            user_policies: ArcSwap::new(Arc::new(CacheEntity::default())),
-            sts_accounts: ArcSwap::new(Arc::new(CacheEntity::default())),
-            sts_policies: ArcSwap::new(Arc::new(CacheEntity::default())),
-            groups: ArcSwap::new(Arc::new(CacheEntity::default())),
-            user_group_memberships: ArcSwap::new(Arc::new(CacheEntity::default())),
-            group_policies: ArcSwap::new(Arc::new(CacheEntity::default())),
+            state: ArcSwap::new(Arc::new(CacheState::default())),
+            write_lock: Mutex::new(()),
         }
     }
 }
 
+pub(crate) type CacheSnapshot = Guard<Arc<CacheState>>;
+
 impl Cache {
-    pub fn ptr_eq<Base, A, B>(a: A, b: B) -> bool
-    where
-        A: AsRaw<Base>,
-        B: AsRaw<Base>,
-    {
-        let a = a.as_raw();
-        let b = b.as_raw();
-        ptr::eq(a, b)
+    pub(crate) fn snapshot(&self) -> CacheSnapshot {
+        self.state.load()
     }
 
-    fn exec<T: Clone>(target: &ArcSwap<CacheEntity<T>>, t: OffsetDateTime, mut op: impl FnMut(&mut CacheEntity<T>)) {
-        let mut cur = target.load();
-        loop {
-            // If the current update time is later than the execution time,
-            // the background task is loaded and the current operation does not need to be performed.
-            if cur.load_time >= t {
-                return;
-            }
-
-            let mut new = CacheEntity::clone(&cur);
-            op(&mut new);
-
-            // Replace content with CAS atoms
-            let prev = target.compare_and_swap(&*cur, Arc::new(new));
-            let swapped = Self::ptr_eq(&*cur, &*prev);
-            if swapped {
-                return;
-            } else {
-                cur = prev;
-            }
+    pub(crate) fn with_write_lock<R>(&self, f: impl FnOnce(&mut LockedCache) -> R) -> R {
+        let _guard = self.write_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
+        let current = self.state.load_full();
+        let mut locked = LockedCache {
+            state: CacheState::clone(&current),
+            current_ptr: Arc::as_ptr(&current),
+            dirty: false,
+        };
+        let ret = f(&mut locked);
+        if locked.dirty {
+            self.state.store(Arc::new(locked.state));
         }
+        ret
     }
 
-    pub fn add_or_update<T: Clone>(target: &ArcSwap<CacheEntity<T>>, key: &str, value: &T, t: OffsetDateTime) {
-        Self::exec(target, t, |map: &mut CacheEntity<T>| {
+    pub fn add_or_update_policy_doc(&self, key: &str, value: &PolicyDoc, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_policy_doc(key, value, t));
+    }
+
+    pub fn add_or_update_user(&self, key: &str, value: &UserIdentity, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_user(key, value, t));
+    }
+
+    pub fn add_or_update_user_policy(&self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_user_policy(key, value, t));
+    }
+
+    pub fn add_or_update_sts_account(&self, key: &str, value: &UserIdentity, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_sts_account(key, value, t));
+    }
+
+    pub fn add_or_update_sts_policy(&self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_sts_policy(key, value, t));
+    }
+
+    pub fn add_or_update_group(&self, key: &str, value: &GroupInfo, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_group(key, value, t));
+    }
+
+    pub fn add_or_update_user_group_membership(&self, key: &str, value: &HashSet<String>, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_user_group_membership(key, value, t));
+    }
+
+    pub fn add_or_update_group_policy(&self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.add_or_update_group_policy(key, value, t));
+    }
+
+    pub fn delete_policy_doc(&self, key: &str, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.delete_policy_doc(key, t));
+    }
+
+    pub fn delete_user(&self, key: &str, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.delete_user(key, t));
+    }
+
+    pub fn delete_user_policy(&self, key: &str, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.delete_user_policy(key, t));
+    }
+
+    pub fn delete_sts_account(&self, key: &str, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.delete_sts_account(key, t));
+    }
+
+    pub fn delete_sts_policy(&self, key: &str, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.delete_sts_policy(key, t));
+    }
+
+    pub fn delete_group(&self, key: &str, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.delete_group(key, t));
+    }
+
+    pub fn delete_group_policy(&self, key: &str, t: OffsetDateTime) {
+        self.with_write_lock(|cache| cache.delete_group_policy(key, t));
+    }
+}
+
+pub(crate) struct LockedCache {
+    state: CacheState,
+    current_ptr: *const CacheState,
+    dirty: bool,
+}
+
+impl LockedCache {
+    pub(crate) fn state(&self) -> &CacheState {
+        &self.state
+    }
+
+    pub(crate) fn matches_snapshot(&self, snapshot: &CacheSnapshot) -> bool {
+        ptr::eq(self.current_ptr, Arc::as_ptr(snapshot))
+    }
+
+    fn exec<T: Clone>(target: &mut Arc<CacheEntity<T>>, t: OffsetDateTime, mut op: impl FnMut(&mut CacheEntity<T>)) -> bool {
+        if target.load_time >= t {
+            return false;
+        }
+
+        let mut new = CacheEntity::clone(target);
+        op(&mut new);
+        *target = Arc::new(new);
+        true
+    }
+
+    fn replaced<T>(value: CacheEntity<T>) -> Arc<CacheEntity<T>> {
+        Arc::new(value.update_load_time())
+    }
+
+    pub(crate) fn replace_policy_docs(&mut self, value: CacheEntity<PolicyDoc>) {
+        self.state.policy_docs = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn replace_users(&mut self, value: CacheEntity<UserIdentity>) {
+        self.state.users = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn replace_user_policies(&mut self, value: CacheEntity<MappedPolicy>) {
+        self.state.user_policies = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn replace_sts_accounts(&mut self, value: CacheEntity<UserIdentity>) {
+        self.state.sts_accounts = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn replace_sts_policies(&mut self, value: CacheEntity<MappedPolicy>) {
+        self.state.sts_policies = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn replace_groups(&mut self, value: CacheEntity<GroupInfo>) {
+        self.state.groups = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn replace_group_policies(&mut self, value: CacheEntity<MappedPolicy>) {
+        self.state.group_policies = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn replace_user_group_memberships(&mut self, value: CacheEntity<HashSet<String>>) {
+        self.state.user_group_memberships = Self::replaced(value);
+        self.dirty = true;
+    }
+
+    pub(crate) fn add_or_update_policy_doc(&mut self, key: &str, value: &PolicyDoc, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.policy_docs, t, |map| {
             map.insert(key.to_string(), value.clone());
-        })
+        });
     }
 
-    pub fn delete<T: Clone>(target: &ArcSwap<CacheEntity<T>>, key: &str, t: OffsetDateTime) {
-        Self::exec(target, t, |map: &mut CacheEntity<T>| {
+    pub(crate) fn add_or_update_user(&mut self, key: &str, value: &UserIdentity, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.users, t, |map| {
+            map.insert(key.to_string(), value.clone());
+        });
+    }
+
+    pub(crate) fn add_or_update_user_policy(&mut self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.user_policies, t, |map| {
+            map.insert(key.to_string(), value.clone());
+        });
+    }
+
+    pub(crate) fn add_or_update_sts_account(&mut self, key: &str, value: &UserIdentity, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.sts_accounts, t, |map| {
+            map.insert(key.to_string(), value.clone());
+        });
+    }
+
+    pub(crate) fn add_or_update_sts_policy(&mut self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.sts_policies, t, |map| {
+            map.insert(key.to_string(), value.clone());
+        });
+    }
+
+    pub(crate) fn add_or_update_group(&mut self, key: &str, value: &GroupInfo, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.groups, t, |map| {
+            map.insert(key.to_string(), value.clone());
+        });
+    }
+
+    pub(crate) fn add_or_update_user_group_membership(&mut self, key: &str, value: &HashSet<String>, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.user_group_memberships, t, |map| {
+            map.insert(key.to_string(), value.clone());
+        });
+    }
+
+    pub(crate) fn add_or_update_group_policy(&mut self, key: &str, value: &MappedPolicy, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.group_policies, t, |map| {
+            map.insert(key.to_string(), value.clone());
+        });
+    }
+
+    pub(crate) fn delete_policy_doc(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.policy_docs, t, |map| {
             map.remove(key);
-        })
+        });
     }
 
-    pub fn build_user_group_memberships(&self) {
-        let groups = self.groups.load();
+    pub(crate) fn delete_user(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.users, t, |map| {
+            map.remove(key);
+        });
+    }
+
+    pub(crate) fn delete_user_policy(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.user_policies, t, |map| {
+            map.remove(key);
+        });
+    }
+
+    pub(crate) fn delete_user_group_membership(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.user_group_memberships, t, |map| {
+            map.remove(key);
+        });
+    }
+
+    pub(crate) fn delete_sts_account(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.sts_accounts, t, |map| {
+            map.remove(key);
+        });
+    }
+
+    pub(crate) fn delete_sts_policy(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.sts_policies, t, |map| {
+            map.remove(key);
+        });
+    }
+
+    pub(crate) fn delete_group(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.groups, t, |map| {
+            map.remove(key);
+        });
+    }
+
+    pub(crate) fn delete_group_policy(&mut self, key: &str, t: OffsetDateTime) {
+        self.dirty |= Self::exec(&mut self.state.group_policies, t, |map| {
+            map.remove(key);
+        });
+    }
+
+    pub(crate) fn build_user_group_memberships(&mut self) {
+        let groups = Arc::clone(&self.state.groups);
         let mut user_group_memberships = HashMap::new();
         for (group_name, group) in groups.iter() {
             for user_name in &group.members {
@@ -112,49 +334,19 @@ impl Cache {
                     .insert(group_name.clone());
             }
         }
-        self.user_group_memberships
-            .store(Arc::new(CacheEntity::new(user_group_memberships)));
+        self.replace_user_group_memberships(CacheEntity::new(user_group_memberships));
     }
 }
 
 impl CacheInner {
     #[inline]
     pub fn get_user(&self, user_name: &str) -> Option<&UserIdentity> {
-        self.users.get(user_name).or_else(|| self.sts_accounts.get(user_name))
+        self.snapshot
+            .users
+            .get(user_name)
+            .or_else(|| self.snapshot.sts_accounts.get(user_name))
     }
 
-    // fn get_policy(&self, _name: &str, _groups: &[String]) -> crate::Result<Vec<Policy>> {
-    //     todo!()
-    // }
-
-    // /// Return Ok(Some(parent_name)) when the user is temporary.
-    // /// Return Ok(None) for non-temporary users.
-    // fn is_temp_user(&self, user_name: &str) -> crate::Result<Option<&str>> {
-    //     let user = self
-    //         .get_user(user_name)
-    //         .ok_or_else(|| Error::NoSuchUser(user_name.to_owned()))?;
-
-    //     if user.credentials.is_temp() {
-    //         Ok(Some(&user.credentials.parent_user))
-    //     } else {
-    //         Ok(None)
-    //     }
-    // }
-
-    // /// Return Ok(Some(parent_name)) when the user is a temporary identity.
-    // /// Return Ok(None) when the user is not temporary.
-    // fn is_service_account(&self, user_name: &str) -> crate::Result<Option<&str>> {
-    //     let user = self
-    //         .get_user(user_name)
-    //         .ok_or_else(|| Error::NoSuchUser(user_name.to_owned()))?;
-
-    //     if user.credentials.is_service_account() {
-    //         Ok(Some(&user.credentials.parent_user))
-    //     } else {
-    //         Ok(None)
-    //     }
-    // }
-
     // todo
     pub fn is_allowed_sts(&self, _args: &Args, _parent: &str) -> bool {
         warn!("policy cache STS check path is not implemented");
@@ -223,30 +415,14 @@ impl<T> CacheEntity<T> {
     }
 }
 
-pub type G<T> = Guard<Arc<CacheEntity<T>>>;
-
 pub struct CacheInner {
-    pub policy_docs: G<PolicyDoc>,
-    pub users: G<UserIdentity>,
-    pub user_policies: G<MappedPolicy>,
-    pub sts_accounts: G<UserIdentity>,
-    pub sts_policies: G<MappedPolicy>,
-    pub groups: G<GroupInfo>,
-    pub user_group_memberships: G<HashSet<String>>,
-    pub group_policies: G<MappedPolicy>,
+    snapshot: CacheSnapshot,
 }
 
 impl From<&Cache> for CacheInner {
     fn from(value: &Cache) -> Self {
         Self {
-            policy_docs: value.policy_docs.load(),
-            users: value.users.load(),
-            user_policies: value.user_policies.load(),
-            sts_accounts: value.sts_accounts.load(),
-            sts_policies: value.sts_policies.load(),
-            groups: value.groups.load(),
-            user_group_memberships: value.user_group_memberships.load(),
-            group_policies: value.group_policies.load(),
+            snapshot: value.snapshot(),
         }
     }
 }
@@ -255,98 +431,139 @@ impl From<&Cache> for CacheInner {
 mod tests {
     use std::sync::Arc;
 
-    use arc_swap::ArcSwap;
     use futures::future::join_all;
+    use rustfs_policy::auth::UserIdentity;
     use time::OffsetDateTime;
 
-    use super::CacheEntity;
     use crate::cache::Cache;
+    use crate::store::MappedPolicy;
 
     #[tokio::test]
     async fn test_cache_entity_add() {
-        let cache = ArcSwap::new(Arc::new(CacheEntity::<usize>::default()));
+        let owner = Arc::new(Cache::default());
 
         let mut f = vec![];
 
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            let c = &cache;
+            let owner = Arc::clone(&owner);
             f.push(async move {
-                Cache::add_or_update(c, &key, &index, OffsetDateTime::now_utc());
+                let user = UserIdentity {
+                    version: index as i64,
+                    ..Default::default()
+                };
+                owner.add_or_update_user(&key, &user, OffsetDateTime::now_utc());
             });
         }
         join_all(f).await;
 
-        let cache = cache.load();
+        let cache = owner.snapshot();
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            assert_eq!(cache.get(&key), Some(&index));
+            assert_eq!(cache.users.get(&key).map(|user| user.version), Some(index as i64));
         }
     }
 
     #[tokio::test]
     async fn test_cache_entity_update() {
-        let cache = ArcSwap::new(Arc::new(CacheEntity::<usize>::default()));
+        let owner = Arc::new(Cache::default());
 
         let mut f = vec![];
 
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            let c = &cache;
+            let owner = Arc::clone(&owner);
             f.push(async move {
-                Cache::add_or_update(c, &key, &index, OffsetDateTime::now_utc());
+                let user = UserIdentity {
+                    version: index as i64,
+                    ..Default::default()
+                };
+                owner.add_or_update_user(&key, &user, OffsetDateTime::now_utc());
             });
         }
         join_all(f).await;
 
-        let cache_load = cache.load();
+        let cache_load = owner.snapshot();
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            assert_eq!(cache_load.get(&key), Some(&index));
+            assert_eq!(cache_load.users.get(&key).map(|user| user.version), Some(index as i64));
         }
 
         let mut f = vec![];
 
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            let c = &cache;
+            let owner = Arc::clone(&owner);
             f.push(async move {
-                Cache::add_or_update(c, &key, &(index * 1000), OffsetDateTime::now_utc());
+                let user = UserIdentity {
+                    version: (index * 1000) as i64,
+                    ..Default::default()
+                };
+                owner.add_or_update_user(&key, &user, OffsetDateTime::now_utc());
             });
         }
         join_all(f).await;
 
-        let cache_load = cache.load();
+        let cache_load = owner.snapshot();
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            assert_eq!(cache_load.get(&key), Some(&(index * 1000)));
+            assert_eq!(cache_load.users.get(&key).map(|user| user.version), Some((index * 1000) as i64));
         }
     }
 
     #[tokio::test]
     async fn test_cache_entity_delete() {
-        let cache = ArcSwap::new(Arc::new(CacheEntity::<usize>::default()));
+        let owner = Arc::new(Cache::default());
 
         let mut f = vec![];
 
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            let c = &cache;
+            let owner = Arc::clone(&owner);
             f.push(async move {
-                Cache::add_or_update(c, &key, &index, OffsetDateTime::now_utc());
+                let user = UserIdentity {
+                    version: index as i64,
+                    ..Default::default()
+                };
+                owner.add_or_update_user(&key, &user, OffsetDateTime::now_utc());
             });
         }
         join_all(f).await;
 
-        let cache_load = cache.load();
+        let cache_load = owner.snapshot();
         for (index, key) in (0..100).map(|x| x.to_string()).enumerate() {
-            assert_eq!(cache_load.get(&key), Some(&index));
+            assert_eq!(cache_load.users.get(&key).map(|user| user.version), Some(index as i64));
         }
+        drop(cache_load);
 
         let mut f = vec![];
 
         for key in (0..100).map(|x| x.to_string()) {
-            let c = &cache;
+            let owner = Arc::clone(&owner);
             f.push(async move {
-                Cache::delete(c, &key, OffsetDateTime::now_utc());
+                owner.delete_user(&key, OffsetDateTime::now_utc());
             });
         }
         join_all(f).await;
 
-        let cache_load = cache.load();
-        assert!(cache_load.is_empty());
+        let cache_load = owner.snapshot();
+        assert!(cache_load.users.is_empty());
+    }
+
+    #[tokio::test]
+    async fn test_cache_snapshot_reads_one_published_state() {
+        let cache = Cache::default();
+        let before = cache.snapshot();
+        let user = UserIdentity {
+            version: 7,
+            ..Default::default()
+        };
+        let policy = MappedPolicy::new("readwrite");
+
+        cache.with_write_lock(|cache| {
+            let now = OffsetDateTime::now_utc();
+            cache.add_or_update_user("snapshot-user", &user, now);
+            cache.add_or_update_user_policy("snapshot-user", &policy, now);
+        });
+
+        assert!(!before.users.contains_key("snapshot-user"));
+        assert!(!before.user_policies.contains_key("snapshot-user"));
+
+        let after = cache.snapshot();
+        assert!(after.users.contains_key("snapshot-user"));
+        assert!(after.user_policies.contains_key("snapshot-user"));
     }
 }

crates/iam/src/oidc.rs

@@ -295,6 +295,7 @@ pub struct OidcProviderConfig {
     pub roles_claim: String,
     pub email_claim: String,
     pub username_claim: String,
+    pub hide_from_ui: bool,
 }
 
 #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
@@ -422,7 +423,7 @@ impl OidcSys {
         !self.configs.is_empty()
     }
 
-    /// Return provider summaries for the console UI.
+    /// List all providers (including hidden ones). Used by site-replication and admin config.
     pub fn list_providers(&self) -> Vec<OidcProviderSummary> {
         self.configs
             .values()
@@ -433,6 +434,18 @@ impl OidcSys {
             .collect()
     }
 
+    /// List only visible providers (excludes those with `hide_from_ui = true`).
+    pub fn list_visible_providers(&self) -> Vec<OidcProviderSummary> {
+        self.configs
+            .values()
+            .filter(|c| !c.hide_from_ui)
+            .map(|c| OidcProviderSummary {
+                provider_id: c.id.clone(),
+                display_name: c.display_name.clone(),
+            })
+            .collect()
+    }
+
     /// Build the PKCE authorization URL for a provider, store state in the state store.
     pub async fn authorize_url(
         &self,
@@ -918,6 +931,19 @@ impl OidcSys {
         configs
     }
 
+    /// Parse a string as an `EnableState` boolean.
+    /// Returns `default_if_empty` when the input is empty, and `default_on_error`
+    /// when parsing fails.
+    fn parse_enable_state(value: &str, default_if_empty: bool, default_on_error: bool) -> bool {
+        if value.is_empty() {
+            return default_if_empty;
+        }
+        value
+            .parse::<EnableState>()
+            .map(|s| s.is_enabled())
+            .unwrap_or(default_on_error)
+    }
+
     /// Parse a single provider's config from env vars with the given suffix.
     fn parse_single_provider(env_suffix: &str, id: &str) -> Option<OidcProviderConfig> {
         let get_env = |base: &str| -> String { std::env::var(format!("{base}{env_suffix}")).unwrap_or_default() };
@@ -930,11 +956,7 @@ impl OidcSys {
             return None;
         }
 
-        let enabled = enable_val.is_empty()
-            || enable_val
-                .parse::<rustfs_config::EnableState>()
-                .map(|s| s.is_enabled())
-                .unwrap_or(false);
+        let enabled = Self::parse_enable_state(&enable_val, true, false);
 
         let scopes_str = get_env(ENV_IDENTITY_OPENID_SCOPES);
         let scopes = if scopes_str.is_empty() {
@@ -951,12 +973,7 @@ impl OidcSys {
             .map(|s| s.to_string())
             .collect();
 
-        let redirect_uri_dynamic_str = get_env(ENV_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC);
-        let redirect_uri_dynamic = redirect_uri_dynamic_str.is_empty()
-            || redirect_uri_dynamic_str
-                .parse::<rustfs_config::EnableState>()
-                .map(|s| s.is_enabled())
-                .unwrap_or(true);
+        let redirect_uri_dynamic = Self::parse_enable_state(&get_env(ENV_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC), true, true);
 
         let claim_name = {
             let v = get_env(ENV_IDENTITY_OPENID_CLAIM_NAME);
@@ -1003,6 +1020,7 @@ impl OidcSys {
             let v = get_env(ENV_IDENTITY_OPENID_CLIENT_SECRET);
             if v.is_empty() { None } else { Some(v) }
         };
+        let hide_from_ui = Self::parse_enable_state(&get_env(ENV_IDENTITY_OPENID_HIDE_FROM_UI), false, false);
 
         Some(OidcProviderConfig {
             id: id.to_string(),
@@ -1022,6 +1040,7 @@ impl OidcSys {
             roles_claim,
             email_claim,
             username_claim,
+            hide_from_ui,
         })
     }
 
@@ -1031,12 +1050,7 @@ impl OidcSys {
             return None;
         }
 
-        let enabled = kvs
-            .lookup(ENABLE_KEY)
-            .unwrap_or_else(|| EnableState::Off.to_string())
-            .parse::<EnableState>()
-            .map(|s| s.is_enabled())
-            .unwrap_or(false);
+        let enabled = Self::parse_enable_state(&kvs.lookup(ENABLE_KEY).unwrap_or_default(), false, false);
 
         let scopes_str = kvs.get(OIDC_SCOPES);
         let scopes = if scopes_str.is_empty() {
@@ -1053,12 +1067,8 @@ impl OidcSys {
             .map(|s| s.to_string())
             .collect();
 
-        let redirect_uri_dynamic = kvs
-            .lookup(OIDC_REDIRECT_URI_DYNAMIC)
-            .unwrap_or_else(|| EnableState::On.to_string())
-            .parse::<EnableState>()
-            .map(|s| s.is_enabled())
-            .unwrap_or(true);
+        let redirect_uri_dynamic =
+            Self::parse_enable_state(&kvs.lookup(OIDC_REDIRECT_URI_DYNAMIC).unwrap_or_default(), true, true);
 
         let claim_name = kvs
             .lookup(OIDC_CLAIM_NAME)
@@ -1078,6 +1088,7 @@ impl OidcSys {
         let display_name = kvs.lookup(OIDC_DISPLAY_NAME).unwrap_or_else(|| id.to_string());
         let redirect_uri = kvs.lookup(OIDC_REDIRECT_URI).filter(|v| !v.is_empty());
         let client_secret = kvs.lookup(OIDC_CLIENT_SECRET).filter(|v| !v.is_empty());
+        let hide_from_ui = Self::parse_enable_state(&kvs.lookup(OIDC_HIDE_FROM_UI).unwrap_or_default(), false, false);
 
         Some(OidcProviderConfig {
             id: id.to_string(),
@@ -1097,6 +1108,7 @@ impl OidcSys {
             roles_claim,
             email_claim,
             username_claim,
+            hide_from_ui,
         })
     }
 
@@ -1582,6 +1594,7 @@ mod tests {
             roles_claim: String::new(),
             email_claim: "email".to_string(),
             username_claim: "username".to_string(),
+            hide_from_ui: false,
         }
     }
 
@@ -1961,9 +1974,198 @@ mod tests {
             roles_claim: String::new(),
             email_claim: "email".to_string(),
             username_claim: "preferred_username".to_string(),
+            hide_from_ui: false,
         }
     }
 
+    #[test]
+    fn test_parse_enable_state_on() {
+        assert!(OidcSys::parse_enable_state("on", false, false));
+    }
+
+    #[test]
+    fn test_parse_enable_state_off() {
+        assert!(!OidcSys::parse_enable_state("off", true, true));
+    }
+
+    #[test]
+    fn test_parse_enable_state_empty_returns_default() {
+        assert!(OidcSys::parse_enable_state("", true, false));
+        assert!(!OidcSys::parse_enable_state("", false, true));
+    }
+
+    #[test]
+    fn test_parse_enable_state_invalid_returns_error_default() {
+        assert!(!OidcSys::parse_enable_state("garbage", true, false));
+        assert!(OidcSys::parse_enable_state("garbage", false, true));
+    }
+
+    #[test]
+    fn test_list_visible_providers_hides_hidden_provider() {
+        let visible = test_config("dex");
+        let mut hidden = test_config("kubernetes");
+        hidden.hide_from_ui = true;
+
+        let sys = make_test_sys(vec![visible, hidden]);
+        let listed = sys.list_visible_providers();
+
+        assert_eq!(listed.len(), 1);
+        assert!(listed.iter().any(|p| p.provider_id == "dex"));
+        assert!(!listed.iter().any(|p| p.provider_id == "kubernetes"));
+    }
+
+    #[test]
+    fn test_hidden_provider_still_resolvable_for_sts() {
+        let visible = test_config("dex");
+        let mut hidden = test_config("kubernetes");
+        hidden.hide_from_ui = true;
+
+        let sys = make_test_sys(vec![visible, hidden]);
+
+        assert!(sys.get_provider_config("kubernetes").is_some());
+        assert!(sys.get_provider_config("dex").is_some());
+    }
+
+    #[test]
+    fn test_list_providers_includes_hidden_for_replication() {
+        let visible = test_config("dex");
+        let mut hidden = test_config("kubernetes");
+        hidden.hide_from_ui = true;
+
+        let sys = make_test_sys(vec![visible, hidden]);
+
+        // Unfiltered list returns all (used by site-replication)
+        assert_eq!(sys.list_providers().len(), 2);
+        // UI-filtered list hides the hidden one
+        assert_eq!(sys.list_visible_providers().len(), 1);
+    }
+
+    #[test]
+    fn test_list_providers_all_visible_by_default() {
+        let a = test_config("okta");
+        let b = test_config("dex");
+
+        let sys = make_test_sys(vec![a, b]);
+        let listed = sys.list_visible_providers();
+
+        assert_eq!(listed.len(), 2);
+    }
+
+    #[test]
+    fn test_list_visible_providers_all_hidden() {
+        let mut a = test_config("k8s-a");
+        a.hide_from_ui = true;
+        let mut b = test_config("k8s-b");
+        b.hide_from_ui = true;
+
+        let sys = make_test_sys(vec![a, b]);
+        let listed = sys.list_visible_providers();
+
+        assert!(listed.is_empty());
+        assert!(sys.has_providers());
+    }
+
+    #[test]
+    fn test_hide_from_ui_default_is_false() {
+        let config = test_config("default");
+        assert!(!config.hide_from_ui);
+    }
+
+    #[test]
+    fn test_parse_persisted_hide_from_ui_off_is_false() {
+        let mut cfg = ServerConfig::new();
+        let mut kvs = KVS(vec![
+            rustfs_ecstore::config::KV {
+                key: ENABLE_KEY.to_string(),
+                value: EnableState::On.to_string(),
+                hidden_if_empty: false,
+            },
+            rustfs_ecstore::config::KV {
+                key: OIDC_CONFIG_URL.to_string(),
+                value: "https://example.com/.well-known/openid-configuration".to_string(),
+                hidden_if_empty: false,
+            },
+            rustfs_ecstore::config::KV {
+                key: OIDC_CLIENT_ID.to_string(),
+                value: "console".to_string(),
+                hidden_if_empty: false,
+            },
+        ]);
+        kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::Off.to_string());
+
+        cfg.0
+            .entry(IDENTITY_OPENID_SUB_SYS.to_string())
+            .or_default()
+            .insert(DEFAULT_DELIMITER.to_string(), kvs);
+
+        let parsed = OidcSys::parse_persisted_configs(&cfg);
+        assert_eq!(parsed.len(), 1);
+        assert!(!parsed[0].hide_from_ui);
+    }
+
+    #[test]
+    fn test_parse_persisted_hide_from_ui_missing_defaults_false() {
+        let mut cfg = ServerConfig::new();
+        let kvs = KVS(vec![
+            rustfs_ecstore::config::KV {
+                key: ENABLE_KEY.to_string(),
+                value: EnableState::On.to_string(),
+                hidden_if_empty: false,
+            },
+            rustfs_ecstore::config::KV {
+                key: OIDC_CONFIG_URL.to_string(),
+                value: "https://example.com/.well-known/openid-configuration".to_string(),
+                hidden_if_empty: false,
+            },
+            rustfs_ecstore::config::KV {
+                key: OIDC_CLIENT_ID.to_string(),
+                value: "console".to_string(),
+                hidden_if_empty: false,
+            },
+        ]);
+
+        cfg.0
+            .entry(IDENTITY_OPENID_SUB_SYS.to_string())
+            .or_default()
+            .insert(DEFAULT_DELIMITER.to_string(), kvs);
+
+        let parsed = OidcSys::parse_persisted_configs(&cfg);
+        assert_eq!(parsed.len(), 1);
+        assert!(!parsed[0].hide_from_ui);
+    }
+
+    #[test]
+    fn test_parse_persisted_hide_from_ui() {
+        let mut cfg = ServerConfig::new();
+        let mut kvs = KVS(vec![
+            rustfs_ecstore::config::KV {
+                key: ENABLE_KEY.to_string(),
+                value: EnableState::On.to_string(),
+                hidden_if_empty: false,
+            },
+            rustfs_ecstore::config::KV {
+                key: OIDC_CONFIG_URL.to_string(),
+                value: "https://example.com/.well-known/openid-configuration".to_string(),
+                hidden_if_empty: false,
+            },
+            rustfs_ecstore::config::KV {
+                key: OIDC_CLIENT_ID.to_string(),
+                value: "console".to_string(),
+                hidden_if_empty: false,
+            },
+        ]);
+        kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::On.to_string());
+
+        cfg.0
+            .entry(IDENTITY_OPENID_SUB_SYS.to_string())
+            .or_default()
+            .insert(DEFAULT_DELIMITER.to_string(), kvs);
+
+        let parsed = OidcSys::parse_persisted_configs(&cfg);
+        assert_eq!(parsed.len(), 1);
+        assert!(parsed[0].hide_from_ui);
+    }
+
     #[test]
     fn test_map_claims_to_policies_with_provider() {
         let mut config = test_config("okta");
@@ -2056,6 +2258,7 @@ mod tests {
             roles_claim: String::new(),
             email_claim: "email".to_string(),
             username_claim: "preferred_username".to_string(),
+            hide_from_ui: false,
         };
 
         assert_eq!(config.id, "test");

crates/iam/src/store/object.rs

@@ -362,6 +362,7 @@ impl ObjectStore {
         let (tx, mut rx) = mpsc::channel::<ObjectInfoOrErr>(100);
 
         let path = prefix.to_owned();
+        let sender_on_error = sender.clone();
         tokio::spawn(async move {
             if let Err(err) = store
                 .walk(ctx.clone(), Self::BUCKET_NAME, &path, tx, WalkOptions::default())
@@ -378,6 +379,12 @@ impl ObjectStore {
                     error = %err,
                     "system path walk failed"
                 );
+                let _ = sender_on_error
+                    .send(StringOrErr {
+                        item: None,
+                        err: Some(err.into()),
+                    })
+                    .await;
             }
         });
 
@@ -1018,6 +1025,7 @@ impl Store for ObjectStore {
     }
 
     async fn load_all(&self, cache: &Cache) -> Result<()> {
+        let cache_snapshot = cache.snapshot();
         let listed_config_items = self.list_all_iamconfig_items().await?;
 
         let mut policy_docs_cache = CacheEntity::new(get_default_policyes());
@@ -1059,16 +1067,12 @@ impl Store for ObjectStore {
             }
         }
 
-        cache.policy_docs.store(Arc::new(policy_docs_cache.update_load_time()));
-
         let mut user_items_cache = CacheEntity::default();
 
         // users
         if let Some(item_name_list) = listed_config_items.get(USERS_LIST_KEY) {
             let mut item_name_list = item_name_list.clone();
 
-            // let mut items_cache = CacheEntity::default();
-
             loop {
                 if item_name_list.len() < 32 {
                     let items = self.load_user_concurrent(&item_name_list, UserType::Reg).await?;
@@ -1099,11 +1103,10 @@ impl Store for ObjectStore {
 
                 item_name_list = item_name_list.split_off(32);
             }
-
-            // cache.users.store(Arc::new(items_cache.update_load_time()));
         }
 
         // groups
+        let mut groups_cache = None;
         if let Some(item_name_list) = listed_config_items.get(GROUPS_LIST_KEY) {
             let mut items_cache = CacheEntity::default();
 
@@ -1115,10 +1118,11 @@ impl Store for ObjectStore {
                 };
             }
 
-            cache.groups.store(Arc::new(items_cache.update_load_time()));
+            groups_cache = Some(items_cache);
         }
 
         // user policies
+        let mut user_policies_cache = None;
         if let Some(item_name_list) = listed_config_items.get(POLICY_DB_USERS_LIST_KEY) {
             let mut item_name_list = item_name_list.clone();
 
@@ -1159,10 +1163,11 @@ impl Store for ObjectStore {
                 item_name_list = item_name_list.split_off(32);
             }
 
-            cache.user_policies.store(Arc::new(items_cache.update_load_time()));
+            user_policies_cache = Some(items_cache);
         }
 
         // group policy
+        let mut group_policies_cache = None;
         if let Some(item_name_list) = listed_config_items.get(POLICY_DB_GROUPS_LIST_KEY) {
             let mut items_cache = CacheEntity::default();
 
@@ -1177,7 +1182,7 @@ impl Store for ObjectStore {
                 };
             }
 
-            cache.group_policies.store(Arc::new(items_cache.update_load_time()));
+            group_policies_cache = Some(items_cache);
         }
 
         let mut sts_policies_cache = CacheEntity::default();
@@ -1212,11 +1217,8 @@ impl Store for ObjectStore {
 
             // Merge items_cache to user_items_cache
             user_items_cache.extend(items_cache);
-
-            // cache.users.store(Arc::new(items_cache.update_load_time()));
         }
 
-        cache.build_user_group_memberships();
         let mut sts_items_cache = CacheEntity::default();
         // sts users
         if let Some(item_name_list) = listed_config_items.get(STS_LIST_KEY) {
@@ -1245,159 +1247,29 @@ impl Store for ObjectStore {
             }
         }
 
-        cache.users.store(Arc::new(user_items_cache.update_load_time()));
-        cache.sts_accounts.store(Arc::new(sts_items_cache.update_load_time()));
-        cache.sts_policies.store(Arc::new(sts_policies_cache.update_load_time()));
+        cache.with_write_lock(|cache| {
+            if cache.matches_snapshot(&cache_snapshot) {
+                cache.replace_policy_docs(policy_docs_cache);
+                if let Some(groups_cache) = groups_cache {
+                    cache.replace_groups(groups_cache);
+                }
+                if let Some(user_policies_cache) = user_policies_cache {
+                    cache.replace_user_policies(user_policies_cache);
+                }
+                if let Some(group_policies_cache) = group_policies_cache {
+                    cache.replace_group_policies(group_policies_cache);
+                }
+                cache.replace_users(user_items_cache);
+                cache.replace_sts_accounts(sts_items_cache);
+                cache.replace_sts_policies(sts_policies_cache);
+                cache.build_user_group_memberships();
+            } else {
+                warn!("skip IAM full reload cache commit because one or more IAM caches changed during reload");
+            }
+        });
 
         Ok(())
     }
-
-    // /// load all and make a new cache.
-    // async fn load_all(&self, cache: &Cache) -> Result<()> {
-    //     let _items = &[
-    //         "policydb/",
-    //         "policies/",
-    //         "groups/",
-    //         "policydb/users/",
-    //         "policydb/groups/",
-    //         "service-accounts/",
-    //         "policydb/sts-users/",
-    //         "sts/",
-    //     ];
-
-    //     let items = self.list_iam_config_items("config/iam/").await?;
-    //     debug!("all iam items: {items:?}");
-
-    //     let (policy_docs, users, user_policies, sts_policies, sts_accounts) = (
-    //         Arc::new(tokio::sync::Mutex::new(CacheEntity::new(Self::get_default_policyes()))),
-    //         Arc::new(tokio::sync::Mutex::new(CacheEntity::default())),
-    //         Arc::new(tokio::sync::Mutex::new(CacheEntity::default())),
-    //         Arc::new(tokio::sync::Mutex::new(CacheEntity::default())),
-    //         Arc::new(tokio::sync::Mutex::new(CacheEntity::default())),
-    //     );
-
-    //     // Read 32 elements at a time
-    //     let iter = items
-    //         .iter()
-    //         .map(|item| item.trim_start_matches("config/iam/"))
-    //         .map(|item| split_path(item, item.starts_with("policydb/")))
-    //         .filter_map(|(list_key, trimmed_item)| {
-    //             debug!("list_key: {list_key}, trimmed_item: {trimmed_item}");
-
-    //             if list_key == "format.json" {
-    //                 return None;
-    //             }
-
-    //             let (policy_docs, users, user_policies, sts_policies, sts_accounts) = (
-    //                 policy_docs.clone(),
-    //                 users.clone(),
-    //                 user_policies.clone(),
-    //                 sts_policies.clone(),
-    //                 sts_accounts.clone(),
-    //             );
-
-    //             Some(async move {
-    //                 match list_key {
-    //                     "policies/" => {
-    //                         let trimmed_item = dir(trimmed_item);
-    //                         let name = trimmed_item.trim_end_matches('/');
-    //                         let policy_doc = self.load_policy(name).await?;
-    //                         policy_docs.lock().await.insert(name.to_owned(), policy_doc);
-    //                     }
-    //                     "users/" => {
-    //                         let name = dir(trimmed_item);
-    //                         if let Some(user) = self.load_user_identity(UserType::Reg, &name).await? {
-    //                             users.lock().await.insert(name.to_owned(), user);
-    //                         };
-    //                     }
-    //                     "groups/" => {}
-    //                     "policydb/users/" | "policydb/groups/" => {
-    //                         let name = trimmed_item.strip_suffix(".json").unwrap_or(trimmed_item);
-    //                         let mapped_policy = self
-    //                             .load_mapped_policy(UserType::Reg, name, list_key == "policydb/groups/")
-    //                             .await?;
-    //                         if !mapped_policy.policies.is_empty() {
-    //                             user_policies.lock().await.insert(name.to_owned(), mapped_policy);
-    //                         }
-    //                     }
-    //                     "service-accounts/" => {
-    //                         let trimmed_item = dir(trimmed_item);
-    //                         let name = trimmed_item.trim_end_matches('/');
-    //                         let Some(user) = self.load_user_identity(UserType::Svc, name).await? else {
-    //                             return Ok(());
-    //                         };
-
-    //                         let parent = user.credentials.parent_user.clone();
-
-    //                         {
-    //                             users.lock().await.insert(name.to_owned(), user);
-    //                         }
-
-    //                         if users.lock().await.get(&parent).is_some() {
-    //                             return Ok(());
-    //                         }
-
-    //                         match self.load_mapped_policy(UserType::Sts, parent.as_str(), false).await {
-    //                             Ok(m) => sts_policies.lock().await.insert(name.to_owned(), m),
-    //                             Err(Error::EcstoreError(e)) if is_err_config_not_found(&e) => return Ok(()),
-    //                             Err(e) => return Err(e),
-    //                         };
-    //                     }
-    //                     "sts/" => {
-    //                         let name = dir(trimmed_item);
-    //                         if let Some(user) = self.load_user_identity(UserType::Sts, &name).await? {
-    //                             warn!("sts_accounts insert {}, user {:?}", name, &user.credentials.access_key);
-    //                             sts_accounts.lock().await.insert(name.to_owned(), user);
-    //                         };
-    //                     }
-    //                     "policydb/sts-users/" => {
-    //                         let name = trimmed_item.strip_suffix(".json").unwrap_or(trimmed_item);
-    //                         let mapped_policy = self.load_mapped_policy(UserType::Sts, name, false).await?;
-    //                         if !mapped_policy.policies.is_empty() {
-    //                             sts_policies.lock().await.insert(name.to_owned(), mapped_policy);
-    //                         }
-    //                     }
-    //                     _ => {}
-    //                 }
-
-    //                 Result::Ok(())
-    //             })
-    //         });
-
-    //     let mut all_futures = Vec::with_capacity(32);
-
-    //     for f in iter {
-    //         all_futures.push(f);
-
-    //         if all_futures.len() == 32 {
-    //             try_join_all(all_futures).await?;
-    //             all_futures = Vec::with_capacity(32);
-    //         }
-    //     }
-
-    //     if !all_futures.is_empty() {
-    //         try_join_all(all_futures).await?;
-    //     }
-
-    //     if let Some(x) = Arc::into_inner(users) {
-    //         cache.users.store(Arc::new(x.into_inner().update_load_time()))
-    //     }
-
-    //     if let Some(x) = Arc::into_inner(policy_docs) {
-    //         cache.policy_docs.store(Arc::new(x.into_inner().update_load_time()))
-    //     }
-    //     if let Some(x) = Arc::into_inner(user_policies) {
-    //         cache.user_policies.store(Arc::new(x.into_inner().update_load_time()))
-    //     }
-    //     if let Some(x) = Arc::into_inner(sts_policies) {
-    //         cache.sts_policies.store(Arc::new(x.into_inner().update_load_time()))
-    //     }
-    //     if let Some(x) = Arc::into_inner(sts_accounts) {
-    //         cache.sts_accounts.store(Arc::new(x.into_inner().update_load_time()))
-    //     }
-
-    //     Ok(())
-    // }
 }
 
 #[cfg(test)]

crates/iam/src/sys.rs

@@ -834,8 +834,27 @@ impl<T: Store> IamSys<T> {
         self.store.policy_db_get(name, groups).await
     }
 
+    /// Check whether a policy name from a JWT claim is safe to resolve against the IAM store.
+    ///
+    /// Allowed characters: `[a-zA-Z0-9_:.-]`
+    /// - Colons: K8s service account `sub` claims (`system:serviceaccount:ns:sa`)
+    /// - Dots: OIDC group names from providers like Okta/Entra (`org.team.role`)
+    ///
+    /// Requirements:
+    /// - At least one ASCII alphanumeric character (prevents meaningless names
+    ///   like `.`, `..`, `-`, `___`, or `:.:`).
+    /// - No characters outside the allowed set (helps mitigate path traversal
+    ///   and injection when names are used in storage keys or log output).
     fn is_safe_claim_policy_name(policy: &str) -> bool {
-        !policy.is_empty() && policy.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
+        let mut has_alnum = false;
+        for c in policy.bytes() {
+            if c.is_ascii_alphanumeric() {
+                has_alnum = true;
+            } else if !matches!(c, b'_' | b'-' | b':' | b'.') {
+                return false;
+            }
+        }
+        has_alnum
     }
 
     // JWT policy claims carry canned policy names only; policy documents are resolved by IAM store.
@@ -1325,7 +1344,7 @@ mod tests {
     use rustfs_policy::policy::action::{Action, AdminAction, S3Action};
     use rustfs_policy::policy::policy_uses_existing_object_tag_conditions;
     use serde_json::Value;
-    use std::collections::HashMap;
+    use std::collections::{HashMap, HashSet};
     use time::OffsetDateTime;
 
     #[test]
@@ -1410,6 +1429,10 @@ mod tests {
         }
 
         async fn load_user(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> {
+            if name == "deleted-notify-user" {
+                return Err(Error::NoSuchUser(name.to_string()));
+            }
+
             if user_type == UserType::Reg && name == "load-failure-user" {
                 return Err(Error::Io(std::io::Error::other("load user failed")));
             }
@@ -1442,7 +1465,10 @@ mod tests {
             Err(Error::InvalidArgument)
         }
 
-        async fn load_group(&self, _name: &str, _m: &mut HashMap<String, GroupInfo>) -> Result<()> {
+        async fn load_group(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
+            if name == "notify-group" {
+                m.insert(name.to_string(), GroupInfo::new(vec!["notify-user".to_string()]));
+            }
             Ok(())
         }
 
@@ -1495,6 +1521,9 @@ mod tests {
             if user_type == UserType::Reg && !is_group && name == "notify-user" {
                 m.insert(name.to_string(), MappedPolicy::new("readwrite"));
             }
+            if user_type == UserType::Sts && !is_group && name == "notify-sts-parent" {
+                m.insert(name.to_string(), MappedPolicy::new("readwrite"));
+            }
             Ok(())
         }
 
@@ -1512,9 +1541,6 @@ mod tests {
             let custom_claim_policy =
                 Policy::parse_config(CUSTOM_STS_CLAIM_POLICY_JSON.as_bytes()).expect("custom STS claim policy should parse");
             policy_docs.insert(CUSTOM_STS_CLAIM_POLICY.to_string(), PolicyDoc::new(custom_claim_policy));
-            cache
-                .policy_docs
-                .store(Arc::new(CacheEntity::new(policy_docs).update_load_time()));
 
             if self.empty_policies {
                 const PARENT_USER: &str = "sts-empty-parent-policy-test";
@@ -1537,16 +1563,17 @@ mod tests {
                 };
                 let mut users = HashMap::new();
                 users.insert(PARENT_USER.to_string(), parent_identity);
-                cache.users.store(Arc::new(CacheEntity::new(users).update_load_time()));
 
-                cache.groups.store(Arc::new(CacheEntity::default().update_load_time()));
-                cache
-                    .group_policies
-                    .store(Arc::new(CacheEntity::default().update_load_time()));
-                cache.user_policies.store(Arc::new(CacheEntity::default().update_load_time()));
-                cache.sts_accounts.store(Arc::new(CacheEntity::default().update_load_time()));
-                cache.sts_policies.store(Arc::new(CacheEntity::default().update_load_time()));
-                cache.build_user_group_memberships();
+                cache.with_write_lock(|cache| {
+                    cache.replace_policy_docs(CacheEntity::new(policy_docs));
+                    cache.replace_users(CacheEntity::new(users));
+                    cache.replace_groups(CacheEntity::default());
+                    cache.replace_group_policies(CacheEntity::default());
+                    cache.replace_user_policies(CacheEntity::default());
+                    cache.replace_sts_accounts(CacheEntity::default());
+                    cache.replace_sts_policies(CacheEntity::default());
+                    cache.build_user_group_memberships();
+                });
                 return Ok(());
             }
 
@@ -1572,24 +1599,25 @@ mod tests {
             };
             let mut users = HashMap::new();
             users.insert(PARENT_USER.to_string(), parent_identity);
-            cache.users.store(Arc::new(CacheEntity::new(users).update_load_time()));
 
             let group = GroupInfo::new(vec![PARENT_USER.to_string()]);
             let mut groups = HashMap::new();
             groups.insert(GROUP_NAME.to_string(), group);
-            cache.groups.store(Arc::new(CacheEntity::new(groups).update_load_time()));
 
             let group_policy = MappedPolicy::new("readwrite");
             let mut group_policies = HashMap::new();
             group_policies.insert(GROUP_NAME.to_string(), group_policy);
-            cache
-                .group_policies
-                .store(Arc::new(CacheEntity::new(group_policies).update_load_time()));
 
-            cache.user_policies.store(Arc::new(CacheEntity::default().update_load_time()));
-            cache.sts_accounts.store(Arc::new(CacheEntity::default().update_load_time()));
-            cache.sts_policies.store(Arc::new(CacheEntity::default().update_load_time()));
-            cache.build_user_group_memberships();
+            cache.with_write_lock(|cache| {
+                cache.replace_policy_docs(CacheEntity::new(policy_docs));
+                cache.replace_users(CacheEntity::new(users));
+                cache.replace_groups(CacheEntity::new(groups));
+                cache.replace_group_policies(CacheEntity::new(group_policies));
+                cache.replace_user_policies(CacheEntity::default());
+                cache.replace_sts_accounts(CacheEntity::default());
+                cache.replace_sts_policies(CacheEntity::default());
+                cache.build_user_group_memberships();
+            });
 
             Ok(())
         }
@@ -1950,7 +1978,10 @@ mod tests {
             parent_user: parent_user.to_string(),
             ..Default::default()
         });
-        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+        iam_sys
+            .store
+            .cache
+            .add_or_update_sts_account(sts_access_key, &sts_user, OffsetDateTime::now_utc());
 
         let mut claims = HashMap::new();
         claims.insert(POLICYNAME.to_string(), Value::String(CUSTOM_STS_CLAIM_POLICY.to_string()));
@@ -1991,7 +2022,10 @@ mod tests {
             parent_user: parent_user.to_string(),
             ..Default::default()
         });
-        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+        iam_sys
+            .store
+            .cache
+            .add_or_update_sts_account(sts_access_key, &sts_user, OffsetDateTime::now_utc());
 
         let mut claims = HashMap::new();
         claims.insert(
@@ -2035,7 +2069,10 @@ mod tests {
             parent_user: parent_user.to_string(),
             ..Default::default()
         });
-        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+        iam_sys
+            .store
+            .cache
+            .add_or_update_sts_account(sts_access_key, &sts_user, OffsetDateTime::now_utc());
 
         let mut claims = HashMap::new();
         claims.insert(POLICYNAME.to_string(), Value::String(CUSTOM_STS_CLAIM_POLICY.to_string()));
@@ -2227,6 +2264,113 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn test_group_notification_populates_new_membership_entry() {
+        let store = StsTestMockStore { empty_policies: false };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        iam_sys.store.cache.with_write_lock(|cache| {
+            cache.replace_user_group_memberships(CacheEntity::default());
+        });
+
+        iam_sys.load_group("notify-group").await.unwrap();
+
+        let cache = iam_sys.store.cache.snapshot();
+        let memberships = &cache.user_group_memberships;
+        assert!(
+            memberships
+                .get("notify-user")
+                .is_some_and(|groups| groups.contains("notify-group")),
+            "group notification must create a reverse membership entry for first-time members"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_sts_policy_mapping_notification_updates_sts_policy_cache() {
+        let store = StsTestMockStore { empty_policies: false };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        iam_sys
+            .load_policy_mapping("notify-sts-parent", UserType::Sts, false)
+            .await
+            .unwrap();
+
+        let cache = iam_sys.store.cache.snapshot();
+        let sts_policies = &cache.sts_policies;
+        assert!(
+            sts_policies.contains_key("notify-sts-parent"),
+            "STS policy mapping notifications must update sts_policies instead of deleting them"
+        );
+    }
+
+    #[tokio::test]
+    async fn test_missing_user_notification_cleans_related_cache_state() {
+        let store = StsTestMockStore { empty_policies: false };
+        let cache_manager = IamCache::new(store).await.unwrap();
+        let iam_sys = IamSys::new(cache_manager);
+
+        const USER: &str = "deleted-notify-user";
+        const GROUP: &str = "deleted-notify-group";
+        const SVC_CHILD: &str = "deleted-notify-service-child";
+        const STS_CHILD: &str = "deleted-notify-sts-child";
+        const OTHER_USER: &str = "deleted-notify-other-user";
+
+        let user = UserIdentity::from(Credentials {
+            access_key: USER.to_string(),
+            secret_key: "longenoughsecret".to_string(),
+            status: ACCOUNT_ON.to_string(),
+            ..Default::default()
+        });
+
+        let mut service_claims = HashMap::new();
+        service_claims.insert(iam_policy_claim_name_sa(), Value::String(INHERITED_POLICY_TYPE.to_string()));
+        let service_child = UserIdentity::from(Credentials {
+            access_key: SVC_CHILD.to_string(),
+            secret_key: "longenoughsecret".to_string(),
+            status: ACCOUNT_ON.to_string(),
+            parent_user: USER.to_string(),
+            claims: Some(service_claims),
+            ..Default::default()
+        });
+
+        let sts_child = UserIdentity::from(Credentials {
+            access_key: STS_CHILD.to_string(),
+            secret_key: "longenoughsecret".to_string(),
+            session_token: "session-token".to_string(),
+            status: ACCOUNT_ON.to_string(),
+            parent_user: USER.to_string(),
+            ..Default::default()
+        });
+
+        let membership = HashSet::from([GROUP.to_string()]);
+        let group = GroupInfo::new(vec![USER.to_string(), OTHER_USER.to_string()]);
+        let mapped_policy = MappedPolicy::new("readwrite");
+
+        iam_sys.store.cache.with_write_lock(|cache| {
+            let now = OffsetDateTime::now_utc();
+            cache.add_or_update_user(USER, &user, now);
+            cache.add_or_update_user_policy(USER, &mapped_policy, now);
+            cache.add_or_update_group(GROUP, &group, now);
+            cache.add_or_update_user_group_membership(USER, &membership, now);
+            cache.add_or_update_user(SVC_CHILD, &service_child, now);
+            cache.add_or_update_sts_account(STS_CHILD, &sts_child, now);
+        });
+
+        iam_sys.load_user(USER, UserType::Reg).await.unwrap();
+
+        let cache = iam_sys.store.cache.snapshot();
+        assert!(!cache.users.contains_key(USER));
+        assert!(!cache.user_policies.contains_key(USER));
+        assert!(!cache.user_group_memberships.contains_key(USER));
+        assert!(!cache.users.contains_key(SVC_CHILD));
+        assert!(!cache.sts_accounts.contains_key(STS_CHILD));
+        let group = cache.groups.get(GROUP).expect("group should remain after member removal");
+        assert!(!group.members.contains(&USER.to_string()));
+        assert!(group.members.contains(&OTHER_USER.to_string()));
+    }
+
     #[tokio::test]
     async fn test_check_key_propagates_cache_miss_load_failure() {
         let store = StsTestMockStore { empty_policies: false };
@@ -2281,7 +2425,10 @@ mod tests {
             parent_user: "sts-empty-parent-policy-test".to_string(),
             ..Default::default()
         });
-        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+        iam_sys
+            .store
+            .cache
+            .add_or_update_sts_account(sts_access_key, &sts_user, OffsetDateTime::now_utc());
 
         let mut claims = HashMap::new();
         claims.insert(
@@ -2395,7 +2542,10 @@ mod tests {
             parent_user: "sts-empty-parent-policy-test".to_string(),
             ..Default::default()
         });
-        Cache::add_or_update(&iam_sys.store.cache.sts_accounts, sts_access_key, &sts_user, OffsetDateTime::now_utc());
+        iam_sys
+            .store
+            .cache
+            .add_or_update_sts_account(sts_access_key, &sts_user, OffsetDateTime::now_utc());
 
         let session_policy_json = r#"{
   "Version":"2012-10-17",
@@ -2445,12 +2595,10 @@ mod tests {
             claims: Some(service_account_claims),
             ..Default::default()
         });
-        Cache::add_or_update(
-            &iam_sys.store.cache.users,
-            service_account_access_key,
-            &service_identity,
-            OffsetDateTime::now_utc(),
-        );
+        iam_sys
+            .store
+            .cache
+            .add_or_update_user(service_account_access_key, &service_identity, OffsetDateTime::now_utc());
 
         let mut request_claims = HashMap::new();
         request_claims.insert("parent".to_string(), Value::String(parent_user.to_string()));
@@ -2538,4 +2686,64 @@ mod tests {
             policy_info.policy
         );
     }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_allows_k8s_sa_sub() {
+        assert!(IamSys::<StsTestMockStore>::is_safe_claim_policy_name(
+            "system:serviceaccount:my-namespace:my-sa"
+        ));
+    }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_allows_dotted_names() {
+        assert!(IamSys::<StsTestMockStore>::is_safe_claim_policy_name("org.team.policy-name"));
+    }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_allows_simple_names() {
+        assert!(IamSys::<StsTestMockStore>::is_safe_claim_policy_name("readwrite"));
+        assert!(IamSys::<StsTestMockStore>::is_safe_claim_policy_name("my-custom_policy"));
+        assert!(IamSys::<StsTestMockStore>::is_safe_claim_policy_name("Policy123"));
+    }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_rejects_empty() {
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name(""));
+    }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_rejects_path_traversal() {
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("../etc/passwd"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("policy/name"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("policy\\name"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("..\\etc\\passwd"));
+    }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_rejects_whitespace() {
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("my policy"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("policy\tname"));
+    }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_rejects_special_chars() {
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("policy;drop"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("pol$icy"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("name{bad}"));
+    }
+
+    #[test]
+    fn test_is_safe_claim_policy_name_rejects_no_alphanumeric() {
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("."));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name(".."));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name(":"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("..."));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name(".:.:"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("-"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("_"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("__"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("---"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("-_-"));
+        assert!(!IamSys::<StsTestMockStore>::is_safe_claim_policy_name("-.:_"));
+    }
 }

crates/io-core/src/deadlock_detector.rs

@@ -155,7 +155,7 @@ impl DeadlockDetector {
     /// Register a new lock.
     pub fn register_lock(&self, lock_type: LockType) -> u64 {
         let id = {
-            let mut next = self.next_lock_id.lock().unwrap();
+            let mut next = self.next_lock_id.lock().unwrap_or_else(|e| e.into_inner());
             *next += 1;
             *next
         };
@@ -243,7 +243,10 @@ impl DeadlockDetector {
             return None;
         }
 
-        let graph = self.wait_graph.lock().unwrap();
+        let graph = match self.wait_graph.lock() {
+            Ok(g) => g,
+            Err(_) => return None,
+        };
 
         // Build adjacency list
         let mut adj: HashMap<u64, Vec<u64>> = HashMap::new();
@@ -310,7 +313,10 @@ impl DeadlockDetector {
             return Vec::new();
         }
 
-        let locks = self.locks.lock().unwrap();
+        let locks = match self.locks.lock() {
+            Ok(l) => l,
+            Err(_) => return Vec::new(),
+        };
         let mut result = Vec::new();
 
         for (&id, info) in locks.iter() {
@@ -349,13 +355,13 @@ impl DeadlockDetector {
 
     /// Get lock info.
     pub fn get_lock_info(&self, lock_id: u64) -> Option<LockInfo> {
-        let locks = self.locks.lock().unwrap();
+        let locks = self.locks.lock().ok()?;
         locks.get(&lock_id).cloned()
     }
 
     /// Get total number of registered locks.
     pub fn lock_count(&self) -> usize {
-        let locks = self.locks.lock().unwrap();
+        let locks = self.locks.lock().unwrap_or_else(|e| e.into_inner());
         locks.len()
     }
 }

crates/io-core/src/pool.rs

@@ -226,8 +226,9 @@ impl BytesPool {
     pub async fn acquire_buffer(&self, size: usize) -> PooledBuffer {
         let tier = self.select_tier(size);
         let mut buffer = tier.acquire_buffer(size, &self.metrics).await;
-        // Set tier reference for return on drop
-        buffer.tier = Some(Arc::clone(tier));
+        if buffer._permit.is_some() {
+            buffer.tier = Some(Arc::clone(tier));
+        }
         buffer
     }
 
@@ -304,12 +305,12 @@ impl PoolTier {
     }
 
     fn set_metrics(&self, metrics: Arc<BytesPoolMetrics>) {
-        *self.metrics.lock().unwrap() = Some(metrics);
+        *self.metrics.lock().unwrap_or_else(|e| e.into_inner()) = Some(metrics);
     }
 
     fn take_or_allocate_buffer(&self, size: usize, pool_metrics: &BytesPoolMetrics) -> (BytesMut, bool) {
         let buffer_opt = {
-            let mut available = self.available_buffers.lock().unwrap();
+            let mut available = self.available_buffers.lock().unwrap_or_else(|e| e.into_inner());
             available.pop()
         };
         let was_reused = buffer_opt.is_some();
@@ -368,10 +369,20 @@ impl PoolTier {
 
     async fn acquire_buffer(&self, size: usize, pool_metrics: &BytesPoolMetrics) -> PooledBuffer {
         // Acquire semaphore permit (owned for storage in PooledBuffer)
-        let permit = Arc::clone(&self.semaphore).acquire_owned().await.unwrap();
+        let permit = match Arc::clone(&self.semaphore).acquire_owned().await {
+            Ok(p) => p,
+            Err(_) => {
+                let buffer = BytesMut::with_capacity(size.max(self.buffer_size));
+                return PooledBuffer {
+                    buffer: ManuallyDrop::new(buffer),
+                    tier: None,
+                    _permit: None,
+                };
+            }
+        };
 
         // Use the pool's shared metrics for recording
-        let _metrics_lock = self.metrics.lock().unwrap();
+        let _metrics_lock = self.metrics.lock().unwrap_or_else(|e| e.into_inner());
         let _metrics = _metrics_lock.as_ref().unwrap();
 
         // Record acquisition
@@ -394,7 +405,7 @@ impl PoolTier {
         let permit = Arc::clone(&self.semaphore).try_acquire_owned().ok()?;
 
         // Use the pool's shared metrics for recording
-        let _metrics_lock = self.metrics.lock().unwrap();
+        let _metrics_lock = self.metrics.lock().unwrap_or_else(|e| e.into_inner());
         let _metrics = _metrics_lock.as_ref().unwrap();
 
         // Record acquisition
@@ -414,11 +425,11 @@ impl PoolTier {
 
     /// Return a buffer to the pool for reuse.
     fn return_buffer(&self, buffer: BytesMut) {
-        let mut available = self.available_buffers.lock().unwrap();
+        let mut available = self.available_buffers.lock().unwrap_or_else(|e| e.into_inner());
         // Limit the size of the pool to prevent unbounded growth
         if available.len() < self.max_buffers {
             available.push(buffer);
-            if let Some(ref metrics) = *self.metrics.lock().unwrap() {
+            if let Some(ref metrics) = *self.metrics.lock().unwrap_or_else(|e| e.into_inner()) {
                 metrics.available_buffers.fetch_add(1, Ordering::Relaxed);
             }
         } else {
@@ -428,7 +439,7 @@ impl PoolTier {
                     Some(current.saturating_sub(released_bytes))
                 })
                 .ok();
-            if let Some(ref metrics) = *self.metrics.lock().unwrap() {
+            if let Some(ref metrics) = *self.metrics.lock().unwrap_or_else(|e| e.into_inner()) {
                 metrics
                     .current_allocated_bytes
                     .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
@@ -447,11 +458,10 @@ impl Drop for PooledBuffer {
     // buffer moves it exactly once into the pool when a tier still owns it.
     #[allow(unsafe_code)]
     fn drop(&mut self) {
-        // Return buffer to pool if tier reference exists
+        // Return buffer to pool if tier reference exists.
+        // Otherwise, drop the standalone fallback buffer normally.
+        let buffer = unsafe { ManuallyDrop::take(&mut self.buffer) };
         if let Some(ref tier) = self.tier {
-            // SAFETY: We're in drop(), so this is the last use of the buffer
-            // ManuallyDrop allows us to take the value without running BytesMut's drop
-            let buffer = unsafe { ManuallyDrop::take(&mut self.buffer) };
             tier.return_buffer(buffer);
         }
         // The permit is automatically dropped here, releasing the semaphore slot
@@ -503,7 +513,10 @@ impl std::fmt::Debug for PoolTier {
             .field("buffer_size", &self.buffer_size)
             .field("max_buffers", &self.max_buffers)
             .field("available_permits", &self.semaphore.available_permits())
-            .field("available_buffers", &self.available_buffers.lock().unwrap().len())
+            .field(
+                "available_buffers",
+                &self.available_buffers.lock().unwrap_or_else(|e| e.into_inner()).len(),
+            )
             .finish()
     }
 }
@@ -526,6 +539,20 @@ mod tests {
         assert!(buffer.capacity() >= 2048);
     }
 
+    #[tokio::test]
+    async fn test_acquire_buffer_after_shutdown_is_unpooled() {
+        let pool = BytesPool::new_tiered();
+        pool.small_pool.semaphore.close();
+
+        let buffer = pool.acquire_buffer(2048).await;
+
+        assert!(buffer.tier.is_none());
+        assert!(buffer._permit.is_none());
+        assert!(buffer.capacity() >= pool.small_pool.buffer_size);
+        drop(buffer);
+        assert_eq!(pool.available_buffers(), 0);
+    }
+
     #[tokio::test]
     async fn test_tier_selection() {
         let pool = BytesPool::new_tiered();

crates/io-metrics/src/lib.rs

@@ -112,7 +112,8 @@ pub use deadlock_metrics::{
 // Lock metrics exports
 pub use lock_metrics::{
     LockMetricsSummary, record_contention_event, record_early_release, record_lock_hold_time, record_lock_optimization_enabled,
-    record_spin_attempt, record_spin_count_change,
+    record_object_lock_diag_acquire_duration, record_object_lock_diag_enabled, record_object_lock_diag_hold_duration,
+    record_object_lock_diag_slow_acquire, record_object_lock_diag_slow_hold, record_spin_attempt, record_spin_count_change,
 };
 
 pub use process_lock_metrics::{

crates/io-metrics/src/lock_metrics.rs

@@ -62,6 +62,61 @@ pub fn record_contention_event() {
     counter!("rustfs_lock_contentions").increment(1);
 }
 
+/// Record object namespace lock diagnostics being enabled.
+#[inline(always)]
+pub fn record_object_lock_diag_enabled(enabled: bool) {
+    use metrics::gauge;
+    gauge!("rustfs_object_lock_diag_enabled").set(if enabled { 1.0 } else { 0.0 });
+}
+
+/// Record object namespace lock acquire duration.
+#[inline(always)]
+pub fn record_object_lock_diag_acquire_duration(op: &'static str, mode: &'static str, duration: Duration) {
+    use metrics::histogram;
+    histogram!(
+        "rustfs_object_lock_diag_acquire_duration_seconds",
+        "op" => op,
+        "mode" => mode
+    )
+    .record(duration.as_secs_f64());
+}
+
+/// Record object namespace lock hold duration.
+#[inline(always)]
+pub fn record_object_lock_diag_hold_duration(op: &'static str, mode: &'static str, duration: Duration) {
+    use metrics::histogram;
+    histogram!(
+        "rustfs_object_lock_diag_hold_duration_seconds",
+        "op" => op,
+        "mode" => mode
+    )
+    .record(duration.as_secs_f64());
+}
+
+/// Record an object namespace lock slow-acquire event.
+#[inline(always)]
+pub fn record_object_lock_diag_slow_acquire(op: &'static str, mode: &'static str) {
+    use metrics::counter;
+    counter!(
+        "rustfs_object_lock_diag_slow_acquire_total",
+        "op" => op,
+        "mode" => mode
+    )
+    .increment(1);
+}
+
+/// Record an object namespace lock slow-hold event.
+#[inline(always)]
+pub fn record_object_lock_diag_slow_hold(op: &'static str, mode: &'static str) {
+    use metrics::counter;
+    counter!(
+        "rustfs_object_lock_diag_slow_hold_total",
+        "op" => op,
+        "mode" => mode
+    )
+    .increment(1);
+}
+
 /// Lock statistics summary.
 #[derive(Debug, Clone, Default)]
 pub struct LockMetricsSummary {
@@ -108,6 +163,97 @@ impl LockMetricsSummary {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use metrics::{Counter, CounterFn, Gauge, GaugeFn, Histogram, HistogramFn, Key, KeyName, Metadata, SharedString, Unit};
+    use std::sync::{Arc, Mutex};
+
+    #[derive(Clone, Default)]
+    struct SeenMetricsRecorder {
+        counters: Arc<Mutex<Vec<Key>>>,
+        gauges: Arc<Mutex<Vec<Key>>>,
+        histograms: Arc<Mutex<Vec<Key>>>,
+    }
+
+    impl SeenMetricsRecorder {
+        fn saw_counter_named(&self, name: &str) -> bool {
+            self.counters
+                .lock()
+                .expect("counter key collection should be lockable")
+                .iter()
+                .any(|key| key.name() == name)
+        }
+
+        fn saw_gauge_named(&self, name: &str) -> bool {
+            self.gauges
+                .lock()
+                .expect("gauge key collection should be lockable")
+                .iter()
+                .any(|key| key.name() == name)
+        }
+
+        fn saw_histogram_named(&self, name: &str) -> bool {
+            self.histograms
+                .lock()
+                .expect("histogram key collection should be lockable")
+                .iter()
+                .any(|key| key.name() == name)
+        }
+    }
+
+    impl metrics::Recorder for SeenMetricsRecorder {
+        fn describe_counter(&self, _key: KeyName, _unit: Option<Unit>, _description: SharedString) {}
+
+        fn describe_gauge(&self, _key: KeyName, _unit: Option<Unit>, _description: SharedString) {}
+
+        fn describe_histogram(&self, _key: KeyName, _unit: Option<Unit>, _description: SharedString) {}
+
+        fn register_counter(&self, key: &Key, _metadata: &Metadata<'_>) -> Counter {
+            self.counters
+                .lock()
+                .expect("counter key collection should be lockable")
+                .push(key.clone());
+            Counter::from_arc(Arc::new(NoopCounter))
+        }
+
+        fn register_gauge(&self, key: &Key, _metadata: &Metadata<'_>) -> Gauge {
+            self.gauges
+                .lock()
+                .expect("gauge key collection should be lockable")
+                .push(key.clone());
+            Gauge::from_arc(Arc::new(NoopGauge))
+        }
+
+        fn register_histogram(&self, key: &Key, _metadata: &Metadata<'_>) -> Histogram {
+            self.histograms
+                .lock()
+                .expect("histogram key collection should be lockable")
+                .push(key.clone());
+            Histogram::from_arc(Arc::new(NoopHistogram))
+        }
+    }
+
+    struct NoopCounter;
+
+    impl CounterFn for NoopCounter {
+        fn increment(&self, _value: u64) {}
+
+        fn absolute(&self, _value: u64) {}
+    }
+
+    struct NoopGauge;
+
+    impl GaugeFn for NoopGauge {
+        fn increment(&self, _value: f64) {}
+
+        fn decrement(&self, _value: f64) {}
+
+        fn set(&self, _value: f64) {}
+    }
+
+    struct NoopHistogram;
+
+    impl HistogramFn for NoopHistogram {
+        fn record(&self, _value: f64) {}
+    }
 
     #[test]
     fn test_record_lock_optimization_enabled() {
@@ -143,6 +289,60 @@ mod tests {
         record_contention_event();
     }
 
+    #[test]
+    fn test_record_object_lock_diag_enabled() {
+        let recorder = SeenMetricsRecorder::default();
+        metrics::with_local_recorder(&recorder, || {
+            record_object_lock_diag_enabled(true);
+            record_object_lock_diag_enabled(false);
+        });
+        assert!(
+            recorder.saw_gauge_named("rustfs_object_lock_diag_enabled"),
+            "expected object lock diagnostics enabled gauge to be emitted"
+        );
+    }
+
+    #[test]
+    fn test_record_object_lock_diag_acquire_duration() {
+        let recorder = SeenMetricsRecorder::default();
+        metrics::with_local_recorder(&recorder, || {
+            record_object_lock_diag_acquire_duration("get_object", "read", Duration::from_millis(10));
+        });
+        assert!(
+            recorder.saw_histogram_named("rustfs_object_lock_diag_acquire_duration_seconds"),
+            "expected object lock diagnostics acquire histogram to be emitted"
+        );
+    }
+
+    #[test]
+    fn test_record_object_lock_diag_hold_duration() {
+        let recorder = SeenMetricsRecorder::default();
+        metrics::with_local_recorder(&recorder, || {
+            record_object_lock_diag_hold_duration("put_object_commit", "write", Duration::from_millis(20));
+        });
+        assert!(
+            recorder.saw_histogram_named("rustfs_object_lock_diag_hold_duration_seconds"),
+            "expected object lock diagnostics hold histogram to be emitted"
+        );
+    }
+
+    #[test]
+    fn test_record_object_lock_diag_slow_events() {
+        let recorder = SeenMetricsRecorder::default();
+        metrics::with_local_recorder(&recorder, || {
+            record_object_lock_diag_slow_acquire("get_object_info", "read");
+            record_object_lock_diag_slow_hold("complete_multipart_upload_commit", "write");
+        });
+        assert!(
+            recorder.saw_counter_named("rustfs_object_lock_diag_slow_acquire_total"),
+            "expected object lock diagnostics slow-acquire counter to be emitted"
+        );
+        assert!(
+            recorder.saw_counter_named("rustfs_object_lock_diag_slow_hold_total"),
+            "expected object lock diagnostics slow-hold counter to be emitted"
+        );
+    }
+
     #[test]
     fn test_lock_metrics_summary() {
         let mut summary = LockMetricsSummary::new();